Feb 3, 2025

I'm making the two query changes discussed in the meeting a few weeks ago. The process doesn't take too much RAM. It always uses all available RAM. The confirmation queries, on lower RAM machines, can result in thrashing (quadratic paging) because of the need to read all over certain tables. memory maps can only be read from disk at the granularity of a page, even if just reading one byte. Controlling paging is the central issue in a memory map store design. This can be done by improving the efficiency of queries, which takes various forms. The query changes I'm implementing consist of (1) using a intermediate table to cache the store identifiers of prevouts spent by inputs, while the inputs are being validated against the prevouts. This precludes the need to perform the prevout lookup twice, in exchange for a modest amount of data. The data can be stored and retrieved in a unitary memory chunk per block, with no search paging. This part is complete and has improved speed, but as expected is not sufficient to prevent thrashing on the mini. The other (2) change is to consolidate the "point" table into the "spend" table. Point is a lookup table to spent point hashes. This is a tradeoff in query complexity in exchange for reduced storage space (about 50GB saved). This is because point hashes are redundant (multiple outputs per tx, indexed by 32 byte hash in the spending inputs). However the optimization results in thrashing on the mini, and slows sync somewhat on the maxi, so we haven't been using it. Removing it will significantly reduce paging and may therefore eliminate thrashing on the mini.