Jan 3, 2025

I was able to simplify this confirmation caching design a lot. The tx locking (prevent duplicates while confirming) idea is out, too costly and complex, and not necessary. By merging the point table back into the spend table things also get simpler, and we save 19GB on the current store and drop a table. The only downsides are we can't optionally reduce store size by 50GB through point hash consolidation, but it's a little costly during sync, and prohibitive on small machines. Hopefully the significantly increased size of the spend table wouldn't create a new paging issue, but I don't think so. It's less data and fewer queries overall. In that consolidation we drop a 4 byte link and 4 byte reference for every spend, so we save 19GB total, which incidentally is the same size of the new cache. However the new cache can be dropped after validation. So this saves us an actual 19GB over the uncompressed store state, at the cost of not being able to compress out another 31GB. That puts us very close to Core store size (we were 25 GB larger or smaller depending on the consolidation option). The cache itself becomes the id of the prevout tx (4 bytes) and the id of the spend of the prevout (4 bytes). With these two values there are no hashmap searches apart from searching for the confirming (strong) block of the tx. Basically everything is done in validation, which is parallel by block, except for this last step, which cannot be parallelized because the strong relation must be done in order by block while confirming. So I feel pretty good about the optimization being right at this point, and that it will be very effective.