Apr 10, 2025

The redesign of filters split the filter head and filter body computations into the validate and confirm chasers. When validating we have the data necessary to populate the filter body, which is the block and its populated previous outputs. These are hugely expensive to obtain, so we don't ever want to have to do it twice. If we aren't validating, but are filtering, then we just bypass the validation aspects in the chaser. Filter body creation is entirely independent of the filter header creation, and has the same ordering constraint as validation. The result is written to an arraymap. This means there is no hashmap, just an array header which positions the link to the body element in the array element corresponding to the block surrogate key. Since blocks are keyed sequentially (by insert) there is no chance of a conflict, even if we happened to write the same block twice. That would just lead to two bodies and the last writer winning in the array head. This technique was developed to avoid using hashmaps for the prevout cache table. Filter header creation hash the same ordering constraint as block confirmation. The previous filter header must exist to create the next. And of course the body must have been created (during validation). So confirmation reads out the body and the parent block filter header, computes the current header and writes it to the filter header arraymap. This table is also an arraymap because it's also block-aligned, and the indexing needs to be independent of the body table because of their independent creations along with the table append-only rule. But the heads for these are very small, so independent tables with no hashmap lookup is very efficient. Given that filer header creation must read from both the filter header and body tables, avoiding hashmaps in favor of arraymaps is a big savings. So that's all done and checked in now. Sizing and perf numbers coming next.