Oct 24, 2024

The "best" chain is the strongest (work) valid header chain we are seeing. Currency is a configurable time from (order than) the host's wall clock. This defaults to 1 hour (ie approximately 6 blocks from the top is "current").

#3 (block.check) isn't strictly limited to static (context free) checks, there are some that require context (fork activations).

#4 is part of #3 (validity at this point is limited to block/tx.check)

#5 There is a table for block validation/confirmability state [validated_bk.data], and one for tx [validated_tx.data], as well as a table for tx confirmedness [strong_tx.data] - which provides the reverse navigation from tx to block (only for confirmed txs). validated_tx is really only an optimization for pool/compact block processing. It records distinct validation states for potentially incompatible contexts (such as validated for the pool, but needed to be validated again because the pool context was no longer consistent with a block context).

#6 we don't wait until all are complete, we will validate concurrently with parallel downloading and confirm concurrently with validation and downloading - as applicable. We will likely wait until complete for lower capacity machines, as otherwise they can get into mmap thrashing. But the big box handles all three processes concurrently quite well.

And of course header sync is always ongoing concurrently as well. All of these processes are continuous and adapt to reorganization and disorganization (where a block is found to be invalid/unconfirmable).

I can't say specifically where the big performance difference is. There are a lot of things that are necessary to make this level of concurrency feasible, such as a very efficient write-concurrent store. But the ability to process downloads and validations highly-concurrently is certainly the main issue. With respect to lower level isolated optimizations, I expect Core is better. We're running this without SHANI (Core has it in our benchmark) and with a 3 year old copy of libsecp256k1 for example. Since those don't affect the architecture they've been deferred.

The easiest way to see it is in the resource underutilization. Core is barely using anything on my big box. It should always have at least one resource maxed out. During download that should generally be bandwidth, and during validation should generally be CPU. Since there is significant excess CPU during download, we validate concurrently to capture that.

Thanks. I'm still missing important details with regard to your data structures. Conceptually you need to keep an index of txid to transaction to be able to validate a block and an index of outpoint to spend status to be able to confirm it. There is this block validation/confirmability table which you explained above, but as far as i understand it points to which transactions are validated/confirmed, not which of their outputs are spent or not. You also referred to "context" a few times which i'm not sure what it is concretely. If i remember correctly you mentioned you could reorg for ~ free, so this might be related.

Yes, we have complete indexes in addition to validation metadata. The two "validated" tables are disposable, and in the case of reorgs would just get generated as necessary. Essentially they are state repositories that assist with concurrency. The "strong" table is not disposable, and only provides the reverse index from (confirmed) tx to its confirming block. This gets overwritten (to negative) in the case of a reorg out of the block. Navigation is possible from candidate or confirmed height to header, header to parent, header to txs (if block is stored), txs to spends (sequence) and outputs (script/value), spend to input (contains script/witness), input to point (output natural key), output to containing tx, spend to containing tx, tx to confirming header (and position in its block).

All of these navigations are basically just table-relative memory pointer indirections, apart from the natural (hashmap) key indexation, which exists for header[hash], txs[header.hash] (txs is an association table between header and its tx entries), tx[hash], point[hash] (point is a table of prevout hashes used to collapse the redundant values to surrogate keys), spend[composite key of point surrogate key and output index], and optionally the electrum table of address[output script hash].

The tables are either blob or array (variable or fixed record size), and can be associated directly by surrogate key, and/or indexed by a hashmap (header with conflict set links internal to the records). You can see those in database/primitives, along with an iterator.

Given this structure we can navigate all aspects bidirectionally in constant time. The point table saves about 50 GB of redundant hash storage, but requires a large memory machine during sync. Despite all of the indexation it's actually very space efficient.

meta: forgot to click it this time 😐

These tables are all written concurrently. My machine writes 64 blocks at a time, spanning all of the tables while doing it. But at the same time, we have the ability to safely hot backup and ensure data consistency in the case of an unexpected process termination. These were the biggest problems with prototyping work I did back in 2017-18, the table headers and bodies were attached, so we had no way to assure fault recovery.

Correction, the txs table uses the header surrogate key as it's search key, not the header.hash (much smaller and total hashmap collision control).

Similarly, the validated_tx, validated_bk and strong_tx hashmaps all use surrogate search keys.

Context is the chain-relative data necessary to validate a block/txs, and the assumed-next-block context when validating for the tx pool (which may turn out to be incorrect). This includes active forks, mtp, height. We also have a concept of chain_state (prob not the same as Core), which is used to retain the information necessary to validate headers, including the accumulation of history necessary for things like mtp and signal accumulation. This allows us to roll it forward without hitting the store, and from it we generate context for each block.

Yeah for Core, chainstate generally refers to the UTxO database.

Since the context for a given header can never change, we store the essential values with the header when the candidate header is committed to the store. Then context is available later for independent block.check/accept/connect/confirm transitions.

Why do you keep an index of transaction to their spends?

Oh, is that how you detect it is unspent, by the absence of an entry in the spend index ("sequence")?

Spends is just an intermediate table between tx and input, it carries the sequence and pointers. Sequence is just the tx.sequence field. The index is not stored, it's just an array offset in the puts table, which associates tx->spends->inputs and tx->outputs. It's important to factor output/input script/witness from the other tables due to their size, as that dramatically reduces paging/reads. input.sequence and output.value are required at different validation phases, which is they aren't stored symmetrically.

There can be any number of spends associated to an output. The spends table is read as a multimap against a given point. So for spentness we iterate the spending txs (spend.parent_tx surrogate key) and determine if they are confirmed by search of the strong_tx table with those keys (pointers). So by unconfirming (setting false in strong_tx) all spends of that tx automatically become unconfirmed spends (still spent, but not confirmed spent).

We do not need to know if a previous output is confirmed, or if there are multiple spenders of an output, when we validate. If we have the prevout (just in the store for any reason) we validate. Spentness and maturity are determined in a completely decoupled step, which is just a query. That confirmability step is the most strongly ordered, and download is the least strongly ordered.

Yes, i've now grasped the separation between validation and confirmation in Libbitcoin. I'm asking about the confirmability step. Using what table/index do you check the inputs of each transaction in a block?

(I've seen the doc you shared, will read it now.)

strong_tx indicates txs associated to confirmed branch blocks. We call then strong vs. confirmed, because they aren't actually confirmed until in the confirmed height index.

These are not part of "archive" data because they aren't necessary to store/retrieve txs or blocks, they are metadata about them.