Bitcoin full node and query server
Oct 27, 2024
View in context in the libbitcoin Slack #general mirror.
A
Antoine Poinsot19:42
Here is my "Libbitcoin for Core people" writeup, which intends to conceptually summarizes how Libbitcoin works for people familiar with Bitcoin Core. https://gist.github.com/darosior/97958327835eeef4119418e4d3b6d27f
I intend it to post it to Delving Bitcoin (probably tomorrow) so people can discuss it and you can participate if you're so inclined. I tried to keep it a high-level overview, so i've omitted some details you've shared here. These can be brought up if people ask about a certain area in the comments.
If you have the time to skim through it please let me know if i got anything wrong or feel i'm misrepresenting something.
51 replies · jungly, Antoine Poinsot, evoskuil · last Oct 28, 2024
jungly19:53
I think people might interpret the description of the database to mean it's a relational database. It might be worth while to add a sentence or two mentioning it's a custom in memory database.
A
Antoine Poinsot19:55
But you could have a relational database backend as well right? What i understood from the document was that it's abstracted, but represented as a relational database.
jungly19:55
Also, Eric has this event driven architecture in how the threads coordinate work. I think that is a big plus for libbitcoin too. Worth highlighting it? Eric probably is the best judge on this, if it's worthwhile etc.
jungly19:57
We could switch to a relational db, yes. But I think that is for not so performant set ups. A big driver of the performance is the custom db, and the event driven architecture. That's my understanding, Eric to keep me honest 🙂
evoskuil20:00
Logically the structure is relational, and is designed so that a SQL-based implementation could be dropped in (two interns have actually been working on SQLite). But people might interpret it as using a generalized SQL store, which would not perform like libbitcoin-database.
💥 1
jungly20:00
Having an architecture that allows using all the cores ... is just mind blowing, and the magic sauce!
👍🏻 1
A
Antoine Poinsot20:01
Ok, i had understood that but i can see how my notes don't make it clear. I'll try to clarify that before posting.
🙏🏾 1
evoskuil20:02
The network implements the asynchronous proactor pattern using BOOST ASIO. I've rebuild this, but this was originally Amir's choice and implementation, as was the mmap hash tables.
(edited)evoskuil20:09
skips all validation
This isn't entirely accurate. There is full header validation and tx commitment and witness commitment validations (both Merkle roots, guarded against malleation).
evoskuil20:11
once the best headers chain tip is current
Maybe too much detail, but headers are also committed whenever a checkpoint is encountered.
evoskuil20:14
When received, all checks which don't require strict ordering are performed and transactions are stored in a global transaction index.
Under milestone only the commitments are checked. Otherwise block.check is performed. This may requires check context (for forks), but it does not require prevouts.
// Header state is checked by organize.
code protocol_block_in_31800::check(const chain::block& block,
const chain::context& ctx, bool bypass) const NOEXCEPT
{
code ec{};
if (bypass)
{
// Only identity is required under bypass.
if (((ec = block.identify())) || ((ec = block.identify(ctx))))
return ec;
}
else
{
// Identity is not correct if error::invalid_transaction_commitment.
if ((ec = block.check()))
{
if ((ec != system::error::invalid_transaction_commitment) &&
block.is_malleated())
return system::error::block_malleated;
return ec;
}
if ((ec = block.check(ctx)))
return ec;
}
return system::error::block_success;
} (edited)evoskuil20:21
Validation may run concurrently with block requests, chasing gaps and performing the aspects of validation that require prevouts (and context). For this validation blocks and prevouts are deserialized from the store.
Confirmability may run concurrently with both download (block.check) and validate (block.accept and block.connect). This is a set of database queries that verify maturity, spendability and spentness.
evoskuil20:22
Kinds feels like you've conflated validation with confirmation. There are three distinct workflows that may or may not run concurrently with each other. On a big machine it is most optimal to run them all concurrently.
evoskuil20:24
Block request/check/store (download) is unordered. Block.accept/connect (validation) is partially ordered (all preceding blocks must be stored). Confirmability is strictly ordered (the preceding block must be confirmable before the next is tested).
A
Antoine Poinsot20:24
I have not conflated them in my understanding. Hmm maybe i should just properly structure my writeup instead of being notes...
A
Antoine Poinsot20:25
The separation of steps in function of ordering is the main thing that i want to convey i think, so i should make sure it's clear
👍🏻 1
evoskuil20:26
The fully validated chain is then persisted
Makes it sound like the blocks aren't stored yet. Everything except the validation/confirmation/height index entries are persisted as blocks arrive.
evoskuil20:28
There is a "spender" relation between an output and its spending transaction(s).
Prob call that an index vs. relation.
evoskuil20:30
There is a "confirmed" relation between a transaction and its confirmation height
confirmation header
(edited)evoskuil20:34
When checking a block for confirmability it will for every transaction fetch its prevout transaction(s) (which must have confirmed state, otherwise the block confirmability check fails) and for each prevout it will mark the spender corresponding to the transaction being checked for confirmability as "confirmed" (if there exists any other spender whose related transaction is "confirmed", this is a double spend, the block confirmability check fails).
Validation "fetches" prevouts, confirmability is just a boolean result query.
There is no marking of spends or spenders or outputs. Confirmed txs are associated to their containing header (their block), only.
evoskuil20:35
wouldn't be able to reorg past the pruning point.
Technically we could, but it requires data requests to recover missing txs.
evoskuil20:36
The transaction table is also used to persist unconfirmed transactions.
Transactions span all tables in the archive except the header and txs (association) table.
(edited)evoskuil20:37
by default the node requires a transaction has at least 1sat/vb.
There is no default as this isn't yet implemented, but presumably zero value would just disable the pool altogether.
(edited)evoskuil20:39
For conflicts the node requires the transaction pays at least the minimum feerate for itself plus for the whole graph it conflicts with, plus an incremental feerate (1sat/vb too by default).
There's no implied additional increment. Self paying for self graph and conflict graph implies that the full amount is offset.
👍 1
evoskuil21:24
timelocks
Input time locks are checked in the confirmability query, as it requires the context (height and mtp) of the previous output (also required for the coinbase maturity check).
(edited)👍 1
evoskuil21:41
Those could be checked in validation, which would benefit from block parallelism, but it requires some amount of redundancy (which may make sense as an optimization). I think that because the data is queried anyway, the two checks are essentially free in the current implementation.
(edited)A
Antoine Poinsot17:39
There is a "confirmed" relation between a transaction and its confirmation height
confirmation header (edited)
Oh, is that a difference between the document and what you ended up implementing?
A
Antoine Poinsot17:47
Then why wiping the transaction -> confirmation header index on reorg? Technically it's true the transaction is associated with this header, just the header isn't part of the strongest chain anymore?
A
Antoine Poinsot17:53
Other than that i should have corrected everything, and i re-wrote the whole thing in a more bearable way (which also highlights the important main point of concurrency).
👍🏻 1
evoskuil17:53
Not a difference, just a bit more detail than is available in the doc. The header height indices (candidate and confirmed) exist at trivial cost, almost zero. These are the final step in exposing a confirmation (remember the store is always fully read exposed). We have headers and transactions. The only thing that constitutes a "block" is the txs association table, that associates a header to the transactions it "contains". headers are populated first, then when blocks are downloaded, transactions are populated, and then the txs association, and then the indexation of the txs association. At that point the block is archived. So all "blocks" are headers that associate their txs. However these txs do not associate their headers. That would require a costly one to many association, for which there isn't a significant use case. So we implement the 1:0..1 association between a tx and its confirming block (or no block). These associations are set in order, block block by block, during confirmation.
evoskuil17:55
This association is sufficient to determine that the tx is confirmed, and to find it's confirming block (which includes the block height as context, since all blocks have a height whether confirmed or not). The height index is also set, but this is not used in the confirmation process, it is just to provide an index of confirmed blocks by height.
(edited)evoskuil17:58
The tx->confirming-block ("strong") association is necessary during confirmation as otherwise we would have to perform a linear scan of the entire store to know whether a previous output is confirmed and at what height.
(edited)evoskuil18:06
It also gives us a way to in constant time distinguish between confirmed and unconfirmed txs for things like the tx pool and general public API query.
evoskuil18:12
Note that the association of a transaction to a block inherently implies the tx is no longer in the tx pool. The existence of a confirmed spend conflict also implies that an unconfirmed tx is not in the tx pool. Thus the graph necessary for rapid generation of block templates can be directly and efficiently populated from the store in its present form.
A
Antoine Poinsot19:03
Hmm, can you remind me how you mark a transaction as validated again? (Not confirmed, just the partially-ordered checks)
evoskuil20:19
TLDR: we don't mark block txs.
We can write an entry into the
validated_tx table with the tx state and validation context. We don't do this during block download because blocks are validated as a whole, so there is no need to relay the transaction state, block state is sufficient for the next (confirmation) checks, which are necessarily block-based. We considered (and designed for) individual transaction-based validation stage fan-out, which could either use this table or synchronize on the block. However, performance testing showed that as long as there are as many blocks as threads it is much faster to validate whole blocks. The same fan-out was tested for confirmability (within a block) and was also found to be not worth the fan-out cost (although was much closer because it's a single block at a time, and there may be configurations where it makes sense). And the tx validation entries are not necessary apart from validating the chain (the plan was to purge the file periodically). So this will probably end up being used only for transactions that enter via the pool. However validated_bk is written with current state (no context required, as block context is inherent and stored with the header), which allows the confirmation chaser to proceed across validated blocks (changing them to confirmable or unconfirmable. A confirmable block may cycle between confirmed and unconfirmed, but the confirmability state is final.evoskuil20:58
validated_bk data is also designed so that it may be purged (it would just have to be regenerated if the block returned via a re-reorg, which would happen automatically. But there are so few blocks and these entries are small (no context) that it's not really worth automating it.evoskuil21:02
So really there is no "marking" whatsoever. The closest we come is maintaining an association from tx and from height to header (candidate and confirmed), which just naturally reflect candidate/confirmation state. We did perform marking in a prototype, and that was abandoned.
👍 1