July 2021

Hi Eric, I don’t quite understand what is the meaning of “current” as you said. What is the logic to decide when a header (or... a chain?) is current? Does LB consider a header is “current” based on the node’s local clock?

Time minus time stamp < configured gap.

is there some logic in LB that will wait for the header sync to *finish* before it starts asking for blocks e.g. at all

Header sync doesn't finish.

(until there is a header that is close to present time)

Ok, this config should be changed inside bn.cfg, right? What is the name of the option I should change there? (I have skimmed through it, but it was not clear to me which one it is)

ok, thank you!

Dear Eric, I experience reproducibly that full_node.close() never ends. Please find here the stack trace of my full_node.close() call as well as the stack trace of all of LB’s worker threads: https://0bin.net/paste/uv1N4MCg#GeqLghy7InX5MAlvVnzT68qrdaBkmZ2Ygs9Li-hyJOO

Yup

ooh, ok! Thanks for pointing that out!

Version?

I’m using v4

Dear Eric just curious, my log shows Block #179900 [00000000000003e21e4c75bb543ac7d6859fe9e27c7ce5872287391ea955ce37] (snip) Valid #065100 [000000000777c9285c621edc2889207de4b5c9b718c3f006dc2107a0b4d69b18] Is it that block validation is fully decouped from block download, why is it, is there a performance consideration underlying? (I guess internally blocks need to be validated in a sequence, and meanwhile thanks to header first sync actually block download order can be nonsequential. Is that why I get block 1379,900 downloaded before block 65,100 is validated?)

Please see previous explanation of parallel block download.

Eric, In version 4, what happens if you have an LB node that is synced up to block X, then suddenly, there’s a reorg. X is orphaned. suddenly Y is the new HEAD, however LB has not received the block yet (that is block X), it has only received the header (that is the header of block X). I presume here that as soon as LB receives X such that a chain with more work is known to be arriving, the header subscription handler will be invoked. Will there be a block reorg that will rewind the chain?

There are no reorgs until blocks are populated and validated

Dear Eric, So LB offers a subscribe_headers() and a subscribe_blocks(). (accessible by full_node.chain(). ) - Note that the subscribe_header name is totally illogical as really what it does is header reorg listening, it is not just dispatched at the full_node’s reception of just any header. This is also reflected in the fact that the callback gets the arguments “outgoing” and “incoming” which denote a reorg - So these are for listening for two distinct events, we can call them “header reorg” and “block reorg”. Please explain to me how full_node’s “header reorg” and “block reorg” processes are correlated. Especially, i want to understand, the header reorg callbacks are made *when* in the chain syncing process, and the block reorg callbacks are made *when* in the chain syncing process? Especially then “when”s in the chain sync that I am interested in is, what if a full_node technically has two HEAD blocks on different forks: the header sync process may use as HEAD the header of block X. meanwhile the block sync process may use as HEAD the block Y which is on a different chain from X. this discrepancy would arise when the full_node has full blocks downloaded for genesis ... Y, BUT, it does not have blocks downloaded for genesis ... X yet! (or anyhow in the genesis ... X chain up to a total amount of work bigger than that of Y). (Please note: subscribe_headers should be renamed to subscribe_header_sync_reorg , the current name is confusing and not relevant.)

headers are organized into a chain before blocks are organized. I've described this process previously. this implies that there are two distinct events. these processes run concurrently, as previously described. Potential differences between these chains is not a discrepancy, it's by design. There is only one genesis block/header, which is defined in configuration. Adding "sync_reorg" to the subscriptions does not differentiate them from anything else, as all header and blocks are only obtained through reorganization. There is no published event when network merely obtains a header or block from a peer. "sync" is not a differentiating concept and "reorg" is where blocks and headers come from (hence the presence of incoming and outgoing on all header/block notifications).

If you want to capture the event of a block or header message then you need to operate at the bc::network level, not bc::node. You would do this when implementing your own node. Given the context of bc::node::subscribe_* the names are fully descriptive. You get a notification when the node has organized a header or block.

Guys, in LibBitcoin, “compact filter” does *not* mean “bloom filter”, does it? But what is compact filter then? (Implementation is here: https://github.com/libbitcoin/libbitcoin-system/blob/master/src/message/compact_filter.cpp ) And re bloom filters LibBitcoin’s full_node does *not* support it does it”

I think that is the result of a seed node peer not responding before the timeout. It happens, especially if the configured seeds are offline.

> Given the ................ fully descriptive. I see, fully accepted & agreed. Thanks for clarifying.

Eric thanks a lot for clarifying! That is that a full_node’s subscribe_headers() and subscribe_blocks() only regard headers/blocks where the node has a whole header/block chain already that includes covers said header/block. Apologies if the following appears like asking doubly however I wish to get great clarity on this. To really get it, I’ll describe the context first: So a node firstly receives block headers. This becomes one whole set Htot, of all headers it received. It’s an array sorted in ascending order of time received. The node basically calculates a function most_work_header_chain(Htot) -> Hmostwork . Hmostwork is a deterministic function of Htot. What you are saying then is that subscribe_headers() will produce a callback at any change to Htot. And then, a node receives blocks. A node will only receive blocks that it has headers for, either by a whole block dropping in in one piece (in which case the header would be added to Htot), or through that a node asks for the block that corresponds to a given header. Received blocks become one whole set Btot. It’s an array sorted in ascending order of time received. map(block_get_header,Btot) is always a subset of Htot. subscribe_blocks() will produce a callback at any change of Btot. My question to you now is this: The node tracks a main full blockchain, we can call it Bmostwork. What is the correlation between Hmostwork and Bmostwork actually??? Please explain. For instance, is LB implemented in such a way that map(block_get_header,Bmostwork) is always a subset of Hmostwork - I presume **NOT***? Please confirm? But then, may there be moments when Hmostwork and Bmostwork are different chains altogether? (As in that Hmostwork does not contain block_get_header(tail(Bmostwork,1)) *AND* map(block_get_header,Bmostwork) does not contain tail(Hmostwork,1). On this one I’m unclear. Your clarification of how LB does this would be very much appreciated, thanks!

It works the way Bitcoin works. It’s not clear by what you mean by “altogether”. The only guarantee is that both confirmed headers and confirmed blocks share a common header, even if that’s only genesis.

Bitcoin headers do not form a linked list (chain), they form a tree. We download blocks for the strongest branch of that tree. This changes when there is a reorg. It may just get longer, or may be entirely replaced all the way to genesis.

hi, are there any problems with testnet servers at the moment? or is it just me, I try testnetN.libbitcoin.net:19091 with various N and they all give me connection timed out

is there maybe a place where I can check current status and availability of servers?

thank you

np

unfortunately, I can't find working testnet server, both seem to be down, for me the problem started yesterday, any chance one of them will be up soon?

Sorry, try tcp://testnet2.libbitcoin.net:19091

it works!, thank you very much, thanks for encouragement as I wasn't sure I am not being a pest :-)

Hey! Right now, when one runs a node, if it cannot connect to any peers, it stops. Wouldn’t it be nice if the node waited a bit and then tried connecting again?

It does. Then it stops. Feel free to change the configured timeouts.

oh. I see. I meant like an infinite loop. Anyhow, I’ll have a look to the timeouts

I try to avoid infinite loops

I understand. ok then. Thank you!

@evoskuil So in effect internally in LB, the header chain and block chain are two separate synchronization processes, with the only constraint between them that it always holds true that map(block_get_header,Btot) is a subset of Htot? (Also internally does LB maintain a separate index for the main header chain and for the main block chain?)

Sorry, not following htot, btot

What do you mean by “main”?

There is a strong(est) chain index and a candidate chain index. Headers and blocks are not independently height-indexed, because a block of a given header is of the same height.

Well, thing is, what if there is no network connectivity when you launch the node. I see what you mean however for a daemon/background service type setup, indeed you *want* it to continue infinitely albeit really slow, if connectivity fails. How do I make the full_node do that... if startup fails I just destroy the full_node and create a new one to .start()&.run(), with ever incrementing delays, until it eventually succeeds?

(btot as in all the blocks a full_node stores in one moment. htot as in all the headers a full_node stores in one moment.)

The ‘strong(est) chain index’, that is an index where for each indexed height, the whole block is always available&known&validated, right? Meanwhile, in the candidate chain, may it be that for one indexed height the header is known but the block is not known yet because it has not been downloaded yet? And, you only trig a reorg that is ‘candidate index’ is a chain with more work than the ‘strong(est) chain index’, you do that ONLY when for each height in that whole candidate chain index, the full block has been known&validated ?

The candidate chain is not the strong chain specifically because its blocks have not been validated. Once they are it is reorged into the strong chain.

Oh, I see that technically there are two kinds of connections that could be started: the seed nodes only situation, when the peers database is empty, and then you have the ordinary start procedure where you have a nonempty peers database already, in which case you want to talk to peers from the peers db first and foremost but I guess traversing the seed nodes to wouldn’t be absolutely disallowed. Does LB have any differentiation in respect of auto-reconnect behavior, between these two? I’m contemplating if full_node.start() doesn’t work out then just, after a timer expires, just run it again.. sounds like a plan?

there is no difference in the type of connection between a seed and a peer. The session differs in that the connection is dropped once addresses are exchanged with a designated seed node.

A candidate chain that “wants to” become the “strong(est) chain” (main chain as I called it before), originates when the full_node receives headers that instruct the full_node that a chain aternative to the currently-strong(est) chain, that has more total work, exists. Right?

Yes

Given that a stronger chain may form from any block starting at genesis, the candidate chain may be an entirely new chain from genesis, but will always have the same or greater work than the confirmed chain.

Dear Eric, should we understand the p2p object as an integral part of a full_node* only or could I create my own p2p object to do stuff on th ep2p network for experimental purposes, unrelated to an actual node function?

libbitcoin-network is lower level than node, and it doesn't process blocks or headers (as it has no place to put them). bx uses it for broadcasting transactions.