Bitcoin full node and query server
June 2021
Archive of the libbitcoin Slack #general channel.
Friday, June 11, 2021
neillm11:14
Heads up that the v3 community servers have been having issues lately. Everything is currently up and running, but catching up. With any luck, the hardware will be replaced at some point later this year.
Wednesday, June 16, 2021
P
Phil13:45
hey there!
I’ve noticed that there are some hardcoded settings, as database directory:
settings::settings()
: directory("blockchain"),Is there any interest in making this kind of thing customizable?
The directory structure of the database is fixed: filenames and subdirectory. Configuration controls the directory.
P
Phil13:48
Configuration has this
boost::filesystem::path file;. But is this the directory? I couldn’t find yet its initialization.The configuration is documented on the wiki.
[database] # The blockchain database directory, defaults to 'blockchain'. directory = blockchain
You can specify the path here. It defaults to the "blockchain" subdirectory of the process current directory.
(edited)P
Phil13:59
When I run
bn it starts syncing in the current dir. What if I want it to sync in another dir?The above configuration setting controls the location of the database directory. By default it uses the "blockchain" subdirectory of the process current directory, which by default is the directory in which the binary resides.
Just specify any valid directory or subdirectory for the database.directory config setting.
evoskuil14:01
You can also change the location of the config settings file via the process command line.
Yes
Whatever you specify is just passed on to the operating system.
neillm16:36
I didn't check, but isn't that just the default, but override-able in the config file?
👍 1
^Hah, missed the discussion above
🤦♂️ 1👍 1
Thursday, June 17, 2021
N
narodnik19:50
@evoskuil hey eric I have access to libbitcoin.org libbtc.org and the .com ones as well. Who should I transfer ownership of them to?
evoskuil19:52
That's great, you can transfer them to me. I hold the other related domains. They may not be transferrable if you recently transferred them. But they just need to be unlocked. I'll put in a request through my registrar (pair domains) and they will forward it to yours.
Saturday, June 19, 2021
evoskuil21:02
@narodnik libbitcoin.org shows available for transfer, but libbtc.org doesn't. I put in a transfer request for libbitcoin.
Tuesday, June 29, 2021
neillm01:34
Looks like a lot of projects left freenode. Is there a new #libbitcoin channel somewhere on IRC, or is freenode still the place?
Wednesday, June 30, 2021
P
Phil19:18
Dear Eric
After spending some time lookin at the reorg listener interface in full_node*, I realize that the distinction of the process of the header sync (which is happening conceptually first: header-first sync, right) versus block sync, is not clear to me.
Is LibBitcoin somehow integrating these two into one, so that there is only one reorg listener and event stream, and not two?
How does that work out in practice.
Is it that the reorg logics in LibBitcoin are trigged based on the headers only - and the reorg listener is trigged accordingly - and these logics *presume* that there will be data availability of the blocks in this chain, such that get/fetch/downloaded block listener events are presumed to follow for each block?
Please explain
evoskuil19:40
Hi Phil,
If you are looking at header sync, you are in v4.
In v4 the process is as follows. First, the structure:
The block table retains headers (a bit of a misnomer), the tx table retains txs. There is an association table that retains the associations between headers and their downloaded transactions. The purpose of the association is to allow headers (before obtaining txs) to be stored in an array-table, as the association to txs is fixed-width. There are two height indexes, one for the confirmed chain and one for a candidate chain. The confirmed index is authoritative for the strong branch. The candidate index is the next-strongest visible branch. There are arrays of pointers to block headers, allowing blocks to be queried by height in constant time. There is also a payments index. This provides constant time query from any output script to the transactions that contains it (outputs) and any that reference it (inputs). The indexes are redundant information. The entire chain is retained by the three headers->transactions tables. Headers can exist without associated transactions, and transactions can exist without associating headers. Transactions can be associated by any number of headers. All associations are complete (full block or none)...
8 replies · Phil, evoskuil · last Aug 11, 2021
P
Phil19:02
What do you mean, for there to be an association between one block header and one transaction (and associations are by nature bidirectional), the whole block structure for the said block must be represented in persistence i.e. either ALL or NONE of the transactions belonging to one block are in the persistence?
evoskuil04:33
If the block is “populated” then all of its transactions are stored and linked from the header by the association table. A block header will first be stored without being populated, though any number of its transactions may be stored at any time.
(edited)evoskuil04:34
A block must be populated before being validated and validated before being confirmed. Transactions arrive from the tx pool and from other blocks, which may be in another branch and therefore have some of the same txs.
evoskuil04:36
The associations are not bidirectionally traversible. But each transaction carries its confirmation height, so confirmed txs can be traversed to their header.
evoskuil19:49
Data is stored as soon as the cost of producing invalid data reaches an arbitrary threshold. This is a denial-of-service feature. For headers this is specified in config as an amount of time from the present clock. The header chain must reach this relative height before block downloading begins. This is referred to as being "current". If the header chain is not current, no blocks are downloaded. If there is a current header, all headers preceding it are downloaded in parallel. They are stored as txs and associations as that happens. This is performed entirely independent of order. Standard deviation is used to detect slower peers in the block downloading, which are disconnected if they fall below a configurable threshold. This results in an acceleration of downloading, and unloads peers that appear to be overloaded. This is not done by header downloading, as we want all peers to have an opportunity to define the strong chain, so header download tends to be slow relative to the amount of data processed.
This process is not independent. Header download proceeds alongside block download. But block download only occurs when there are "current" headers that have not been populated by block download. I refer to this as continuous parallel block download.
P
Phil19:55
Specifically, I though, what if a malicious miner would produce many block headers but for blocks that don’t exist.
If LibBitcoin’s header and block sync process are one, where the header sync part decides which is the longest chain and block download is then applied on those blocks, if this is how the block sync works then essentially LibBitcoin block sync is stalled during any such >50% malicious miners attack which does an unavailable blocks attack.
For this reason I originally assumed that LibBitcoin would have two separate sync processes internally: one separate header sync process, which is about finding the longest header chain. But then a separate block sync process which will cause LibBitcoin to use as actually longest chain, the longest chain for which it has blocks, too. This approach would render LibBitcoin resistant to the attack I described.
However, I do see the argument that “header only & block unavailable attack” is unreasonable.
evoskuil19:55
The cost of producing a header for blocks that "don't exist" is the same as that for producing any other block.
And no, this does not create a stall.
Proof of work and parentage is validated before any header is accepted.
So the block "exists". If one cannot find a peer to produce it, the presumption is that there is a stronger chain. The same is true if a transaction within a block turns out to be invalid, despite the valid header.
If there is no stronger chain, or if no peer can produce it, this is not a stall.
The node will continue to request and accept headers, so the stronger chain will be found if it exists. If there is a validated header for a block which cannot be found, height remains at the parent of that block until it is found or a stronger chain is found.
The result is no different than any other node.
The difference is that we rely on the assumption (that all nodes make) that valid headers are costly to produce and as such can be relied upon for the storage cost of the header. This is around 100 bytes.
As blocks are populated a validation "chaser" (pointer) validates blocks in order as gaps are filled. Both the download queue and the chaser are repopulated upon restart of the node.
But at this point headers, transactions and the chain have all been pre-validated (as they are downloaded). The only validation that remains is script validation, as this requires references to preceding blocks.
The objective is of course to maximize parallelism, which is how code scales (improves with more hardware).