Bitcoin full node and query server
October 2024
Archive of the libbitcoin Slack #general channel.
Thursday, October 3, 2024
Huddle with pmienk, evoskuil · 11 min20:04
Wednesday, October 9, 2024
C
cguida22:32
hi libbitcoin people! I installed libbitcoin-node, and I'm looking for how to control/access it. does it have some kind of GUI perhaps, like bitcoin-qt, or a CLI like bitcoin-cli? the docs for libbitcoin-node seem to suggest there's no CLI https://github.com/libbitcoin/libbitcoin-node/wiki
but i'm also not seeing a GUI
39 replies · pmienk, cguida, evoskuil · last Oct 10, 2024
pmienk22:57
bn operates as a bitcoin node supporting the peer to peer protocol. It's configuration controls its operation. What specifically are you seeking to do?
C
cguida22:58
just want to watch its progress, was hoping for something like
(edited)bitcoin-cli getblockchaininfopmienk23:02
bn does not accept rpc commands. This project is not structured in an identical manner to the prototype bitcoin client. bn is a node and supports the p2p protocol. If you're looking for query support, bs - bitcoin server, combines the node with an additional client-server protocol over zmq
pmienk23:05
bx - bitcoin explorer provides a command line tool which implements the client protocol for executing queries against bitcoin-server as well as implementing a variety of p2p queries which can be executed against any bitcoin node implementation
pmienk23:08
Please see the documentation for libbitcoin-explorer for a breakdown of which queries are available and which protocol they rely upon. There are additionally many data manipulation commands available therein
C
cguida23:13
sounds like i need to install
bs and bx instead... but maybe i can tweak my logging settings to see what % of the chain is currently synced?C
cguida23:20
i tried making
bn.cfg and passing it in via -c but i get
$ ~/libbitcoin/bin/bn -c bitcoin/bn.cfg Error: unrecognised option 'verbose'
pmienk23:51
Well, bs instead gets you the additional client-server protocol. bx in addition to either simulate a peer connection or play client to the server.
C
cguida23:53
i really just want to watch sync progress. evoskuil claims that the new version of libbitcoin syncs the chain much faster than bitcoind and i wanted to see for myself!
pmienk23:53
If you're running master instead of version3 branches, the documentation will be incorrect.
pmienk23:56
Well, master does not have updates to server or explorer, so node is the proper starting point. Without providing your intention I have been working under the assumption you were interested in production work.
pmienk23:58
Without additional information related to your configuration file, I would assume you're missing the section header for [log]
pmienk23:59
For additional settings, I would suggest reading the code related to configuration parsing.
pmienk00:00
Working from my phone here. I do not have the bandwidth to locate the file at the moment.
C
cguida00:02
hmm verbose mode does not appear to be reporting synced block height... guess i'll dig into the code..
C
cguida00:09
for the record, i did provide my intention: https://libbitcoin.slack.com/archives/C6LH74ACQ/p1728514731924209?thread_ts=1728513125.238759&cid=C6LH74ACQ
cguida
just want to watch its progress, was hoping for something like
bitcoin-cli getblockchaininfopmienk00:14
Without additional context, the response to pursue libbitcoin-explorer was correct. The context that you wished to evaluate work in progress was not specified.
(edited)
evoskuil03:40
The node has various log contexts, sent both to a rotating pair of log files and the console. Each log context can be toggled on and off with initial values defaulted and controllable via the config file.
evoskuil03:42
Blocks are not downloaded in order, it is highly concurrent. You can see what blocks are being downloaded using the
protocol log context, but this will slow down sync, which is much faster than the console. The console does not block the node, but it does consume resources.evoskuil03:44
There is also a high speed event system. We use this to track download status. By default blocks are downloaded in 50k groups. By observing the quantity of entries in the event log you can see the number of blocks downloaded.
(edited)evoskuil03:45
Downloading will slow as the 50k group is closing out, and then accelerate again. The group size is configurable.
C
cguida18:27
Ok, thanks for the info. What would be the most straightforward way to gauge how long the sync will take to complete, without just starting the node and wait till it stops syncing. Is there even a message that indicates when the node has finished syncing?
C
cguida18:28
Sounds like maybe this "high speed event system" will generate a log every time 50k blocks is downloaded? Is there some special configuration to turn that on?
evoskuil22:55
The number of lines in the event log tells you the number of blocks. You can also use a compand line option to view table stats while running (or not). The number of tx table rows is fairly linear in total time as is byte size. I can get back to you with tx counts at each 50k interval.
evoskuil22:56
The event system is always on. It writes events as they occur (ie each block), not waiting to 50k increments.
👍 1
Thursday, October 10, 2024
evoskuil03:36
Yes, the configuration file can specify an alternate. Use -c (sir/file) to specify the file. It uses an “ini” format. Use -s to dump the settings names.
There is no gui, command line only.
Use -h for help (list of cl options).
👍 1
evoskuil12:37
It is likely that the default settings will be suboptimal for your machine, even resulting in thrashing. If you post your basic specs I can make suggestions. One of the aspects we are working on is automatically tuning what is presently manual configuration for given platform.
Also note that if your CPU supports sha native (shani), which is about 5x faster individual hashing, it will distort realistic benchmarking, since shani is not yet integrated into the hashing library.
75 replies · cguida, evoskuil · last Oct 19, 2024
C
cguida18:30
just a pretty average laptop from a few years ago
20GB RAM
Intel i5-10210U CPU @ 1.60GHz
4TB NMVe SSD
(edited)evoskuil18:55
I don’t test below 32gb RAM with 8 threads, but the default config won’t be good. The initial table sizing needs to be reduced, and the configured thread and peer counts. The node has a command line option to tell you the relevant CPU features. I’m not able to look at config until tomorrow, out with guests now.
evoskuil18:57
My top test machine, which consistently achieves 60 minutes, is 256GB with 32 dual cores (7 years old). I scale it down to 32 GB and 10 cores to get close to matching Lopp’s test platform. I forget the time, I think 2-3 hours.
C
cguida19:08
Wow, that's a beast! No wonder you are not seeing much issue with the utxoset growing faster than Moore's Law; it will be a long time before it will start having an impact on that. I'm concerned with low-resource machines being able to sync the chain in a reasonable amount of time. The average person cannot afford a computer with 256GB of RAM (or even 32GB).
The canary in the coalmine is the RPi 4B, which used to be able to sync in a reasonable amount of time a couple of years ago, but now takes over a month.
Would it be fair to say that low-resource machines are not a priority for libbitcoin?
evoskuil22:42
That is not why the utxo set is not an issue. We do not store a utxo set, we only store blocks, and that growth is linear.
evoskuil22:46
I have never been focused on low resource machines, because they will eventually not exist. I started coding on a 16k RAM machine with no disk drives or network capability. I take the longer term view. Scalability matters. Core does not scale. Its performance does not increase relative to hardware increment. Consequently it will eventually collapse. bn performs well on a machine equivalent to Lopp’s benchmark, which is at the lower end of my target range. The market will reduce the cost of the hardware, but that will only work for Bitcoin if the software scales.
evoskuil22:52
I’ve been spending a good amount of time optimizing various aspects for low RAM. If I assumed 512GB RAM Libbitcoin would sync in 45 minutes and Core would prob take 10-20 hours. I assume this because it does not use even a small fraction of my current RAM or CPU or network or disk capacity. bn always maxes out one aspect. bn can always perform better because of this, even with lower RAM machines, but that requires work to avoid mmap thrashing.
Eventually the distinction between SSD and RAM will not be significant and the mmap will not be necessary, and network will dominate and with no risk of thrashing.
C
cguida23:29
I'm curious: how much money do you think it's reasonable to ask a merchant who wants to accept lightning to pay for their node?
evoskuil00:20
Value is subjective, reasonable depends on every individual. But, nobody is asking. People do it if it is worthwhile. Eventually nodes will be practically free and sync in seconds. That’s the future I work toward. Also, bitcoin is absolutely not a white market money, so selling coffee isn’t a concern of mine.
evoskuil03:38
I don’t mean to imply that small value permissionless txs cannot have utility. I do not work toward white market scenarios, as they are pointless, ultimately given a complete lack of security. Micropayments is an independent question. It implies either high volume or a material expense to make a small (or large) payment. Bitcoin (node) does not provide for high volume (but can of course accommodate high cost small payments). Lightning is in some ways problematic, but under its assumption of infrequent on-chain transaction, I don’t see any conflict with the above observations.
evoskuil03:44
The question above implies that cost to maintain the same sync time will necessarily increase over time. That is absolutely not the case. The cost is only increasing due to implementation. Our effort is not to merely maintain a constant cost, it is to radically reduce that cost. That requires a very different architecture from Core, which was designed to fit a certain hardware profile and does not scale beyond it. Hardware has already outgrown that architecture, which is obvious from resource underutilization. We do this because it has been clear for some time that node design has been a significant centralizing force.
C
cguida22:40
Bitcoin (node) does not provide for high volume (but can of course accommodate high cost small payments). Lightning is in some ways problematic, but under its assumption of infrequent on-chain transaction, I don’t see any conflict with the above observations.
The trouble is that anyone who wants to make money running a darknet microservices business (for example) would need to spend thousands of dollars on a server. I just think this limits the potential base of merchants who would do this to a few elites, and certainly does nothing to incentivize more people to run nodes, which of course threatens bitcoin's decentralization.
C
cguida22:42
I think bitcoin has the ability to allow anyone the option to connect their lightning node with some microservice-buying-and-selling plugins, and earn a passive income proportional to their invested capital. Ignoring low-resource devices makes the barrier to entry artificially high.
evoskuil23:16
Why do you assume that? I can easily find a min spec laptop for $300, and that’s today. Not to say lower spec wouldn’t work, just slower. Nodes will become almost instant and practically free over time. Nobody is ignoring low resource, but failing to scale will never get there.
evoskuil23:17
FWIW, I haven’t seen the passive LN thing working, and not because nodes are expensive.
evoskuil23:21
And keep in mind that libbitcoin is just one project. We have specific objectives. We don’t stop anyone else from making nodes that optimize for a pi with 4MB. But there is no reason a modest PC should take forever to sync.
C
cguida01:55
The point is that the utxoset ballooning from 4GB to 12GB has made it no longer feasible to use a RPi 4B as a bitcoin node, whereas before the recent spam attack, it worked just fine.
Keeping a high-availability utxoset in RAM makes the sync much faster on low-resource devices. On higher-end devices, sure, perhaps the difference is not massive. Anyway it obviously has no effect on devices with more than <utxoset size> GB of RAM.
So if a 12GB utxoset can make a $100 device (the RPi 4B) incapable of syncing the blockchain in a reasonable amount of time, and the utxoset can grow at 40GB/year (the current limit), then in a few years the utxoset might be larger than 100GB, at which point only high-end devices (such as the ones you use to test libbitcoin) will be able to sync the chain in a reasonable amount of time.
FWIW, I haven’t seen the passive LN thing working, and not because nodes are expensive.
It will happen, but we need more time to build automated Lightning agents. These already exist but are primitive.
And keep in mind that libbitcoin is just one project. We have specific objectives. We don’t stop anyone else from making nodes that optimize for a pi with 4MB. But there is no reason a modest PC should take forever to sync.
No one can make the chain sync quickly on a RPi 4B anymore, because the utxoset does not fit in RAM. There are some promising research directions such as utreexo and assumeutxo, but these seem to be far from production-ready and user-friendly.
The only actual fixes, that I know of, are to filter spam, make the blocks smaller, or both.
Or just give up on low-resource devices being able to sync the chain, until Moore's Law catches up to the 40GB/year utxoset growth cap, in several decades.
evoskuil02:01
Again, there is no need for outputs to persist in RAM. We are talking about sync times that are absurdly high without even performing script validation. There is no reason for scripts to be consuming RAM. Query over existence and spentness is sufficient.
evoskuil02:04
When your store is slow and non-concurrent, and your processing is unduly ordered, RAM feels like the only solution.
evoskuil02:16
In any case, I have no interest in debating block size or spam. I pointed out that Core does not scale and the Libbitcoin does, and that linear chain growth will be easily overcome by non-linear hardware cost reduction. This is what we work toward, which is reasonable and useful. If you want to work on a fork with smaller blocks and no “spam”, have at it.
C
cguida16:49
Again, there is no need for outputs to persist in RAM. We are talking about sync times that are absurdly high without even performing script validation. There is no reason for scripts to be consuming RAM. Query over existence and spentness is sufficient.
And again, this precludes low-resource devices from being able to sync bitcoin. This is demonstrated by the fact that libbitcoin is much slower to sync on devices with <12GB of RAM than bitcoin core.
You're just ignoring an entire class of users, until Moore's Law catches up with blockchain/utxoset growth.
If we just throw up our hands and say "well I guess low-resource nodes will never work", then node centralization will accelerate to the extreme. If only people can afford a machine that has 256GB of RAM can sync bitcoin, then there will only be a few people actually running nodes, and these will be quickly co-opted by the state, long before Moore's Law allows computers to add 40GB RAM/year with no additional cost.
When your store is slow and non-concurrent, and your processing is unduly ordered, RAM feels like the only solution.
RAM is the only solution for low-resource devices. If there were no bitcoin implementations with a RAM utxoset, there would not be any devices worth less than $1000 running bitcoin full nodes.
If we don't bring down the cost of running a node, very few people will actually do it, and very soon bitcoin will not be around.
Anyway, that's fine if libbitcoin wants to be the "high-resource-nodes-only" implementation. Everyone's got a niche.
evoskuil16:54
Core does not scale (up). That’s a problem going forward. To the extent that libbitcoin does not scale down (which remains to be seen), that’s a problem that will go away on its own. This is not a discussion regarding whether bitcoind has been useful, but whether it can survive. Ultimately it has to adapt, which is what we are doing. So it’s not at all clear to me what your issue is. The cost of hardware for a node with the specs I gave you is around $300 at most.
(edited)evoskuil17:03
I realize you are invested in this idea that Bitcoin itself cannot continue to operate on affordable hardware, but this is just not the case. This is something you should be celebrating.
C
cguida17:18
In what year do you expect libbitcoin to overtake bitcoind in terms of sync speed, on a device that costs $300?
C
cguida17:20
Ok, interesting, have you run the benchmark on such a device? I am trying to do a comparison between bitcoind and libbitcoin on my laptop, but so far I've been unable to figure out how to determine what block I'm synced to in
bnevoskuil17:23
As I mentioned earlier, I sometimes swap out my RAM down to 32GB and disable all but 10 CPU threads (roughly equivalent to the newer 32/8 thread machine that JL uses for benchmarking). With proper config this is still way faster than Core on the same machine.
evoskuil17:32
I gave you info on how to determine block count (the sync is not ordered). The node emits events to
(edited)events.log. I believe current build emits only the block_archived event (with block height and time in seconds from startup). This will show the individual blocks archived since the node started up. You can also type i <enter> while the node is running and it will dump store stats to the console. Finally, you can type p <enter> and see individual blocks being archived at the console. This is a toggle, type again to disable.✅ 1
evoskuil17:45
This shows that 232,416 blocks have been downloaded and stored (archived) 146 seconds since startup, and the last block reported is at height 231,110
evoskuil17:46
This is an [i]nfo dump to the console while running:
2024-10-15T17:45:38Z.0 Body sizes... header :84847420 txs :192929304 tx :2921586630 point :4068475272 input :14381059167 output :4461494048 puts :1088675541 candidate :2597370 confirmed :3 spend :2719974744 strong_tx :565389996 valid_tx :0 valid_bk :0 address :0 neutrino :0 2024-10-15T17:45:38Z.0 Table records... header :865790 tx :47122365 point :113013202 candidate :865790 confirmed :1 spend :113332281 strong_tx :47115833 address :0 2024-10-15T17:45:38Z.0 Head buckets... header :565129 txs :565129 tx :681968291 point :638910752 spend :1739749749 strong_tx :681968291 valid_tx :681968291 valid_bk :565129 address :1 neutrino :1 2024-10-15T17:45:38Z.0 Collision rates... header :1.53202 txs :1.53202 tx :0.0690976 point :0.176884 spend :0.0651429 strong_tx :0.069088 valid_tx :0.0690976 valid_bk :1.53202 address :0 neutrino :0
evoskuil17:46
tx: 47,122,365 is a much better indicator of progress than block height, as it is close to a linear progression.
(edited)C
cguida17:47
ok so i am not synced to 524493 here? the
block_archived event would show a much lower number?
2024-10-15T17:44:33Z.0 Body sizes... header :84847420 txs :189308 tx :655464 point :28296 input :162756 output :856072 puts :98121 candidate :2597370 confirmed :3 spend :270960 strong_tx :126840 valid_tx :0 valid_bk :0 address :0 neutrino :0 2024-10-15T17:44:33Z.0 Table records... header :865790 tx :10572 point :786 candidate :865790 confirmed :1 spend :11290 strong_tx :10571 address :0 2024-10-15T17:44:33Z.0 Head buckets... header :524493 txs :524493 tx :551320125 point :546188501 spend :1459791875 strong_tx :551320125 valid_tx :551320125 valid_bk :524493 address :1 neutrino :1 2024-10-15T17:44:33Z.0 Collision rates... header :1.65072 txs :1.65072 tx :1.91758e-05 point :1.43906e-06 spend :7.73398e-06 strong_tx :1.91758e-05 valid_tx :1.91758e-05 valid_bk :1.65072 address :0 neutrino :0
evoskuil17:49
This is a recent run which completed 840,000 in 61 minutes. The transaction count at 50k block intervals is in column B. This is how I typically track progress.
evoskuil18:03
You have synced 10,572 txs, which is basically nothing (see above, the first 50,780 txs takes about 7 seconds after header sync completes at 48 seconds).
(edited)C
cguida18:11
ahh here we go
$ tail -f events.log block_archived...... 768617 17 block_archived...... 757842 19
evoskuil18:13
since it's default configured for a 50k block window, you should be somewhere around 750k blocks. But that does not jibe with the info dump above.
C
cguida18:21
right, the info dump above is from a new sync i accidentally started.
here's the new one
2024-10-15T18:18:55Z.0 Body sizes... header :84847910 txs :3127058952 tx :48304299758 point :73453346676 input :181148659338 output :75376308520 puts :19056712575 candidate :2597385 confirmed :3 spend :48987159240 strong_tx :9349219308 valid_tx :0 valid_bk :0 address :0 neutrino :0 2024-10-15T18:18:55Z.0 Table records... header :865795 tx :779101609 point :2040370741 candidate :865795 confirmed :1 spend :2041131635 strong_tx :779101609 address :0 2024-10-15T18:18:55Z.0 Head buckets... header :524493 txs :524493 tx :551320125 point :546188501 spend :1459791875 strong_tx :551320125 valid_tx :551320125 valid_bk :524493 address :1 neutrino :1 2024-10-15T18:18:55Z.0 Version Information... libbitcoin-node: 4.0.0 libbitcoin-database: 4.0.0 libbitcoin-network: 4.0.0 libbitcoin-system: 4.0.0 2024-10-15T18:18:55Z.0 Press CTRL-C to stop the node.
C
cguida18:22
btw i'd love to know where you found a machine with 32GB RAM and 10 processor cores for $300
https://libbitcoin.slack.com/archives/C6LH74ACQ/p1729013014026969?thread_ts=1728563873.913659&cid=C6LH74ACQ
(edited)Eric Voskuil
As I mentioned earlier, I sometimes swap out my RAM down to 32GB and disable all but 10 CPU threads (roughly equivalent to the newer 32/8 thread machine that JL uses for benchmarking). With proper config this is still way faster than Core on the same machine.
evoskuil18:23
779,101,609 txs out of 991,032,192 (at block 840,000). But your config prob isn't set to stop at 840k.
👍 1
evoskuil18:25
It wouldn't need 10 today, that's because mine is 7 years old and that's how I approximate JL's 8 faster threads. 32/8 is what you'd be looking for. Also I have slowish RAM and SSD because of the old bus.
evoskuil18:27
I've got my PC back up now (been putting a new floor into my house), so I can help you out with some config options a little later (it's a tax day today).
evoskuil18:42
[log] path = X:/ application = true news = true session = true protocol = false quitting = false proxy = false remote = true fault = true verbose = false #objects = false # debug builds only [database] path = X:/bitcoin/ minimize = false [bitcoin] # defaults to all checkpoints. #checkpoint = 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f:0 minimum_work = 000000000000000000000000000000000000000052b2559353df4117b7348b64 milestone = 0000000000000000000320283a032748cef8227873ff4872689bf23f1cda83a5:840000 [forks] [node] headers_first = true allowed_deviation = 1.5 sample_period_seconds = 10 currency_window_minutes = 60 maximum_concurrency = 50,000 maximum_height = 840,000 # stops block sync (not header sync) snapshot_bytes = 0 snapshot_valid = 0 allocation_multiple = 20 [network] threads = 16 outbound_connections = 200 connect_batch_size = 5 host_pool_capacity = 10,000 enable_loopback = false self = xxx.xxx.xxx.xxx:8333 # advertised addresses inbound_connections = 0 # Use 1 for loopback user_agent = "/libbitcoin /" enable_ipv6 = false enable_alert = false # log alert messages enable_reject = false # send reject messages enable_address = true # in/out address messages enable_transaction = false # peer should not relay txs validate_checksum = false # validate inbound message checksum invalid_services = 176 # shitcoins services_minimum = 9 # witness node services_maximum = 9 # witness node protocol_maximum = 70012 # current default (send_headers). protocol_minimum = 31800 # Satoshi:22.0.0 minimum (headers). minimum_buffer = 1,000,000 # network payload buffer (per channel) retry_timeout_seconds = 5 # 1s, 2s, 7s f/m/s loopback, 5s for large connect_timeout_seconds = 5 # 1s, 3s, 5s f/m/s loopback, 10s for large handshake_timeout_seconds = 5 # time to complete handshake. channel_heartbeat_minutes = 5 # ping channel_inactivity_minutes = 10 # receive channel_expiration_minutes = 60 # unconditional [database] # TODO: These require tuning for low RAM machines. header_buckets = 565,129 header_size = 21,000,000 header_rate = 5 txs_buckets = 565,129 txs_size = 1,000,000,000 txs_rate = 5 tx_buckets = 681,968,291 tx_size = 16,000,000,000 tx_rate = 5 # compressed size point_buckets = 638,910,752 point_size = 9,000,000,000 point_rate = 5 spend_buckets = 1,739,749,749 spend_size = 16,000,000,000 spend_rate = 5 puts_size = 7,000,000,000 puts_rate = 5 input_size = 92,000,000,000 input_rate = 5 output_size = 25,000,000,000 output_rate = 5 candidate_size = 2,568,513 candidate_rate = 5 confirmed_size = 2,568,513 confirmed_rate = 5 strong_tx_buckets = 681,968,291 strong_tx_size = 3,000,000,000 strong_tx_rate = 5 validated_tx_buckets = 681,968,291 validated_tx_size = 1 validated_tx_rate = 5 validated_bk_buckets = 565,129 validated_bk_size = 2,000,000 validated_bk_rate = 5 address_buckets = 0 address_size = 1 address_rate = 0 neutrino_buckets = 0 neutrino_size = 1 neutrino_rate = 0
This is similar to what I run. The initial [database] table sizes definitely need to be reduced for low RAM. I would cut these
_size values down to 1/4 to 1/10 of above for lower RAM. This is required to prevent mmap thrashing. Also the growth _rate values must be small. I run at 15, but these are probably okay for low RAM at 5.evoskuil18:44
There are a bunch of other settings, which you can see from the command line using
(edited)--settings or -s.evoskuil18:45
You can use
m <enter> at runtime to see the runtime options and --help or -h at the command line to see the CL options (there's some overlap).evoskuil18:47
This just completed since we've been chatting. 3671 seconds (including 40 sec header sync) to block 840k.
evoskuil19:33
This is 8 dual cores (16 threads) at 4.75 GHz. That's better than my machine (2.1GHz) running at 32 threads. The RAM is twice as fast as mine (4800KHz vs. 2400), as is the SSD bus (PCIe4 vs. 3). This could probably sync the full chain on my WAN in ~1.5 hours, for $208 and free shipping.
(edited)
🔥 1
evoskuil20:08
Looks like the price is $356 for this model, never used Temu before. But as you can see, this is more than twice the machine that I minimum benchmark on.
C
cguida20:11
If you want people to switch from core to libbitcoin, "it syncs in 2.5 hours on a $300 box" is an amazing selling point
🎉 1
C
cguida20:15
Would you be willing to share the link to this? My "searching for cheap hardware" skills clearly need work... https://libbitcoin.slack.com/archives/C6LH74ACQ/p1729020838107709?thread_ts=1728563873.913659&cid=C6LH74ACQ
Eric Voskuil
This is 8 dual cores (16 threads) at 4.75 GHz. That's better than my machine (2.1GHz) running at 32 threads. The RAM is twice as fast as mine (4800KHz vs. 2400), as is the SSD bus (PCIe4 vs. 3). This could probably sync the full chain on my WAN in ~1.5 hours, for $208 and free shipping.
evoskuil20:19
evoskuil02:26
Appears to be fully as advertised and working well. Started a sync on the provided Windows 11 OS, using my big machine config (prob not optimal config but getting a baseline).
evoskuil03:19
Holy shit. This thing synced as fast as my 256GB/32 Dual Core 2.1 MHz box. No mmap thrashing despite big box config. This is prob because the disk and RAM are twice as fast as mine.
$356 delivered, syncs in one hour on Windows (without even having SHA-NI implemented, which will be 5x faster hashing). I noticed it does have AVX512, which libbitcoin supports, but the build I’m testing with is compiled for AVX2, which computes big Merkle roots in twice the time of AVX512). So the baseline time on this machine with 2Gbps Internet is going to beat one hour.
evoskuil03:30
Much faster than big machine in the header sync and before blocks get full, but almost the same after that. The bigger block speed on this box will also be faster with AVX512 and SHA-NI. It is absolutely maxing out my current Internet speed at 2300 Gbps.
🚀 2
Tuesday, October 15, 2024
evoskuil17:03
replied to a thread: view
I realize you are invested in this idea that Bitcoin itself cannot continue to operate on affordable hardware, but this is just not the case. This is something you should be celebrating.
evoskuil19:33
replied to a thread: view
This is 8 dual cores (16 threads) at 4.75 GHz. That's better than my machine (2.1GHz) running at 32 threads. The RAM is twice as fast as mine (4800KHz vs. 2400), as is the SSD bus (PCIe4 vs. 3). This could probably sync the full chain on my WAN in ~1.5 hours, for $208 and free shipping.
(edited)🔥 1
Thursday, October 17, 2024
Saturday, October 19, 2024
evoskuil03:35
I just completed my first sync on a new $365 Windows mini PC (32 GB RAM, 16 dual cores). It slightly outperformed my big machine (256 GB, 32 dual cores) at 61 minutes to block 840k. And it's capable of AVX512 and SHA-NI, which will make it even faster once those are compiled in. I don't see any reason to work toward lesser machine support given that this is so cheap, and will continue to drop in price.
🔥 2❤️ 1🎉 1
12 replies · jungly, evoskuil, cguida · last Oct 19, 2024
evoskuil13:14
Each mini thread is 62% faster than big machine threads, so 40% of the big machine processing power. The mini makes up for the difference by a much higher utilization (close to 100% vs big at around 35%). That nicely shows the implementation scaling.
evoskuil13:16
The most extraordinary aspect however, is the complete lack of mmap thrashing - even with big machine config. This has to be a consequence of the 2x faster RAM and 2x faster SSD, since the RAM is only 12.5% of the big machine.
(edited)evoskuil13:18
There may also be a benefit from Windows 11 vs 10, since mmap is just a layer over Win32 APIs.
evoskuil14:11
Yes, I did. This is with a “milestone” of block 840,000 - so comparable to Core’s “assume valid”.
However, this also includes a full bidirectional constant time index for all chain objects (Core has the option to index tx hashes).
evoskuil14:12
Also this was with no config tuning (I used my big box config) and no SHA-NI and no AVX512. Also note that libbitcoin contains no C code and no Assembly, 100% C++.
evoskuil14:14
There’s still work to do, I try to not overpromote. We are presently working on full validation optimization. That was struggling on smaller machines due to mmap thrashing. It may turn out that we can just ignore that by setting this as min spec.
🔥 1
evoskuil14:16
But it’s a great sign. What I have told people for some time is that the performance problems people see are largely due to bitcoind architecture, not an inherent problem of capacity.
(edited)evoskuil14:06
My big machine cost $15,000 almost 8 years ago (and my Internet cost remains constant), and the mini cost me $356. So the cost of syncing 840,000 blocks has been cut in half more than 5 times in eight years. That exceeds the Moore’s Law projection.
🔥 2
1 reply · evoskuil · last Oct 21, 2024
evoskuil14:08
Correction on the spec above, the mini is 8 dual cores (not 16), for a total of 16 threads.
evoskuil17:27
I’m now running Bitcoin Core v28 on the same machine. After 45 minutes it is just passing block 250,000. Notice from the table above that BN reaches that point in under 2 minutes.
evoskuil17:50
This is Core passing block 328,865 (< 5%) after 1 hour. Tx indexing is not even turned on, default config. I wonder if there's any config that can speed it up, like more peers.
M
Marnix18:57
For your libbitcoin sync, did you use the default 8 outbound connections? or more to optimize?
A couple years ago I ran a lot of tests on how long it takes to get to a targeted number of peers. I was often running up to 5,000 and 10,000 with results posted here. During that time it was all about rapid connection and reconnection, which is really tight. We can pick up 100 in a few seconds.
Of course since we are dividing the work among them, and dropping the slow peers in the context of block retrieval, we are load balancing much better than a slow node that hogs other slow nodes.
Sunday, October 20, 2024
2 replies · evoskuil, Charlie Spears · last Oct 21, 2024
Monday, October 21, 2024
/usr/bin/ld: /home/ava/bitcoin/libbitcoin/install/lib/libbitcoin-system.so: undefined reference to `tbb::detail::r1::spawn(tbb::detail::d1::task&, tbb::detail::d1::task_group_context&)' /usr/bin/ld: /home/ava/bitcoin/libbitcoin/install/lib/libbitcoin-system.so: undefined reference to `tbb::detail::r1::allocate(tbb::detail::d1::small_object_pool*&, unsigned long)' /usr/bin/ld: /home/ava/bitcoin/libbitcoin/install/lib/libbitcoin-system.so: undefined reference to `tbb::detail::r1::deallocate(tbb::detail::d1::small_object_pool&, void*, unsigned long, tbb::detail::d1::execution_data const&)' /usr/bin/ld: /home/ava/bitcoin/libbitcoin/install/lib/libbitcoin-system.so: undefined reference to `tbb::detail::r1::destroy(tbb::detail::d1::task_group_context&)' /usr/bin/ld: /home/ava/bitcoin/libbitcoin/install/lib/libbitcoin-system.so: undefined reference to `tbb::detail::r1::allocate(tbb::detail::d1::small_object_pool*&, unsigned long, tbb::detail::d1::execution_data const&)' /usr/bin/ld: /home/ava/bitcoin/libbitcoin/install/lib/libbitcoin-system.so: undefined reference to `tbb::detail::r1::execute_and_wait(tbb::detail::d1::task&, tbb::detail::d1::task_group_context&, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&)' /usr/bin/ld: /home/ava/bitcoin/libbitcoin/install/lib/libbitcoin-system.so: undefined reference to `tbb::detail::r1::initialize(tbb::detail::d1::task_group_context&)' /usr/bin/ld: /home/ava/bitcoin/libbitcoin/install/lib/libbitcoin-system.so: undefined reference to `tbb::detail::r1::is_group_execution_cancelled(tbb::detail::d1::task_group_context&)' /usr/bin/ld: /home/ava/bitcoin/libbitcoin/install/lib/libbitcoin-system.so: undefined reference to `tbb::detail::r1::execution_slot(tbb::detail::d1::execution_data const*)' /usr/bin/ld: /home/ava/bitcoin/libbitcoin/install/lib/libbitcoin-system.so: undefined reference to `tbb::detail::r1::max_concurrency(tbb::detail::d1::task_arena_base const*)' /usr/bin/ld: /home/ava/bitcoin/libbitcoin/install/lib/libbitcoin-system.so: undefined reference to `tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, long)' /usr/bin/ld: /home/ava/bitcoin/libbitcoin/install/lib/libbitcoin-system.so: undefined reference to `tbb::detail::r1::notify_waiters(unsigned long)' collect2: error: ld returned 1 exit status make: *** [Makefile:1228: console/bn] Error 1
any ideas?
couldn't find any reference to
tbb in libbitcoin-system which is confusing mepmienk16:34
I know I've seen this before, but can't recall off the top of my head. Could you provide some additional information, like platform and install command?
Ava Chow16:36
./install.sh --build-boost --build-dir=/home/ava/bitcoin/libbitcoin/build --prefix=/home/ava/bitcoin/libbitcoin/installarch does not appear to have a separate
-dev package for itpmienk16:41
It looks like it's presence may alter some common library and require its inclusion for linking but we don't require it ourselves and so do not check or link it. You can either add it via flags or remove the package to avoid the necessity
i guess it's something gcc is pulling in
👍 1👍🏻 1
evoskuil22:04
I ordered another little test box so I can run Ubuntu and Windows SxS. Unfortunately these boxes don't have AVX512, which we have implemented (though do have SHANI, which we don't have implemented). So same as my big box presently, just ASVX2.
I'm consistently hitting 46 minutes of a full chain (866k blocks) sync with default config on the Windows mini, and 61 minutes with my big box config. I'm working out what the difference is. Have shaved off 4 minutes by reducing the table growth rates to 5% (default) vs. 15%, and by reducing the outbound peer count from 200 to the default of 100. I suspect the rest is turning on automatic snapshotting, which I'll run now.
🔥 3
3 replies · evoskuil · last Oct 22, 2024
evoskuil17:51
Well I tracked this down. In our configuration we have settings for min and max services.
services_minimum = 9 # witness node services_maximum = 9 # witness node
This is how I've had my config file set up for a long time. However running with default config, the min is set to 1. Because this results in the node not requiring witness capable peers, this implies the node is not a witness node, and everything operates pre-witness serialization. This drops a large amount of data from the input table, where the witness is stored. I first noticed it there, and then figured out that node default value was 1, so when running without config we shaved off 15 minutes. So the reported 45 minutes is legit, but it's for non-witness (legacy node). The witness-included sync remains at about 60 minutes.
(edited)👍 2
evoskuil18:55
So I just completed a full run with the witness config, verified the result, and it came in at 47 minutes (to benchmark block 840k), probably because I took out the 5 snapshots from default config. I also recomputed the theoretical limit based on block download alone (no other messages up or down) of [609,888,208,331] bytes, on my (measured) 2.3Gbps Internet to the top block [866,881]. It comes out to 36 minutes, where that block took 52 minutes. So we're still only 68% efficient.
(edited)Tuesday, October 22, 2024
Дмитрий Дмитриевич14:00
libbitcoin-network
hi please help me make
you need to subscribe to inv messages and filter tx announcements and then query tx for the relay you want and subscribe to tx messages to get them
Maximal resource utilization looks something like this.
Дмитрий Дмитриевич17:47
// Single Outbound Channel.
bc::network::settings settings(bc::config::settings::mainnet);
settings.seeds.clear();
settings.seeds.push_back({ "seed.bitcoin.wiz.biz",8333 });
settings.seeds.push_back({ "dnsseed.bitcoin.dashjr.org",8333 });
settings.seeds.push_back({ "seed.btc.petertodd.org",8333 });
settings.inbound_connections = 0; // default setting = 0.
settings.outbound_connections = 2; // default setting = 8.
settings.host_pool_capacity = 100; // default setting = 0.
settings.threads = 3; // default setting = 0.
settings.services = bc::message::version::service::none; // default none.
5 replies · evoskuil, Дмитрий Дмитриевич · last Oct 23, 2024
evoskuil18:11
You'll need to instantiate an instance of the
p2p class, and implement a way to suspend the main thread until it completes (p2p starts its own thread pool). There are examples of this in the p2p class test cases.evoskuil18:12
Then you'll need to implement a transaction protocol. There are protocols implemented in the network library, but not for tx (there's no place to store them). You can use the others as examples, but a better example might be to look at libbitcoin-node protocols, as there are implementations for incoming blocks. You can make a much simpler one than those, but same idea.
evoskuil17:51
replied to a thread: view
Well I tracked this down. In our configuration we have settings for min and max services.
services_minimum = 9 # witness node services_maximum = 9 # witness node
This is how I've had my config file set up for a long time. However running with default config, the min is set to 1. Because this results in the node not requiring witness capable peers, this implies the node is not a witness node, and everything operates pre-witness serialization. This drops a large amount of data from the input table, where the witness is stored. I first noticed it there, and then figured out that node default value was 1, so when running without config we shaved off 15 minutes. So the reported 45 minutes is legit, but it's for non-witness (legacy node). The witness-included sync remains at about 60 minutes.
(edited)👍 2
evoskuil18:11
replied to a thread: view
You'll need to instantiate an instance of the
p2p class, and implement a way to suspend the main thread until it completes (p2p starts its own thread pool). There are examples of this in the p2p class test cases.evoskuil18:12
replied to a thread: view
Then you'll need to implement a transaction protocol. There are protocols implemented in the network library, but not for tx (there's no place to store them). You can use the others as examples, but a better example might be to look at libbitcoin-node protocols, as there are implementations for incoming blocks. You can make a much simpler one than those, but same idea.
evoskuil18:55
replied to a thread: view
So I just completed a full run with the witness config, verified the result, and it came in at 47 minutes (to benchmark block 840k), probably because I took out the 5 snapshots from default config. I also recomputed the theoretical limit based on block download alone (no other messages up or down) of [609,888,208,331] bytes, on my (measured) 2.3Gbps Internet to the top block [866,881]. It comes out to 36 minutes, where that block took 52 minutes. So we're still only 68% efficient.
(edited)evoskuil20:42
This is what I like to see when validating. 32 dual cores (64) at absolutely 100%. On the machine I'm posting this from. 🤷🏻♂️
(edited)
🎉 1
evoskuil20:59
It's interesting that we've been milestone syncing in under an hour for a year now, but significant interest came when I posted that we are doing it on a $356 machine. As if doing it on a 7 year old machine wasn't good. 🤔 But getting requests for validation stats now. So I've patched up the work I was doing in August and running it now on the big machine for a baseline.
The above finished the first 295,000 blocks in 382 seconds. I had the blocks downloaded from an old run, and by turning off checkpoints I can force it to validate them, but since milestone state is stored with the block, these cannot be bypassed. Starting a clean run now, but that is 3.88% of the 840k block txs, which puts it on pace to complete in 2.7 hours if linear. But the overhead in small blocks is much higher, so it's on pace for around 2 hours (excluding download).
We have the option to validate and download concurrently, which is optimal on big machines, because CPU is otherwise underutilized. On smaller machines it's probably best to only validate when downloads are "current". Then of course there's the confirmation step. Also, it's pointless to be validating checkpointed blocks, but it's sort of a benchmarking thing.
👍 1
Wednesday, October 23, 2024
evoskuil01:25
This is the flame graph during full chain download and validation on the big box. Everything starting with secp256 is signature verifications by the NuGet build of libsecp256k1. The smaller unnamed stacks to the right are (1) reading blocks to validate, (2) reading prevouts for validation, (3) block destruct, (4) block.accept(), (5/7) store block, (6/8) deserialize block.
(edited)On a big machine (fast/high RAM, fast SSD) it's clearly not worth attempting to cache deserialized downloaded blocks. Much better to just read them from the store as they are required.
Signature hashing is some of the tiny boxes to the right of signature verification stacks.
But as you can see it still adds up.
So SHANI will be a small boost for validation. An optimized build of libsecp is far more significant.
(edited)👍 2
evoskuil03:54
Despite massive work backlog, validation by block remains very consistent. The inflection point is just the transition from empty blocks to full blocks. This would be linear by tx.
(edited)
🔥 1
Thursday, October 24, 2024
evoskuil04:32
Did a little more validation performing evaluation today. We have a lot of flexibility in terms of how to run it concurrently with download and/or confirmability. Presently it looks like concurrent download/validation is optimal for high CPU machines (64 thread maxi) and not for low (16 thread mini). But this is relative to how much CPU is unused during download, which is in turn a function of RAM/SSD/WAN. I've achieved 4.5 hours with maxi concurrent validation (no confirmability checks) and have hit thrashing attempting the same on the mini. With CPU generally pegged on the mini just for download, it makes more sense to defer validation until download is current. This prevents the heavy reading of validation conflicting with the heavy writing of download (and therefore thrashing due to moderate RAM), which is not achieving any boost because CPU is already maxed out.
evoskuil04:38
Basically we want to make use of excess (wasted) CPU during download, as long as it doesn't cause thrashing due to low/slow RAM and/or slow SSD. Otherwise we want to wait for download currency and put everything into validation. Validation itself will not cause thrashing because it's around 99.9% read (it will read about 130% of the store, all txs and all prevouts) and .1% write (validation states). Also it may make sense to eliminate tx validation states for block validation, as the granularity is probably a net loss apart from pool txs, since we are concurrently validating at the block level. This would save a lot of the writing, which presently isn't ever getting read.
Validating after download currency on the maxi cost an extra 1.5 hours, more than just the 1 hour to download first. possibly due to lower read proximity. So the concurrent model works, as long as there is excess CPU and sufficient RAM.
I suspect we may get different numbers on Linux and macOS due to the differences in mmap and also possibly CPU optimization in lipsecp256k1. The Windows builds are using a fairly old NuGet build that we maintain. So I'm going to get my new Linux mini set up, my old Linux maxi, and my monster Mac that I've hardly ever used.
👍 1
evoskuil04:55
Will also perform a confirmation run just to see where we are. IIRC it was like 30 minutes or so on the maxi box and very excessive when I trimmed back to 32GB/10 cores. So it will be nice to test that out on the mini, which I think will perform well since it's all query. The mini is outperforming the maxi when it's not CPU bound, which confirmability is not. So with confirmability my guess is that we'll be at around 5 hours for full validation with download on the maxi. On the mini the CPU is about 38% of the maxi (16 x 3.2 vs. 64 x 2.1), and download is additive, so a full dl/validate/confirm run should take 1 + (5/.38) + .5 = 14.7 hrs. Of course there's still SHANI to add, which will reduce that. Core takes 15 hours to sync on the mini, without script validation.
🔥 1
evoskuil16:23
Latest maxi milestone run to block 840k with full Electrum indexation is 1.6 hours, or 39 minutes over the average non-Electrum run. This is over the 2.3Gbps Internet concurrent with full chain sync, and adds 116,938,826,676 bytes plus the 7,298,959,380 byte header (sized for 1.9 collision rate) to the store.
A snapshot of the full store took 68 seconds cold and 37 seconds warm, and full stop took 35 secs. Restore from snapshot took 22 seconds. For those who are new to BN, snapshots can be performed hot (without a stop), and the full store can be hot-backed up to a snapshot point(s).
(edited)6 replies · cguida, evoskuil · last Oct 25, 2024
evoskuil21:29
A snapshot is a flushed copy of the table heads, concurrent with the flush to disk of all table bodies. A snapshot looks exactly like the “head” subdirectory and rotates between a “primary” and “secondary” folder. This ensures that once there is a good snapshot, a failed future snapshot can always revert to the previous good one.
The purpose is to allow the store to revert to the point where the snapshot was taken, in the case of a process fault (uncontrolled termination). I also use it in development to roll the store back to certain points (such as post-sync and pre-validate, by storing a snapshot once synced, though on big machines we sync and validate concurrently).
The node knows for sure when a previous shutdown has not been completed, and when it starts up will automatically recover from the most recent snapshot. If there is no snapshot it will currently require a manual restart after clearing the store. This gives us full fault tolerance, which leveldb (and most general database systems) cannot actually provide. This is pretty extraordinary given that the store is fully write/write and read/write concurrent.
(edited)👍 1
evoskuil21:33
The other benefit of snapshots is that the store can be hot-backed up. The snapshot is written using file system atomicity, which ensures that if there is a snapshot directory it is good. So for example the store does not need to be stopped for backup to another drive by a backup system.
evoskuil21:37
Automatically snapshotting 6 times during a sync adds about 5-10 minutes to the total time and ensures that recovery in a run that would take 60 minutes never loses more than 10 minutes.
🔥 1
evoskuil21:45
Of course that can be a much more significant time savings on a slower machine/network.
evoskuil16:32
Fault detection is assured and recovery from the most recent snapshot is automatic upon restart. There is no spot-checking of the store for corruption at startup, as the store cannot become corrupt without detection, short of hardware unreliability. However, if the hardware is unreliable, nothing reliable can be gleaned from using the hardware and certainly spot-checking (e.g. Bitcoin Core at startup) is also an unreliable technique.
A
Antoine Poinsot18:45
Does Libbitcoin have a fixed number of outbound peers that it tries to maintain like Bitcoin Core?
2 replies · pmienk, evoskuil · last Oct 25, 2024
evoskuil00:24
Yes, the default is 100 and that's what I've been benchmarking with. Headers are competitively downloaded from all, and then peer performance is managed for download across the set using standard deviation. We will probably allow the count to vary based on whether the node is current or not (and ideally based on more robust perf measurement), but presently it is fixed via config.
A
Antoine Poinsot19:36
I'm trying to understand the process of catching up to the latest tip implemented in Libbitcoin, from the description in this thread: https://x.com/evoskuil/status/1847736160317042874.
- From the current tip of the node, it will request headers from all its peers (which will likely be redundant), extending the best chain of headers.
- As soon as the best chain of headers is updated (extended or reorged), the node will query the block content. Block content queries are spread among all the node's peers.
- Upon reception, blocks are statically checked (block size, coinbase tx, etc..).
- The node then checks whether all transactions in the block are valid and if they are it adds them to a global index of valid transactions.
- ? I suppose you have a mechanism to mark blocks as fully validated in the header chain, to be able to trigger the final step of validation after all blocks before a certain point were fully validated ?
- Once all blocks in the best headers chain are checked (valid block + valid txs), you go through the chain to check there were no double spend.
Is that correct, as a general description? The large speedup compared to Core comes from the fact that steps 2. to 4. can be performed concurrently. Now i still need to more precisely document what indexes you keep to be able to perform this concurrently, but i wanted to make sure i had the big picture right already.
Huddle with pmienk, evoskuil · 82 min19:58
31 replies · evoskuil, Antoine Poinsot · last Oct 26, 2024
evoskuil00:27
The "best" chain is the strongest (work) valid header chain we are seeing. Currency is a configurable time from (order than) the host's wall clock. This defaults to 1 hour (ie approximately 6 blocks from the top is "current").
evoskuil00:29
#3 (block.check) isn't strictly limited to static (context free) checks, there are some that require context (fork activations).
evoskuil00:36
#5 There is a table for block validation/confirmability state [
validated_bk.data], and one for tx [validated_tx.data], as well as a table for tx confirmedness [strong_tx.data] - which provides the reverse navigation from tx to block (only for confirmed txs). validated_tx is really only an optimization for pool/compact block processing. It records distinct validation states for potentially incompatible contexts (such as validated for the pool, but needed to be validated again because the pool context was no longer consistent with a block context).evoskuil00:38
#6 we don't wait until all are complete, we will validate concurrently with parallel downloading and confirm concurrently with validation and downloading - as applicable. We will likely wait until complete for lower capacity machines, as otherwise they can get into mmap thrashing. But the big box handles all three processes concurrently quite well.
evoskuil00:38
And of course header sync is always ongoing concurrently as well. All of these processes are continuous and adapt to reorganization and disorganization (where a block is found to be invalid/unconfirmable).
(edited)evoskuil00:44
I can't say specifically where the big performance difference is. There are a lot of things that are necessary to make this level of concurrency feasible, such as a very efficient write-concurrent store. But the ability to process downloads and validations highly-concurrently is certainly the main issue. With respect to lower level isolated optimizations, I expect Core is better. We're running this without SHANI (Core has it in our benchmark) and with a 3 year old copy of libsecp256k1 for example. Since those don't affect the architecture they've been deferred.
evoskuil00:47
The easiest way to see it is in the resource underutilization. Core is barely using anything on my big box. It should always have at least one resource maxed out. During download that should generally be bandwidth, and during validation should generally be CPU. Since there is significant excess CPU during download, we validate concurrently to capture that.
(edited)A
Antoine Poinsot21:27
Thanks. I'm still missing important details with regard to your data structures. Conceptually you need to keep an index of txid to transaction to be able to validate a block and an index of outpoint to spend status to be able to confirm it.
There is this block validation/confirmability table which you explained above, but as far as i understand it points to which transactions are validated/confirmed, not which of their outputs are spent or not.
You also referred to "context" a few times which i'm not sure what it is concretely. If i remember correctly you mentioned you could reorg for ~ free, so this might be related.
evoskuil21:36
meta: sure, we don't have that much traffic, so I like to surface good public explanations to general. But up to you.
👍 1
evoskuil21:46
Yes, we have complete indexes in addition to validation metadata. The two "validated" tables are disposable, and in the case of reorgs would just get generated as necessary. Essentially they are state repositories that assist with concurrency. The "strong" table is not disposable, and only provides the reverse index from (confirmed) tx to its confirming block. This gets overwritten (to negative) in the case of a reorg out of the block.
Navigation is possible from candidate or confirmed height to header, header to parent, header to txs (if block is stored), txs to spends (sequence) and outputs (script/value), spend to input (contains script/witness), input to point (output natural key), output to containing tx, spend to containing tx, tx to confirming header (and position in its block).
evoskuil21:52
All of these navigations are basically just table-relative memory pointer indirections, apart from the natural (hashmap) key indexation, which exists for header[hash], txs[header.hash] (txs is an association table between header and its tx entries), tx[hash], point[hash] (point is a table of prevout hashes used to collapse the redundant values to surrogate keys), spend[composite key of point surrogate key and output index], and optionally the electrum table of address[output script hash].
evoskuil21:55
The tables are either blob or array (variable or fixed record size), and can be associated directly by surrogate key, and/or indexed by a hashmap (header with conflict set links internal to the records). You can see those in database/primitives, along with an iterator.
evoskuil21:57
Given this structure we can navigate all aspects bidirectionally in constant time. The point table saves about 50 GB of redundant hash storage, but requires a large memory machine during sync. Despite all of the indexation it's actually very space efficient.
evoskuil22:01
These tables are all written concurrently. My machine writes 64 blocks at a time, spanning all of the tables while doing it. But at the same time, we have the ability to safely hot backup and ensure data consistency in the case of an unexpected process termination. These were the biggest problems with prototyping work I did back in 2017-18, the table headers and bodies were attached, so we had no way to assure fault recovery.
evoskuil22:04
Correction, the txs table uses the header surrogate key as it's search key, not the header.hash (much smaller and total hashmap collision control).
(edited)evoskuil22:07
Similarly, the validated_tx, validated_bk and strong_tx hashmaps all use surrogate search keys.
(edited)evoskuil22:35
Context is the chain-relative data necessary to validate a block/txs, and the assumed-next-block context when validating for the tx pool (which may turn out to be incorrect). This includes active forks, mtp, height. We also have a concept of chain_state (prob not the same as Core), which is used to retain the information necessary to validate headers, including the accumulation of history necessary for things like mtp and signal accumulation. This allows us to roll it forward without hitting the store, and from it we generate context for each block.
(edited)👍 1
A
Antoine Poinsot22:37
prob not the same as Core
Yeah for Core, chainstate generally refers to the UTxO database.
👍🏻 1
evoskuil22:42
Since the context for a given header can never change, we store the essential values with the header when the candidate header is committed to the store. Then context is available later for independent block.check/accept/connect/confirm transitions.
👍 1
A
Antoine Poinsot22:50
Oh, is that how you detect it is unspent, by the absence of an entry in the spend index ("sequence")?
evoskuil23:35
Spends is just an intermediate table between tx and input, it carries the sequence and pointers. Sequence is just the tx.sequence field. The index is not stored, it's just an array offset in the puts table, which associates tx->spends->inputs and tx->outputs. It's important to factor output/input script/witness from the other tables due to their size, as that dramatically reduces paging/reads. input.sequence and output.value are required at different validation phases, which is they aren't stored symmetrically.
(edited)👍 1
evoskuil23:39
There can be any number of spends associated to an output. The spends table is read as a multimap against a given point. So for spentness we iterate the spending txs (spend.parent_tx surrogate key) and determine if they are confirmed by search of the strong_tx table with those keys (pointers). So by unconfirming (setting false in strong_tx) all spends of that tx automatically become unconfirmed spends (still spent, but not confirmed spent).
evoskuil23:43
We do not need to know if a previous output is confirmed, or if there are multiple spenders of an output, when we validate. If we have the prevout (just in the store for any reason) we validate. Spentness and maturity are determined in a completely decoupled step, which is just a query. That confirmability step is the most strongly ordered, and download is the least strongly ordered.
A
Antoine Poinsot01:24
Yes, i've now grasped the separation between validation and confirmation in Libbitcoin. I'm asking about the confirmability step. Using what table/index do you check the inputs of each transaction in a block?
evoskuil01:25
strong_tx indicates txs associated to confirmed branch blocks. We call then strong vs. confirmed, because they aren't actually confirmed until in the confirmed height index.evoskuil01:26
These are not part of "archive" data because they aren't necessary to store/retrieve txs or blocks, they are metadata about them.
evoskuil21:29
replied to a thread: view
A snapshot is a flushed copy of the table heads, concurrent with the flush to disk of all table bodies. A snapshot looks exactly like the “head” subdirectory and rotates between a “primary” and “secondary” folder. This ensures that once there is a good snapshot, a failed future snapshot can always revert to the previous good one.
The purpose is to allow the store to revert to the point where the snapshot was taken, in the case of a process fault (uncontrolled termination). I also use it in development to roll the store back to certain points (such as post-sync and pre-validate, by storing a snapshot once synced, though on big machines we sync and validate concurrently).
The node knows for sure when a previous shutdown has not been completed, and when it starts up will automatically recover from the most recent snapshot. If there is no snapshot it will currently require a manual restart after clearing the store. This gives us full fault tolerance, which leveldb (and most general database systems) cannot actually provide. This is pretty extraordinary given that the store is fully write/write and read/write concurrent.
(edited)👍 1
evoskuil21:33
replied to a thread: view
The other benefit of snapshots is that the store can be hot-backed up. The snapshot is written using file system atomicity, which ensures that if there is a snapshot directory it is good. So for example the store does not need to be stopped for backup to another drive by a backup system.
Friday, October 25, 2024
evoskuil00:24
replied to a thread: view
Yes, the default is 100 and that's what I've been benchmarking with. Headers are competitively downloaded from all, and then peer performance is managed for download across the set using standard deviation. We will probably allow the count to vary based on whether the node is current or not (and ideally based on more robust perf measurement), but presently it is fixed via config.
evoskuil00:27
replied to a thread: view
The "best" chain is the strongest (work) valid header chain we are seeing. Currency is a configurable time from (order than) the host's wall clock. This defaults to 1 hour (ie approximately 6 blocks from the top is "current").
evoskuil00:29
replied to a thread: view
#3 (block.check) isn't strictly limited to static (context free) checks, there are some that require context (fork activations).
evoskuil00:30
replied to a thread: view
#4 is part of #3 (validity at this point is limited to block/tx.check)
evoskuil00:36
replied to a thread: view
#5 There is a table for block validation/confirmability state [
validated_bk.data], and one for tx [validated_tx.data], as well as a table for tx confirmedness [strong_tx.data] - which provides the reverse navigation from tx to block (only for confirmed txs). validated_tx is really only an optimization for pool/compact block processing. It records distinct validation states for potentially incompatible contexts (such as validated for the pool, but needed to be validated again because the pool context was no longer consistent with a block context).evoskuil00:38
replied to a thread: view
#6 we don't wait until all are complete, we will validate concurrently with parallel downloading and confirm concurrently with validation and downloading - as applicable. We will likely wait until complete for lower capacity machines, as otherwise they can get into mmap thrashing. But the big box handles all three processes concurrently quite well.
evoskuil00:38
replied to a thread: view
And of course header sync is always ongoing concurrently as well. All of these processes are continuous and adapt to reorganization and disorganization (where a block is found to be invalid/unconfirmable).
(edited)evoskuil00:44
replied to a thread: view
I can't say specifically where the big performance difference is. There are a lot of things that are necessary to make this level of concurrency feasible, such as a very efficient write-concurrent store. But the ability to process downloads and validations highly-concurrently is certainly the main issue. With respect to lower level isolated optimizations, I expect Core is better. We're running this without SHANI (Core has it in our benchmark) and with a 3 year old copy of libsecp256k1 for example. Since those don't affect the architecture they've been deferred.
evoskuil00:47
replied to a thread: view
The easiest way to see it is in the resource underutilization. Core is barely using anything on my big box. It should always have at least one resource maxed out. During download that should generally be bandwidth, and during validation should generally be CPU. Since there is significant excess CPU during download, we validate concurrently to capture that.
(edited)evoskuil00:54
I'm seeing some warnings coming up in GCC 13.2 (default Ubuntu). We're running 11.4 in CI, which isn't much order (2 months) in terms of release dates. @pmienk do we have an upper bound on GCC support presently?
150 replies · pmienk, evoskuil, parsevalbtc · last Oct 29, 2024
pmienk01:01
I remember either clang or gcc has an issue after a certain version. I thought it was clang though
evoskuil01:29
Looks like all of the warnings are either in Boost (mostly std::unary_function<> deprecated) or in std::lib.
evoskuil02:05
Everything is running, and seems blazing fast, but I'm getting frequent stalling on blocks.
evoskuil02:10
IOW I believe that in the hardware avx512 implies avx2 implies sse4. It's not imposed by my code, they are independent flags. So I assume it's probably that way for yours.
pmienk02:18
yes, passively passed. they are independent flags from the perspective of configuration. while hardware-wise i'm unaware of any system which has avx512 but not avx2, there's no ripple checking done so if the code assumes avx512 means avx2, the flag for avx2 wouldn't have been set and the test wouldn't have been run
evoskuil02:19
This stalling is frustrating. Never happens in Windows builds, ever. Really have no tools to troubleshoot it.
pmienk02:24
translates to:
avx512 - avx512f, avx512bw
avx2 - avx, avx2
shani - msse4, msha
sse4 - sse4.1
evoskuil02:37
AH, maybe that's the only difference. I'll try that now with the hardware flags rebuild.
👍 1
evoskuil02:43
Still one warning (twice) in system, but that's coming out of my use of std lib, so I'll track that down, otherwise clean.
(edited)evoskuil02:46
This is coming up in all of the repos except for system. But each of the libs needs to compile with the options right?
pmienk02:47
yeah, that'll spit out for every repo that doesn't recognize them, which at min will be secp256k1, but probably some of our repos above system
evoskuil02:49
But does that mean these will be affected in each repo? --enable-avx2, --enable-sse41...
pmienk02:50
no. it means if someone were to manually try to build a project and already had system with those options, then the system options would be used. no point in having two configuration flags passed that disagree
evoskuil02:53
Ok, that's what I meant. The system options derived from our the args passed to system will cascade so that all libs get the same compiler options (and our preprocessor symbols).
evoskuil02:56
checking --enable-avx2 option... yes checking --enable-avx512 option... no checking --enable-sse41 option... yes checking --enable-shani option... no checking --enable-ndebug option... yes checking --enable-isystem option... yes
evoskuil02:57
$ ./bn --hardware Intrinsics... arm..... platform:0. intel... platform:1. avx512.. platform:0 compiled:0. avx2.... platform:1 compiled:0. sse41... platform:1 compiled:0. shani... platform:1 compiled:0. neon.... platform:0 compiled:0.
pmienk03:04
no, and because of the depth of one dependency, they get squished out. which would imply anything passed that way would require restatement in the pc file
pmienk03:07
configure.ac only references explicit dependencies from the project. node reaches to network and database, network reaches to system
pmienk03:09
the xml is great as is. the configure.ac and Makefile.am just need to explicitly reference the full dependency tree
pmienk03:10
and they're there. the pc file just isn't coming in to the dependencies that aren't explicit.
pmienk03:11
yeah, thought it was fixed. thought it was verified. must have been for the database project and so this got missed
evoskuil03:13
K, I'll go back to building my performance lab. You can verify the PP symbols by running
bn --hardwarepmienk03:44
i build with
--enable-avx2 --enable-sse41 and
$ ./bn --hardware Intrinsics... arm..... platform:0. intel... platform:1. avx512.. platform:0 compiled:0. avx2.... platform:1 compiled:1. sse41... platform:1 compiled:1. shani... platform:0 compiled:0. neon.... platform:0 compiled:0.
pmienk03:55
well, as long as there is nothing special between the two scripts, which there shouldn't be, all i did was
--enable-avx2 --enable-sse41 --enable-isystempmienk03:58
a parameter within the package config pc file which is how the system pc file is supposed to be brought in
evoskuil04:13
Are you sure that the
-m arguments are making it into each/any lib, given that they don't show in any of the pc files?evoskuil04:14
I guess I could test by compiling for avx512 and letting it crash, but that wouldn't tell me for sure it got into more than just system.
evoskuil04:17
The issue is that we would get the optimizations in system only, and can't really tell without inspecting the disassembly.
pmienk04:17
i guess the question is, are there headers/templates that would require it or is this a linking to the static/dynamic lib that was generated with the optimizations already?
pmienk04:18
thought we had a discussion about this before, and because of that we concluded that the compiler flags were excluded but the declares were passed on
evoskuil04:20
The declares initially were not passed on, and I thought it was enough to just have them in system, but the prob is that the system header gets included in the other builds, and the symbols in the system header are defined independently for each lib. So I added the --hardware option to verify it in the node project.
evoskuil04:21
But the compiler flags are opaque to me, I have to assume each library must be compiled with the flags, and otherwise those optimizations will just not be present in the libs without the flags.
evoskuil04:25
Unrelated: I can't find it in the log. Do we incorporate default compiler optimizations into the script?
pmienk04:28
no, the install scripts don't. we'd have a collision detection problem if someone passed anything via CFLAGS/CXXFLAGS
evoskuil04:30
Right, and in v4 it's a critical setting. So gotta make sure we document. I think we decided -O3 right?
evoskuil04:33
So now I'm up to:
--build-boost --disable-shared --enable-avx2 --enable-sse41 --enable-isystem CFLAGS=-O3 CXXFLAGS=-O3
pmienk04:34
well,
(edited)CFLAGS=-O3 CXXFLAGS=-O3 ./install.sh --build-boost --disable-shared --enable-avx2 --enable-sse41 --enable-isystempmienk04:35
I don't believe it would pick up the flags if they're declared after parameters. needs to prefix the script execution
evoskuil04:37
configure options: --with-boost=/home/nixmini/Prefix --with-pkgconfigdir=/home/nixmini/Prefix/lib/pkgconfig --prefix=/home/nixmini/Prefix --disable-shared --enable-avx2 --enable-sse41 --enable-isystem CFLAGS=-O3 CXXFLAGS=-O3 --enable-static
evoskuil04:38
That's from our build log, and also libsecp:
onfigure: WARNING: unrecognized options: --enable-avx2, --enable-sse41, --enable-isystem Build Options: with external callbacks = no with benchmarks = yes with tests = no with ctime tests = no with coverage = no with examples = no module ecdh = yes module recovery = yes module extrakeys = yes module schnorrsig = yes module ellswift = yes asm = x86_64 ecmult window size = 15 ecmult gen table size = 86 KiB valgrind = no CC = gcc CPPFLAGS = -w SECP_CFLAGS = -O2 -std=c89 -pedantic -Wno-long-long -Wnested-externs -Wshadow -Wstrict-prototypes -Wundef -Wno-overlength-strings -Wall -Wno-unused-function -Wextra -Wcast-align -Wcast-align=strict -fvisibility=hidden CFLAGS = -O3 LDFLAGS =
evoskuil04:39
So it's not complaining about the FLAGS as unknown, and reports them, and also libsecp is outputting the CFLAGS. If you do it prior to install it sets the process environment, but I think the way we are passing args it also would work.
(edited)pmienk04:40
I'm surprised, as I don't know of that being documented. only the environment declearations
evoskuil04:41
This is what libscp output on the run without me setting them:
Build Options: with external callbacks = no with benchmarks = yes with tests = no with ctime tests = no with coverage = no with examples = no module ecdh = yes module recovery = yes module extrakeys = yes module schnorrsig = yes module ellswift = yes asm = x86_64 ecmult window size = 15 ecmult gen table size = 86 KiB valgrind = no CC = gcc CPPFLAGS = -w SECP_CFLAGS = -O2 -std=c89 -pedantic -Wno-long-long -Wnested-externs -Wshadow -Wstrict-prototypes -Wundef -Wno-overlength-strings -Wall -Wno-unused-function -Wextra -Wcast-align -Wcast-align=strict -fvisibility=hidden CFLAGS = -g -O2 LDFLAGS =
evoskuil04:45
Libbitcoin boost configuration. -------------------------------------------------------------------- variant : release threading : multi toolset : boost cxxflags : cxxflags=-Wno-enum-constexpr-conversion boost linkflags : link : static boost.locale.iconv : on boost.locale.posix : on -sNO_BZIP2 : 1 -sNO_ZSTD : 1 -sICU_PATH : -j : 16 -d0 : [supress informational messages] -q : [stop at the first error] --reconfigure : [ignore cached configuration] --prefix : /home/nixmini/Prefix BOOST_OPTIONS : --with-chrono --with-iostreams --with-json --with-locale --with-program_options --with-regex --with-system --with-thread --with-test cxxflags (ignored) : --------------------------------------------------------------------
evoskuil04:45
libbitcoin-node installer configuration. -------------------------------------------------------------------- OS : Linux PARALLEL : 16 CC : CXX : CPPFLAGS : CFLAGS : CXXFLAGS : LDFLAGS : LDLIBS : WITH_ICU : BUILD_ICU : BUILD_BOOST : yes BOOST_ROOT : /home/nixmini/Prefix BUILD_DIR : build-libbitcoin-node PREFIX : /home/nixmini/Prefix DISABLE_SHARED : yes DISABLE_STATIC : with_boost : --with-boost=/home/nixmini/Prefix with_pkgconfigdir : --with-pkgconfigdir=/home/nixmini/Prefix/lib/pkgconfig --------------------------------------------------------------------
pmienk04:48
# If coverage is enabled, and the user has not overridden CFLAGS, # override Autoconf's value "-g -O2" with "-g". Otherwise we'd end up # with "-O0 --coverage -g -O2".
pmienk04:49
for them to have picked up in boost, it had to treat them as environment because the parameters wouldn't have made it to the relevant calls that way
evoskuil04:53
There's no positive log feedback in the libbitcoin repos, but it's def in libsec and boost.
pmienk04:57
I don't know what switched the order on master vs version3, that wasn't how it was documented when the docs were scrubbed
pmienk05:10
ha, no idea. i never touched system/master README.md, so maybe it's a correction I made to version3
P
parsevalbtc08:16
I remember I had something similar back this summer. Eventually @pmienk solved the issue, at least for me. https://github.com/libbitcoin/libbitcoin-node/issues/654
2 replies · cguida · last Oct 25, 2024
C
cguida03:56
Wow, the deserialized blocks are huge! I guess this is because of all the metadata and indexes
evoskuil02:29
@pmienk today updated the build scripts (install.sh) to pull from latest libsecp256k1 in the origin repo, as we are able now to phase out our versioning branch (they now version it). This should be a nice performance boost for validation, since the version we've been using is 3 years old.
I got a second mini box and today installed Ubuntu latest. I pulled down a build script and completed and started up just fine. However it is stalling frequently, which I have not seen over the past year in the msvc build. Also the events.log is created but not populated. So have to track those down. Otherwise everything appears to be great, and speed looks impressive.
For anyone building for Linux or macOS, the install.sh accepts hardware optimization parameters:
--enable-avx512 --enable-avx2 --enable-sse41
There is also one for --enable-shani, but that is not yet implemented. These are implemented and incorporated into the test matrix for libbitcoin-system. These significantly reduce Merkle hashing time, which materially affects milestone runs (as well as full validation). My mini boxes don't have avx512. To determine what level of support is on a platform, and what level is compiled into bn, you can run bn -d and it will tell you. But it you attempt to run for hardware you do not have, the process will fail on startup.I'm setting up my monster Mac for performance testing no as well, should have some info on that tomorrow.
👀 1👍 1
evoskuil06:11
Also, for any install.sh build, maximal compiler optimizations (
-O3) are performance critical. Also --enable-isystem cleans up some boost warnings.
$ CFLAGS=-O3 CXXFLAGS=-O3 ./install.sh --build-boost --disable-shared --enable-avx512 --enable-avx2 --enable-sse41 --enable-isystem(edited)
pmienk06:23
It should be noted that on systems with multiple side-by-side compiler versions,
(edited)--enable-isystem may cause failure. This is due to a directory precedence issue in compilers.👍🏻 1
evoskuil14:24
I found that the stall in the linux runs is temporary, apparently at the end of each download window. It completed overnight. I'm also not seeing any entries in the event log. But by setting
(edited)maximum_concurrency = 0 (full download concurrency) the run does not appear to stall. Hard to tell speed without the events, but I'll at least get a ballpark completion time.evoskuil16:19
Looks like there's still a serious level of work starvation going on in the linux build. Work per channel is way too low and it's running fast but with large gaps (see disk). So I think there's something going on in work allocation.
At this rate it will take about 3 hours to complete. But the mmap performance seems significantly better. Store build, backup/restore, shutdown... all significantly faster than the Windows runs.
👍 3
evoskuil19:00
Spent an hour tracking down and documenting three new bogus gcc 13.2
Wstringop-overflow warnings:
https://github.com/libbitcoin/libbitcoin-system/issues/1539
https://github.com/libbitcoin/libbitcoin-system/issues/1540
https://github.com/libbitcoin/libbitcoin-system/issues/1539
And one new bogus gcc 13.2 copy warning:
https://github.com/libbitcoin/libbitcoin-system/issues/1538
and closed the previous ones from gcc12 which have disappeared on their own:
https://github.com/libbitcoin/libbitcoin-system/issues/1461
Never impressed with the gcc static bounds checker.So the full BN build is "clean" on Ubuntu latest with default gcc, apart from 4 bogus warnings in system. Now back to work. 🙄
(edited)evoskuil19:46
I figured out why the Linux build was stalling (not emitting notifications) and not emitting events to the high speed event log. The two lines that emit the event and the notification were commented out in the current build, and I had been running the Windows builds one PR ahead of master. So nothing to do with Linux/Windows, same behavior.
I cancelled that larger PR and the fix is now merged here:
https://github.com/libbitcoin/libbitcoin-node/pull/689
evoskuil21:29
Still seeing a high level of starvation in the Linux build, similar to previously with
(edited)maximum_concurrency = 0. But the stall and lack of events are resolved. I don't have any tools set up for concurrency and performance analysis on Linux, so I've probably gotta work that out. Setting up my monster mac now also.
In the mean time I pulled down Bitcoin Core latest Linux binary to get a benchmark. It's doing significantly better than it did on the same box for a Windows build (15 hours). After 30 minutes it's completed about 10% of the chain (by txs), and projecting 6 hours (and falling). So will probably hit 5 hrs. Interesting how it can be 3x longer on the same box for Windows vs. Linux.25 replies · Antoine Poinsot, evoskuil, cguida, parsevalbtc · last Oct 29, 2024
evoskuil22:18
Today is the first time I've attempted a Linux sync with the v4 architecture. I work almost exclusively on Windows, because of the tooling, and because my habit patterns were set well before Linux existed. 😅
I think that Core still builds for Windows using mingw, which is sort of an afterthought approach, and only supports CMake now. We support native Windows (msvc/win32), macOS (clang/xcode), Linux (gcc/clang) with generated build artifacts, CI/coverage and build scripts for Automake and CMake, native VC++ solutions for Visual Studio Community, and presets for VSCode. If we had CI platforms for Big Endian and for ARM we'd have those supported too (working on it).
👀 1
A
Antoine Poinsot22:20
I think Core has a msvc build system, but i'm not the right person to ask about it. 🙂
evoskuil22:20
So I'm not at all surprised that my Linux box is not < 1hr yet. I was happy just to see it complete a sync the first time it was ever attempted!
evoskuil22:21
Actually, yes, I think that was a product of switching to CMake, since it can generate a msvc build. Not sure if the latest binaries are built with that, but seems to be a pretty significant difference.
A
Antoine Poinsot22:23
We switched to cmake after the branch-off for v29 (the latest release). But i'm pretty sure there was already a msvc build supported. But maybe not for releases, i'd have to check.
👍🏻 1
A
Antoine Poinsot14:37
Sharing your bench on Core's IRC i got more info about that FWIW.
10:31 <laanwj> and it makes sense, bitcoind is optimized for various POSIX platforms, not windows, which is different in many aspects; for example leveldb, doesn't even really support windows by itself; the win32 API backend it uses is something i worked on. also the P2P networking doesn't use any i/o polling primitives optimized for windows. Disk i/o is also slower on a lot of windows machines due to windows defender etc getting in the way. To get any level of 10:31 <laanwj> performance there would need a lot of work. Still, it shouldn't be that slow... 10:34 <laanwj> compiling with MSVC instead of mingw also used to be an improvement in speed, but this was 2013, no idea about now, and we can't use that in deterministic builds anyhow 10:36 <hebasto> MSVC builds lack HW-acceleration for SHA256 -- https://github.com/bitcoin/bitcoin/pull/24773 10:36 <darosior> He did mention MSVC vs mingw and i quickly got out of my depth. :) 10:36 <darosior> What do we use for release builds? 10:36 <hebasto> mingw 10:36 <laanwj> mingw, because that's the only thing we can use in side guix 10:36 <laanwj> well we could use clang that would be something to investigate, many seem to prefer that for windows nowadays 10:37 <hebasto> right
👍🏻 1
evoskuil14:44
Thanks, good read. Mostly what I assumed. This is mostly for us, I don’t really have the bandwidth to work on Core as well. But if our observations can help that’s great.
Note that the Linux build slowed considerably, so it finished 840k blocks in about 9.5 hours, and the windows build was 15.3 hours. So not as different as it looked initially.
Also note that BN does not have native SHA256 implemented yet, and both the Windows and Linux builds of BC reported in the log that it was active for them.
A
Antoine Poinsot14:47
Yes, noted that. Plus libsecp256k1 had significant perf improvements in the past couple years.
evoskuil14:49
Right, and my Windows build uses my NuGet prebuilt version of libsecp256k1, which I think is three years old now.
evoskuil14:53
All of my validation performance testing so far has been based on this old version of libsecp, and without native sha256 acceleration, on Windows.
(edited)👍 1
evoskuil15:38
Ah, I misread. The MSVC builds don’t have SHANI (but the mingw builds do, which explains the log).
Yeah, SHANI, SHA vectorization, and algorithm optimization is a fun basket of optimizations. I ended up implementing all SHA, MD, HMAC, and PBKD from spec as C++ templates, to eliminate all of the redundancy and generalize the optimizations. Still have to drop ARM vectorization and ARM/Intel SHA into the SHA template.
For this reason the libbitcoin build is highly sensitive to compiler optimization. With maximal speed optimization it slightly outperforms the BC C implementations and optimizations (where SHANI is excluded), but without -O3 it underperforms. (edited)
… and Scrypt from spec, and the hash accumulator. https://github.com/libbitcoin/libbitcoin-system/tree/master/include/bitcoin/system/impl/hash
evoskuil16:43
This was also part of an academic exercise for me, to show that proper C++ can outperform C. In this case there are performance advantages in autovectorization when using std::array as opposed to raw pointers.
C
cguida00:26
Do I understand from this message that the mini pc you bought for $350 did a full validation sync in 5 hours?
https://libbitcoin.slack.com/archives/C6LH74ACQ/p1729891755834889
Eric Voskuil
Still seeing a high level of starvation in the Linux build, similar to previously with
maximum_concurrency = 0. But the stall and lack of events are resolved. I don't have any tools set up for concurrency and performance analysis on Linux, so I've probably gotta work that out. Setting up my monster mac now also.
In the mean time I pulled down Bitcoin Core latest Linux binary to get a benchmark. It's doing significantly better than it did on the same box for a Windows build (15 hours). After 30 minutes it's completed about 10% of the chain (by txs), and projecting 6 hours (and falling). So w…C
cguida00:28
Ah ok it slowed down to 9.5 hours
https://libbitcoin.slack.com/archives/C6LH74ACQ/p1729953853698889?thread_ts=1729891755.834889&cid=C6LH74ACQ
Eric Voskuil
Thanks, good read. Mostly what I assumed. This is mostly for us, I don’t really have the bandwidth to work on Core as well. But if our observations can help that’s great.
Note that the Linux build slowed considerably, so it finished 840k blocks in about 9.5 hours, and the windows build was 15.3 hours. So not as different as it looked initially.
Also note that BN does not have native SHA256 implemented yet, and both the Windows and Linux builds of BC reported in the log that it was active for them.
C
cguida00:29
Was that with full validation on the $350 machine? Because that's still very impressive
(edited)evoskuil00:56
No, all of the above benchmarks are with a milestone (what BC calls "assume valid") set at the benchmark height. This the default configuration of BC.
(edited)C
cguida02:41
Ah, gotcha. So the corresponding benchmark for BN is 1 hour? Or has the Linux test not completed yet?
evoskuil02:45
The only benchmark for BN is Windows, while I've benchmarked Windows, Linux, and macOS for BC. I had never run BN v4 on Linux or macOS until this week. It runs fine on both, but we need to performance tune it. We've been putting the tools together to do that over the last few days. In the mean time I've rerun the BC benchmarks to verify consistency, and will put out a graph soon.
🔥 1
evoskuil02:53
If you scroll back a ways you'll see s simpler version. Also, I did discuss my most recent BN validation runs on my big box, which came in about 4.5 hours (without confirming queries, which on that machine are prob another 30 mins at least). I haven't attempted any BC full validation runs yet. I also haven't attempted BC with txindex, which is at least a little closer to our level of indexing.
P
parsevalbtc08:38
Yeah, I was facing starvation when I tried on my linux 8GB RAM machine some months ago. Will be following the improvements on this!
👍🏻 1
evoskuil22:18
replied to a thread: view
Today is the first time I've attempted a Linux sync with the v4 architecture. I work almost exclusively on Windows, because of the tooling, and because my habit patterns were set well before Linux existed. 😅
I think that Core still builds for Windows using mingw, which is sort of an afterthought approach, and only supports CMake now. We support native Windows (msvc/win32), macOS (clang/xcode), Linux (gcc/clang) with generated build artifacts, CI/coverage and build scripts for Automake and CMake, native VC++ solutions for Visual Studio Community, and presets for VSCode. If we had CI platforms for Big Endian and for ARM we'd have those supported too (working on it).
👀 1
Saturday, October 26, 2024
evoskuil00:07
This is a little dated (two years old now), some small differences in current table structure, but pretty close.
1 reply · Antoine Poinsot · last Oct 27, 2024
A
Antoine Poinsot00:31
In the transaction table, you don't store the index of the transaction in the block? How do you keep the order of transactions?
evoskuil00:17
4 hours and 20 minutes into my Linux baseline sync of Bitcoin Core v27 (no script validation). It's now 50% by tx count to 840k (45% by its counter) and showing 4 hours to go, for ~8.5 hrs total. So looks like it's slowed substantially, as it was looking like 5 hours total early on. I ran a quick speed test on the box with BC running, just in case it was the network, but it's still showing above 2Gbps bandwidth.
(edited)1 reply · evoskuil · last Oct 26, 2024
evoskuil02:45
Almost 6 hours and about 70% to 840k (by tx count), and still slowing. Reporting 4 hours to completion, so looking like 9-10 total now.
BN is on the right. The periods of download correspond with CPU and disk activity. These should be nearly continuous, but something is wrong with that presently. But what's really interesting is the difference in CPU discordance (and peak utilization) between BC and BN. I can't yet say I know how best to interpret that.
(edited)evoskuil00:34
It seems like at any given moment BC is using a few threads, but never all of them. BN is always using all of them, and always in the same proportion.
These two green threads are interesting as well, as they appear coupled in opposition to each other, as if when one does more work the other is forced to do the same amount less. This seems to imply a locking codependency that essentially reduces two threads to one.
👍 1
This segment shows that is fairly consistent. Like two dependent threads at a time are being taken from the pool.
(edited)
evoskuil01:22
I suspect that there is a task being divided into two partially-dependent parts and then synchronized. If consolidated onto a single thread, the task would complete faster.
(edited)I got it's referring to Libbitcoin, but i couldn't figure out the acronym.
evoskuil01:30
Libbitcoin: libbitcoin-system, libbitcoin-database, libbitcoin-network, libbitcoin-node [and BN executable project], libbitcoin-protocol, libbitcoin-server [and BS executable project], libbitcoin-client, libbitcoin-explorer [and BX executable project]. BN is really just a developer-level executable, really we only ship BS and BX as applications. The others are libs.
(edited)BitcoinServer
Bitcoin eXplorer
evoskuil02:21
Yea, the "lib" is dropped for the non-library (application/executable) builds, and then we use the acronym.
evoskuil02:45
replied to a thread: view
Almost 6 hours and about 70% to 840k (by tx count), and still slowing. Reporting 4 hours to completion, so looking like 9-10 total now.
evoskuil04:47
I didn't save the log from the BC-Windows-11 run, but it was just over 900 minutes to 840k. I also don't yet have a good read on BN-Ubuntu-24.
(edited)
The inflection is not just from the change in block fullness, there is also a material reduction in BC performance on a per tx basis, even when blocks are consistently-sized, as a function of total store size. This is more noticeable when BN is removed for scale...
(edited)
In other words, the rate of slowing increases with chain size. In order for Moore's Law to resolve node scaling, node performance must remain linear in relation to the chain size. The low early rate is just a consequence of empty blocks. But since blocks started becoming full, there is a significant decline on a per tx basis. It should be getting faster on a per tx basis as the blocks become more full (just as in the early stage). But non-scalability in relation to chain size is dominating.
👍 1
1 reply · evoskuil · last Oct 26, 2024
evoskuil22:37
This is a misstatement.... linear would be awful, I meant constant (independent of chain size).
(edited)👍 1
A
Antoine Poinsot14:37
replied to a thread: view
Sharing your bench on Core's IRC i got more info about that FWIW.
10:31 <laanwj> and it makes sense, bitcoind is optimized for various POSIX platforms, not windows, which is different in many aspects; for example leveldb, doesn't even really support windows by itself; the win32 API backend it uses is something i worked on. also the P2P networking doesn't use any i/o polling primitives optimized for windows. Disk i/o is also slower on a lot of windows machines due to windows defender etc getting in the way. To get any level of 10:31 <laanwj> performance there would need a lot of work. Still, it shouldn't be that slow... 10:34 <laanwj> compiling with MSVC instead of mingw also used to be an improvement in speed, but this was 2013, no idea about now, and we can't use that in deterministic builds anyhow 10:36 <hebasto> MSVC builds lack HW-acceleration for SHA256 -- https://github.com/bitcoin/bitcoin/pull/24773 10:36 <darosior> He did mention MSVC vs mingw and i quickly got out of my depth. :) 10:36 <darosior> What do we use for release builds? 10:36 <hebasto> mingw 10:36 <laanwj> mingw, because that's the only thing we can use in side guix 10:36 <laanwj> well we could use clang that would be something to investigate, many seem to prefer that for windows nowadays 10:37 <hebasto> right
👍🏻 1
evoskuil14:44
replied to a thread: view
Thanks, good read. Mostly what I assumed. This is mostly for us, I don’t really have the bandwidth to work on Core as well. But if our observations can help that’s great.
Note that the Linux build slowed considerably, so it finished 840k blocks in about 9.5 hours, and the windows build was 15.3 hours. So not as different as it looked initially.
Also note that BN does not have native SHA256 implemented yet, and both the Windows and Linux builds of BC reported in the log that it was active for them.
evoskuil15:38
replied to a thread: view
Ah, I misread. The MSVC builds don’t have SHANI (but the mingw builds do, which explains the log).
Yeah, SHANI, SHA vectorization, and algorithm optimization is a fun basket of optimizations. I ended up implementing all SHA, MD, HMAC, and PBKD from spec as C++ templates, to eliminate all of the redundancy and generalize the optimizations. Still have to drop ARM vectorization and ARM/Intel SHA into the SHA template.
For this reason the libbitcoin build is highly sensitive to compiler optimization. With maximal speed optimization it slightly outperforms the BC C implementations and optimizations (where SHANI is excluded), but without -O3 it underperforms. (edited)
… and Scrypt from spec, and the hash accumulator. https://github.com/libbitcoin/libbitcoin-system/tree/master/include/bitcoin/system/impl/hash
evoskuil17:36
Building for the first time ever on the monster Mac, which I bought for this purpose over 3 years ago :/. The $350 mini computers have the same thread count and CPU speed. :|, don’t wanna say what I paid for this thing. But at least it still has enough RAM to store the entire chain with room to spare.
evoskuil21:34
Got a clean bn build on the big mac, clang16 with not too much boost hassle. The node runs as expected, with about the same behavior as the linux node. It's not stalling, but using very few resources. So I've got three working test boxes, with running nodes on each, for Windows-11, macOS-15, and Ubuntu-24, and prebuilt versions of Bitcoin Core benchmarked on all but the Big Mac, so will fire that up now.
I ran another milestone sync just now of the win mini box, to 850k this time, completed in 50 minutes and 695GB.
evoskuil22:37
replied to a thread: view
This is a misstatement.... linear would be awful, I meant constant (independent of chain size).
(edited)👍 1
evoskuil23:04
I really didn't think this could be possible, but it looks like this monster mac Core benchmark is going to come in slower than the Linux mini benchmark (9.5 hours). After 1.3 hours it's passing block 400k (BN passes that at 7 minutes).
I tested the Big Mac at over 2Gbps before starting... the specs are crazy and it's basically a new slick machine/OS. It's not shani according to BN, and the BC log. BC is reporting AVX2 optimization (despite the box also supporting AVX512). But these are tiny issues relative to these times.
Sunday, October 27, 2024
evoskuil00:00
Presently estimating 11 hrs to go, on top of the 2:15 passed, and experience shows that the estimates are low, so this may actually exceed the BC Windows mini time (15.3 hrs).
(edited)9 replies · evoskuil · last Oct 27, 2024
evoskuil03:55
6 hours into this run, with 8 hours estimated and rate decreasing. This monster Mac pro run will end up taking longer than the Windows mini run.
evoskuil12:52
15 hours into the monster mac pro Bitcoin Core sync and reporting 6 hours to go. Significantly worse than both the 15.3 hour windows mini sync, and the 9.5 hour Linux mini sync. All on a measured 2.3Gbps Internet, no script validation (assume valid) and resource unconstrained.
evoskuil13:45
The lack of shani probably explains a big part of the difference between the big Mac and the Linux and Windows minis. However, BN does not even implement shani yet, and completes this in under an hour on a 32GB/16 thread windows mini, whereas this is a 728GB/32 thread machine.
Both BC and BN are doing the same amount of hashing, because both skip script execution and are performing all other hashing (all headers, txs, with and without witness, merkle roots for tx and witness commitments).
The difference is that BN is doing it on all threads, close to 100% on the mini, where BC looks more like a single-threaded app on this network and database aspect. As such these local optimizations like shani make a big difference.
But this is really at the center of one of the main architectural distinctions. BC is at its heart a single threaded application that uses threads, and only really fans out for script validation. BN is only improved by shani and hash vectorization when it becomes CPU bound. This is why shani is deferred work - it can be dropped into the sha template with no change to any other code.
(edited)evoskuil13:49
Since the theoretical min time is 38 minutes, and we are at 47-50, and generally CPU bound on the mini, I suspect that with shani we will see around a 10% reduction. The rest is a consequence of RAM flushing to disk, and this can only be entirely eliminated with full RAM (eg the big Mac). At that point we should be under 40 minutes, and cannot get past that without higher bandwidth.
evoskuil13:53
10 Gbps Internet is already being deployed to consumers in the US. That will drop the theoretical limit to under 10 minutes, even with chain growth.
(edited)evoskuil14:39
If you can download the full chain in 10 minutes, how long does it take to validate? Well once it's downloaded, the full chain can be validated concurrently, and it's all read. So as long as the SSD/RAM can keep up, it's limited only by CPU count. We already run validation at 100% CPU on 64 cores. While shani and other local optimizations can help, it's really just Moore's Law at this point.
evoskuil15:49
The Core big Mac sync has reached block 850k at 18.1 hours.
I was previously using 840k for benchmarking, but I'm updating it to 850k to keep it current and because we are using 50k sampling, so this will make accurate plotting easier.
evoskuil19:01
Completed the BC monster Mac Pro (no shani) sync to 867,619 in 23.4 hours (default config, no tx index, some validation near the top).
(edited)A
Antoine Poinsot00:31
replied to a thread: view
In the transaction table, you don't store the index of the transaction in the block? How do you keep the order of transactions?
evoskuil00:33
Index of input, output, tx, confirmed block, and candidate block are never explicitly stored. The only place a position exists explicitly is as part of the point surrogate key.
(edited)All indices are derived from relative positions.
evoskuil00:34
You’ll get the tx “position” from the txs association table, and input/output positions from the puts association table (it does both).
7 replies · Antoine Poinsot, evoskuil · last Oct 27, 2024
evoskuil00:41
We have a much more detailed design doc. Copies of it are shared way back in #general. But it’s a bit much when starting out.
evoskuil00:44
A put index can be obtained from the put by transitioning to the parent and scanning the put list for the put’s PK. Since the list is bounded it remains constant time. And since that is not needed for sync or validation or block creation or population, it’s more optimal for those to avoid the storage and recovery time.
evoskuil00:46
I have a spreadsheet that was used to predict the storage size from the schema and tuning parameters. Spent a long time optimizing the store for size and time tradeoffs. So some of the schema changed.
A
Antoine Poinsot00:36
replied to a thread: view
Although for the puts you do store the index explicitly, at least per the PDF?
evoskuil00:39
replied to a thread: view
I wish it was current, but too much to do. It’s pretty close tho.
evoskuil00:41
replied to a thread: view
We have a much more detailed design doc. Copies of it are shared way back in #general. But it’s a bit much when starting out.
evoskuil00:44
replied to a thread: view
A put index can be obtained from the put by transitioning to the parent and scanning the put list for the put’s PK. Since the list is bounded it remains constant time. And since that is not needed for sync or validation or block creation or population, it’s more optimal for those to avoid the storage and recovery time.
evoskuil00:46
replied to a thread: view
I have a spreadsheet that was used to predict the storage size from the schema and tuning parameters. Spent a long time optimizing the store for size and time tradeoffs. So some of the schema changed.
A
Antoine Poinsot02:01
So confirmability of a block is performed by querying each transaction's previous transaction(s), making sure they are confirmed (this is not printing an output out of thin air), then for each previous output making sure no spender is marked as confirmed (this is not a double spend) and marking the spend corresponding to the transaction being checked for confirmability as confirmed.
This means you can technically support pruning, but cannot reorganize past the height at which you've pruned, or i'm missing something? This would be performed by dropping all the outputs which have a spend entry which corresponds to a transaction which is marked as confirmed.
2 replies · evoskuil · last Oct 27, 2024
evoskuil02:35
We don’t perform marking. We only associate the txs of a confirmed block to their header, and the header to its height. Everything else is inferable. And yes, all of the necessary confirmable checks are performed. There is also maturity and bip30 cecks.
evoskuil04:02
As for pruning, we can definitely do it structurally, but the store is append only, so this implies a local rewrite of the store to reduce it's size (i.e. compression). I documented the queries to make sure it was possible. Inclusive of purging unconfirmed. But it's probably faster to just resync for that, and not worth it because it's so small. But really I don't see pruning as important. It's bad for the network, the storage cost for the full chain is currently like $5 and dropping (even with growth considered), it adds complexity and limits functionality, it will eventually be totally inconsequential.
Yes, no pruned node can reorg past its pruning point.
A
Antoine Poinsot02:13
So for reorgs to unconfirm a block you just wipe all the confirmed entry for each of the transactions?
👍🏻 1
Ok, if i got those right, i'd be interested to learn about how you handle the mempool. Unconfirmed transactions storage / validation (this i can envision, just a non-confirmed spend entry i suppose). But also unconfirmed transaction replacement (RBF or whatever). I remember some discussions on Twitter some time ago where you said you took an approach very different from Core.
18 replies · evoskuil, Antoine Poinsot · last Oct 27, 2024
evoskuil02:40
We use PoW as the block DoS guard, and fees for txs. They are validated in the context of a next block (the actual block may have an incompatible context, but almost never, and we revalidate as necessary). We don’t have a mempool, we have a tx pool (unconfirmed txs, either from weak (reorged or disgorged) blocks or as unconfirmed txs relay). The store is append only, once we write a tx it never changes and they are all stored I. The same manner/place.
evoskuil02:46
We don’t care about “replacement”, or any of the policy and pool antics that are currently ongoing. We just store a tx if it has sufficient fees (configurable) to guard the store.
This implies that if a tx creates a conflict with a stored tx, the net mineable fee must be a min increase for it to be accepted, strictly for DoS protect. Packaging is this also considered, but not in the way Core has implemented it. I posted our concept of that to bitcoin-dev a couple years ago.
A
Antoine Poinsot02:47
I'll dig that up. To be clear here by fee you mean absolute fee, not feerate right?
A
Antoine Poinsot02:56
Is that what you are referring to? https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-May/020518.html
A
Antoine Poinsot02:58
Ah, no it must be this https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-June/020544.html.
evoskuil03:23
The tx DoS protection is a min fee rate (sats/byte), and for conflicts is the same idea, but must “pay for” both itself and the graph it conflicts with.
evoskuil03:30
Since txs are stored to disk and paid for, and constant time accessible, there is little concern for size. The purpose of them is for miners to build the most optimal templates. They also speed block announcement, also strictly for miners.
For a merchant to “see” unconfirmed txs doesn’t require anything but querying the store for the one’s of interest. Address subscript was a v3 feature, and that is now Electrum-based indexing, which covers all outputs for all confirmed, unconfirmed, and conflicting txs.
For mining we will eventually run a chaser to maintain the minimal state (graph) necessary to generate a block template (much smaller than txs in a “pool”, which we store to disk).
A
Antoine Poinsot15:27
This implies that if a tx creates a conflict with a stored tx, the net mineable fee must be a min increase for it to be accepted, strictly for DoS protect.
The tx DoS protection is a min fee rate (sats/byte), and for conflicts is the same idea, but must “pay for” both itself and the graph it conflicts with.
Am i correct this minimum feerate is 1 sat/vbyte, as per https://github.com/libbitcoin/libbitcoin-blockchain/blob/2be874a62181fd73f93a4b5feb3e9f162b39dd78/src/settings.cpp#L29 ? Is that also the value used for the min increase?
So for instance if an incoming 100k vbytes transaction conflicts with an in-pool, 1sat/vb, 500k vbytes transaction, then it needs to be at least 7sat/vb (pays the min fee for both itself and the transaction it conflicts with, plus the 1sat/vb min increase).
evoskuil15:33
No, blockchain is an obsolete v3 project (and fee rate was configurable for v3). But that's the idea. It's not limited to individual transactions, but any unconfirmed subgraph(s) in conflict. You won't find this in implementation for v4.
(edited)A
Antoine Poinsot15:37
Ok, i see. I'll make the graph thing clear in the example i'll use in my writeup.
evoskuil15:41
No, there is no limit to the size of individual transactions except that they must be confirmable. For a package to be considered it must be independently confirmable as well (see comments in bitcoin-dev post). There is an inherent rough patch there at the limit, due the the variable nature of coinbases, which could be exploited if misconfigured, but that's really only an issue for miners, who have the coinbase under their control.
evoskuil15:43
The approach is simple, allow anything that is valid while protecting against disk overflow (using PoW for blocks and fees for unconfirmed txs). There's no policy necessary apart from the fee rate required for DoS protection.
A
Antoine Poinsot15:44
There is an inherent rough patch there at the limit, due the the variable nature of coinbases, which could be exploited if misconfigured, but that's really only an issue for miners, who have the coinbase under their control.
That's where i was getting at. 🙂 ok, i'll read these posts before asking more questions.
evoskuil15:57
Yeah, the idea is that the coinbase size is configured by default with a large enough value that the maximal tx/subgraph can get mined. A miner who wants to maximize revenue could take more risk (given his low probability of mining it himself, his own coinbase size isn't necessarily the primary consideration).
A
Antoine Poinsot02:27
Finally i'd be interesting in learning how you manage your peers. You mentioned a few times how you had put a lot of work into this, i guess it would be interesting to include in my writeup. Then i'll stop bothering you all with all my questions.
18 replies · evoskuil, Antoine Poinsot · last Oct 27, 2024
evoskuil02:52
The questions are fine, no worries.
We don’t ban, but we drop and connect aggressively. We currently have a target outbound peer count, inbound limit, and explicit “manual” connections that are maintained (reconnected on drop). At startup there are also seed connections, dropped after seeding. These are managed by four specialized “sessions”, and a base class. The C++ API also allows for addition of manual connections, and for subscription to various callbacks, such as connection and messages by type. These are used by the many protocol specializations.
There are then “chaser” specializations that impose order for the chain building process, as we’ve been discussing.
evoskuil03:02
The protocols are attached to each channel and the block download protocol supports performance metrics (block byte totals) and feeds them to the outbound session (we don’t manage inbound or manual for block sync perf), which maintains a rate table. Configurable timers are used to measure download rates on a per channel basis, and to perform standard deviation computation across the set (if more than 2). Stalled channels are dropped (even if less that 3 or if SD is not enabled by config). Slow channels are defined by a configurable negative deviation from the SD and dropped.
Drops are very common, so rapid connection is important. We can very rapidly connect even thousands of peers, but best sync on a 2Gbps network is 100-200 peers. Rapid connection is part address book hygiene and part concurrency. To connect 1 we attempt 5 (by default) with a race established between them, dropping all but the first to complete handshake. We find about 1/5 addresses are good on average.
evoskuil03:05
Completion for speed is not used for header sync, we want to see all peers, not just the fast ones. But with a strong header chain we just want the blocks as fast as we can get them. Dividing work up among a larger set of peers doesn’t increase load on the network, and rate management moves that load to where it is more available (load balancing). We will eventually make the total connection count more dynamic, so that it throttles back once the chain is “current” and back up again if it falls behind (eg after a shutdown period).
evoskuil03:15
The node rapidly finds the fastest available peers on the network and uses very little of their resources because of work splitting. There are no redundant block requests. Work is managed by the check chaser such that channels request work (hashes to download) and if the channel is dropped, the hashes are returned to the chaser. As channels become starved, more work is generated, and if no more is pending, the slowest channels are dropped (to prevent redundant downloads, since there is no way to withdraw a request made to a peer, and we always request the full batch of work) and their work divided up. This can cause a few seconds of delay in each download “window” as the last work is shuffled around, but a peer cannot delay the sync. The window can also be configured to be the whole chain, but breaking it up is good for preventing validation delay.
evoskuil03:18
Full DoS protect is not implemented, but that consists mainly of rate limiting requests based on config. Otherwise, as long as the protocol is complied with the peer is valid, and if not is immediately dropped. We do have ip/host configurable white/black listing as well, but it’s not really a DoS thing and I don’t use it while perf testing sync.
A
Antoine Poinsot16:09
At startup there are also seed connections, dropped after seeding.
Where do you seed from? DNS seeds like Core? Fixed ones? For seeding Core has an indirection, it gets ip addresses from either the DNS or fixed seeds, queries these ips for addresses, and syncs from those final received addresses.
There are no redundant block requests.
So: (highly) redundant but not-monitored header requests, and non-redundant but monitored block requests?
evoskuil16:13
For seeding we just connect to the configured set of peers and drop on completion and timers. These can be IP addresses or DNS names, and port. The defaults are currently all DNS names that point to libbitcoin.net, which allows us to redirect them for testing. However these are not typically operational (until we release v4).
(edited)evoskuil16:15
The node operator can specify any node(s) for this in config, and/or can prepopulate the address cache file with the same values. The node won't start unless it as [batch x outbound] unique addresses in the cache, either from seeding or otherwise. Seeding is bypassed if the address cache is sufficiently populated.
(edited)evoskuil16:16
So: (highly) redundant but not-monitored header requests, and non-redundant but monitored block requests?
yes
A
Antoine Poinsot16:22
Slow channels are defined by a configurable negative deviation from the SD and dropped.
What's the default for this value?
evoskuil17:07
evoskuil17:07
evoskuil17:07
evoskuil18:54
There is internal optimization in the redundant header downloads. Headers that are already stored get discarded as they are redundantly received, after identification via hash. But we download the full header chain from each peer at first startup. However once there is a candidate chain (often just a few seconds) a populated locator is provided to peers, so unless they disagree, there is no redundancy.
evoskuil02:35
replied to a thread: view
We don’t perform marking. We only associate the txs of a confirmed block to their header, and the header to its height. Everything else is inferable. And yes, all of the necessary confirmable checks are performed. There is also maturity and bip30 cecks.
evoskuil02:40
replied to a thread: view
We use PoW as the block DoS guard, and fees for txs. They are validated in the context of a next block (the actual block may have an incompatible context, but almost never, and we revalidate as necessary). We don’t have a mempool, we have a tx pool (unconfirmed txs, either from weak (reorged or disgorged) blocks or as unconfirmed txs relay). The store is append only, once we write a tx it never changes and they are all stored I. The same manner/place.
evoskuil02:46
replied to a thread: view
We don’t care about “replacement”, or any of the policy and pool antics that are currently ongoing. We just store a tx if it has sufficient fees (configurable) to guard the store.
This implies that if a tx creates a conflict with a stored tx, the net mineable fee must be a min increase for it to be accepted, strictly for DoS protect. Packaging is this also considered, but not in the way Core has implemented it. I posted our concept of that to bitcoin-dev a couple years ago.
A
Antoine Poinsot02:47
replied to a thread: view
I'll dig that up. To be clear here by fee you mean absolute fee, not feerate right?
evoskuil02:52
replied to a thread: view
The questions are fine, no worries.
We don’t ban, but we drop and connect aggressively. We currently have a target outbound peer count, inbound limit, and explicit “manual” connections that are maintained (reconnected on drop). At startup there are also seed connections, dropped after seeding. These are managed by four specialized “sessions”, and a base class. The C++ API also allows for addition of manual connections, and for subscription to various callbacks, such as connection and messages by type. These are used by the many protocol specializations.
There are then “chaser” specializations that impose order for the chain building process, as we’ve been discussing.
A
Antoine Poinsot02:58
replied to a thread: view
Ah, no it must be this https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-June/020544.html.
evoskuil03:02
replied to a thread: view
The protocols are attached to each channel and the block download protocol supports performance metrics (block byte totals) and feeds them to the outbound session (we don’t manage inbound or manual for block sync perf), which maintains a rate table. Configurable timers are used to measure download rates on a per channel basis, and to perform standard deviation computation across the set (if more than 2). Stalled channels are dropped (even if less that 3 or if SD is not enabled by config). Slow channels are defined by a configurable negative deviation from the SD and dropped.
Drops are very common, so rapid connection is important. We can very rapidly connect even thousands of peers, but best sync on a 2Gbps network is 100-200 peers. Rapid connection is part address book hygiene and part concurrency. To connect 1 we attempt 5 (by default) with a race established between them, dropping all but the first to complete handshake. We find about 1/5 addresses are good on average.
evoskuil03:05
replied to a thread: view
Completion for speed is not used for header sync, we want to see all peers, not just the fast ones. But with a strong header chain we just want the blocks as fast as we can get them. Dividing work up among a larger set of peers doesn’t increase load on the network, and rate management moves that load to where it is more available (load balancing). We will eventually make the total connection count more dynamic, so that it throttles back once the chain is “current” and back up again if it falls behind (eg after a shutdown period).
evoskuil03:15
replied to a thread: view
The node rapidly finds the fastest available peers on the network and uses very little of their resources because of work splitting. There are no redundant block requests. Work is managed by the check chaser such that channels request work (hashes to download) and if the channel is dropped, the hashes are returned to the chaser. As channels become starved, more work is generated, and if no more is pending, the slowest channels are dropped (to prevent redundant downloads, since there is no way to withdraw a request made to a peer, and we always request the full batch of work) and their work divided up. This can cause a few seconds of delay in each download “window” as the last work is shuffled around, but a peer cannot delay the sync. The window can also be configured to be the whole chain, but breaking it up is good for preventing validation delay.
evoskuil03:18
replied to a thread: view
Full DoS protect is not implemented, but that consists mainly of rate limiting requests based on config. Otherwise, as long as the protocol is complied with the peer is valid, and if not is immediately dropped. We do have ip/host configurable white/black listing as well, but it’s not really a DoS thing and I don’t use it while perf testing sync.
evoskuil03:23
replied to a thread: view
The tx DoS protection is a min fee rate (sats/byte), and for conflicts is the same idea, but must “pay for” both itself and the graph it conflicts with.
evoskuil03:30
replied to a thread: view
Since txs are stored to disk and paid for, and constant time accessible, there is little concern for size. The purpose of them is for miners to build the most optimal templates. They also speed block announcement, also strictly for miners.
For a merchant to “see” unconfirmed txs doesn’t require anything but querying the store for the one’s of interest. Address subscript was a v3 feature, and that is now Electrum-based indexing, which covers all outputs for all confirmed, unconfirmed, and conflicting txs.
For mining we will eventually run a chaser to maintain the minimal state (graph) necessary to generate a block template (much smaller than txs in a “pool”, which we store to disk).
evoskuil03:55
replied to a thread: view
6 hours into this run, with 8 hours estimated and rate decreasing. This monster Mac pro run will end up taking longer than the Windows mini run.
evoskuil04:02
replied to a thread: view
As for pruning, we can definitely do it structurally, but the store is append only, so this implies a local rewrite of the store to reduce it's size (i.e. compression). I documented the queries to make sure it was possible. Inclusive of purging unconfirmed. But it's probably faster to just resync for that, and not worth it because it's so small. But really I don't see pruning as important. It's bad for the network, the storage cost for the full chain is currently like $5 and dropping (even with growth considered), it adds complexity and limits functionality, it will eventually be totally inconsequential.
Yes, no pruned node can reorg past its pruning point.
evoskuil12:52
replied to a thread: view
15 hours into the monster mac pro Bitcoin Core sync and reporting 6 hours to go. Significantly worse than both the 15.3 hour windows mini sync, and the 9.5 hour Linux mini sync. All on a measured 2.3Gbps Internet, no script validation (assume valid) and resource unconstrained.
evoskuil13:45
replied to a thread: view
The lack of shani probably explains a big part of the difference between the big Mac and the Linux and Windows minis. However, BN does not even implement shani yet, and completes this in under an hour on a 32GB/16 thread windows mini, whereas this is a 728GB/32 thread machine.
Both BC and BN are doing the same amount of hashing, because both skip script execution and are performing all other hashing (all headers, txs, with and without witness, merkle roots for tx and witness commitments).
The difference is that BN is doing it on all threads, close to 100% on the mini, where BC looks more like a single-threaded app on this network and database aspect. As such these local optimizations like shani make a big difference.
But this is really at the center of one of the main architectural distinctions. BC is at its heart a single threaded application that uses threads, and only really fans out for script validation. BN is only improved by shani and hash vectorization when it becomes CPU bound. This is why shani is deferred work - it can be dropped into the sha template with no change to any other code.
(edited)evoskuil13:49
replied to a thread: view
Since the theoretical min time is 38 minutes, and we are at 47-50, and generally CPU bound on the mini, I suspect that with shani we will see around a 10% reduction. The rest is a consequence of RAM flushing to disk, and this can only be entirely eliminated with full RAM (eg the big Mac). At that point we should be under 40 minutes, and cannot get past that without higher bandwidth.
evoskuil13:53
replied to a thread: view
10 Gbps Internet is already being deployed to consumers in the US. That will drop the theoretical limit to under 10 minutes, even with chain growth.
(edited)evoskuil14:39
replied to a thread: view
If you can download the full chain in 10 minutes, how long does it take to validate? Well once it's downloaded, the full chain can be validated concurrently, and it's all read. So as long as the SSD/RAM can keep up, it's limited only by CPU count. We already run validation at 100% CPU on 64 cores. While shani and other local optimizations can help, it's really just Moore's Law at this point.
A
Antoine Poinsot15:27
replied to a thread: view
This implies that if a tx creates a conflict with a stored tx, the net mineable fee must be a min increase for it to be accepted, strictly for DoS protect.
The tx DoS protection is a min fee rate (sats/byte), and for conflicts is the same idea, but must “pay for” both itself and the graph it conflicts with.
Am i correct this minimum feerate is 1 sat/vbyte, as per https://github.com/libbitcoin/libbitcoin-blockchain/blob/2be874a62181fd73f93a4b5feb3e9f162b39dd78/src/settings.cpp#L29 ? Is that also the value used for the min increase?
So for instance if an incoming 100k vbytes transaction conflicts with an in-pool, 1sat/vb, 500k vbytes transaction, then it needs to be at least 7sat/vb (pays the min fee for both itself and the transaction it conflicts with, plus the 1sat/vb min increase).
evoskuil15:33
replied to a thread: view
No, blockchain is an obsolete v3 project (and fee rate was configurable for v3). But that's the idea. It's not limited to individual transactions, but any unconfirmed subgraph(s) in conflict. You won't find this in implementation for v4.
(edited)A
Antoine Poinsot15:38
replied to a thread: view
Do you have a limit on the size of transactions you accept in the pool?
evoskuil15:41
replied to a thread: view
No, there is no limit to the size of individual transactions except that they must be confirmable. For a package to be considered it must be independently confirmable as well (see comments in bitcoin-dev post). There is an inherent rough patch there at the limit, due the the variable nature of coinbases, which could be exploited if misconfigured, but that's really only an issue for miners, who have the coinbase under their control.
evoskuil15:43
replied to a thread: view
The approach is simple, allow anything that is valid while protecting against disk overflow (using PoW for blocks and fees for unconfirmed txs). There's no policy necessary apart from the fee rate required for DoS protection.
evoskuil15:49
replied to a thread: view
The Core big Mac sync has reached block 850k at 18.1 hours.
I was previously using 840k for benchmarking, but I'm updating it to 850k to keep it current and because we are using 50k sampling, so this will make accurate plotting easier.
evoskuil15:54
replied to a thread: view
For comparison with the previous two BC runs, the time to 840k was 17.35 hours.
A
Antoine Poinsot16:09
replied to a thread: view
At startup there are also seed connections, dropped after seeding.
Where do you seed from? DNS seeds like Core? Fixed ones? For seeding Core has an indirection, it gets ip addresses from either the DNS or fixed seeds, queries these ips for addresses, and syncs from those final received addresses.
There are no redundant block requests.
So: (highly) redundant but not-monitored header requests, and non-redundant but monitored block requests?
evoskuil16:13
replied to a thread: view
For seeding we just connect to the configured set of peers and drop on completion and timers. These can be IP addresses or DNS names, and port. The defaults are currently all DNS names that point to libbitcoin.net, which allows us to redirect them for testing. However these are not typically operational (until we release v4).
(edited)evoskuil16:15
replied to a thread: view
The node operator can specify any node(s) for this in config, and/or can prepopulate the address cache file with the same values. The node won't start unless it as [batch x outbound] unique addresses in the cache, either from seeding or otherwise. Seeding is bypassed if the address cache is sufficiently populated.
(edited)evoskuil16:16
replied to a thread: view
So: (highly) redundant but not-monitored header requests, and non-redundant but monitored block requests?
yes
evoskuil18:54
replied to a thread: view
There is internal optimization in the redundant header downloads. Headers that are already stored get discarded as they are redundantly received, after identification via hash. But we download the full header chain from each peer at first startup. However once there is a candidate chain (often just a few seconds) a populated locator is provided to peers, so unless they disagree, there is no redundancy.
evoskuil19:01
replied to a thread: view
Completed the BC monster Mac Pro (no shani) sync to 867,619 in 23.4 hours (default config, no tx index, some validation near the top).
(edited)A
Antoine Poinsot19:42
Here is my "Libbitcoin for Core people" writeup, which intends to conceptually summarizes how Libbitcoin works for people familiar with Bitcoin Core. https://gist.github.com/darosior/97958327835eeef4119418e4d3b6d27f
I intend it to post it to Delving Bitcoin (probably tomorrow) so people can discuss it and you can participate if you're so inclined. I tried to keep it a high-level overview, so i've omitted some details you've shared here. These can be brought up if people ask about a certain area in the comments.
If you have the time to skim through it please let me know if i got anything wrong or feel i'm misrepresenting something.
51 replies · jungly, Antoine Poinsot, evoskuil · last Oct 28, 2024
jungly19:53
I think people might interpret the description of the database to mean it's a relational database. It might be worth while to add a sentence or two mentioning it's a custom in memory database.
A
Antoine Poinsot19:55
But you could have a relational database backend as well right? What i understood from the document was that it's abstracted, but represented as a relational database.
jungly19:55
Also, Eric has this event driven architecture in how the threads coordinate work. I think that is a big plus for libbitcoin too. Worth highlighting it? Eric probably is the best judge on this, if it's worthwhile etc.
jungly19:57
We could switch to a relational db, yes. But I think that is for not so performant set ups. A big driver of the performance is the custom db, and the event driven architecture. That's my understanding, Eric to keep me honest 🙂
evoskuil20:00
Logically the structure is relational, and is designed so that a SQL-based implementation could be dropped in (two interns have actually been working on SQLite). But people might interpret it as using a generalized SQL store, which would not perform like libbitcoin-database.
💥 1
jungly20:00
Having an architecture that allows using all the cores ... is just mind blowing, and the magic sauce!
👍🏻 1
A
Antoine Poinsot20:01
Ok, i had understood that but i can see how my notes don't make it clear. I'll try to clarify that before posting.
🙏🏾 1
evoskuil20:02
The network implements the asynchronous proactor pattern using BOOST ASIO. I've rebuild this, but this was originally Amir's choice and implementation, as was the mmap hash tables.
(edited)evoskuil20:09
skips all validation
This isn't entirely accurate. There is full header validation and tx commitment and witness commitment validations (both Merkle roots, guarded against malleation).
evoskuil20:11
once the best headers chain tip is current
Maybe too much detail, but headers are also committed whenever a checkpoint is encountered.
evoskuil20:14
When received, all checks which don't require strict ordering are performed and transactions are stored in a global transaction index.
Under milestone only the commitments are checked. Otherwise block.check is performed. This may requires check context (for forks), but it does not require prevouts.
// Header state is checked by organize.
code protocol_block_in_31800::check(const chain::block& block,
const chain::context& ctx, bool bypass) const NOEXCEPT
{
code ec{};
if (bypass)
{
// Only identity is required under bypass.
if (((ec = block.identify())) || ((ec = block.identify(ctx))))
return ec;
}
else
{
// Identity is not correct if error::invalid_transaction_commitment.
if ((ec = block.check()))
{
if ((ec != system::error::invalid_transaction_commitment) &&
block.is_malleated())
return system::error::block_malleated;
return ec;
}
if ((ec = block.check(ctx)))
return ec;
}
return system::error::block_success;
} (edited)evoskuil20:21
Validation may run concurrently with block requests, chasing gaps and performing the aspects of validation that require prevouts (and context). For this validation blocks and prevouts are deserialized from the store.
Confirmability may run concurrently with both download (block.check) and validate (block.accept and block.connect). This is a set of database queries that verify maturity, spendability and spentness.
evoskuil20:22
Kinds feels like you've conflated validation with confirmation. There are three distinct workflows that may or may not run concurrently with each other. On a big machine it is most optimal to run them all concurrently.
evoskuil20:24
Block request/check/store (download) is unordered. Block.accept/connect (validation) is partially ordered (all preceding blocks must be stored). Confirmability is strictly ordered (the preceding block must be confirmable before the next is tested).
A
Antoine Poinsot20:24
I have not conflated them in my understanding. Hmm maybe i should just properly structure my writeup instead of being notes...
A
Antoine Poinsot20:25
The separation of steps in function of ordering is the main thing that i want to convey i think, so i should make sure it's clear
👍🏻 1
evoskuil20:26
The fully validated chain is then persisted
Makes it sound like the blocks aren't stored yet. Everything except the validation/confirmation/height index entries are persisted as blocks arrive.
evoskuil20:28
There is a "spender" relation between an output and its spending transaction(s).
Prob call that an index vs. relation.
evoskuil20:30
There is a "confirmed" relation between a transaction and its confirmation height
confirmation header
(edited)evoskuil20:34
When checking a block for confirmability it will for every transaction fetch its prevout transaction(s) (which must have confirmed state, otherwise the block confirmability check fails) and for each prevout it will mark the spender corresponding to the transaction being checked for confirmability as "confirmed" (if there exists any other spender whose related transaction is "confirmed", this is a double spend, the block confirmability check fails).
Validation "fetches" prevouts, confirmability is just a boolean result query.
There is no marking of spends or spenders or outputs. Confirmed txs are associated to their containing header (their block), only.
evoskuil20:35
wouldn't be able to reorg past the pruning point.
Technically we could, but it requires data requests to recover missing txs.
evoskuil20:36
The transaction table is also used to persist unconfirmed transactions.
Transactions span all tables in the archive except the header and txs (association) table.
(edited)evoskuil20:37
by default the node requires a transaction has at least 1sat/vb.
There is no default as this isn't yet implemented, but presumably zero value would just disable the pool altogether.
(edited)evoskuil20:39
For conflicts the node requires the transaction pays at least the minimum feerate for itself plus for the whole graph it conflicts with, plus an incremental feerate (1sat/vb too by default).
There's no implied additional increment. Self paying for self graph and conflict graph implies that the full amount is offset.
👍 1
evoskuil21:24
timelocks
Input time locks are checked in the confirmability query, as it requires the context (height and mtp) of the previous output (also required for the coinbase maturity check).
(edited)👍 1
evoskuil21:41
Those could be checked in validation, which would benefit from block parallelism, but it requires some amount of redundancy (which may make sense as an optimization). I think that because the data is queried anyway, the two checks are essentially free in the current implementation.
(edited)A
Antoine Poinsot17:39
There is a "confirmed" relation between a transaction and its confirmation height
confirmation header (edited)
Oh, is that a difference between the document and what you ended up implementing?
A
Antoine Poinsot17:47
Then why wiping the transaction -> confirmation header index on reorg? Technically it's true the transaction is associated with this header, just the header isn't part of the strongest chain anymore?
A
Antoine Poinsot17:53
Other than that i should have corrected everything, and i re-wrote the whole thing in a more bearable way (which also highlights the important main point of concurrency).
👍🏻 1
evoskuil17:53
Not a difference, just a bit more detail than is available in the doc. The header height indices (candidate and confirmed) exist at trivial cost, almost zero. These are the final step in exposing a confirmation (remember the store is always fully read exposed). We have headers and transactions. The only thing that constitutes a "block" is the txs association table, that associates a header to the transactions it "contains". headers are populated first, then when blocks are downloaded, transactions are populated, and then the txs association, and then the indexation of the txs association. At that point the block is archived. So all "blocks" are headers that associate their txs. However these txs do not associate their headers. That would require a costly one to many association, for which there isn't a significant use case. So we implement the 1:0..1 association between a tx and its confirming block (or no block). These associations are set in order, block block by block, during confirmation.
evoskuil17:55
This association is sufficient to determine that the tx is confirmed, and to find it's confirming block (which includes the block height as context, since all blocks have a height whether confirmed or not). The height index is also set, but this is not used in the confirmation process, it is just to provide an index of confirmed blocks by height.
(edited)evoskuil17:58
The tx->confirming-block ("strong") association is necessary during confirmation as otherwise we would have to perform a linear scan of the entire store to know whether a previous output is confirmed and at what height.
(edited)evoskuil18:06
It also gives us a way to in constant time distinguish between confirmed and unconfirmed txs for things like the tx pool and general public API query.
evoskuil18:12
Note that the association of a transaction to a block inherently implies the tx is no longer in the tx pool. The existence of a confirmed spend conflict also implies that an unconfirmed tx is not in the tx pool. Thus the graph necessary for rapid generation of block templates can be directly and efficiently populated from the store in its present form.
A
Antoine Poinsot19:03
Hmm, can you remind me how you mark a transaction as validated again? (Not confirmed, just the partially-ordered checks)
evoskuil20:19
TLDR: we don't mark block txs.
We can write an entry into the
validated_tx table with the tx state and validation context. We don't do this during block download because blocks are validated as a whole, so there is no need to relay the transaction state, block state is sufficient for the next (confirmation) checks, which are necessarily block-based. We considered (and designed for) individual transaction-based validation stage fan-out, which could either use this table or synchronize on the block. However, performance testing showed that as long as there are as many blocks as threads it is much faster to validate whole blocks. The same fan-out was tested for confirmability (within a block) and was also found to be not worth the fan-out cost (although was much closer because it's a single block at a time, and there may be configurations where it makes sense). And the tx validation entries are not necessary apart from validating the chain (the plan was to purge the file periodically). So this will probably end up being used only for transactions that enter via the pool. However validated_bk is written with current state (no context required, as block context is inherent and stored with the header), which allows the confirmation chaser to proceed across validated blocks (changing them to confirmable or unconfirmable. A confirmable block may cycle between confirmed and unconfirmed, but the confirmability state is final.evoskuil20:58
validated_bk data is also designed so that it may be purged (it would just have to be regenerated if the block returned via a re-reorg, which would happen automatically. But there are so few blocks and these entries are small (no context) that it's not really worth automating it.evoskuil21:02
So really there is no "marking" whatsoever. The closest we come is maintaining an association from tx and from height to header (candidate and confirmed), which just naturally reflect candidate/confirmation state. We did perform marking in a prototype, and that was abandoned.
👍 1
Monday, October 28, 2024
evoskuil13:21
Ran another BC Linux mini sync to obtain the log. 9.9 hours to 850k, 9.4 hours to 840k. Essentially the same as the previous same run.
The BC store size (full chain, /blocks and /chainstate) is 708 GB. That's without tx indexing. BN store size (full chain) is 729GB, and that's with full bidirectional indexing, and I'm pretty sure the BC txindex is bigger than 21GB. So BN is more space efficient. BN Point compression knocks off around 50GB IIRC, though that requires big RAM machine or it will increase sync.
I'll get a Linux mini run of BC with txindex on for a proper comparison, but presently rerunning BC winmini to ensure consistency and to also get the detailed log for graphing comparison.
🎉 1
evoskuil13:29
I've set up vscode on Linux and macOS, and am going to investigate the Tracy profiler, which works on all three platforms and integrates with vscode. It's the closest I can find to the Visual Studio performance profiler, possibly even better. This will give me a visual debugger and profiler in all three environments. Then I'll be working through what is causing the much lower resource utilization *nix.
(edited)evoskuil16:48
Decided to get a third mini (they're back in stock at Temu) to make a hackintosh. This will allow me to perform consistent benchmarking across all three operating systems and all with SHANI (but not AVX512). While libbitcoin does support AVX512, Core does not. This takes away one of our advantages, and (currently) gives Core a significant advantage in that it supports SHANI and BN (currently) does not.
(edited)evoskuil17:00
Was worried about overloading this circuit, but with three PC’s, a UPS, two routers, a dual monitor, and a switch, I’m at 2.5A (out of 15A). That’s with BC syncing on mini. I’ll measure it again with BN when that completes.
(edited)
evoskuil17:53
replied to a thread: view
Not a difference, just a bit more detail than is available in the doc. The header height indices (candidate and confirmed) exist at trivial cost, almost zero. These are the final step in exposing a confirmation (remember the store is always fully read exposed). We have headers and transactions. The only thing that constitutes a "block" is the txs association table, that associates a header to the transactions it "contains". headers are populated first, then when blocks are downloaded, transactions are populated, and then the txs association, and then the indexation of the txs association. At that point the block is archived. So all "blocks" are headers that associate their txs. However these txs do not associate their headers. That would require a costly one to many association, for which there isn't a significant use case. So we implement the 1:0..1 association between a tx and its confirming block (or no block). These associations are set in order, block block by block, during confirmation.
evoskuil17:55
replied to a thread: view
This association is sufficient to determine that the tx is confirmed, and to find it's confirming block (which includes the block height as context, since all blocks have a height whether confirmed or not). The height index is also set, but this is not used in the confirmation process, it is just to provide an index of confirmed blocks by height.
(edited)evoskuil17:58
replied to a thread: view
The tx->confirming-block ("strong") association is necessary during confirmation as otherwise we would have to perform a linear scan of the entire store to know whether a previous output is confirmed and at what height.
(edited)evoskuil18:06
replied to a thread: view
It also gives us a way to in constant time distinguish between confirmed and unconfirmed txs for things like the tx pool and general public API query.
evoskuil18:12
replied to a thread: view
Note that the association of a transaction to a block inherently implies the tx is no longer in the tx pool. The existence of a confirmed spend conflict also implies that an unconfirmed tx is not in the tx pool. Thus the graph necessary for rapid generation of block templates can be directly and efficiently populated from the store in its present form.
A
Antoine Poinsot19:10
Published "Libbitcoin for Core people" https://delvingbitcoin.org/t/libbitcoin-for-core-people/1222
Delving Bitcoin
Recently Eric Voskuil shared a benchmark of doing IBD with Libbitcoin versus Bitcoin Core, showing that Libbitcoin could perform IBD 15x faster than Core with -assumevalid. Libbitcoin’s approach is very different from Bitcoin Core. This writeup intends to summarize how Libbitcoin functions for someone familiar with Bitcoin Core’s approach. This discusses the behaviour of upcoming, not yet released, version 4 of Libbitcoin. Libbitcoin is event-based. It will kick-off multiple asynchronous tasks...
🔥 1
There might still be some inaccuracies but i think the big picture is there, and those can be discussed in the thread.
👍🏻 1
evoskuil20:19
replied to a thread: view
TLDR: we don't mark block txs.
We can write an entry into the
validated_tx table with the tx state and validation context. We don't do this during block download because blocks are validated as a whole, so there is no need to relay the transaction state, block state is sufficient for the next (confirmation) checks, which are necessarily block-based. We considered (and designed for) individual transaction-based validation stage fan-out, which could either use this table or synchronize on the block. However, performance testing showed that as long as there are as many blocks as threads it is much faster to validate whole blocks. The same fan-out was tested for confirmability (within a block) and was also found to be not worth the fan-out cost (although was much closer because it's a single block at a time, and there may be configurations where it makes sense). And the tx validation entries are not necessary apart from validating the chain (the plan was to purge the file periodically). So this will probably end up being used only for transactions that enter via the pool. However validated_bk is written with current state (no context required, as block context is inherent and stored with the header), which allows the confirmation chaser to proceed across validated blocks (changing them to confirmable or unconfirmable. A confirmable block may cycle between confirmed and unconfirmed, but the confirmability state is final.evoskuil20:45
With some help from @pmienk I'm now building in vscode on Linux using his autogenerated cmake presets. It's been shipping for a couple years now and I've finally tried it. Very cool. Now working on the same on macOS.
🔥 2
evoskuil20:58
replied to a thread: view
validated_bk data is also designed so that it may be purged (it would just have to be regenerated if the block returned via a re-reorg, which would happen automatically. But there are so few blocks and these entries are small (no context) that it's not really worth automating it.evoskuil21:02
replied to a thread: view
So really there is no "marking" whatsoever. The closest we come is maintaining an association from tx and from height to header (candidate and confirmed), which just naturally reflect candidate/confirmation state. We did perform marking in a prototype, and that was abandoned.
👍 1
Tuesday, October 29, 2024
evoskuil00:13
My second BC run on the Windows 11 mini is 11.2 hours in and estimating 3 to go, so looks like it will also be about the same as the initial run.
👀 1
evoskuil03:26
BC-Win11-Mini is passing the 840k benchmark block now at 14.5 hours, almost exactly as in the previous run.
3 replies · cguida, evoskuil · last Oct 29, 2024
C
cguida12:38
Wow, why is Mac so slow? What kind of processor is on that? Wild that it’s slower than a machine with 32GB
evoskuil14:16
I suspect it’s largely a result of not having intrinsic sha256. It doesn’t really matter how much bandwidth, RAM, CPU or SSD speed you have, Bitcoin Core operates more like a single threaded application. The Mac is faster until blocks start filling up, then it falls way behind.
evoskuil14:19
Note that BN also does not have intrinsic sha256, but since it fully uses all 16 threads that’s imperceptible.
👍 1
evoskuil08:17
I'll add a ratio of time/bytes soon, just need to pull out cumulative wire serialization sizes. Also currently pulling a bc-linux sync with
txindex turned on (we always index).👍 1
evoskuil14:16
replied to a thread: view
I suspect it’s largely a result of not having intrinsic sha256. It doesn’t really matter how much bandwidth, RAM, CPU or SSD speed you have, Bitcoin Core operates more like a single threaded application. The Mac is faster until blocks start filling up, then it falls way behind.
evoskuil14:19
replied to a thread: view
Note that BN also does not have intrinsic sha256, but since it fully uses all 16 threads that’s imperceptible.
👍 1
evoskuil16:09
Finishing up the overnight sync of BC with tx indexing turned on. Looks like it will have added about 30 minutes and over 50GB. Libbitcoin always has tx indexing enabled (and also spender indexing, which is almost 3x as many entries). Nevertheless, with
(edited)txindex enabled the BC store is about 25GB larger than BN. I'll add a line to the graph(s) when I get a chance.evoskuil17:36
I've been thinking a bit on Bitcoin Core (really all bitcoind-based implementations). Obviously the above paints a bleak picture, and maybe Libbitcoin is showing that failure isn't necessary. I think that, realizing the performance issues but not realizing that they are implementation-based vs. inherent/necessary, has driven development in a certain direction for years. There are SPV wallets and pruning, but also "assume valid" and "assume utxo". It is becoming increasingly likely that few people will actually validate or be able to bootstrap others (really already the case). Development is moving toward greater and greater reliance on trust to as the only apparent solution.
💯 2
But the central problem these are all trying to solve, the cost/time of running a node, resolves itself over time in accordance with Moore's Law. This is something that Satoshi articulated, and something that I've believed since starting work on libbitcoin. But most core devs really do not seem to believe it. I'm older than most, if not all. I've seen it. I started coding on a TRS80 Model 1 - with 16KB of RAM and no hard drives anywhere except companies with Winchester drives (some didn't even have floppy drives, if you know what those are). Now we text each other high def videos over the air for free.
jungly17:42
It's not just the limitations at sync time. I think how we think about the possibilities of using bitcoin changes too. Take for example silent payments. I think they are really cool, but can they scale with Core is the question I saw being discussed. Maybe there are solutions now, but you feel like you are in a corner. Basically, how we can use bitcoin can change if we have a properly designed system.
TBH, libbitcoin has been a beacon of hope for people like me, who just couldn't find the resources to commit time to the project.
(edited)❤️ 1
evoskuil17:43
It cannot resolve itself unless the node software scales. Scaling means that when you add hardware to bounded resource, performance improves proportionally (until hitting the next hardware limit). So libbitcoin is designed to do that, and we ignore the workarounds for the problem that should not exist. That allows Moore's Law to apply to full node performance.
💯 1
This is really just background commentary for my actual thought. It seems that people think that bitcoind architectures like Bitcoin Core can just adopt some feature or technique that we have discovered. I've thought about this, like what would I do if I had the job of improving bitcoind performance now that it's been shown to be possible for nodes to scale...
jungly17:50
Yup, you need a fundamental rethink from time to time. By locking yourself into a single design you are really making life hard, to put it politely.
👍🏻 1
Windows 3.0, windows xp, windows NT, windows 10
Fundamental rethinks, not patches!
1 reply · evoskuil · last Oct 29, 2024
evoskuil17:53
So you might start with just parallel block download. Create a robust asynchronous and lock-free network stack that distributes work and manages peers, to get candidate chain blocks down quickly. So you get them and store them out of order. But you cannot validate them out of order, and you can't cache them in RAM unless the concurrency window is very small, defeating the advantage (I implemented and measured it)...
So you chase the downloads with a validator, that spins up max threads to validate blocks as the population gaps in the chain are filled. Given that block 1 can be a gap until the very end, you introduce a window that must close out every 50k blocks or so (we tried different sizes). Then you can keep the validator moving with the downloader. Or on smaller machines, you just defer validation until downloads are all complete (we've meander both approaches)...
But then you realize that since you are bitcoind, you have some other issues to deal with. A utxo store provides a material advantage in terms of validation, however it cannot be populated out of order. So you must populate it in the validation chaser, as download gaps are filled. You read each block back out of the store, deserialize it in full, and push all the outputs, and then pop all of those that are spent in that block. Then you perform validation on the block...
But then you realize that the validation of one block must complete before starting the validation of the next, since otherwise you have lost the required ordering of output spends that was provided by the utxo store. So you are back to linear validation (one block after the other). At which point the benefit that arises from the high cost storing and then reading each block is wiped out by the fact that validation has not been parallelized, and it looks pretty much like what you started with.
That's when you realize that the utxo store is not an actual optimization, it's a premature optimization.
💥 1
evoskuil18:08
So then you toss that out and search for another way. Eventually you realize that you must store all outputs with the blocks anyway, and if you need validation to be disordered, then you don't care about output ordering at that point. So you just toss out the utxo store, and all is great. However, you know that outputs must therefore be indexed by their spenders (more than one tx can spend an output, unless you have a store that supports random access delete/write - which is a big additional cost for a state store that has almost zero change over time). And it obviously needs to be constant time, so you construct a hashmap of all outputs by input point. And not wanting to store all outputs twice (that's about 1/3 of the store size), you factor the block into a relational data model.
evoskuil18:17
Now you can read all previous outputs for all inputs for all txs in any block as long as all previous blocks are already stored, which the validation chaser ensures. So far so good. But then you realize that you are going to need to be able to determine chain-relative things like height, mtp, active forks, etc. (context) in order to validate. And for a given tx, this context can change, as txs end up in different blocks (and no block). So you realize you need to store this context for every bock with the block header so that its available out of block order. Or at least, the validator needs to maintain this context for the candidate branch and issue it to each block validation job. But since you have already validated the headers, which required the same context, and have already downloaded and checked the blocks out of order, which also required some of the same context, and you need it for other things, you don't want to keep regenerating it, you decide to store context with each header.
evoskuil18:23
All is now good, and it's time to deal with that issue of tossing out the utxo store, which inherently assures confirmed and not double spent previous outputs. This then requires that as concurrent validation jobs complete, something must synchronize them (put them in order) for the purpose of verifying output confirmability. So another chaser is in order, this one requiring a total order over blocks (one block verified at a time). But then another problem. How do you know that a previous output recovered from the input point search is confirmed (prior, since the order is imposed by the chaser)? There are blocks that are being downloaded (and/or validated) well past the height of this confirmability point, that's what makes it concurrent! So you must be able to navigate from the previous output to its confirmed state (these states being set for future blocks during this same phase).
evoskuil18:33
So now you need navigation from header->txs->tx->inputs->input->output->tx->header->confirmation_state. Of course you already needed navigation from height->header->txs->tx->inputs and ->tx->outputs even if just so that a block can be reconstructed. But you also need to parse out scripts and witnesses as otherwise store queries, such as this confirmability verification, would be horrendous as they don't require these massive amounts of data (scripts/witnesses are most of the chain). So tack on input->script/witness, and output->script as additional factoring (probably have to be independent tables). Somewhere back there (I was in Milan when it came to me) you realized that there needs to be an association table between header and tx, because headers get stored before blocks (transactions) arrive (or you could just keep all headers in memory until blocks arrive :{, and you have no way to know how many txs will arrive. Which is why there is a txs association table. If only you had a random access store you could write the association into the header table, and not have to implement an index for txs, based on the header surrogate key. But alas, append only makes the store fast, so tough call.
Then you realize you need to store this confirmation_state somewhere, and realize it's just an association table between confirmed tx and header. So that's great, you also get reverse navigation for confirmed transactions, a query server bonus. Then you realize, holy crap that's a really big query resulting from elimination of the utxo store. Yep, that's where the true cost of validation concurrency rears its ugly head. So you think hard about how to make a store that is ridiculously fast. You cone to grips with the fact that no generalized database software is going to do that nearly as fast as a specialized append only set of in-memory hashmaps. And then you decided that mmap will make this possible, but has some serious drawbacks.
evoskuil18:45
So you spend about 8 years wrestling with that issue, lots of prototyping, making progress in other areas. And finally if you are lucky enough to be on a call with @jungly , it comes to you that you can solve the mmap problem. All you need to do is start over from scratch, redesign the from-scratch store, from scratch again, splitting the headers from the bodies so that you can be truly append only, have true fault detection and recovery, at no material cost to performance. So you build that whole thing and no shit, it really works. So you're back to making that confirmability query perform in constant time per block, because getting rid of that utxo store was the central issue, and this is the resolution. And that works by then there's the mmap paging issue, all over the place. So you work on that, realizing that in the end, old Gordon Moore is coming to the rescue. With the full chain in RAM, all queries are stupid fast, and you can do things like download, parse, hash, check, index and store 350,000 txs/sec on a $350 mini pc.
(edited)Then you realize that your node is running faster on $350 pc than the $12,300 pc you bought 7 years ago so that you could test to see future performance. So you wonder why the hell anyone is worrying about linear chain growth. So you fire up some nodes and do some perf comparisons. Holy shit, maybe it was worth it, maybe we can save Bitcoin.
🙇🏾 1
This is the very abbreviated version of what someone would go through to get to where Libbitcoin is, of course this is the largely self-funded version, but you probably get the point. It's not likely something that can be bolted on.
evoskuil18:56
evoskuil19:09
Some of the other areas of importance not discussed are, mmap: store must be fully write/write, read/write, and read/read concurrent and lock-free. mmaps grow and must be independently (by file) remapped, but that invalidates every outstanding pointer into the map. heavily concurrent reading and writing can cause mmaps to thrash. process termination with a mmap unflushed corrupts the file data. snapshotting (or pausing to snapshot) a live mmap implies transactional commit. automatic snapshotting, fault detection, and recover, as well as hot backup are crazy features to be able to provide given above, not to mention that it takes about a minute to snapshot the store while running.
evoskuil19:15
Query performance is critical, and my tests show that we are orders of magnitude faster than ElectrumX and Fulcrum (and no doubt BC but I haven't tested). Making a node useful means providing features that are always being bolted on to bitcoind. We provide the full chain Electrum address index during sync, for about 30 mins extra presently. That takes ElectrumX and Fulcrum up to a week when operating against a fully-synced and local bitcoind. So where is the performance issue? probably both. The ability to expose the node to high volume query is important, as otherwise yet again it requires full extraction to a capable query server making the node much less usable. We worked through this in BS in versions 1-3 using ZeroMQ and it's mature. Yet another one of Amir's solid design decisions (same with proactor async and mmap hashmap store).
(edited)evoskuil19:34
Libbitcoin isn't just a node, it's a developer toolkit. It has significant public API. The code is intended to document itself, including the behavior of Bitcoin over its history. We don't remove old behavior. New forks are additive, so that they can be disabled/enable by the user/developer. This preserves the history and exposes the behavior. These are significant undertakings in addition to just writing a node, server, or client stack.
The script machine is place to learn exactly how Bitcoin works, which few people would ever be able to do using bitcoind, and there are no compromises on performance. In order to get more out of script, we implemented a variant stack, which eliminates state stack type conversions in all standard scripts, and templatized the stack data structure (and compile with both std::vector and std::list), which resolves the quadratic op_roll issue.
To simplify maintenance and centralize optimization of hashing functions, we implemented the entire hashing stack from spec in C++ templates, and benchmarked it against bitcoind optimizations. It tends to outperform as a consequence of improved use of compiler auto-vectorization. There's no longer any C code in the library (though we do embed qrencode, which is C). Dropping in SSE, AVX2, and AVX512 was trivial (I don't believe Core has AVX512 yet), and we'll get SNANI and ARM variants added as well, in the same template.
These are investments that are starting to pay off. The code has been absurdly reliable, which was not the case in the old days. This has made it possible to keep pushing forward without having to look back as we used to have to do a lot. But it took years to get to this point. We still have a way to go, but people are starting to notice. And that's good, because there needs to be some group that speaks for maintaining the decentralized nature of the design even if it isn't quite ready yet. I have spoken up about these types of design issues for years. But it goes nowhere, which I understand. Code talks, and Cypherpunks write code.
❤️ 2
evoskuil21:03
Here are the promised charts with the addition of Bitcoin Core sync on the Linux mini with
txindex enabled...
🔥 1
As the first chart shows, time to block 850k increased by over an hour (1.35 hours).
Total store size increased by 57GB (to current top, 850k would be around 55GB). So libbitcoin is significantly more space efficient, given that we also always index txs and spends, and spends are output scale vs. tx scale.
(edited)evoskuil21:35
Something interesting I noticed about Core is that the Windows store (/blocks and /chainstate) is 659GB (full chain, node stopped), vs 709GB on Linux. 🤷🏻♂️
The disk activity is a consequence of insufficient RAM to hold the chain, otherwise the disk would be entirely silent for the whole run.
14.8 minutes to block 500k so far. I'm doing this open window run to see how it affects the tx rate. It should be more evenly distributed, but I notice that since the work is parceled out initially in order, and the first couple hundred thousand blocks finish in seconds, it's still weighted toward the smaller blocks up front.
(edited)evoskuil22:27
14.8 minutes is below par for a typical run, but since the blocks are more broadly dispersed, the rate will be flatter. so slower up front but faster overall. Passing 650k (58%) now and it's almost caught up to par at 37 mins.
evoskuil23:16
Sometimes a relearn things while performance testing. Smaller memory machines like the mini can get into thrashing with the blocks too dispersed in terms of height. This increases read paging for hashmap conflict lists, blocking writes. So an open concurrency window is only for big memory machines, like my dev box (256GB) or the big mac (768GB).
Still finished in a respectable 84 minutes, but this is a < 60 minute box and thrashing is bad. This is what it looks like in comparison to a typical run.
Wednesday, October 30, 2024
evoskuil02:17
Ran the same basic open window (unlimited block download concurrency) with a fix to the work allocation so that each peer can get a much larger chunk (up to the protocol limit of 50k). Since the new benchmark is another 50k large blocks, it's doing a lot more work, but finished in 57 minutes, so still under 1 hour. But also got a somewhat more balanced block distribution.
1 reply · evoskuil · last Oct 31, 2024
evoskuil04:43
*another 10k large blocks (bumped the benchmark up from 840k [halving threshold] to 850k)
The big box open window run is overlaid on the two previous. A couple observations... the slight uptick in slope as each run is completing is a consequence of fishing around to get the last blocks as other channels start to close out. In the previous runs the 50k windows are visible, as the rate slows to close out the window and then reopens. This is not the case with the fully open window run in green, it's smoother - which means faster.
The green line is both shallower and straighter. Shallower because it's faster (see above) and straighter because the blocks are more randomly distributed (still very ordered by accident in the beginning, because they are so fast and complete almost a soon as they are assigned, as the others are still being assigned. I'd have to do some work to randomize that, which isn't ideal for production, since the loose ordering generally helps performance.
This also creates some slight curvature even between 400k and 800k. The hash maps are not actually constant time unless they are full memory resident (paging increases with store size and insufficient memory), and there are fewer hash collisions with empty tables. So there will also be some slight amount of non-linearity as the chain builds, but query for a full chain will remain constant with sufficient RAM, regardless of the chain size (assuming proper hashmap bucketization).
evoskuil02:53
I layered that last run into the previous charts in order to see how it looks when we aren't chunking up downloads into a 50k window. As expected, the lines smooth out and the sync rate actually continues to increase right up until completion. IOW the synch is getting FASTER as the store getting BIGGER. This isn't as evident when the blocks are following a particular pattern along then chain, but when they are more randomly obtained it becomes obvious.
The light orange line is the sync with a 50k window, and as can be seen, that's actually faster (higher rate shown throughout) than the fully open window sync on the much larger RAM machine. This is because the mini has much faster RAM and SSD and so syncs about 5-10% faster. But the important point is that the greater block randomization shows that the store remains constant time, if not FASTER as the store gets bigger!
😮 1
evoskuil03:45
Validating now on the big box. This is a test of validation after download (small box target) with read ahead caching of 1024 blocks, to ensure that the CPU is never starved.
64 threads at 100% (
block.accept() and block.connect()), disk reading at 100% (block read and prevout population) and writing almost nothing (block state entry). [first image]
As block size increases (in order), memory will fill up with table pages but will not thrash (reading only) and disk reads will vary to maintain the read-ahead block/prevout cache. [second image]
All that work we did up to this point is to get here so we can validate at 100%, because it doesn't get any faster than that.
2 replies · evoskuil · last Oct 30, 2024
evoskuil05:28
It's not actually 1024 blocks, it's 1024 * 5000 txs (simulating 1024 very full blocks by tx count, even if blocks are empty).
evoskuil13:15
On Windows at least, the OS will just retain the maps until something else wants the memory. No point at all in freeing it up. So the query response remains optimal even when it's been idle overnight.
evoskuil03:47
This is a validation of the open download window run, which means more disk paging due to lower store locality, but that shouldn't slow the CPU much.
Despite all threads at high priority (probably overkill) and full utilization, the console is still responsive and spits out store stats almost instantly when queried using
(edited)i <enter>.This is where SNANI and less-than-three-year-old copy of libsecp256k1 would provide some benefit, could be significant. Will get on updating that NuGet build with libsecp latest, but SHANI is on the back burner. Taking volunteers 🙂.
5 replies · josie, evoskuil · last Nov 05, 2024
josie09:18
fyi, i think this would be useful for you guys, as well: https://github.com/bitcoin-core/secp256k1/issues/702
#702 is something im planning to pick up early 2025 and work on, would be happy to help with SHANI stuff here a) for my own benefit to get hands on and learn and b) to dogfood the implementation in libsecp256k1 with a real use case
evoskuil13:21
Thanks for the offer. You should check out the libbitcoin-system hash library, specifically the sha templates.
https://github.com/libbitcoin/libbitcoin-system/blob/master/include/bitcoin/system/impl/hash/sha/algorithm.ipp
This integrates native vectorizations
https://github.com/libbitcoin/libbitcoin-system/blob/master/include/bitcoin/system/impl/hash/sha/algorithm_vectorization.ipp
and has shani/neon (compression) stubbed in and commented.
https://github.com/libbitcoin/libbitcoin-system/blob/master/include/bitcoin/system/impl/hash/sha/algorithm_compression.ipp
evoskuil13:29
Also, WRT Delving, if you scroll back here in General you can see that yesterday I commented many of the Delving comments. I’m personally avoiding getting into a protracted broader discussion about design and differences with Core at this point. At some point I’d love to do that, but I know it will take too much focus away from things getting done here. If you would like to ask questions here and translate to others that’s fine and we’re happy to support it. Your initial observations are generally sound.
josie18:24
awesome, thanks for the pointers re: sha, will take a look! also WRT to delving, i totally understand. these discussions can take a lot of effort. from my side, im simply trying to avoid needing to be in all the chat rooms following all the discussions, so would rather have it one place in the public sphere. like i said on delving, in the short term im happy to have discussions here to plug my gaps, and then translate it back to delving; just be charitable if i fuck up the explanation, its certainly not malicious 😄
evoskuil18:28
No worries, it's not simple stuff, and so very different than what people are familiar with!
😅 1
evoskuil03:56
The high memory consumption here is misleading, and we want max CPU (can of course reduce thread priority if we want), and the disk activity is not excessive. As the blocks and prevouts are paged through RAM the OS keeps as much in RAM as it thinks is optimal, but can and will dump any amount of it at any time. It's just lurking there in case it gets read again, not blocking other applications.
evoskuil04:03
To validate the full chain requires reading all blocks (99%) and all prevouts (15%) of the store body (not including heads and paging through conflict lists). But the disk should generally be much faster than the CPU can validate (which is apparent in low disk utilization), in which case it is only a performance factor in the CPU utilization in read/parse.
evoskuil04:18
In the heat map we can see that secp256k1_ecdsa_verify/tweak_add are taking 69.24% of the entire run, and sha256 compress and schedule (for signature hashing) are taking 8.94% for 78% of all time. This is not Merkle hashing, so it will be reduced to about 1/5th by SHANI, to less than 2%.
evoskuil04:30
99.8% of the run is contained within
node::chaser::validate_block and 85.6 is in chain::block::connect , leaving the rest to block read/deserialize, populate prevouts and context, block.accept() validation, storing the block validation state, logging and deallocation.
evoskuil05:07
Of
block.connect, which executes all script and is 85.6% of the entire run, 98.7% of it is in op_check_sig_verify and op_check_multisig_verify , which is pretty extraordinary. All other opcodes fit into that remaining 1.3%. These two ops are of course dominated by libsecp and sha256, as well as a small amount for tx serialization for signature hashing.So as it stands, our two major optimizations for validation are implementing SHANI and updating libsecp (for win32, the other builds are current).
evoskuil13:15
replied to a thread: view
On Windows at least, the OS will just retain the maps until something else wants the memory. No point at all in freeing it up. So the query response remains optimal even when it's been idle overnight.
evoskuil13:38
5.9 hours to validate the open window run from big box cold storage (previously synced and closed) to block 850k, with ~85% in
(edited)op_check_sig_verify and op_check_multisig_verify . This machine can sync and validate the full chain in about 5 hours with the current build, but running after sync is better for smaller machines that already have CPU maxed out and/or are subject to thrashing in this heavy read/write scenario due to low/slow RAM/SSD.
So I'm going to make that choice a boolean config option, and while I'm at it will add the same for enabling confirmability checks. Since I've per-analyzed the variations on fan out, and a lot of tuning on both of these aspects, automating that choice is really the last big item in sync.evoskuil15:31
This shows a very consistent time difference (lag) between the buffering of each job and the completion of validation of the net block. The buffering progression is initially ordered from 1-850k, but the validator is asynchronous, so completion is out of order within what is buffered.
Initially with very small blocks, the validator is fast enough to preclude any backlog (note that the x axis is not time-based but block height-based, so the first 1/5th of the graph completes in one minute).
The backlog starts accumulating until it hits the arbitrary 1024 * 5000 tx limit (configured). Then it is cycling back to zero delay before reaccumulating a backlog. This is because the validator owns the CPU while it works off all of the jobs, and its completion handlers don't execute until all the backlogged jobs are complete.
This implies some amount of waste as the CPU drops off the peg momentarily in order to get the backlog restarted. However without a limit the backlog would include a job for every block. The tx count that imposes the limit is read directly from the store, not from the deserialized block, since of course the limit is imposed prior to the read/validate job.
The waste can be seen infrequently in the perf monitor as narrow CPU slices that drop off the 100% peg. Ideally we would want to be always buffering jobs while validating, but we need a limit and that's hard to measure if the stranded job completion handler invocation is blocked by the ongoing work.
evoskuil15:41
The amount of concurrent validation by block is limited to the physical thread count (or configured if lower), and the block/prevout reads occur in those job executions as well. So the job backlog is just closures held by ASIO, containing nothing but the surrogate key of the block header, so the backlog is very low cost. Still, we limit it to prevent it from buffering the whole chain. That may not be strictly necessary, but I've seen very large ASIO buffers (like billions and even trillions of jobs queued, not t like the ~million height entries) bring the node to a halt.
boost::asio::post(threadpool_.service(), BIND(validate_block, link));
Bottom line is that it may make sense to maintain a large backlog limit, and not bother making that limit a function of actual work (i.e. tx counts, which requires a store hashmap read for each block) as the issue with the backlog isn't CPU work but the number of closures held in this dedicated ASIO validation queue. And I don't think a queue on the order of millions is much of an issue on any machine, as long as it can be drained without doing the work, which we always ensure:
void chaser_validate::validate_block(const header_link& link) NOEXCEPT
{
if (closed())
return;
...
} (edited)evoskuil15:51
The limit above was about 5M txs, which was intended to vary the backlog to be 1024 maximal blocks. And that's actually about 180K blocks initially (as is evident in the plot). So this run handles that .18M ASIO message backlog no problem.
evoskuil15:57
So the conclusion is that we could increase the buffering and reduce the CPU valleys, but the buffering itself occurs on a single threaded loop and that itself is part of the valley, which would get wider if less frequent. But if there's any optimization in the aspect, it's in minimizing this synchronization cost. Keeping in mind that this graph doesn't actually present that cost, but when backlog delay goes to zero, it implies the backlog has been drained and rebuilt.
evoskuil17:15
After resetting the same store to the pre-validation state (using
-r) and using a build that changes backlog measurement to blocks vs. tx count, the visible CPU valleys are gone. This makes sense because the query to obtain a block's tx count, as it is queued and again dequeued (the deque could have been passed via closure as an optimization), is a hashmap search, and has a material cost. This also changes the buffer limit from a hardwired tx-based value of about 5M to a simple block window, and uses the concurrency_window setting for that purpose (since the values reasonably related to perf, and this use isn't too significant).evoskuil19:01
This corresponds to the run above. The spikes are also visible. Not sure what's causing those yet. But this shows that validation cost is very consistently constant time by tx (probably input volume), consistent with the perf tool showing 85% costs in tx.connect.
Taproot is not implemented, and given there's a partial ordering by block height, that's interesting. I would expect to see validation cost drop off as the percentage of TR increases.
evoskuil19:22
This is a new partial run with the above-described change to simplify the buffering limit. At the stopping point it is 3.6 minutes ahead of the previous run against the same data. That's at about 50% of the chain by tx count, so saves about 7.2 minutes over a full sync.
evoskuil19:32
This overlay shows that the change is delaying what I think is the memory map reorganization, saving time. This makes sense given the removal of two hashmap queries per block.
evoskuil20:00
Turns out I still had config set to an unlimited window, so the overloaded use of the currency window setting for the buffer resulted in this second run being fully buffered (850k blocks) in the first 6 seconds of the run, and the run was faster. So clearly the buffering level isn't too significant, but the removal of the tx count queries at least saved 7 minutes.
evoskuil22:35
Concurrent validation and download on the big machine is obviously limited by CPU. But interesting that the CPU (2 dual 32 core 2.1GHz) is able to maintain 50% network utilization. This implies that a 4.2GHz (or 64 dual core 2.1GHz) machine would be able to entirely unblock the network (completing in 1 hour). But it also implies that if we can cut validation cost in half (e.g. with shani [10% 5x faster] and updated secp256k1 [75% ?x faster]) we would achieve the same result.
evoskuil22:54
Implementation of shani (though it's not in this pc hardware) would thus drop 8% off the total, meaning that we need 42% from secp256k1, or a 56% improvement in its throughput. Seems unlikely, but I've been testing on my nearly 3 year old Windows build of secp256k1 published via NuGet. Maybe time to update that, or at least build it locally.
(edited)evoskuil23:10
Above isn't entirely accurate. The network is maintaining 50% capacity under validation, but it's getting well past the validator in terms of work completion. Currently have 550k blocks downloaded with 400k blocks validated (in 39 minutes). That's 10.9% of the 850k chain by txs, and bn is fairly constant by tx, so we're looking at completion in 6 hours total (download and validation, not confirmation).
(edited)evoskuil23:18
So a 50% cut in cost of the main APIs would reduce it to 3 hours. However it's also worth considering that there will be about 40% more CPU available once downloads complete, so it should beat that time. The balance between validation and download can be tuned by adjusting the validator thread priority (configurable). A higher priority ensures that validation doesn't get very far ahead of download, but completion total should remain about the same since CPU will be fully utilized either way.
evoskuil23:32
50% downloaded now at at 53 minutes, so should complete download at 1.7h. So
[6h - 1.7h = 4.3h] gets accelerated by 40% [1.7h + 4.3 * 60% = 4.3h]I'd be very happy to get that down to 3h with the single thread optimizations.
Thursday, October 31, 2024
evoskuil17:25
Had to restart the run above after running into an intermittent store bug that was detected as a consequence of full validation. No impact to previously-determined benchmarks.
So after fixing that I started up a concurrent download-validation on the windows mini overnight. That started hot but then CPU utilization dropped to about 60%. I assumed at first that it was thrashing because the drive was close to max. But I decided to increase thread count (set to 16 to match hardware) just in case it was configuration-starved. That bumped it up to about 95%. Increasing it more didn't help, but advancing validator thread priority got it up to 100% for most of the time (both network and validator had been set to normal). So it's making much better time. I'll plot the whole thing so as to see the inflection point.
The best news in the above story is that the mini does not appear to be thrashing during validation, even with concurrent download, which I certainly didn't expect.
evoskuil18:20
I lost a big chunk of the numbers between restarts, but the validation time for a couple big samples is now 65-70% of what it was before the config change, which is what I would expect if it was just straight CPU time and not about thrashing.
evoskuil19:00
After 2 hours no material change at all in rate of tx validation (slight improvement). So looking very constant across the chain. Should complete in about an hour.
(edited)evoskuil19:39
I think I understand the taproot (not implemented) performance behavior. Now that it's in that range the CPU has started to drop off the peg, but SSD remains maxed out. In other words, CPU and SSD are pretty well balanced in terms of being the limiting factor (both at 100%) so as CPU demand falls, rate remains constant. IOW the times on the mini are likely reasonable without the implementation. But that also implies the mini might not benefit much from CPU optimization, apart from not needing priority and/or as many configured threads.
(edited)It's hard to imagine however that it's significantly faster to read all blocks off the network and store them, then it is to read them (with prevouts) all off of the disk in a pre-parsed format. But that's the cost of paging. With sufficient RAM there is no page swapping and we are strictly CPU bound.
Huddle with pmienk, evoskuil, parsevalbtc · 165 min19:57