Bitcoin full node and query server
Oct 10, 2024
View in context in the libbitcoin Slack #general mirror.
evoskuil12:37
It is likely that the default settings will be suboptimal for your machine, even resulting in thrashing. If you post your basic specs I can make suggestions. One of the aspects we are working on is automatically tuning what is presently manual configuration for given platform.
Also note that if your CPU supports sha native (shani), which is about 5x faster individual hashing, it will distort realistic benchmarking, since shani is not yet integrated into the hashing library.
75 replies · cguida, evoskuil · last Oct 19, 2024
C
cguida18:30
just a pretty average laptop from a few years ago
20GB RAM
Intel i5-10210U CPU @ 1.60GHz
4TB NMVe SSD
(edited)evoskuil18:55
I don’t test below 32gb RAM with 8 threads, but the default config won’t be good. The initial table sizing needs to be reduced, and the configured thread and peer counts. The node has a command line option to tell you the relevant CPU features. I’m not able to look at config until tomorrow, out with guests now.
evoskuil18:57
My top test machine, which consistently achieves 60 minutes, is 256GB with 32 dual cores (7 years old). I scale it down to 32 GB and 10 cores to get close to matching Lopp’s test platform. I forget the time, I think 2-3 hours.
C
cguida19:08
Wow, that's a beast! No wonder you are not seeing much issue with the utxoset growing faster than Moore's Law; it will be a long time before it will start having an impact on that. I'm concerned with low-resource machines being able to sync the chain in a reasonable amount of time. The average person cannot afford a computer with 256GB of RAM (or even 32GB).
The canary in the coalmine is the RPi 4B, which used to be able to sync in a reasonable amount of time a couple of years ago, but now takes over a month.
Would it be fair to say that low-resource machines are not a priority for libbitcoin?
evoskuil22:42
That is not why the utxo set is not an issue. We do not store a utxo set, we only store blocks, and that growth is linear.
evoskuil22:46
I have never been focused on low resource machines, because they will eventually not exist. I started coding on a 16k RAM machine with no disk drives or network capability. I take the longer term view. Scalability matters. Core does not scale. Its performance does not increase relative to hardware increment. Consequently it will eventually collapse. bn performs well on a machine equivalent to Lopp’s benchmark, which is at the lower end of my target range. The market will reduce the cost of the hardware, but that will only work for Bitcoin if the software scales.
evoskuil22:52
I’ve been spending a good amount of time optimizing various aspects for low RAM. If I assumed 512GB RAM Libbitcoin would sync in 45 minutes and Core would prob take 10-20 hours. I assume this because it does not use even a small fraction of my current RAM or CPU or network or disk capacity. bn always maxes out one aspect. bn can always perform better because of this, even with lower RAM machines, but that requires work to avoid mmap thrashing.
Eventually the distinction between SSD and RAM will not be significant and the mmap will not be necessary, and network will dominate and with no risk of thrashing.
C
cguida23:29
I'm curious: how much money do you think it's reasonable to ask a merchant who wants to accept lightning to pay for their node?
evoskuil00:20
Value is subjective, reasonable depends on every individual. But, nobody is asking. People do it if it is worthwhile. Eventually nodes will be practically free and sync in seconds. That’s the future I work toward. Also, bitcoin is absolutely not a white market money, so selling coffee isn’t a concern of mine.
evoskuil03:38
I don’t mean to imply that small value permissionless txs cannot have utility. I do not work toward white market scenarios, as they are pointless, ultimately given a complete lack of security. Micropayments is an independent question. It implies either high volume or a material expense to make a small (or large) payment. Bitcoin (node) does not provide for high volume (but can of course accommodate high cost small payments). Lightning is in some ways problematic, but under its assumption of infrequent on-chain transaction, I don’t see any conflict with the above observations.
evoskuil03:44
The question above implies that cost to maintain the same sync time will necessarily increase over time. That is absolutely not the case. The cost is only increasing due to implementation. Our effort is not to merely maintain a constant cost, it is to radically reduce that cost. That requires a very different architecture from Core, which was designed to fit a certain hardware profile and does not scale beyond it. Hardware has already outgrown that architecture, which is obvious from resource underutilization. We do this because it has been clear for some time that node design has been a significant centralizing force.
C
cguida22:40
Bitcoin (node) does not provide for high volume (but can of course accommodate high cost small payments). Lightning is in some ways problematic, but under its assumption of infrequent on-chain transaction, I don’t see any conflict with the above observations.
The trouble is that anyone who wants to make money running a darknet microservices business (for example) would need to spend thousands of dollars on a server. I just think this limits the potential base of merchants who would do this to a few elites, and certainly does nothing to incentivize more people to run nodes, which of course threatens bitcoin's decentralization.
C
cguida22:42
I think bitcoin has the ability to allow anyone the option to connect their lightning node with some microservice-buying-and-selling plugins, and earn a passive income proportional to their invested capital. Ignoring low-resource devices makes the barrier to entry artificially high.
evoskuil23:16
Why do you assume that? I can easily find a min spec laptop for $300, and that’s today. Not to say lower spec wouldn’t work, just slower. Nodes will become almost instant and practically free over time. Nobody is ignoring low resource, but failing to scale will never get there.
evoskuil23:17
FWIW, I haven’t seen the passive LN thing working, and not because nodes are expensive.
evoskuil23:21
And keep in mind that libbitcoin is just one project. We have specific objectives. We don’t stop anyone else from making nodes that optimize for a pi with 4MB. But there is no reason a modest PC should take forever to sync.
C
cguida01:55
The point is that the utxoset ballooning from 4GB to 12GB has made it no longer feasible to use a RPi 4B as a bitcoin node, whereas before the recent spam attack, it worked just fine.
Keeping a high-availability utxoset in RAM makes the sync much faster on low-resource devices. On higher-end devices, sure, perhaps the difference is not massive. Anyway it obviously has no effect on devices with more than <utxoset size> GB of RAM.
So if a 12GB utxoset can make a $100 device (the RPi 4B) incapable of syncing the blockchain in a reasonable amount of time, and the utxoset can grow at 40GB/year (the current limit), then in a few years the utxoset might be larger than 100GB, at which point only high-end devices (such as the ones you use to test libbitcoin) will be able to sync the chain in a reasonable amount of time.
FWIW, I haven’t seen the passive LN thing working, and not because nodes are expensive.
It will happen, but we need more time to build automated Lightning agents. These already exist but are primitive.
And keep in mind that libbitcoin is just one project. We have specific objectives. We don’t stop anyone else from making nodes that optimize for a pi with 4MB. But there is no reason a modest PC should take forever to sync.
No one can make the chain sync quickly on a RPi 4B anymore, because the utxoset does not fit in RAM. There are some promising research directions such as utreexo and assumeutxo, but these seem to be far from production-ready and user-friendly.
The only actual fixes, that I know of, are to filter spam, make the blocks smaller, or both.
Or just give up on low-resource devices being able to sync the chain, until Moore's Law catches up to the 40GB/year utxoset growth cap, in several decades.
evoskuil02:01
Again, there is no need for outputs to persist in RAM. We are talking about sync times that are absurdly high without even performing script validation. There is no reason for scripts to be consuming RAM. Query over existence and spentness is sufficient.
evoskuil02:04
When your store is slow and non-concurrent, and your processing is unduly ordered, RAM feels like the only solution.
evoskuil02:16
In any case, I have no interest in debating block size or spam. I pointed out that Core does not scale and the Libbitcoin does, and that linear chain growth will be easily overcome by non-linear hardware cost reduction. This is what we work toward, which is reasonable and useful. If you want to work on a fork with smaller blocks and no “spam”, have at it.
C
cguida16:49
Again, there is no need for outputs to persist in RAM. We are talking about sync times that are absurdly high without even performing script validation. There is no reason for scripts to be consuming RAM. Query over existence and spentness is sufficient.
And again, this precludes low-resource devices from being able to sync bitcoin. This is demonstrated by the fact that libbitcoin is much slower to sync on devices with <12GB of RAM than bitcoin core.
You're just ignoring an entire class of users, until Moore's Law catches up with blockchain/utxoset growth.
If we just throw up our hands and say "well I guess low-resource nodes will never work", then node centralization will accelerate to the extreme. If only people can afford a machine that has 256GB of RAM can sync bitcoin, then there will only be a few people actually running nodes, and these will be quickly co-opted by the state, long before Moore's Law allows computers to add 40GB RAM/year with no additional cost.
When your store is slow and non-concurrent, and your processing is unduly ordered, RAM feels like the only solution.
RAM is the only solution for low-resource devices. If there were no bitcoin implementations with a RAM utxoset, there would not be any devices worth less than $1000 running bitcoin full nodes.
If we don't bring down the cost of running a node, very few people will actually do it, and very soon bitcoin will not be around.
Anyway, that's fine if libbitcoin wants to be the "high-resource-nodes-only" implementation. Everyone's got a niche.
evoskuil16:54
Core does not scale (up). That’s a problem going forward. To the extent that libbitcoin does not scale down (which remains to be seen), that’s a problem that will go away on its own. This is not a discussion regarding whether bitcoind has been useful, but whether it can survive. Ultimately it has to adapt, which is what we are doing. So it’s not at all clear to me what your issue is. The cost of hardware for a node with the specs I gave you is around $300 at most.
(edited)evoskuil17:03
I realize you are invested in this idea that Bitcoin itself cannot continue to operate on affordable hardware, but this is just not the case. This is something you should be celebrating.
C
cguida17:18
In what year do you expect libbitcoin to overtake bitcoind in terms of sync speed, on a device that costs $300?
C
cguida17:20
Ok, interesting, have you run the benchmark on such a device? I am trying to do a comparison between bitcoind and libbitcoin on my laptop, but so far I've been unable to figure out how to determine what block I'm synced to in
bnevoskuil17:23
As I mentioned earlier, I sometimes swap out my RAM down to 32GB and disable all but 10 CPU threads (roughly equivalent to the newer 32/8 thread machine that JL uses for benchmarking). With proper config this is still way faster than Core on the same machine.
evoskuil17:32
I gave you info on how to determine block count (the sync is not ordered). The node emits events to
(edited)events.log. I believe current build emits only the block_archived event (with block height and time in seconds from startup). This will show the individual blocks archived since the node started up. You can also type i <enter> while the node is running and it will dump store stats to the console. Finally, you can type p <enter> and see individual blocks being archived at the console. This is a toggle, type again to disable.✅ 1
evoskuil17:45
This shows that 232,416 blocks have been downloaded and stored (archived) 146 seconds since startup, and the last block reported is at height 231,110
evoskuil17:46
This is an [i]nfo dump to the console while running:
2024-10-15T17:45:38Z.0 Body sizes... header :84847420 txs :192929304 tx :2921586630 point :4068475272 input :14381059167 output :4461494048 puts :1088675541 candidate :2597370 confirmed :3 spend :2719974744 strong_tx :565389996 valid_tx :0 valid_bk :0 address :0 neutrino :0 2024-10-15T17:45:38Z.0 Table records... header :865790 tx :47122365 point :113013202 candidate :865790 confirmed :1 spend :113332281 strong_tx :47115833 address :0 2024-10-15T17:45:38Z.0 Head buckets... header :565129 txs :565129 tx :681968291 point :638910752 spend :1739749749 strong_tx :681968291 valid_tx :681968291 valid_bk :565129 address :1 neutrino :1 2024-10-15T17:45:38Z.0 Collision rates... header :1.53202 txs :1.53202 tx :0.0690976 point :0.176884 spend :0.0651429 strong_tx :0.069088 valid_tx :0.0690976 valid_bk :1.53202 address :0 neutrino :0
evoskuil17:46
tx: 47,122,365 is a much better indicator of progress than block height, as it is close to a linear progression.
(edited)C
cguida17:47
ok so i am not synced to 524493 here? the
block_archived event would show a much lower number?
2024-10-15T17:44:33Z.0 Body sizes... header :84847420 txs :189308 tx :655464 point :28296 input :162756 output :856072 puts :98121 candidate :2597370 confirmed :3 spend :270960 strong_tx :126840 valid_tx :0 valid_bk :0 address :0 neutrino :0 2024-10-15T17:44:33Z.0 Table records... header :865790 tx :10572 point :786 candidate :865790 confirmed :1 spend :11290 strong_tx :10571 address :0 2024-10-15T17:44:33Z.0 Head buckets... header :524493 txs :524493 tx :551320125 point :546188501 spend :1459791875 strong_tx :551320125 valid_tx :551320125 valid_bk :524493 address :1 neutrino :1 2024-10-15T17:44:33Z.0 Collision rates... header :1.65072 txs :1.65072 tx :1.91758e-05 point :1.43906e-06 spend :7.73398e-06 strong_tx :1.91758e-05 valid_tx :1.91758e-05 valid_bk :1.65072 address :0 neutrino :0
evoskuil17:49
This is a recent run which completed 840,000 in 61 minutes. The transaction count at 50k block intervals is in column B. This is how I typically track progress.
evoskuil18:03
You have synced 10,572 txs, which is basically nothing (see above, the first 50,780 txs takes about 7 seconds after header sync completes at 48 seconds).
(edited)C
cguida18:11
ahh here we go
$ tail -f events.log block_archived...... 768617 17 block_archived...... 757842 19
evoskuil18:13
since it's default configured for a 50k block window, you should be somewhere around 750k blocks. But that does not jibe with the info dump above.
C
cguida18:21
right, the info dump above is from a new sync i accidentally started.
here's the new one
2024-10-15T18:18:55Z.0 Body sizes... header :84847910 txs :3127058952 tx :48304299758 point :73453346676 input :181148659338 output :75376308520 puts :19056712575 candidate :2597385 confirmed :3 spend :48987159240 strong_tx :9349219308 valid_tx :0 valid_bk :0 address :0 neutrino :0 2024-10-15T18:18:55Z.0 Table records... header :865795 tx :779101609 point :2040370741 candidate :865795 confirmed :1 spend :2041131635 strong_tx :779101609 address :0 2024-10-15T18:18:55Z.0 Head buckets... header :524493 txs :524493 tx :551320125 point :546188501 spend :1459791875 strong_tx :551320125 valid_tx :551320125 valid_bk :524493 address :1 neutrino :1 2024-10-15T18:18:55Z.0 Version Information... libbitcoin-node: 4.0.0 libbitcoin-database: 4.0.0 libbitcoin-network: 4.0.0 libbitcoin-system: 4.0.0 2024-10-15T18:18:55Z.0 Press CTRL-C to stop the node.
C
cguida18:22
btw i'd love to know where you found a machine with 32GB RAM and 10 processor cores for $300
https://libbitcoin.slack.com/archives/C6LH74ACQ/p1729013014026969?thread_ts=1728563873.913659&cid=C6LH74ACQ
(edited)Eric Voskuil
As I mentioned earlier, I sometimes swap out my RAM down to 32GB and disable all but 10 CPU threads (roughly equivalent to the newer 32/8 thread machine that JL uses for benchmarking). With proper config this is still way faster than Core on the same machine.
evoskuil18:23
779,101,609 txs out of 991,032,192 (at block 840,000). But your config prob isn't set to stop at 840k.
👍 1
evoskuil18:25
It wouldn't need 10 today, that's because mine is 7 years old and that's how I approximate JL's 8 faster threads. 32/8 is what you'd be looking for. Also I have slowish RAM and SSD because of the old bus.
evoskuil18:27
I've got my PC back up now (been putting a new floor into my house), so I can help you out with some config options a little later (it's a tax day today).
evoskuil18:42
[log] path = X:/ application = true news = true session = true protocol = false quitting = false proxy = false remote = true fault = true verbose = false #objects = false # debug builds only [database] path = X:/bitcoin/ minimize = false [bitcoin] # defaults to all checkpoints. #checkpoint = 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f:0 minimum_work = 000000000000000000000000000000000000000052b2559353df4117b7348b64 milestone = 0000000000000000000320283a032748cef8227873ff4872689bf23f1cda83a5:840000 [forks] [node] headers_first = true allowed_deviation = 1.5 sample_period_seconds = 10 currency_window_minutes = 60 maximum_concurrency = 50,000 maximum_height = 840,000 # stops block sync (not header sync) snapshot_bytes = 0 snapshot_valid = 0 allocation_multiple = 20 [network] threads = 16 outbound_connections = 200 connect_batch_size = 5 host_pool_capacity = 10,000 enable_loopback = false self = xxx.xxx.xxx.xxx:8333 # advertised addresses inbound_connections = 0 # Use 1 for loopback user_agent = "/libbitcoin /" enable_ipv6 = false enable_alert = false # log alert messages enable_reject = false # send reject messages enable_address = true # in/out address messages enable_transaction = false # peer should not relay txs validate_checksum = false # validate inbound message checksum invalid_services = 176 # shitcoins services_minimum = 9 # witness node services_maximum = 9 # witness node protocol_maximum = 70012 # current default (send_headers). protocol_minimum = 31800 # Satoshi:22.0.0 minimum (headers). minimum_buffer = 1,000,000 # network payload buffer (per channel) retry_timeout_seconds = 5 # 1s, 2s, 7s f/m/s loopback, 5s for large connect_timeout_seconds = 5 # 1s, 3s, 5s f/m/s loopback, 10s for large handshake_timeout_seconds = 5 # time to complete handshake. channel_heartbeat_minutes = 5 # ping channel_inactivity_minutes = 10 # receive channel_expiration_minutes = 60 # unconditional [database] # TODO: These require tuning for low RAM machines. header_buckets = 565,129 header_size = 21,000,000 header_rate = 5 txs_buckets = 565,129 txs_size = 1,000,000,000 txs_rate = 5 tx_buckets = 681,968,291 tx_size = 16,000,000,000 tx_rate = 5 # compressed size point_buckets = 638,910,752 point_size = 9,000,000,000 point_rate = 5 spend_buckets = 1,739,749,749 spend_size = 16,000,000,000 spend_rate = 5 puts_size = 7,000,000,000 puts_rate = 5 input_size = 92,000,000,000 input_rate = 5 output_size = 25,000,000,000 output_rate = 5 candidate_size = 2,568,513 candidate_rate = 5 confirmed_size = 2,568,513 confirmed_rate = 5 strong_tx_buckets = 681,968,291 strong_tx_size = 3,000,000,000 strong_tx_rate = 5 validated_tx_buckets = 681,968,291 validated_tx_size = 1 validated_tx_rate = 5 validated_bk_buckets = 565,129 validated_bk_size = 2,000,000 validated_bk_rate = 5 address_buckets = 0 address_size = 1 address_rate = 0 neutrino_buckets = 0 neutrino_size = 1 neutrino_rate = 0
This is similar to what I run. The initial [database] table sizes definitely need to be reduced for low RAM. I would cut these
_size values down to 1/4 to 1/10 of above for lower RAM. This is required to prevent mmap thrashing. Also the growth _rate values must be small. I run at 15, but these are probably okay for low RAM at 5.evoskuil18:44
There are a bunch of other settings, which you can see from the command line using
(edited)--settings or -s.evoskuil18:45
You can use
m <enter> at runtime to see the runtime options and --help or -h at the command line to see the CL options (there's some overlap).evoskuil18:47
This just completed since we've been chatting. 3671 seconds (including 40 sec header sync) to block 840k.
evoskuil19:33
This is 8 dual cores (16 threads) at 4.75 GHz. That's better than my machine (2.1GHz) running at 32 threads. The RAM is twice as fast as mine (4800KHz vs. 2400), as is the SSD bus (PCIe4 vs. 3). This could probably sync the full chain on my WAN in ~1.5 hours, for $208 and free shipping.
(edited)
🔥 1
evoskuil20:08
Looks like the price is $356 for this model, never used Temu before. But as you can see, this is more than twice the machine that I minimum benchmark on.
C
cguida20:11
If you want people to switch from core to libbitcoin, "it syncs in 2.5 hours on a $300 box" is an amazing selling point
🎉 1
C
cguida20:15
Would you be willing to share the link to this? My "searching for cheap hardware" skills clearly need work... https://libbitcoin.slack.com/archives/C6LH74ACQ/p1729020838107709?thread_ts=1728563873.913659&cid=C6LH74ACQ
Eric Voskuil
This is 8 dual cores (16 threads) at 4.75 GHz. That's better than my machine (2.1GHz) running at 32 threads. The RAM is twice as fast as mine (4800KHz vs. 2400), as is the SSD bus (PCIe4 vs. 3). This could probably sync the full chain on my WAN in ~1.5 hours, for $208 and free shipping.
evoskuil20:19
evoskuil02:26
Appears to be fully as advertised and working well. Started a sync on the provided Windows 11 OS, using my big machine config (prob not optimal config but getting a baseline).
evoskuil03:19
Holy shit. This thing synced as fast as my 256GB/32 Dual Core 2.1 MHz box. No mmap thrashing despite big box config. This is prob because the disk and RAM are twice as fast as mine.
$356 delivered, syncs in one hour on Windows (without even having SHA-NI implemented, which will be 5x faster hashing). I noticed it does have AVX512, which libbitcoin supports, but the build I’m testing with is compiled for AVX2, which computes big Merkle roots in twice the time of AVX512). So the baseline time on this machine with 2Gbps Internet is going to beat one hour.
evoskuil03:30
Much faster than big machine in the header sync and before blocks get full, but almost the same after that. The bigger block speed on this box will also be faster with AVX512 and SHA-NI. It is absolutely maxing out my current Internet speed at 2300 Gbps.
🚀 2