Bitcoin full node and query server
Apr 13, 2024
View in context in the libbitcoin Slack #general mirror.
evoskuil17:02
Synced and confirmed 804,000 blocks in 1.5hrs, which is about 90% of my peak network bandwidth for the data download alone. This is with validation bypassed by milestone as that is not currently parallelized. However confirmation was formerly a very costly exercise in previous libbitcoin versions, since it required updates throughout the store. The new design simply writes a record for each confirmed tx and block so is very fast. This could also be accelerated under checkpoints/milestone because it can be done concurrently.
132 replies · giphy, evoskuil, jungly, Marnix · last Apr 14, 2024
evoskuil17:53
That's confirmation, but validation write costs are the same, just a different table. They are distinct because a block/txs can be valid but not confirmed and can become unconfirmed and reconfirmed - and txs can have validation context for different blocks.
evoskuil17:55
The remaining big costs are reading transactions from the store to validate scripts, reading prevouts for the same, and performing that validation. These are all fully parallelizable to the extent that prevouts exist (and in some cases without them). That's the outstanding work right now to complete sync = the parallelizations.
(edited)evoskuil17:59
I'll have to give Core a run on my machine. I've got 64 cores and 256 GB RAM, though it's an old machine.
(edited)evoskuil18:33
7 minutes in and I'm at block 28,181, and these blocks are like 1kb. Libbitcoin downloads, stores and indexes the first 300k blocks faster than that.
(edited)evoskuil18:40
It took 3x longer to sync headers. The pre-header sync thing it does to protect against DoS doubles the time and achieves nothing more than we do natively.
jungly18:42
iirc, there's a small config change you have to make to the default config. Afk, so can't check.
jungly18:47
Ah yes. The connections, they ask you to keep them below 8.l iirc. Maybe cause concurrent can't handle more.
jungly18:48
But I think you should definitely complete a sync with default config.
Lopp's yearly blog post might have some config suggestions
jungly18:50
This is what lopp recommends from 2022
connect=<local node IP address>
assumevalid=0
dbcache=24000
disablewallet=1
evoskuil18:57
It looks like the block overhead is so high that it crushes the download rate for smaller blocks. The rate is going up a bit now that blocks are bigger.
evoskuil18:58
CPU is at zero because there is no validation. This is just indexation and storage.
(edited)evoskuil19:17
The tests that JL has been putting out are super misleading because they don't take into account peer variability, which is the hard issue and real issue.
jungly19:18
An ideal repeatable test is to set up ten peers and connect the node under test to the same ten
evoskuil19:20
The node has to be able to manage the network. 10 peers is WAY too low, and doesn't allow for perf differences.
jungly19:26
I agree that being able to handle more connections is a huge plus.. and setting up all those nodes that you can handle will be a bit hard. Maybe then the test has to be on theb love network, but repeated a good few times. However, we aren't trying to get reproducible numbers, just a good enough ballpark
evoskuil19:27
The network is big and varied enough that you do get consistent numbers if you do it right.
(edited)evoskuil19:29
If he set up 5000 peers with varying speed from 0 to GB and ran each of us against them, we would max out the LAN bandwidth and Core would take whatever it gets.
evoskuil19:36
It's approaching the end of the checkpoint range (295k). We can process the first 300k in just a few minutes.
evoskuil19:41
It's hard to imagine that I started working on this when the top block was less than 300k. That's just a couple percent of the present chain.
(edited)evoskuil19:46
300 is an inflection point, blocks start getting bigger. Because they are sequential this increases the download rate, so now they are hitting maybe 75MB/s (vs. our consistent 980).
evoskuil19:51
Interesting, the number of outbound connections has increased to about 30 and still slow.
evoskuil19:54
Ok, that's 1.5 hours. They are at block 310k, which is about 5% of block 804k, where we are at 1.5 hrs.
evoskuil20:06
Interestingly, their shutdown time at this low store size is also long, like 30-45 seconds.
evoskuil20:07
Since they do not max out any aspect of the hardware, it implies no scalability. We are capped by WAN speed on my machine. If I had a 10Gb/sec network we could probably complete in 10 minutes.
evoskuil20:10
So you see why I wasn't sweating the store locking? it's so small compared to everything else I haven't found it's limit yet.
evoskuil20:13
Bitcoin Stack Exchange
Checkpoints prevented an attack where a node could mine many many low-difficulty blocks from an early point in the blockchain and serve those blocks to syncing nodes who would see their disks fill ...
evoskuil20:15
What huge steaming turd. We just load the headers into a hash table until they are valid and current with sufficient work or are under milestone. Then they organize into the candidate chain on disk. Simple and same result in a single pass.
evoskuil20:16
I've seen this complete in 10 secs, typically more like 30-45 since we don't manage peer perf for this, and we redundantly hit peers.
(edited)evoskuil20:19
The difference is that we could have the hash table blow up if we're getting sybil attacked, but in that case you just drop the table and peers and try again. Because it's a race between multiple peers, only one good peer has to beat the bad.
jungly21:47
I don't fully follow the "disk fill attack". Wouldn't all your peers have to send the same "spam" headers? Basically you need to be eclipsed before you can be attacked, right?
evoskuil21:56
No, a peer can give you weak headers forever and until you find a stronger and current branch (and even after) you have to take them somehow.
evoskuil21:56
Because they can eventually become stronger. The only limit is that they can't pass through a checkpoint. But there are infinite branches possible below and between checkpoints. And now above is feasible too.
(edited)evoskuil21:58
Below and between checkpoints can be resolved by simply enumerating all points up to the top checkpoint. Ugly but works and not a fork.
jungly21:59
Gotcha. So just in case they become stronger later on. But once at checkpoint you can clean them up.
Isn't the answer simpler if you throttle each peer when downloading weak headers? Connect to more peers instead to speed the up?
(edited)evoskuil22:00
You can reject anything below a checkpoint once you have the full set of hashes that reach the checkpoint (like the headers).
jungly22:02
So you just need to get to the first checkpoint in the headers chain and that requires a single honest peer to send you 9mb.
evoskuil22:03
My plan is to allow you to export the hash set up to any point, and then use it to bootstrap a node.
jungly22:04
I was thinking the same. Just the peers can see you 9mb in a single message as headers up to the first checkpoint
evoskuil22:06
And worst case they are blowing up your ram until you dump it (not your disk).
(edited)jungly22:07
You can do relative throttling tbh. Sounds complicated. But tells a spammer to slow down. Problem is slow nodes will slow down your sync
evoskuil22:08
The Core approach is reasonable, it just feels like some serious complexity and over-engineering.
evoskuil22:11
I don't get the 600 block gap though, since an infinite number of blocks fit between any gap of 1.
evoskuil23:47
Some of the poor performance will be coming from my machine not having sha-ni. What this means that that, because I haven't implemented that optimization in our sha template, it's a fair comparison.
evoskuil23:49
IOW when we that's implemented we will also have a 5x improvement of all sha hashing, which is a material reduction in overall time, which should be proportional for Core. If we both got a 20% boost that would knock our 1:30 time for 804k blocks down to 1:15.
💥 1
evoskuil23:51
Together with other optimizations that are still available I'm hopeful that I can get 800k blocks in an hour.
M
Marnix07:13
Can confirm from Core IBD experiences on "normal hardware" (few) peer connections always is an issue
👍🏻 1
evoskuil07:21
It's not that there is an insufficient number peer connections, it's that the peer connections are not perfomance-managed and that they are not fully utilized due to lack of concurrency.
(edited)👍 1
evoskuil07:23
Libbitcoin implements rapid re-connection, standard deviation rate management, and concurrent requests balanced against the peer set - as well as a fully write-concurrent store.
(edited)👍 1