Bitcoin full node and query server

March 2024

Archive of the libbitcoin Slack #general channel.

Sunday, March 3, 2024

evoskuil04:47
Running initial tests of parallel block download. This is not fully optimized yet, but it is fully functional in production quality code. This is all about how fast can we get blocks for the full header chain. There are several tuning parameters in config, such as allowable negative deviations from standard deviation in block download bitrate, threads, channels, sample period, request batch size, etc.
04:52
I was tuning these settings after getting the system operational and was a little frustrated that on my peak 940 Mbps network I was only getting about a 180 Mbps rate in any configuration. Then I realized I had been testing with only a single network thread (for the entire networking stack with the exception of logging and reporting, which each have a dedicated thread). After bumping that to 8 threads I immediately was hitting around 850 Mbps - over 90% saturated.
(edited)
🎉 1
evoskuil04:57
I have some optimization work to do on the distribution of work across peers. Currently there is a ton of copying going on (block hashes with validation context struct). I'll optimize that using a set of sets, where the subsets are moved around and shared pointer owned objects until they are fully depleted. Hopefully I can get to full network saturation.

Tuesday, March 5, 2024

evoskuil16:07
After swapping out the simple std::unordered_map for a multiindex boost table I'm now able to maintain height-indexed requests with constant time hash-based indexing. This tracking is still very inefficient relative to where it will be soon (moving one pointer per channel) around vs. moving millions of records. But I've achieved full saturation of my network at ~940Mbps (peak determined by speedtest.net) as measured internally via block download tracking/logging and externally using a network monitor. I have no reason to believe this would not scale beyond my network capacity. At 940Mbps/8 = 117.5 MBps = 423GBph. I haven't measured the chain in a while, but that should be close to all of it.
🎉 3
10 replies · neillm, evoskuil · last Mar 05, 2024
neillm19:27
Core shows: 
587G blocks
11G	chainstate
47G indexes
evoskuil20:06
Thanks. That should come out to 1.4 hours on my machine then. I'm just finishing up a sync after several starts and stops and that time seems about right.
evoskuil20:35
584 GB for blockchain storage sans indexes.
evoskuil20:35
😅
evoskuil20:37
Actually that includes a lot of the indexation (hash bucket deconfliction lists are in the hash tables, including key values).
evoskuil20:53
The hash table headers are 14 GB
evoskuil20:55
With that we achieve a much more highly indexed and performant query store than Core. Validation metadata will increase that somewhat, and then of course optional indexation such as all payment addresses. But validation metadata can be purged once current.
🙌 1
evoskuil21:00
That should amount to ~40 GB, so it's 638 (or 598 after sync) to 645.
neillm23:26
Wow, awesome!
evoskuil23:34
Yeah, use of surrogate keys offers some good compression.
evoskuil16:08
This of course is writing blocks into RAM, which on this machine is 256GB. I'd expect this to slow down significantly as the map is flushed by the OS.
evoskuil16:14
The deviation tracking is set to a limit of 1.5 sd below the norm. This leads the total speed to accelerate from 200mpbs to 940mps gradually over the course of about 1 minute as slower channels are dropped. Once the peers no longer have work they are dropped from the calculation, and when it reduces below three peers a channel can no longer be dropped (due to lack of deviation). By eventually shifting work among peers until there are only 2 blocks remaining this will be fully optimized.
evoskuil16:42
Blocks are parsed, checked and stored, with the largest cost being hashing. We have merkle tree hash optimization but not the benefit of sha-ni for individual hashing. CPU is less than 10%, but should drop significantly once we have sha-ni (5x faster per hash).
evoskuil21:02
Based on my current full sync and @neillm’s Core numbers, we are 638 GB to their 645 GB. However we can also drop 40GB of validation metadata once sync'd by just clearing three tables.
(edited)
evoskuil21:09
Total block download time should be just over an hour on a 1gbps network (remember the store is bigger than what's downloaded) once remaining optimizations are implemented and it's tuned a bit. And with sufficient RAM it should be able to easily expand to any available bandwidth. Otherwise there's the cost of flushing to the store, as the drive write speed and bandwidth are competing limiting factors once RAM is full.
🙌 1
1 reply · John Doe · last Mar 06, 2024
John Doe00:25
Very nice.

Wednesday, March 6, 2024

evoskuil15:17
Completed a second full scan yesterday. As individual channels become starved for work the aggregate rate slows considerably. This is as expected since I haven't done the load balancing work. This was also observed and resolved in the early prototype. So moving on to that next.
🎉 2

Thursday, March 7, 2024

pmienk20:49
Topic: Dev Weekly Status Time: March 7, 2024 01:00pm Pacific Time (US and Canada) Join Zoom Meeting https://us05web.zoom.us/j/87666370902?pwd=bEZDUlJ0MGJ5V3dVMUhVdHl5QzdFdz09 Meeting ID: 876 6637 0902 Passcode: 85U0bF

Sunday, March 10, 2024

evoskuil16:49
Nice job on the Boost warning fix, I'm now getting these locally: 
1>.../libbitcoin-system/test/intrinsics/haves.cpp(250): warning : in "intrinsics_haves_tests/intrinsics_haves__have_avx512__when_defined__true": condition have_avx512() is not satisfied
1>.../libbitcoin-system/test/intrinsics/haves.cpp(286): warning : in "intrinsics_haves_tests/intrinsics_haves__have_shani__when_defined__true": condition have_shani() is not satisfied
(edited)
evoskuil18:52
Parallel block download completely saturating my 1bps cable, which speedtest.net has never shows above 940mbps.
image.png
18:54
That's a sustained 100 peer connection, with max 5k blocks per peer requested, max 32 threads.
18:56
There's still load balancing work to be done, as this will degrade as channels starve, and there's a material flushing delay as RAM becomes saturated.
19:01
This pc doesn't have avx512, and Merkle root computations are up to twice as fast with it.
(edited)
19:02
We also don't have sha-ni intrinsics integrated into the sha template yet, and individual hashes are about 5x faster with it.
(edited)
19:03
So I would expect CPU utilization to drop significantly on a modern machine once we have sha-ni.
evoskuil19:21
This is immediately following an OS-directed flush to disk due to full RAM. This caused network throughput to fall off significantly, then this quickly recovered to capacity with periodic flushing - as shown.
image.png
19:24
Eventually the SSD becomes the limiting factor, as bandwidth utilization basically drops down to RAM->SSD flush speed. This is limited to about 60% SSD capacity during operation, even though it hits 100% during shutdown.
(edited)
image.png
19:26
That's most likely because there are no reads during shutdown, but the hash table key lookups continue to traverse hash tables being flushed dynamically.
19:28
The fully-indexed store is presently 600GB, so with a little more than twice the RAM this wouldn't be a factor at all.
evoskuil19:55
Shutting down before complete implies that index data spread across tables must be read from SSD, though that is a small fraction of the overall store. This shows up as 100% SSD read, which suppresses network throughput until sufficient indexing is populated in RAM. This only takes a couple minutes. It can be seen here coming to an end.
image.png
19:57
So a full flush to disk, which takes just a few minutes, restores full network capacity download. In other words, it's likely that the snapshot mechanism will serve a dual purpose.
(edited)
evoskuil20:18
Another interesting behavior is table memory remapping. When the table size overflows the preallocated size, the OS remaps the table to memory, which on Windows causes it to be fully flushed to disk. This seems to have the effect of keeping memory under control (< 50%) while not slowing bandwidth at all. For example:
image.png
20:19
Normally I have the tables preallocated to exceed the expected tables sizes, but I haven't updated the allocations in about a year, so toward the end of the sync they start flushing. This is basically the same mmap behavior we would expect from periodic snapshots (which require a full flush).

Monday, March 11, 2024

evoskuil03:20
Finished the work today to handle reorganization with ongoing concurrent download. This includes merging new headers into the download set as the result of a stronger candidate header chain, and the invalidation of a candidate header chain due to block invalidity. A bunch of other simplifications, normalizations, optimizations, comments, style stuff. Synced the full chain 3 times.
M
Marnix12:44
Hi, Great work! Any reason why the source code releases are not PGP signed?
evoskuil15:12
Thanks! Not particularly. You are free to review and sign them.
(edited)

Tuesday, March 12, 2024

evoskuil18:59
Running some performance analysis on header and block sync today. Header sync is 80% bitcoin_hash (double sha256) in debug (unoptimized) builds and 12% in optimized. Implementation of sha-ni should therefore reduce overall header time by 9.5%. Overall processing time looks very evenly distributed and as expected. Locking within the database templates is material, but not crazy. That is very isolated so that we might apply a more efficient technique such as memory fencing in the future. Otherwise I identified one unnecessary line of code that removed 8% of the total run cost during header synch.
🎉 1
19:03
Regarding writer locks in table headers, these are not limiting factors in terms of performance, at least on my 1gbit network. The network would have to be a lot faster to overpower direct memory writes. And of course paging is unrelated to writer locks.
(edited)

Wednesday, March 13, 2024

evoskuil01:27
Just finished the concurrent download load balancer. As we currently run a configured number of channels and distribute downloads to each in a configured set size, it is easy for channels to become starved for work and then just idle until others are done. This results in all but one channel idling toward completion. I resolved this in prototyping by selecting the slowest channel with work for termination, splitting its work into two sets, and notifying channels of new work. Dropping the channel has a small cost, but the channel cannot be retained as that would results in a significant level of duplicate block downloads, or very inefficient single block requests. Since channels are recovered very quickly (earlier work from last year) the cost is trivial.
01:30
This method results in a little inefficiency at the very end, as channels with as low as 2 hashes are dropped to create 2 channels with 1. But prototyping showed that this was a marginal cost and the overall benefit of the simple scheme was large.

Thursday, March 14, 2024

evoskuil01:06
I took most of the day to work off some code debt, just merged. The blocks-first and headers-first chasers are big, consensus critical, and about 95% identical. So I constructed an intermediate chaser templated abstract base class, templated on block vs. header. This reduced the two chasers to only unique functionality and isolated all chain organization code into the one. Blocks first is retained only for legacy channels which really don't exist anymore, but important for the configurable p2p and consensus protocol leveling feature. Appears to be working well, as before the change.
👍 1
evoskuil01:14
Tomorrow I'm going to take a crack at the moving download window. Currently the window is unlimited, but controlling it may provide a significant performance benefit for validation by limiting delays due to download gaps.
pmienk16:00
Topic: Dev Weekly Status Time: March 14, 2024 01:00pm Pacific Time (US and Canada) Join Zoom Meeting https://us05web.zoom.us/j/87666370902?pwd=bEZDUlJ0MGJ5V3dVMUhVdHl5QzdFdz09 Meeting ID: 876 6637 0902 Passcode: 85U0bF
👍🏻 1

Friday, March 15, 2024

evoskuil04:29
The use of undersized store file allocations, to force remapping, as a simulation of snapshotting, works as expected. With an allocation of 1/4 of total expected file sizes, and 10% growth increments, memory allocation during concurrent block download fluctuates around 30% (256gb machine) with periodic flushes to disk slightly impacting download rate, but overall download rate on 200 channels totally saturates bandwidth about 99% of the time.
04:30
Shutdown time is also significantly diminished given the 30% memory allocation (where 10% is idle), it takes under 2 minutes to flush the store (vs 10+ with full RAM).
evoskuil04:37
A rough estimate for total download cost at that pace (on 920Mbps cable) is about 1.5hrs.
(edited)
🚀 1
evoskuil04:48
Didn’t get to the download window today due to some necessary rework in the blocks-first protocol. I was still seeing some heavy message backlogging, which I’ve fully resolved now. Then did some perf testing described above.
evoskuil18:58
These are two runs where only one line of store code differs. This is total bandwidth consumption under identical conditions. I ran is several times both ways to make sure it was the code change.
19:00
The left side is totally saturating my < 1Gbps Internet. The right side results from a single shared pointer allocation moved from outside to inside the store memory allocation critical section.
19:02
Notice also the difference in CPU utilization, more than 50% higher on the right.
evoskuil19:41
After more than a year of flawless store/network operation, I hit a deadlock last night. This showed up because of the intense level of write concurrency while also remapping memory maps as they are expanding (the undersized allocations mentioned above). Within database::map the two member mutexes ended up waiting on each other. It was entirely isolated to the one class and easy to reproduce under load.
19:43
memory_ptr map::get(size_t offset) const NOEXCEPT
{
    const auto ptr = std::make_shared<access>(remap_mutex_);
    if (!loaded_ || is_null(ptr))
        return nullptr;

    ptr->assign(memory_map_ + offset, memory_map_ + size());
    return ptr;
}
19:44
memory_ptr map::get(size_t offset) const NOEXCEPT
{
    const auto logical = size();
    const auto ptr = std::make_shared<access>(remap_mutex_);
    if (!loaded_ || is_null(ptr))
        return nullptr;

    ptr->assign(memory_map_ + offset, memory_map_ + logical);
    return ptr;
}
(edited)
19:47
That's the before and after the fix. Just needed to obtain logical size outside of access construction. That's because of the interaction of these two methods, and the fact that access() takes a shared lock until destruct. 
size_t map::size() const NOEXCEPT
{
    std::shared_lock field_lock(field_mutex_);
    return logical_;
}

size_t map::allocate(size_t chunk) NOEXCEPT
{
    std::unique_lock field_lock(field_mutex_);
    if (!loaded_ || is_add_overflow(logical_, chunk))
        return storage::eof;

    auto end = logical_ + chunk;
    if (end > capacity_)
    {
        const auto size = to_capacity(end);
        std::unique_lock remap_lock(remap_mutex_);
        if (!remap_(size))
            return storage::eof;
    }

    std::swap(logical_, end);
    return end;
}
(edited)
19:51
I ended up simplifying some of the database::map implementation, including a change to remove the boost dependency, avoiding upgrade locks. The result is at least as performant as prior, possibly a little better.
evoskuil21:03
I'm starting to see meaningful GPU utilization during concurrent download (which also includes block.check [simple context-free validation]). I don't remember seeing this before, and I think that's the result of an update to the performance monitor.
image.png
21:04
This is definitely associated with node processing under heavy load. Any ideas how we can identify what is being offloaded to the GPU?
evoskuil21:18
I'm wondering if the extensive use of concurrency-friendly stdlib algorithms is starting to get pushed into GPU processing. Similarly the hash library, which is a huge part of the CPU processing for catch-up, makes use of array copies wherever possible. I've seen these get mapped into AVX2 (I don't have AVX512 on this machine) when stepping through at the machine code level. But GPU processing is another level altogether.
evoskuil21:27
Well, I ran that down pretty quickly... it's the rendering of the rapid console output. :?

Saturday, March 16, 2024

evoskuil04:13
2024-03-16T04:10:49Z.0 Node backup started.
2024-03-16T04:10:49Z.0 backup::flush_table(header_body)
2024-03-16T04:10:49Z.0 backup::flush_table(point_body)
2024-03-16T04:10:54Z.0 backup::flush_table(input_body)
2024-03-16T04:10:59Z.0 backup::flush_table(output_body)
2024-03-16T04:11:07Z.0 backup::flush_table(puts_body)
2024-03-16T04:11:09Z.0 backup::flush_table(tx_body)
2024-03-16T04:11:15Z.0 backup::flush_table(txs_body)
2024-03-16T04:11:15Z.0 backup::flush_table(address_body)
2024-03-16T04:11:15Z.0 backup::flush_table(candidate_body)
2024-03-16T04:11:15Z.0 backup::flush_table(confirmed_body)
2024-03-16T04:11:15Z.0 backup::flush_table(spend_body)
2024-03-16T04:11:20Z.0 backup::flush_table(strong_tx_body)
2024-03-16T04:11:20Z.0 backup::flush_table(bootstrap_body)
2024-03-16T04:11:20Z.0 backup::flush_table(buffer_body)
2024-03-16T04:11:20Z.0 backup::flush_table(neutrino_body)
2024-03-16T04:11:20Z.0 backup::flush_table(validated_bk_body)
2024-03-16T04:11:20Z.0 backup::flush_table(validated_tx_body)
2024-03-16T04:11:20Z.0 backup::backup_table(header_table)
2024-03-16T04:11:20Z.0 backup::backup_table(point_table)
2024-03-16T04:11:20Z.0 backup::backup_table(input_table)
2024-03-16T04:11:20Z.0 backup::backup_table(output_table)
2024-03-16T04:11:20Z.0 backup::backup_table(puts_table)
2024-03-16T04:11:20Z.0 backup::backup_table(tx_table)
2024-03-16T04:11:20Z.0 backup::backup_table(txs_table)
2024-03-16T04:11:20Z.0 backup::backup_table(address_table)
2024-03-16T04:11:20Z.0 backup::backup_table(candidate_table)
2024-03-16T04:11:20Z.0 backup::backup_table(confirmed_table)
2024-03-16T04:11:20Z.0 backup::backup_table(spend_table)
2024-03-16T04:11:20Z.0 backup::backup_table(strong_tx_table)
2024-03-16T04:11:20Z.0 backup::backup_table(bootstrap_table)
2024-03-16T04:11:20Z.0 backup::backup_table(buffer_table)
2024-03-16T04:11:20Z.0 backup::backup_table(neutrino_table)
2024-03-16T04:11:20Z.0 backup::backup_table(validated_bk_table)
2024-03-16T04:11:20Z.0 backup::backup_table(validated_tx_table)
2024-03-16T04:11:23Z.0 backup::dump_table(header_head)
2024-03-16T04:11:23Z.0 backup::dump_table(point_head)
2024-03-16T04:11:29Z.0 backup::dump_table(input_head)
2024-03-16T04:11:29Z.0 backup::dump_table(output_head)
2024-03-16T04:11:29Z.0 backup::dump_table(puts_head)
2024-03-16T04:11:29Z.0 backup::dump_table(tx_head)
2024-03-16T04:11:36Z.0 backup::dump_table(txs_head)
2024-03-16T04:11:36Z.0 backup::dump_table(address_head)
2024-03-16T04:11:36Z.0 backup::dump_table(candidate_head)
2024-03-16T04:11:36Z.0 backup::dump_table(confirmed_head)
2024-03-16T04:11:36Z.0 backup::dump_table(spend_head)
2024-03-16T04:11:54Z.0 backup::dump_table(strong_tx_head)
2024-03-16T04:12:00Z.0 backup::dump_table(bootstrap_head)
2024-03-16T04:12:00Z.0 backup::dump_table(buffer_head)
2024-03-16T04:12:00Z.0 backup::dump_table(neutrino_head)
2024-03-16T04:12:00Z.0 backup::dump_table(validated_bk_head)
2024-03-16T04:12:00Z.0 backup::dump_table(validated_tx_head)
2024-03-16T04:12:09Z.0 Node backup complete.
04:16
Full snapshot with detailed progress callbacks without stopping the process or closing channels. This is just the node integration of the database backup/restore that I implemented over a year ago. Next I'll integrate the restore as a command line option and at startup with fault and snapshot detected.
04:17
Once this is invoked automatically in the background, memory should remain under control and hard shutdown faults are mitigated.
evoskuil04:30
With current bucket configuration the snapshot is 14GB, regardless of store size (currently about 600GB total). That will increase given optional query server indexation (e.g. for all payment addresses).
04:33
2024-03-16T04:30:40Z.0 Node backup started.
2024-03-16T04:30:40Z.0 backup::flush_table(header_body)
2024-03-16T04:30:40Z.0 backup::flush_table(point_body)
2024-03-16T04:31:00Z.0 backup::flush_table(input_body)
2024-03-16T04:31:09Z.0 backup::flush_table(output_body)
2024-03-16T04:31:53Z.0 backup::flush_table(puts_body)
2024-03-16T04:32:04Z.0 backup::flush_table(tx_body)
2024-03-16T04:32:15Z.0 backup::flush_table(txs_body)
2024-03-16T04:32:15Z.0 backup::flush_table(address_body)
2024-03-16T04:32:15Z.0 backup::flush_table(candidate_body)
2024-03-16T04:32:15Z.0 backup::flush_table(confirmed_body)
2024-03-16T04:32:15Z.0 backup::flush_table(spend_body)
2024-03-16T04:32:18Z.0 backup::flush_table(strong_tx_body)
2024-03-16T04:32:18Z.0 backup::flush_table(bootstrap_body)
2024-03-16T04:32:18Z.0 backup::flush_table(buffer_body)
2024-03-16T04:32:18Z.0 backup::flush_table(neutrino_body)
2024-03-16T04:32:18Z.0 backup::flush_table(validated_bk_body)
2024-03-16T04:32:18Z.0 backup::flush_table(validated_tx_body)
2024-03-16T04:32:18Z.0 backup::backup_table(header_table)
2024-03-16T04:32:18Z.0 backup::backup_table(point_table)
2024-03-16T04:32:18Z.0 backup::backup_table(input_table)
2024-03-16T04:32:18Z.0 backup::backup_table(output_table)
2024-03-16T04:32:18Z.0 backup::backup_table(puts_table)
2024-03-16T04:32:18Z.0 backup::backup_table(tx_table)
2024-03-16T04:32:18Z.0 backup::backup_table(txs_table)
2024-03-16T04:32:18Z.0 backup::backup_table(address_table)
2024-03-16T04:32:18Z.0 backup::backup_table(candidate_table)
2024-03-16T04:32:18Z.0 backup::backup_table(confirmed_table)
2024-03-16T04:32:18Z.0 backup::backup_table(spend_table)
2024-03-16T04:32:18Z.0 backup::backup_table(strong_tx_table)
2024-03-16T04:32:18Z.0 backup::backup_table(bootstrap_table)
2024-03-16T04:32:18Z.0 backup::backup_table(buffer_table)
2024-03-16T04:32:18Z.0 backup::backup_table(neutrino_table)
2024-03-16T04:32:18Z.0 backup::backup_table(validated_bk_table)
2024-03-16T04:32:18Z.0 backup::backup_table(validated_tx_table)
2024-03-16T04:32:21Z.0 backup::dump_table(header_head)
2024-03-16T04:32:21Z.0 backup::dump_table(point_head)
2024-03-16T04:32:27Z.0 backup::dump_table(input_head)
2024-03-16T04:32:27Z.0 backup::dump_table(output_head)
2024-03-16T04:32:27Z.0 backup::dump_table(puts_head)
2024-03-16T04:32:27Z.0 backup::dump_table(tx_head)
2024-03-16T04:32:34Z.0 backup::dump_table(txs_head)
2024-03-16T04:32:34Z.0 backup::dump_table(address_head)
2024-03-16T04:32:34Z.0 backup::dump_table(candidate_head)
2024-03-16T04:32:34Z.0 backup::dump_table(confirmed_head)
2024-03-16T04:32:34Z.0 backup::dump_table(spend_head)
2024-03-16T04:32:51Z.0 backup::dump_table(strong_tx_head)
2024-03-16T04:32:58Z.0 backup::dump_table(bootstrap_head)
2024-03-16T04:32:58Z.0 backup::dump_table(buffer_head)
2024-03-16T04:32:58Z.0 backup::dump_table(neutrino_head)
2024-03-16T04:32:58Z.0 backup::dump_table(validated_bk_head)
2024-03-16T04:32:58Z.0 backup::dump_table(validated_tx_head)
2024-03-16T04:33:07Z.0 Node backup complete.
04:36
This is from much larger unflushed memory, about 100GB. The flush time about doubles to around 1.5 mins In both cases the remainder is about 50 secs. That's what we expect as those are constant size operations where the flush varies by the amount of unflushed RAM.
(edited)
evoskuil22:59
$ bn -c f:/bn.cfg --restore
Using config file: "f:/bn.cfg"
Please wait while restoring "F:/bitcoin/" from most recent snapshot...
restore::recover_snapshot(store)
restore::open_file(header_head)
restore::open_file(header_body)
restore::open_file(point_head)
restore::open_file(point_body)
restore::open_file(input_head)
restore::open_file(input_body)
restore::open_file(output_head)
restore::open_file(output_body)
restore::open_file(puts_head)
restore::open_file(puts_body)
restore::open_file(tx_head)
restore::open_file(tx_body)
restore::open_file(txs_head)
restore::open_file(txs_body)
restore::open_file(address_head)
restore::open_file(address_body)
restore::open_file(candidate_head)
restore::open_file(candidate_body)
restore::open_file(confirmed_head)
restore::open_file(confirmed_body)
restore::open_file(spend_head)
restore::open_file(spend_body)
restore::open_file(strong_tx_head)
restore::open_file(strong_tx_body)
restore::open_file(bootstrap_head)
restore::open_file(bootstrap_body)
restore::open_file(buffer_head)
restore::open_file(buffer_body)
restore::open_file(neutrino_head)
restore::open_file(neutrino_body)
restore::open_file(validated_bk_head)
restore::open_file(validated_bk_body)
restore::open_file(validated_tx_head)
restore::open_file(validated_tx_body)
restore::load_file(header_head)
restore::load_file(header_body)
restore::load_file(point_head)
restore::load_file(point_body)
restore::load_file(input_head)
restore::load_file(input_body)
restore::load_file(output_head)
restore::load_file(output_body)
restore::load_file(puts_head)
restore::load_file(puts_body)
restore::load_file(tx_head)
restore::load_file(tx_body)
restore::load_file(txs_head)
restore::load_file(txs_body)
restore::load_file(address_head)
restore::load_file(address_body)
restore::load_file(candidate_head)
restore::load_file(candidate_body)
restore::load_file(confirmed_head)
restore::load_file(confirmed_body)
restore::load_file(spend_head)
restore::load_file(spend_body)
restore::load_file(strong_tx_head)
restore::load_file(strong_tx_body)
restore::load_file(bootstrap_head)
restore::load_file(bootstrap_body)
restore::load_file(buffer_head)
restore::load_file(buffer_body)
restore::load_file(neutrino_head)
restore::load_file(neutrino_body)
restore::load_file(validated_bk_head)
restore::load_file(validated_bk_body)
restore::load_file(validated_tx_head)
restore::load_file(validated_tx_body)
restore::restore_table(header_table)
restore::restore_table(point_table)
restore::restore_table(input_table)
restore::restore_table(output_table)
restore::restore_table(puts_table)
restore::restore_table(tx_table)
restore::restore_table(txs_table)
restore::restore_table(address_table)
restore::restore_table(candidate_table)
restore::restore_table(confirmed_table)
restore::restore_table(spend_table)
restore::restore_table(strong_tx_table)
restore::restore_table(bootstrap_table)
restore::restore_table(buffer_table)
restore::restore_table(neutrino_table)
restore::restore_table(validated_bk_table)
restore::restore_table(validated_tx_table)
restore::unload_file(header_head)
restore::unload_file(header_body)
restore::unload_file(point_head)
restore::unload_file(point_body)
restore::unload_file(input_head)
restore::unload_file(input_body)
restore::unload_file(output_head)
restore::unload_file(output_body)
restore::unload_file(puts_head)
restore::unload_file(puts_body)
restore::unload_file(tx_head)
restore::unload_file(tx_body)
restore::unload_file(txs_head)
restore::unload_file(txs_body)
restore::unload_file(address_head)
restore::unload_file(address_body)
restore::unload_file(candidate_head)
restore::unload_file(candidate_body)
restore::unload_file(confirmed_head)
restore::unload_file(confirmed_body)
restore::unload_file(spend_head)
restore::unload_file(spend_body)
restore::unload_file(strong_tx_head)
restore::unload_file(strong_tx_body)
restore::unload_file(bootstrap_head)
restore::unload_file(bootstrap_body)
restore::unload_file(buffer_head)
restore::unload_file(buffer_body)
restore::unload_file(neutrino_head)
restore::unload_file(neutrino_body)
restore::unload_file(validated_bk_head)
restore::unload_file(validated_bk_body)
restore::unload_file(validated_tx_head)
restore::unload_file(validated_tx_body)
restore::close_file(header_head)
restore::close_file(header_body)
restore::close_file(point_head)
restore::close_file(point_body)
restore::close_file(input_head)
restore::close_file(input_body)
restore::close_file(output_head)
restore::close_file(output_body)
restore::close_file(puts_head)
restore::close_file(puts_body)
restore::close_file(tx_head)
restore::close_file(tx_body)
restore::close_file(txs_head)
restore::close_file(txs_body)
restore::close_file(address_head)
restore::close_file(address_body)
restore::close_file(candidate_head)
restore::close_file(candidate_body)
restore::close_file(confirmed_head)
restore::close_file(confirmed_body)
restore::close_file(spend_head)
restore::close_file(spend_body)
restore::close_file(strong_tx_head)
restore::close_file(strong_tx_body)
restore::close_file(bootstrap_head)
restore::close_file(bootstrap_body)
restore::close_file(buffer_head)
restore::close_file(buffer_body)
restore::close_file(neutrino_head)
restore::close_file(neutrino_body)
restore::close_file(validated_bk_head)
restore::close_file(validated_bk_body)
restore::close_file(validated_tx_head)
restore::close_file(validated_tx_body)
Restored database in 13781 ms.
23:02
The restore from snapshot is now fully integrated with the node. This is the progress indicator with typical time, about 14 secs. Initially it was only a couple seconds, but the restore would swap in the snapshotted index, resulting in one or no snapshots remaining (the system rotate only through 2 snapshots wince they are 14GB). So I changed it to clone back the snapshot after swapping it in. The 14GB copy takes about 10 secs. It will reject the restore unless it sees that the store is corrupted (flush.lock file).
(edited)
evoskuil23:08
So now, nearly a decade after @narodnik announced hot backup, we have something very close. The node can be snapshotted while running, but not externally - it must be invoked on the running command line. This pauses socket reads on all channels, which very quickly results in no threads holding a store transactor. The transactor object encompases all writes, spanning tables and ensuring logical atomicity, as a shared lock (concurrent writes). The backup then proceeds with a unique lock transactor until complete, releasing it and notifying channel resume.
evoskuil23:13
The snapshot flushes table bodies, just to ensure they are written to disk, and then copies headers from memory into a new index directory. So the snapshot constitutes the full set of headers and the corrupted bodies. But as the bodies are append only, they are perfectly valid given the uncorrupted headers. Furthermore the logical body size is retained by the header, so during restore the bodies are truncated. As a result, as long as a live snapshot has been taken an external full backup can be made by copying the snapshot directory and the body directory. This can be done safely against the '/secondary' directory (as opposed to '/primary' since the former is always created by a file system directory rename from primary - ensuring that its files are a consistent and complete set.
(edited)
23:15
The snapshot feature can be placed on automatic based on bytes added to the store, and the restore can be automatically invoked at startup based on the presence of the flush lock - so seamless.
23:20
This all just cleared CI, so please try it out if you are so inclined!
23:20
$ bn -c f:/bn.cfg --help

Usage: bn [-bfhilmrsvwx] [--config value]

Info: Runs a full bitcoin node.

Options (named):

-b [--buckets]       Compute and display all bucket densities.
-c [--config]        Specify path to a configuration settings file.
-f [--flags]         Compute and display all flag transitions.
-h [--help]          Display command line options.
-i [--initchain]     Initialize store in configured directory.
-l [--collisions]    Compute and display hashmap collision stats.
-m [--measure]       Compute and display store measures.
-r [--restore]       Restore from most recent snapshot.
-s [--settings]      Display all configuration settings.
-v [--version]       Display version information.
(edited)
23:21
$ bn -c f:/bn.cfg --version

Version Information
----------------------------
libbitcoin-node:       4.0.0
libbitcoin-database:   4.0.0
libbitcoin-network:    4.0.0
libbitcoin-system:     4.0.0
(edited)
23:22
$ bn -c f:/bn.cfg --flags
Using config file: "f:/bn.cfg"
Please wait while the database is starting...
Press CTRL-C to cancel operation.
Forked from [00000000000000000000000000000000] to [00000000000000100000000010001011] at [00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048:1]
Forked from [00000000000000100000000010001011] to [00000000000000100000000010000011] at [00000000000a4d0a398161ffc163c503763b1f4360639393e0e4c8e300e0caec:91842]
Forked from [00000000000000100000000010000011] to [00000000000000100000000010001011] at [0000000000016ca756e810d44aee6be7eabad75d2209d7f4542d1fd53bafc984:91843]
Forked from [00000000000000100000000010001011] to [00000000000000100000000010000011] at [00000000000743f190a18c5577a3c2d2a1f610ae9601ac046a38084ccb7cd721:91880]
Forked from [00000000000000100000000010000011] to [00000000000000100000000010001011] at [00000000000cf6204ce82deed75b050014dc57d7bb462d7214fcc4e59c48d66d:91881]
Forked from [00000000000000100000000010001011] to [00000000000000100000000010001111] at [00000000000000ce80a7e057163a4db1d5ad7b20fb6f598c9597b9665c8fb0d4:173805]
Forked from [00000000000000100000000010001111] to [00000000000000100000000010010111] at [000000000000024b89b42a942fe0d9fea3bb44ab7bd1b19115dd6a759c0808b8:227931]
Forked from [00000000000000100000000010010111] to [00000000000000100000000010110111] at [00000000000000000379eaa19dce8c9b722d46ae6a57c2f1a988119488b50931:363725]
Forked from [00000000000000100000000010110111] to [00000000000000100000000011110111] at [000000000000000004c2b624ed5d7756c508d90fd0da2c7c679febfa6c4735f0:388381]
Forked from [00000000000000100000000011110111] to [00000000000000100000011111110111] at [000000000000000004a1b34462cb8aeebd5799177f7a29cf28f2d1961716b5b5:419328]
Forked from [00000000000000100000011111110111] to [00000000000000100011111111110111] at [0000000000000000001c8018d9cb3b742ef25114f27563e3fc4a1902167f9893:481824]
23:24
The other options are a bit verbose, but you can try them out.
evoskuil23:48
The high level store redesign was based on achieving this objective, which I saw for the first time today and works flawlessly. This was the result of a conversation with @jungly in 2021, which inspired me to restart work based on the v4 prototype. This finally gives up a fault tolerant store, including hard shutdown and disk overflow, with the side benefit hot backup - and can easily be fully automated.
evoskuil23:54
The key design principle is based on Amir's append only design, however that suffered from hash table headers being co-located within the body files. The headers are random write access and cannot be recovered in case of a fault. Also memory maps must be flushed before they can be relied upon upon shutdown or for hot copy. This of course also required that indexation spanning multiple files was logically atomic. So to resolve these issues we split the headers and bodies, enforced true append only on the bodies, created a transactional commit system using a mutex-based transactor (only locked during backup, so not a performance consideration), created a method to pause channel reads (thereby suspending store writes) without dropping channels, and implemented flush to disk of all files after obtaining the exclusive transactor and before taking index snapshots.
(edited)
23:55
Note that there is no suspension of any reads during snapshotting, so server queries may continue unabated, as can outgoing p2p traffic (such as a long sequence of blocks/headers) that doesn't require incoming communication.
23:58
I put off the download moving window to integrate snapshot/recovery, and I think I'll take it a little further, so that there's an automated/configurable snapshot based on write volume, and automated recovery on startup with corruption detected. These should be one day work combined.

Sunday, March 17, 2024

evoskuil00:10
The store implementation is now fault tolerant (can recover from hardware failures) and available (does not need to be stopped or queries paused to provide fault tolerance), but also provides an assurance that other database-based systems cannot. Absent undetected hardware malfunction, the store is reliable. This is not the case with other database systems, there is no such assurance that a hard shutdown will not result in data corruption. In our case we always detect if there is a hard shutdown, and can always recover to a snapshot with perfect data integrity and low cost. Other systems have no way to know a fault has occurred, which is why Core for a long time has scanned recent blocks at startup (though a failure can occur anywhere).
🙌 1
evoskuil00:54
Another interesting aspect of the memory map store that I forgot to mention is that, when a map for a given file needs to be made larger the operating system can remap it. In other words, invalidate all outstanding pointers into the map. This was another aspect that hadn't been dealt in the original implementation. This requires, on a per file basis, that each read/write pointer obtained from the store is an object that encapsulates a remap mutex. When the map allocation is exceeded for the given file, the store blocks until all shared locks are released and then holds exclusive access during the remap. This requires that all pointers are very short lived and cannot be passed into code that could be waiting on another pointer (deadlock). This was something that I implemented in the prototype, but it wasn't sufficiently safe. So in the current store reads and writes encapsulated within low level store primitives which manage memory access, passing higher level result sets back to the query implementations, and these providing the custom query interface that the store exposes publicly.
evoskuil22:10
I finally took the time to remove the boost::thread dependency. This was the result of using boost::thread_specific_ptr as a workaround for not having thread_static implemented in all platforms. This was precipitated from an exception thrown by boost:shared_mutex from database. I decided to test operation with 500 threads configured, and it crashed as soon as 128 blocked writers accumulated. This was the result of boost using a 7 bit struct bitfield to store that limit. After removing all boost thread and mutex code it's humming along no problem with a 500 thread pool. The main objective with the test was to determine how critical is thread pool sizing. Turns out too few can really slow down the network but too many doesn't really matter. This is also true of channel count.
(edited)
22:13
Also did some refactoring of the recent work to keep protocols simple. Performance control was moved out to an intermediate base class, greatly simplifying protocol_blocks_in_31800.
22:18
1024 threads configured with 1024 channels, connection-batched 5 at a time hums along just fine without a noticeable difference in resource consumption or speed (limited by 1gbps bandwidth). The result is a reduction in average per-channel download rate and/or a lower percentage of configured connections able to keep pace. For example, this configuration is holding at 1gbps with 250 channels connected and an average download rate of 4,000kbps.
(edited)

Tuesday, March 26, 2024

evoskuil22:15
Libbitcoin has been asked and I've tentatively accepted taking on summer college interns via "Summer of Bitcoin 2024". The work period would be May 15 - Aug 15. I'm happy to manage 1 or 2 but would love some additional team involvement. Please ping me or get back here if you are willing to participate.
4 replies · neillm, evoskuil · last Mar 27, 2024
neillm02:53
Depending on the scope, I'll be around. Keep me posted
👍🏻 1
evoskuil04:06
Think you'd be able to run a full sync any time soon?
neillm15:03
For v4? Yes, I could be setup for either later in the weekend or next week if that works.
evoskuil15:23
Sure, thanks.
👍 1
evoskuil22:20
I spent a couple days refactoring, after that last large bunch of chaser work. I've done several full syncs, including live backups and restores, and have added validation. This kicks in above the top checkpoint/milestone (as applicable) and currently runs sequentially as it chases block order with block being rapidly downloaded in parallel. This is working well, but of course sequential validation at the top of the chain is very slow (about 1 block per second). However it is only using about 1-2% CPU to achieve this, so we should be able to get between 50-100 full blocks per second with full parallelization. Keep in mind sha-ni is still not integrated, and that's a 5x improvement in sha256 hashing - which is one of the most significant costs.
evoskuil22:28
With the validation chaser working I'm moving on to the confirmation chaser. That's the last one for full validating sync. The others are tx pooling and block template generation (both for mining and not present in v3). Confirmation is always sequential, so basically the node will be fully functionally soon, and I'll be able to dedicate all efforts toward optimizing concurrent validation. This includes fan-out by tx with result aggregation by block, a moving download window to reduce the maximum blockage waiting on a gapped block download, automation of background snapshots, and automated store sizing configuration. These last two should be able fully manage the problem of excess paging (it's doing great presently).

Thursday, March 28, 2024

Adi Shankara09:54
Hey folks, Eric invited me here. I'm the program lead at Summer of Bitcoin, a non-profit that runs a global internship program for university students to work on bitcoin FOSS. If you'd like to sign up and get a couple of interns work on your favorite project ideas under your mentorship, here's all the info you need: Quick recap on how the program works: 1. Bitcoin open-source devs / designers sign up as mentors. 2. Mentors provide project ideas from their respective projects. 3. University students apply, get screened and submit a project proposal using the project idea as a starting point. 4. Mentors evaluate and select project proposals. 5. Students contribute code / design during the project period, as outlined in their proposals. 6. Mentors guide their mentees wherever they're stuck and help with reviews. On time commitment from mentors: We have typically seen mentors do a 30-60 min call once per week to discuss progress and unblock their interns on critical issues. Most teams simply add the intern to their weekly stand-up and work with them just like any other member of the team. Some also go totally async in their comms, especially when the project is relatively straightforward and the candidate is self-driven. We leave this part completely on both the mentor and their intern to figure out. Timeline: 1. Application period: Feb 1 - Feb 15. 2. Bootcamp: Feb 16 - Mar 31. 3. Proposal period: Apr 1 - Apr 30. 4. Results announced: May 7. 5. Project period: May 15 to Aug 15. What does a project idea look like? Project ideas are scoped for a university-level student with a basic background in CS / UX design and bitcoin fundamentals. Project should be achievable over 12-weeks. Template for a project idea is available here: https://docs.google.com/document/d/1wQV5rhZ5DC5HO89K5-AVWkZLK3-QhqVPjjPQDwHzSZc/edit Some project ideas from prior years for reference:
When do we need project ideas from you? We're looking to publish project ideas on our website by the end of this month, which is quite close already. But feel free to take a couple of additional days if you need. You can submit as many project ideas as you want. Here's a form to submit them: https://www.jotform.com/form/240533114662044. As a token of gratitude, mentors receive a gift of $500 in BTC once the program successfully concludes 🙂 Let me know if you have any further questions!
👋 1👍🏻 1
4 replies · evoskuil, jungly, Adi Shankara · last Apr 03, 2024
evoskuil23:05
I'm going to be traveling for the next 10 days. I'm not sure if anyone else wants to write up a proposal, but in any case I'll try and get it done on flights.
evoskuil23:05
What's the "bootcamp" thing?
jungly08:55
It'll be nice to have someone try out SQLite. I can write an initial draft if that helps.
💯 1🙏 1👍🏻 1
Adi Shankara14:30
The bootcamp consists of series of readings and seminars over a month, to teach bitcoin technicals, mostly using Grokking Bitcoin as a textbook. It culminated last week. It also includes a hard programming assignment on bitcoin technicals that one needs to solve in order to pass the screening.
evoskuil20:27
I'm around now if anyone wants to meet today.
20:29
I've had an interesting experience a couple times this week. I fire up the empty node to test something, then I work for a while, then I notice it's stalled. Then I notice it's not actually stalled, it's fully synced.
20 replies · jungly, evoskuil · last Mar 31, 2024
jungly21:06
That's really cool!
👍🏻 1
jungly21:07
Are you saying, we can try out a working v4?
evoskuil21:08
It's working to the extent it's implemented, but I'm still completing the confirmation stage, and validation does not yet implement fan-out, so it's slow/sequential.
evoskuil21:09
But it does maintain 1gbps sync rate (max on my WAN) until fully populated, which takes somewhere around 1-1.5 hrs.
evoskuil21:10
And the "chasers" are all implemented, which manage the necessary level of ordering for validation/confirmation.
evoskuil21:10
And hot backup/restore works perfectly.
jungly21:10
That is so cool!
evoskuil21:11
And by forcing flush (either via snapshot or undersizing tables, resulting in remap), paging churn is largely mitigated.
(edited)
jungly21:12
Interesting idea!
evoskuil21:12
All that stuff that we talked about 2-3 years ago is finally up and working.
💥 1🎉 2🙌 1
jungly21:12
If I ran it on my "normal" 16gb machine will a test like that give you something useful?
evoskuil21:14
Sure. There is no automatic table sizing or automatic snapshotting presently, but you can configure for it. The big issue you'd hit is paging with tables that exceed your RAM.
evoskuil21:14
I have 256GB GRAM and it's a 600GB store. So I configured all tables to 1/4th of target size.
jungly21:15
Someone here was supposed to try out SQLite for smaller machines 😁
evoskuil21:15
So they get resized (each independently) before exceeding overall RAM capacity, and remap on win32 causes a full flush (remap).
evoskuil21:15
yep, interns
😄 1
evoskuil21:17
So it's still a bit early, but it is all working flawlessly and I can see the remaining path.
evoskuil21:18
One of the most exciting things is the reliability. The investment in the lower level libs, including their testing, has really paid off.
💯 1
evoskuil08:48
Jeesh, burned by fast sync again... spent a half hour trying to figure out what was wrong. lol
🤣 1
jungly08:57
I can't wait to try this out. If we are getting v4 ready, we really need to build braidpool using libbitcoin. There's so many benefits to doing so.
  1. No RPC
  2. Validate any blocks - without confirming them - we need this to validate blocks created by other miners and we need to make sure the shares for those blocks should be rewarded etc.
I want to do a write up and then set up a call with you.
👍🏻 1
evoskuil20:45
Also, spent some time yesterday coding up an un-bipped hard fork introduced by Core in 2016 and an un-bipped soft fork introduced by Core in 2018 (to patch the former). The hard fork was not recognized as such at the time, and was signed off on by just about every active significant Core dev.
2 replies · Marnix, evoskuil · last Mar 29, 2024
M
Marnix22:22
Interesting, do you mind elaborating some more?
evoskuil23:01
https://bitcoin.stackexchange.com/questions/120834/what-is-the-block-1-983-702-problem
Bitcoin Stack Exchange
What is the Block 1,983,702 Problem?
The issue, transition to requiring block height in block coinbases, was discussed back in 2012, however I just stumbled upon it, and would like to understand more about its significance.
👍 1

Friday, March 29, 2024

evoskuil23:01
replied to a thread: view
https://bitcoin.stackexchange.com/questions/120834/what-is-the-block-1-983-702-problem
Bitcoin Stack Exchange
What is the Block 1,983,702 Problem?
The issue, transition to requiring block height in block coinbases, was discussed back in 2012, however I just stumbled upon it, and would like to understand more about its significance.
👍 1

Saturday, March 30, 2024

evoskuil23:05
replied to a thread: view
I'm going to be traveling for the next 10 days. I'm not sure if anyone else wants to write up a proposal, but in any case I'll try and get it done on flights.
evoskuil23:05
replied to a thread: view
What's the "bootcamp" thing?