Bitcoin full node and query server

Mar 5, 2024

View in context in the libbitcoin Slack #general mirror.

evoskuil16:07

After swapping out the simple std::unordered_map for a multiindex boost table I'm now able to maintain height-indexed requests with constant time hash-based indexing. This tracking is still very inefficient relative to where it will be soon (moving one pointer per channel) around vs. moving millions of records. But I've achieved full saturation of my network at ~940Mbps (peak determined by speedtest.net) as measured internally via block download tracking/logging and externally using a network monitor. I have no reason to believe this would not scale beyond my network capacity. At 940Mbps/8 = 117.5 MBps = 423GBph. I haven't measured the chain in a while, but that should be close to all of it.

🎉 3

10 replies · neillm, evoskuil · last Mar 05, 2024

neillm19:27

Core shows:

587G blocks
11G	chainstate
47G indexes

evoskuil20:06

Thanks. That should come out to 1.4 hours on my machine then. I'm just finishing up a sync after several starts and stops and that time seems about right.

evoskuil20:35

584 GB for blockchain storage sans indexes.

evoskuil20:35

😅

evoskuil20:37

Actually that includes a lot of the indexation (hash bucket deconfliction lists are in the hash tables, including key values).

evoskuil20:53

The hash table headers are 14 GB

evoskuil20:55

With that we achieve a much more highly indexed and performant query store than Core. Validation metadata will increase that somewhat, and then of course optional indexation such as all payment addresses. But validation metadata can be purged once current.

🙌 1

evoskuil21:00

That should amount to ~40 GB, so it's 638 (or 598 after sync) to 645.

neillm23:26

Wow, awesome!

evoskuil23:34

Yeah, use of surrogate keys offers some good compression.