Bitcoin full node and query server
June 2024
Archive of the libbitcoin Slack #general channel.
Wednesday, June 5, 2024
evoskuil01:05
It's been a little slow this past, with my son visiting from school. I've implemented confirmations by the downloading protocols when under checkpoint/milestone, so that confirmations are written in parallel. Otherwise this adds about 10 minutes to the run.
I've also implemented a generalized determination of combined checkpoint/milestone (bypass) height by the organizer, which is then broadcasted to the validation and confirmation chasers. This allows them to track and switch modes at almost no cost.
However, the block_in protocol also requires this information, but as protocols are many and come and go constantly with channels, it's not possible to initialize the bypass height at startup as with the others. So I'm working on an efficient mechanism for this. One might assume that they could simply initialize active milestone height from the store at startup, but this presents two problems:
(1) the query is a cost (though very small), and (2) the organizer controls the candidate chain, to which milestones are relative. Checkpoints on the other hand are assured by header validation, so those are trivial. The issue then is how to ensure proper ordering of the milestone to the protocol. I implemented this as a relay, where the protocol requests it at startup and the organizer responds with the corresponding message. However the need to delay the protocol during this communication relay costs about 20 minutes over a full run. So still looking at it.
👍 1
evoskuil02:50
Also, after optimizing everything that I can find using the performance analyzer, and adding in the above-referenced confirmation writes (which leaves the node fully usable during/after a complete milestone/checkpoint sync) the total time is consistently 66-70 minutes on my max machine, including store creation, startup, header sync, block sync/index, and complete shutdown.
evoskuil02:55
The machine is old, so no AVX512 (which will cut the two Merkle roots per block in half) and no SHANI (which will cut most sha256 by 80%). SHANI is also not yet integrated. The Merkle computations are presently only around 0.75% of the run, but sha256 is about 10%. That's are the only material optimizations that remain in the checkpoint/milestone sync realm.
Given periodic remap/flush delays, the run would likely be close to the theoretical limit of 46 minutes with a faster SSD and sufficient RAM to avoid flushing during the run.
eynhaender18:36
I´m trying to get a clean build on a fresh macOS Ventura but keep getting these errors:
In file included from src/chain/script.cpp:19:
In file included from ./include/bitcoin/system/chain/script.hpp:29:
In file included from ./include/bitcoin/system/chain/operation.hpp:29:
In file included from ./include/bitcoin/system/stream/stream.hpp:33:
In file included from ./include/bitcoin/system/stream/streamers/bit_flipper.hpp:27:
In file included from ./include/bitcoin/system/stream/streamers/bit_reader.hpp:27:
In file included from ./include/bitcoin/system/stream/streamers/byte_reader.hpp:28:
In file included from ./include/bitcoin/system/hash/hash.hpp:23:
In file included from ./include/bitcoin/system/hash/accumulator.hpp:173:
./include/bitcoin/system/impl/hash/accumulator.ipp:426:32: error: cannot take address of consteval function 'empty_single' outside of an immediate invocation
constexpr auto empty = empty_single();
^
./include/bitcoin/system/impl/hash/accumulator.ipp:227:1: note: declared here
empty_single() NOEXCEPT
^
./include/bitcoin/system/impl/hash/accumulator.ipp:451:32: error: cannot take address of consteval function 'empty_single' outside of an immediate invocation
constexpr auto empty = empty_single();
13 replies · eynhaender, evoskuil, pmienk · last Jun 10, 2024
eynhaender18:40
----------------------------------------------------------------kernel
Darwin Kernel Version 22.6.0: Mon Apr 22 20:54:28 PDT 2024; root:xnu-8796.141.3.705.2~1/RELEASE_X86_64
----------------------------------------------------------------os
ProductName: macOS
ProductVersion: 13.6.7
BuildVersion: 22G720
----------------------------------------------------------------date
Mi 5 Jun 2024 20:39:36 CEST
----------------------------------------------------------------gcc
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: x86_64-apple-darwin22.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
----------------------------------------------------------------brew
rm: /usr/local/Homebrew/.git/describe-cache/e130e47f23b8b806096f9ec4f2c193213b8ec908: Permission denied
rm: /usr/local/Homebrew/.git/describe-cache: Permission denied
Homebrew >=4.1.0 (shallow or no git repository)
----------------------------------------------------------------autoconf
autoconf (GNU Autoconf) 2.72
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+/Autoconf: GNU GPL version 3 or later
<https://gnu.org/licenses/gpl.html>, <https://gnu.org/licenses/exceptions.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by David J. MacKenzie and Akim Demaille.
----------------------------------------------------------------make
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for i386-apple-darwin11.3.0
evoskuil21:28
evoskuil00:01
FYI, the above is a clang c++20 bug related to this one: https://bugs.llvm.org/show_bug.cgi?id=48886
eynhaender08:18
The problem was me not following the README in detail and my lack of experience with github.
The README at https://github.com/libbitcoin/libbitcoin-system contains this command:
(edited)wget https://raw.githubusercontent.com/libbitcoin/libbitcoin/version3/install.sh
This one works flawlessly.
The README at https://github.com/libbitcoin/libbitcoin-node/tree/version3 does not have that wget fetch. It has links to the correct install.sh though,I just didn´t see it right away.
What I did then, I cloned the repo with the link on the github-page
git clone https://github.com/libbitcoin/libbitcoin-node.git
which gave me a version that doesn´t build. It´s working now.evoskuil14:07
That’s version3, which requires only c++11 and is therefore cool with your compiler version. However that’s also 7 years old.
evoskuil14:08
You can run the install.sh from the /master branch, but will need to upgrade to clang 15.
eynhaender12:13
Ok, that worked but… Now I run into this one https://libbitcoin.slack.com/archives/C6LH74ACQ/p1713913754688469?thread_ts=1713881832.894529&cid=C6LH74ACQ
Are there any news on that?
Rich Phillips
In file included from libs/thread/src/pthread/thread.cpp:11:
In file included from ./boost/thread/thread_only.hpp:17:
In file included from ./boost/thread/pthread/thread_data.hpp:12:
In file included from ./boost/thread/lock_types.hpp:18:
In file included from ./boost/thread/thread_time.hpp:11:
In file included from ./boost/date_time/posix_time/posix_time_types.hpp:16:
In file included from ./boost/date_time/posix_time/posix_time_duration.hpp:15:
In file included from ./boost/numeric/conversion/cast.hpp:33:
In file included from ./boost/numeric/conversion/converter.hpp:13:
In file included fro…
evoskuil13:00
@pmienk do you recall what resolved the above llvm/clang-boost issue in our macOS builds?
pmienk05:05
The only recent change I recall was related to std::unary, -DBOOST_NO_CXX98_FUNCTION_BASE parameterized on the call in CI.
pmienk05:09
unless, is this boost erroring during it's own build because it's escalated warning to error? we had to build the bjam tool and to disable that behavior and then execute it. we also suppressed warnings.
Thursday, June 6, 2024
pmienk19:50
Topic: Dev Weekly Status
Time: June 6, 2024 01:00pm Pacific Time (US and Canada)
Join Zoom Meeting
https://us05web.zoom.us/j/87666370902?pwd=bEZDUlJ0MGJ5V3dVMUhVdHl5QzdFdz09
Meeting ID: 876 6637 0902
Passcode: 85U0bF
Friday, June 7, 2024
evoskuil01:36
I struggled with the above ordering issue, and another similar issue related to deconfirming in the case of an invalidated block under a valid header chain, and another related to block hash malleability. I ended up implementing a relatively simple concept between the check chaser (block download with block.check() context free validation) and the channel download protocol that fans out to provide concurrent download/check. Basically the problem was that there was no way to invoke code at the completion of a purge in the downloading as a consequence of a reorganization or disorganization (which I refer to as regression - where the chain moves backward before moving forward again). I added a new race class to manage this, which has almost no cost.
The optimal node is maximal disorder, which implies some necessary level of order. The chasers exist to impose this order, and in working with this model over the past few months its become apparent that it is ideally suited for imposing the minimal level of order required while not imposing unnecessary costs to achieve it. But in order to avoid the inevitable race conditions certain general implementation rules have become apparent. These conceptually reduce to ensuring ownership over the several different aspects of ordering, while preserving the ability to provide the minimal necessary order between them.
This allows us to safely implement complex but reasonable/optimal things. For example, the data model requires that each tx be associated to its confirming block header. This must be implemented in order as it is relied upon when confirmability queries are executed. However, when under checkpoint/milestone (validation/confirmation bypass) these associations can be written for all blocks under bypass as they are downloaded (out of order). As there are 1 billion txs, this is a material optimization. About 10 minutes over a full 65 minute run (would otherwise be 75 minutes).
But milestones (unlike checkpoints) are subject to reorganization by a stronger branch. So what to do when an unknown set of blocks are confirmed and all of the sudden some of them have to become unconfirmed. Computing the potential set isn't so hard, but there needs to be a point when no more are being written from before the reorg, and none are yet being written from after the reorg. And in the midst of this there is the possibility that some blocks are malleable and therefore cannot be bypassed for validation/confirmation, and may be invalid even under bypass, and therefore require a download reinitialization. Managing optimal concurrency is non-trivial to say the least. But by following some simple rules can be made safe.
(edited)🔥 2
Sunday, June 9, 2024
Tuesday, June 11, 2024
evoskuil20:58
Sync is now running at under an hour on my 2Gbps Internet, and I've done a few tests of unoptimized concurrent validation, which completes in 3 hours. So I've spent the past few days on confirmation queries.
Download concurrency is limited by bandwidth - with no block ordering required. Validation concurrency is limited by CPU threads, and requires contiguous blocks but can validate any number concurrently. Confirmation is pure query, limited by disk speed (or exclusively RAM there is enough to hold the store), and can confirm blocks only one at a time in order - however transactions/inputs can be concurrently processed within each block.
evoskuil21:06
Starting at block 840,000 I ran some performance numbers on confirmation. Most of the confirmation cost, like 99.5% is determining confirmability. Updating state is trivial in comparison. On a cold store confirmation queries were averaging about 1-2 seconds per block. This gave me an opportunity to run the performance tools on intense querying for the first time. I was able to identify several areas where denormalization would produce material benefits. After a couple days of tuning store primitives, such as the
iterator and the hashmap and the head , as well as simple modifications to several queries, the confirmation rate has increased to 5 per second, or roughly 5-10x faster than previously.evoskuil21:18
Generally this only affects the tail of the chain, like validation, above the milestone block. However for a full chain validation that's too slow. Keep in mind that the first 300k blocks or so are trivial. These numbers are for full blocks, about 5k txs each. So 1 billion txs (at current heights) and 5,000 tx/block and 5 blocks/sec you get 25k txs/sec or 40,000 secs total (11.1 hours). If that can be divided in full by my 64 cores, it reduces to 10.4 minutes. However, disk speed is likely to be a limiting factor (CPU utilization is trivial). Moving on to that next.
(edited)evoskuil23:18
Confirmation reads are sprawling, but there is very little data actually collected. The input and output tables, which contain the bulk of blockchain data aren't touched, since they are not hash tables (just blob tables referenced from others). So while there is a large amount of querying, data quickly populates RAM and disk activity slows significantly. It's dropped from 300MiB/s to 50MiB/s, with a disk capacity of 530MiB/s.
Over a span of the top 7,500 blocks, confirmation speed accelerated to 6.85 blocks/sec. Concurrency at full disk capacity (only about 1% of CPU is currently being used, on single thread) would put this at about 68 blocks/sec or 2,941 sec (49 minutes) for the full 1B txs). That assumes a good amount of RAM to limit page swaps. I had reached about 100GiB (with no swapping because I have 128GiB). But it's a low end SSD and in this example requires only 10 threads. It also assumes no concurrency with download and/or validation.
Thursday, June 13, 2024
evoskuil17:44
Also, for those testing out current master builds, you should be aware that all libbitcoin libs, but most importantly -system, are significantly impacted by compiler optimizations. In the visual studio builds optional settings are defaulted into the .props files for debug and release builds.
HOWEVER, the other builds require that you set these optimizations in the environment. This can be done as follows:
$ CFLAGS="-03" CXXFLAGS="-03" ./install.sh --build-boost --enable-ndebug --disable-shared --prefix=<path>
Because all hash functions are written in C++, you will get obscenely poor performance if you do not let the compiler fully optimize. With this setting the optimized code outperforms C implementations.👍 1
1 reply · parsevalbtc · last Jun 16, 2024
P
parsevalbtc11:14
The flag is with capital o not with zero:
$ CFLAGS="-O3" CXXFLAGS="-O3" ./install.sh --build-boost --enable-ndebug --disable-shared --prefix=<path>👍🏻 1
evoskuil17:56
There are also significant CPU-specific optimizations that you can set via the command line. These provide hash function vectorization at different levels. For example:
--enable-sse4 is the most basic and should be available on any supported CPU.
--enable-avx2 and --enable-avx512 are also extremely common. I have been testing on a 7 year old machine that has sse4 and avx2 but not avx512. That allows 4 and and 8 lane vectorization, but not 16 lanes (avx512). If you set these then the build will enable them.
However, these are implemented such that portability is still enabled. This hurts performance, and so I have made a minor modification to one file locally...(Also note that
shani and neon (ARM) are not yet fully implemented.)inline bool have_avx512() NOEXCEPT
{
if constexpr (with_avx512)
return try_avx512();
else
return false;
}
inline bool have_avx2() NOEXCEPT
{
if constexpr (with_avx2)
return true;//// try_avx2();
else
return false;
}
inline bool have_sse41() NOEXCEPT
{
if constexpr (with_sse41)
return true;//// try_sse41();
else
return false;
} (edited)I'm going to make this the normal build soon.
evoskuil18:07
Also, there is a command line option on bn that allows you to see what your compiler supports (without the need to compile with the option of course)... platform zero means not available, compiled zero means you didn't compile for it.
$ bn -c f:/bn.cfg --hardware Intrinsics... arm..... platform:0. intel... platform:1. avx512.. platform:0 compiled:0. avx2.... platform:1 compiled:1. sse41... platform:1 compiled:1. shani... platform:0 compiled:0. neon.... platform:0 compiled:0.
21 replies · parsevalbtc, evoskuil, pmienk · last Jun 16, 2024
P
parsevalbtc11:25
got avx2 and sse4 as well. Will re-compile also including additional flags reported in the messages above and let you know hows test will be going.
$ ./bn -c ../etc/libbitcoin/libbitcoin.conf -d Intrinsics... arm..... platform:0. intel... platform:1. avx512.. platform:0 compiled:0. avx2.... platform:1 compiled:0. sse41... platform:1 compiled:0. shani... platform:0 compiled:0. neon.... platform:0 compiled:0.
P
parsevalbtc16:51
Unfortunately, it seems that by adding $ CFLAGS="-03" CXXFLAGS="-03" to ./install.sh ... the compilation stops at:
configure:3495: error: in `/home/user11/gitprojects2/libbitcoin4/build-libbitcoin-node/secp256k1': configure:3497: error: C compiler cannot create executables See `config.log' for more details(edited)
P
parsevalbtc16:56
By removing these optimization "-03" flags the compliation process goes smoothly till the end. However, looking at the messages:
configure: WARNING: unrecognized options: --enable-avx2, --enable-sse4evoskuil23:11
Yes. The install.sh arguments get passed to each build, but these only apply to the libbitcoin-system build, so you get warnings on the others.
👍 1
P
parsevalbtc12:04
Ok the compilation went fine with -O3 flags. However
avx2 and ss4 optimization where not compiled:
$ ./bn -c ../libbitcoin.conf -d Intrinsics... arm..... platform:0. intel... platform:1. avx512.. platform:0 compiled:0. avx2.... platform:1 compiled:0. sse41... platform:1 compiled:0. shani... platform:0 compiled:0. neon.... platform:0 compiled:0.
evoskuil13:59
I did extensive testing, including disassembly for comparison with C implementations, and detailed performance testing. The implementation relies heavily on compiler optimization. That’s the nature of C++. The developer manages the code such that the compiler can reduce it to efficient machine code while retaining the advantageous abstractions. Hashing (for example) is like 10x slower without full optimization.
(edited)👍 1
P
parsevalbtc14:04
yeah latest build:
libbitcoin-node$ git log
commit 0e7853934e778f7ba03e74b887b6a90c004596d1 (grafted, HEAD -> master, origin/master, origin/HEAD)
Author: Eric Voskuil <eric@voskuil.org>
Date: Wed Jun 12 21:02:09 2024 -0400
Merge pull request #643 from evoskuil/master
libbitcoin-system$ git log
commit b619ff2dd6c2e77679ce935cb84a177fbb7ae918 (grafted, HEAD -> master, origin/master, origin/HEAD)
Author: Eric Voskuil <eric@voskuil.org>
Date: Thu Jun 13 21:12:39 2024 -0400
Merge pull request #1480 from evoskuil/master
libbitcoin-network$ git log
commit 19177766c968917e3d38306e84f0d7c398d5c7e6 (grafted, HEAD -> master, origin/master, origin/HEAD)
Author: Eric Voskuil <eric@voskuil.org>
Date: Thu Jun 6 18:04:51 2024 -0400
Merge pull request #414 from evoskuil/master
libbitcoin-database$ git log
commit ed3e0a26b7072266d8b3b81134497ea9f575525b (grafted, HEAD -> master, origin/master, origin/HEAD)
Author: Eric Voskuil <eric@voskuil.org>
Date: Thu Jun 13 00:47:13 2024 -0400
Merge pull request #493 from evoskuil/master
P
parsevalbtc14:31
From a quick look it seems that the compiler has embedded the flags in the compilation stream:
checking whether C++ compiler accepts -mavx -mavx2... yes checking whether C++ compiler accepts -msse4.1... yes
evoskuil15:59
FYI in a recent -system change I removed all of the runtime conditionality, so the builds are not portable but are maximally benefit from hash vectorization. The conditionality is only used now to provide bn command line feedback on the availability of the optimizations.
evoskuil16:01
The sse4 and avx2 options cut CPU by 4x and 8x respectively for merkle root computations. avx512 would be 16x.
P
parsevalbtc19:29
here is the log @pmienk. The launched coommand was:
CFLAGS="-O3" CXXFLAGS="-O3" ./install.sh --prefix=/home/tester/gitprojects2/libbitcoin4 --build-boost --enable-ndebug --disable-shared --enable-avx2 --enable-sse4
Friday, June 14, 2024
evoskuil01:23
Eric Voskuil
Today's meeting is canceled, but I'll be around if people need questions answered.
Saturday, June 15, 2024
P
parsevalbtc11:25
replied to a thread: view
got avx2 and sse4 as well. Will re-compile also including additional flags reported in the messages above and let you know hows test will be going.
$ ./bn -c ../etc/libbitcoin/libbitcoin.conf -d Intrinsics... arm..... platform:0. intel... platform:1. avx512.. platform:0 compiled:0. avx2.... platform:1 compiled:0. sse41... platform:1 compiled:0. shani... platform:0 compiled:0. neon.... platform:0 compiled:0.
P
parsevalbtc16:51
replied to a thread: view
Unfortunately, it seems that by adding $ CFLAGS="-03" CXXFLAGS="-03" to ./install.sh ... the compilation stops at:
configure:3495: error: in `/home/user11/gitprojects2/libbitcoin4/build-libbitcoin-node/secp256k1': configure:3497: error: C compiler cannot create executables See `config.log' for more details(edited)
Sunday, June 16, 2024
Tuesday, June 18, 2024
evoskuil15:40
I've created an initial implementation for concurrent evaluation of confirmability. In libbitcoin validation and confirmability are distinct checks as they have distinct ordering constraints. Download is unordered by block, validation is partially ordered by block whereas confirmability is totally ordered by block. Yet, as with validation, confirmation is unordered by tx, and fuller blocks have around 4 thousand txs. This provides an opportunity to concurrently determine confirmability within a block, similarly to how we do with validation, which is concurrent by tx across multiple blocks.
🔥 1
evoskuil15:48
I've run the concurrent-by-tx confirmability checks both after completing download and during download. Interestingly there is almost no increase in total download time, 5-10% at the most but this is within the typical range of download variation. 840,000 blocks are downloaded and archived in 69 minutes while at the same time the first 381,395 have been checked for confirmability. The total run completes in 2.5hrs, which is slightly faster than if all confirmation is performed at the completion of downloads. Notably, confirmation reads a huge amount of data (and writes very little). So on low memory machines it will suffer, and performing it during download is likely to produce terrible page churn.
evoskuil15:59
I've already optimized the related queries using the performance analyzer. There's not much left to work with in that regard, the confirmation process has been reduced almost entirely to pure disk reads, like 90% of it initially. So the process accelerates as the RAM starts to fill (from a cold start), and is eventually screaming (over 100 blocks per second) with no disk reads and 90% RAM full.
evoskuil16:09
This process is what allows us to operate without the redundancy of a UTXO store, relying on just the block data and confirmation metadata. The two dominating costs of the confirmability checks are the
spent_prevout(...) and unspendable_prevout(...) queries.evoskuil16:19
Also, above checks are with no checkpoints or milestones. Checkpoints are consensus so should always be used, but these blocks are small and the impact is minimal, maybe 5 minutes.
Wednesday, June 19, 2024
evoskuil20:09
I decided to take another look at modest memory/CPU configuration as it pertains to the confirmability query. On my max machine confirmability takes a total of 2.5 hrs running concurrently with 1 hr download (with no checkpoints/milestone). However on a 32GiB machine it blows up due to paging, taking at least 1 second per block on average. This is a huge cost that I don't presently have any mitigation for. As it's a consequence of disk read, concurrency provides only a modest improvement.
🧡 1
evoskuil20:16
This is a flame graph of the complete process performing continuous confirmability checks on the 32GiB RAM machine. The yellow areas are all mmap paging from disk. These shrink to almost zero on my 256GiB RAM machine.
The left section is performing
unspendable_prevout reads, most of which is consumed by to_strong, which in turn is almost entirely spent in the tx hash table lookup. Since tx keys are 32 byte values, this is a (relatively) costly search. The tables is configured to 1.5 conflicts on average, but this can still cause significant paging of the body. The total actual distribution of conflict list lengths in the tx tables is:
tx: 1 frequency: 228206539 tx: 2 frequency: 171211514 tx: 3 frequency: 85657769 tx: 4 frequency: 32136386 tx: 5 frequency: 9642087 tx: 6 frequency: 2413019 tx: 7 frequency: 517115 tx: 8 frequency: 97139 tx: 9 frequency: 16368 tx: 10 frequency: 2436 tx: 11 frequency: 306 tx: 12 frequency: 31 tx: 13 frequency: 3
evoskuil20:26
The right section of the flame graph is the
spent_prevout query, most of which is consumed by a search of the spend hash table for the 7 byte input point (composite key of 4 bytes for the tx pk and 3 for the input index). This is a much smaller key, but because the table correlates to inputs, it is much larger than the tx table and therefore results in a similar search cost. Spend conflicts are also 1.5 with the following real (observed) distribution:
spend: 0 frequency: 130555225 spend: 1 frequency: 860714907 spend: 2 frequency: 542083651 spend: 3 frequency: 167287129 spend: 4 frequency: 33528526 spend: 5 frequency: 4944542 spend: 6 frequency: 576384 spend: 7 frequency: 54768 spend: 8 frequency: 4321 spend: 9 frequency: 276 spend: 10 frequency: 18 spend: 11 frequency: 1
evoskuil20:54
It's likely that with more constrained RAM there is a paging cycle between these two tables, as they compete for RAM. The query performs on one block at a time (in order) and one tx at a time within the block (either in order or concurrently), with the spendable and spent subqueries within the tx query. This could be rotated to perform all spendable queries in the block before (or after) all spent, significantly reducing memory contention.
👍 1
Friday, June 21, 2024
evoskuil01:16
I divided the per block confirmable queries into three natural sections, iterating over each in full before moving to the next. This reduces page swapping between tables. I then applied concurrency to each of the loops, using C++17 concurrency constructs. This results in at least a 300% improvement in confirmability for the baseline machine of 32GiB/10threads, averaging about 3 blocks per second, but this is still unacceptably slow. It appears that it has also modestly sped up the max machine, which previously completed in 2.5hrs, though I haven't evaluated that fully.
With sufficient RAM to hold the indexes, and with the indexes fully loaded (as during a full sync), confirmability runs faster than concurrent download, and can run coincident with it. This implies no net cost, though its inherent ordering implies at least some small increase in total time over checkpoint/milestone sync.
🔥 3
Tuesday, June 25, 2024
evoskuil19:22
After a lot of performance analysis on the confirmation process, I've concluded that the current table design is too costly for lower RAM machines, unless maybe there is a very high speed SSD. Mine is pretty typical at about 500MB/sec r/w. With 256GiB RAM the entire thing runs in-memory, disk activity is trivial. Otherwise it is 100% (thrashing), even at 32GiB RAM, I didn't bother below that. So supporting full chain confirmation on low RAM machines (part of validation) will likely require additional indexation (or will take at least a day if not several).
(edited)👍 1
evoskuil19:27
On a 128GiB RAM machine with 64 cores, validating confirmability (without script validation) of all blocks requires about 90 minutes. Full chain validation (without confirmability) has independently tested out at 3 hours. That's before any caching or optimization work. I expect to be able to bring that down. Running these steps independently places a full chain download/validation/confirmation at under 6 hours, though on big machines these can be combined.
evoskuil19:46
So I'm moving on to optimizing concurrent validation. This will consist of maintaining downloaded/parsed blocks in memory (cache) for the validator, and prefetching prevouts. Both the block cache and disk can be used to populate prevouts for validation, which implies that the cache is not only block-ordered, but output-indexed. Protecting the cache and allowing it to operate on spare cycles ahead of the validator implies that it requires its own thread(s), so it's another chaser.
Just as a reminder, the current full block sync under milestone is 65 minutes on my machine. That's to synch a fully usable, highly indexed chain from the Internet. I've done this at least 100 times and it only varies by a few minutes for any given run. A low memory/core (8GiB/8thread) milestone run is closer to 3 hours.
(edited)👍 3🎉 1
Thursday, June 27, 2024
evoskuil01:00
There are several ways that costly loops can be parallelized, assuming that the design allows the concurrency. There are four loops that can be independently parallel, and setup synchronization can be accomplished via standard library execution policy on loops such as
(edited)std::all_of(...) or via BOOST ASIO.
As std lib execution is only useful for simple loops, we use ASIO for network concurrency. I've learned in the past that the cost of using std lib execution policy can be very high. I tested it for simple two lane parallelization in RIPEMD and the result was horrific compared to just sequential execution. The only place we use it is in concurrency for scrypt hashing execution. In that case it provides a huge benefit with no ASIO complexities or dependency. The database queries are complex and costly, so I figured it was worth a look.
Turns out that using an ASIO thread pool, posting closures to the work queue, and tracking completion, absolutely crushes the much simpler loop vectorization. std lib takes 133 mins to ASIO's 79 mins.
I'm pretty sure that this type of performance difference would also impact C++20 co-routines. When I started on the network stack rewrite I looked at coroutines, but unlike the the rest of C++20, the reviews weren't good. A thread pool provide significant performance advantages, and these are now very visible.1 reply · evoskuil · last Jun 27, 2024
evoskuil16:11
Another simple optimization got the above 79 minute time for confirmability and confirmation, with no checkpoints/milestones, down to 73 minutes. So we're close to 1 hour for sync and 1 hr for confirmability, which can run concurrently on large machines. With checkpoints/milestone this is 65 minutes total combined.
(edited)pmienk19:32
Topic: Dev Weekly Status
Time: June 27, 2024 01:00pm Pacific Time (US and Canada)
Join Zoom Meeting
https://us05web.zoom.us/j/87666370902?pwd=bEZDUlJ0MGJ5V3dVMUhVdHl5QzdFdz09
Meeting ID: 876 6637 0902
Passcode: 85U0bF
Huddle with evoskuil, pmienk, Shashank Shekhar Singh · 2 min20:57