Bitcoin full node and query server
February 2025
Archive of the libbitcoin Slack #general channel.
Saturday, February 1, 2025
evoskuil04:10
I'm now running a full validation/confirmation sync on the same machine. This is with only the prevout table conformation small memory optimization. Presently about 1/3 complete (by tx count). It's configured for a 5k block window, which slows the early blocks (this can be sped up some later by expanding the early blocks window, but it's not a major factor).
I expected this run to thrash, but the confirmation rate is holding very steady at 60,000 transactions per second, for the past 200,000 blocks (450,000-550,000). If that holds until completion then the run will take 12.5 hours. For reference Core full validation on this machine (8 dual core, 32GB RAM, Win11, shani, avx) is 15 hours.
(edited)💥 1
1 reply · evoskuil · last Feb 01, 2025
evoskuil04:14
For additional context, the same run takes 4.7 hours on the maxi, on which Core takes 18 hours. Core gets slower on the big box because it's not shani, and we get faster despite that because we can take advantage of the extra resources (32 dual cores, 256GB RAM).
👍 1
As you can see the limiting fact here is RAM, with the SSD therefore maxed out reading blocks and prevouts for validation, and reading/writing indexes for confirmation. CPU is minimal despite validation. Without the confirmation query (in other tests) CPU is maxed out, as it is on the maxi. So the objective is to further reduce the cost of the confirmation query, reducing RAM cost and SSD activity.
(edited)
👀 1
So now that everything is cleaned up from the last phase, and we've got updated benchmarks to work from, I'll finally be implementing the point table (point.hash lookup table) consolidation into the spend table. That should significantly reduce confirmation query cost, which should raise CPU utilization for validation.
(edited)👍 1
evoskuil set the topic: Discuss libbitcoin here.04:23
evoskuil05:00
Here's the current trajectory, actually speeding up a bit.
Note that the first 3.5% of the chain is checkpointed, so by consensus these are not validated. The total time for those is about 10 minutes with the 5k window. To validate them takes about 30 minutes (so 20 additional minutes overall if we were to do that, but that should not be done since it's 100% waste).
The rate for the subsequent 30% of the chain has been very consistent, even speeding up a bit from segwit activation in the last segment. I'll update this in the morning. And it appears that this is not going to thrash, which is already a huge win.
(edited)
evoskuil06:45
A clarification on the perf console above. The full validation runs tend to cycle within the window between download, validate and confirm. Download is low CPU but validate does max out CPU:
Also I had a material math error in the spreadsheet for the plot above:
So this run is projecting 15.4 hours presently, if current rate sustains, and the rate isn't as constant. The prevout plot was dividing cumulative txs by time, vs. txs in the 50k block window.
evoskuil22:46
There was a stall due to a race condition in the check_chaser (it assumed confirmation notifications arrive in order (they are sent in order but may not). That's been resolved, and the run is almost complete:
The confirmation rate per transaction has dropped from over 35,000 per second to 6,000 per second. The logical cost of each tx is essentially the same (and stays constant on the big box). So thrashing is increasing as the cost. If the confirmation rate stayed at the initial 35,000 txs/sec then the full run would have completed in 8.3 hours, but is presently looking like it will be around 18hrs.
Sunday, February 2, 2025
evoskuil06:41
Trend is pretty much the same as before the prevout table change. Thrashing increases over time, probably as a consequence of the growth of the strong_tx table, which is both written and read throughout. The reading is constant time, but the paging cost increases as the table grows.
On the maxi there is no thrashing and per tx performance remains constant throughout. So the goal is really to get the confirmation disk I/O down to the point where it doesn't thrash, which should allow completion in around 8-10 hrs on the mini.
🔥 1
Monday, February 3, 2025
C
cguida17:53
What is the strategy for reducing thrashing? Is the idea just that the whole process is taking up too much RAM?
evoskuil18:35
I'm making the two query changes discussed in the meeting a few weeks ago. The process doesn't take too much RAM. It always uses all available RAM. The confirmation queries, on lower RAM machines, can result in thrashing (quadratic paging) because of the need to read all over certain tables. memory maps can only be read from disk at the granularity of a page, even if just reading one byte.
Controlling paging is the central issue in a memory map store design. This can be done by improving the efficiency of queries, which takes various forms. The query changes I'm implementing consist of (1) using a intermediate table to cache the store identifiers of prevouts spent by inputs, while the inputs are being validated against the prevouts. This precludes the need to perform the prevout lookup twice, in exchange for a modest amount of data. The data can be stored and retrieved in a unitary memory chunk per block, with no search paging. This part is complete and has improved speed, but as expected is not sufficient to prevent thrashing on the mini.
The other (2) change is to consolidate the "point" table into the "spend" table. Point is a lookup table to spent point hashes. This is a tradeoff in query complexity in exchange for reduced storage space (about 50GB saved). This is because point hashes are redundant (multiple outputs per tx, indexed by 32 byte hash in the spending inputs). However the optimization results in thrashing on the mini, and slows sync somewhat on the maxi, so we haven't been using it. Removing it will significantly reduce paging and may therefore eliminate thrashing on the mini.
(edited)👍 1
Tuesday, February 4, 2025
evoskuil00:26
In the past two days I've completed most of the merge of the point lookup table into the spend table. I should be done updating tests today and hopefully can get an initial test run in.
🎉 1
evoskuil06:28
Completed test updates, running milestone on the maxi for a baseline now. Running slightly faster than previously. That's expected due to the code simplification and write of slightly less data overall. But no significant change is expected in milestone runs on either maxi or mini.
evoskuil23:42
Here's a more representative picture of concurrency within the node, while downloading, storing, validating and confirming blocks:
Wednesday, February 5, 2025
evoskuil23:47
I've completed evaluations of consolidation of the point and spend tables. Turns out this is a bust on both big and small boxes. It takes about 1.5x on the big box, even before becoming memory bound. The extra time is spent in comparisons of search keys. This amounts to memory I/O to perform the comparisons (not even paging because the entire store is memory resident). One of the advantages of splitting the tables was to make search against either of them less costly due to reduced paging and increased locality, despite it taking about 20GiB more space when not utilizing hash consolidation. Turns out that's a significant benefit. On the mini the performance hit is the same up until thrashing kicks in. So I'll stick this work in a dead branch and revert, while cherry picking a couple of small improvements.
(edited)🫤 1
Thursday, February 6, 2025
evoskuil00:02
I immediately went back to first principles to see if there is some other way to approach it. There are 2 queries that make up the confirmation process (3 if bip30 is active, which it never is). One of the queries is inclusive and the other is exclusive. Inclusive means we check to ensure that the prevouts are confirmed. One confirmation is all we expect/need, and we find it quickly. The other is to ensure that there is no confirmed spender of each external (not in-the-same-block) prevout. This is exclusive - we must exclude all possible spenders. Since we allow conflicting and redundant blocks and txs, and we store them all in the same space, we must ensure that none of them that spend the prevout are presently confirmed (this is during confirmation, so there is no race).
This exhaustive search is to identify any confirmed spend is costly, as the natural point (hash/index), and is the only thing that correlates the spends. Spends are in another table, indexed by the point that they spend - but not the natural point, what I call a "foreign point" which is the primary key of the tx hash (4) in the point table, and the index (3) or 7 bytes. So finding all of the confirmed spenders of a point logically requires reading the spend, transiting to the point table, getting the hash (32) and then searching the point table for all identical hashes, and obtaining the set of point primary keys. Then the index is used to formulate the set of foreign points, and these are used to search the spend hash table. In doing so we exclude the spend that we are presently evaluating. Any other spends are then transitioned to their parent tx primary, which is then searched in the strong_tx table (for confirmedness)...
evoskuil00:09
A recent change I had made to the first query was to expect success (prevout confirmed) and only if it is not confirmed, search for other possible identical transactions (same but with other ids) in the strong_tx table. This is costly, but rare. In a normal run there are only two duplicate txs (bip30 exceptions) and these don't spend. So performing the extra step as necessary is essentially free.
The second query is a different problem, because the search must be exhaustive. The issue then is that txs with the same hash don't have the same primary key, so natural keys (tx hash) must be searched to correlate both spends and confirmations. But we cannot ensure that no duplicates enter the store, as that requires a search for every insert. That tight coupling makes archival (download) very slow. However, the hashmap hash of each identical hashmap element must of course be the same. These duplicates show up as hashmap conflicts and we simply walk the conflict list using an iterator to key on matching search keys, which makes any hash table a multimap.
evoskuil00:17
Spend search keys are the primary key of the point table, concatenated to the 3 byte point index. The point pk of course varies even for identical point hashes. It's just the record offset in the point table. However, if we index the spend table not on the primary key of the point, but instead on the search key of the point (point table hash function of the point hash), all spends of the same point would have the same search key, and would therefore be directly identifiable in the spend table iterator. However, conflicts could not be resolved without reading the actual point hash from the point table.
evoskuil00:27
Our hashmap functions for sha256-hash keyed hashmaps is a no-op. It's just the low order bytes of the hashes. So not only is it a good distribution, but we can obtain it at no cost. But this isn't the most interesting aspect. By making the spend table search key a function of the point hash, all conflicting spends would necessarily share the same spend table search key. So it is unnecessary to read the point table at all unless there is a conflict. It's not necessary to know the point hash, as the spend for which we are investigating conflicts is already indexed with the foreign point that all conflicting spends must have. This would be the case if the point table was guaranteed to be unique, but it cannot be. So if there are no conflicts with the foreign point of the spend (i.e. a search finds only itself) then there is never a read of the point table during the search. And if there is a conflict in the spend search, the spend can directly identify its point (which it presently does via the foreign point). This implies extra bytes per spend to account for the hash function value in the search key while retaining the point pk.
evoskuil00:36
The cost of reading the point table to check conflicts is no higher than searching it up front (as a hashmap) for conflicts, and in fact lower because no conflicts are paged through and its header is not accessed. The spend table is searched for the same sized key as before, and conflicts are expected as before.
Spends (inputs below) are indexed on a foreign point (point pk and point index), which is very regular as well, as primary keys are sequential. This would remain similar using the sha256 truncation (as above) in place of the sequential point pk.
evoskuil00:54
In the above table, there would be over 300k spends that would encounter no conflict (14%). This means that the point table would now be unsearched for each of these - never even accessed. Singular entries in the spend table multimap would imply the current spend is the only spend (which is verified at no cost by comparing the spend pks). The remainder would compare hashes in the point table for each conflict that is not self. That reduces to n-1 hash compare for each n conflicts. So for 2 conflicts above (41% of all spends) that's one sha256 compare in the point table. This drops off logarithmically, 31% require 2 compares, 12% require 3 , and the rest is inconsequential. And again, no searching and comparing all conflicts in the point table. The cost tradeoff is 4 bytes extra per spend. That's not trivial, presently 49GiB (7% increase).
(edited)Keep in mind also that this is not required on the big box. The node currently syncs with full validation/confirmation in under 5 hours. This is really about eliminating thrashing on the small target.
👍 1
evoskuil01:13
Another related consideration is that of strong_tx indexing. Presently the tx table pk is the search key for the strong_tx table. This means that the same tx can be strong (confirmed) under any of the pks under which it may be stored. This means that the exhaustive search above must exhaust them all. This means searching the tx hashmap (using the sha256 hash of the tx) to obtain the pks, until all are excluded. I've considered indexing this table on the hashmap hash of the tx's sha256 hash a well. This would also require secondary conflict resolution, but would preclude searching the tx table or using the 32 byte search key. This resolution could be satisfied by storing the pk of the specific tx confirmed in the row. This would then allow for an immediate search using information immediately available in the spend row (the hashmap hash of the tx sha256 hash is the spend search key), which would eliminate any tx table searching. The implied storage cost is 4 bytes per tx, presently 3.8GiB.
Note that on the big box, this ~54GiB storage cost can be offset using point table hash consolidation, for a modest sync time increase.
The above changes are far less disruptive than the table consolidation that I just implemented over the past 5 days (complete with tests :|). I should be able to get a test of it running in a few days.
👀 1
C
cguida15:58
@pmienk are cmake presets supposed to work in vscode? struggling to build the nix-gnu-debug-static preset
[main] Building folder: /home/cguida/code/libbitcoin/src/libbitcoin-node/obj/nix-gnu-debug-static [main] Configuring project: libbitcoin-node [proc] Executing command: /usr/bin/cmake -DBUILD_SHARED_LIBS:BOOL=OFF "-DCMAKE_C_FLAGS= -Og -g --coverage" "-DCMAKE_CXX_FLAGS= -Og -g --coverage" -Denable-ndebug:BOOL=OFF -DCMAKE_PREFIX_PATH:PATH=/home/cguida/code/libbitcoin/src/libbitcoin-node/builds/../../../prefix/nix-gnu-debug-static -DCMAKE_LIBRARY_PATH:PATH=/home/cguida/code/libbitcoin/src/libbitcoin-node/builds/../../../prefix/nix-gnu-debug-static/lib: -DCMAKE_INSTALL_PREFIX=/home/cguida/code/libbitcoin/prefix/nix-gnu-debug-static -S/home/cguida/code/libbitcoin/src/libbitcoin-node/builds/cmake -B/home/cguida/code/libbitcoin/src/libbitcoin-node/obj/nix-gnu-debug-static [cmake] CMake Warning (dev) at CMakeLists.txt:149 (find_package): [cmake] Policy CMP0167 is not set: The FindBoost module is removed. Run "cmake [cmake] --help-policy CMP0167" for policy details. Use the cmake_policy command to [cmake] set the policy and suppress this warning. [cmake] [cmake] This warning is for project developers. Use -Wno-dev to suppress it. [cmake] [cmake] -- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.83.0/BoostConfig.cmake (found suitable version "1.83.0", minimum required is "1.76.0") found components: unit_test_framework [cmake] -- Checking for module 'libbitcoin-database >= 4.0.0' [cmake] -- Package 'libbitcoin-database', required by 'virtual:world', not found [cmake] -- Configuring incomplete, errors occurred! [cmake] CMake Error at /usr/share/cmake-3.31/Modules/FindPkgConfig.cmake:645 (message): [cmake] The following required packages were not found: [cmake] [cmake] - libbitcoin-database >= 4.0.0 [cmake] [cmake] Call Stack (most recent call first): [cmake] /usr/share/cmake-3.31/Modules/FindPkgConfig.cmake:873 (_pkg_check_modules_internal) [cmake] modules/FindBitcoin-Database.cmake:58 (pkg_check_modules) [cmake] CMakeLists.txt:167 (find_package) [cmake] [cmake] [proc] The command: /usr/bin/cmake -DBUILD_SHARED_LIBS:BOOL=OFF "-DCMAKE_C_FLAGS= -Og -g --coverage" "-DCMAKE_CXX_FLAGS= -Og -g --coverage" -Denable-ndebug:BOOL=OFF -DCMAKE_PREFIX_PATH:PATH=/home/cguida/code/libbitcoin/src/libbitcoin-node/builds/../../../prefix/nix-gnu-debug-static -DCMAKE_LIBRARY_PATH:PATH=/home/cguida/code/libbitcoin/src/libbitcoin-node/builds/../../../prefix/nix-gnu-debug-static/lib: -DCMAKE_INSTALL_PREFIX=/home/cguida/code/libbitcoin/prefix/nix-gnu-debug-static -S/home/cguida/code/libbitcoin/src/libbitcoin-node/builds/cmake -B/home/cguida/code/libbitcoin/src/libbitcoin-node/obj/nix-gnu-debug-static exited with code: 1(edited)
9 replies · pmienk, evoskuil, cguida · last Feb 06, 2025
pmienk18:08
Yes, but they do not impose a build order, require their dependencies (as I believe the above indicates as a failure), and depend on a specified directory structure.
👍 1
pmienk18:13
The above is failing because the database project hasn't been built and installed to the expected prefix path.
👍 1
pmienk21:05
I got it working, then it went to the back burner since I work gsl more than anything else and vscode doesn't help me there
Huddle with pmienk, Marnix, cguida, evoskuil · 163 min20:57
Friday, February 7, 2025
evoskuil00:52
Correction on my sizing estimate today. The increase to the spend table by adding the 4 byte point primary key is 10GiB.
Saturday, February 8, 2025
evoskuil20:03
I completed the confirmation change last night that we discussed at the last meeting. That took about 24 hours and worked out a little simpler than I expected. Turns out there is no index change necessary to optimize strong_tx queries during confirmation, those can remain indexed against the tx table primary key, since we must identify all spends of the same point by hash, and those resolve directly to the spending tx pk. The run completed on the maxi in similar time to previous runs without the change. Testing now on the mini.
Also, I'm starting to see that there is a way to get more concurrency out of confirmation, and probably also resolve the thrashing for certain. Spending some time on that now.
👀 1
evoskuil20:26
On the maxi I took another look at the performance analyzer during confirmation on the maxi. There are three concurrent loops for each block being confirmed: get-spends, is-spendable, is-spent. I previously spent a bunch of time organizing these in different groups with parallelism and without, and this was the most optimal. But still, the std lib loops are very inefficient. About 75% of all confirmation time on the maxi is spent in lock contention. The locks aren't guarding read/write - they are only providing thread synchronization for the loops.
The database mutexes for hashmap headers, table allocators, and remap safety are all in place as well, but they are inconsequential - because there are very few writes occurring. So is appears that confirmation (absent thrashing) can be significantly optimized during IBD by performing more of the calculations concurrently by block (i.e. as part of validation) and using more storage (as we've done with the prevout table). The only aspect of confirmation that must be sequential by block is checking for spentness (i.e. query against the strong_tx table). So in theory we can produce an ordered list of tx ids to check for spentness by block, followed by writing spentness for each block.tx.
Double spend checks are most expensive, as they require exhaustive search, however they rarely require a check for spentness, because double spends (duplicate tx storage or actual invalidity) are rare. So this is almost no storage and can be made concurrent (*). Confirmedness of prevouts just requires identifying all of the prevout tx pks and storing them until the confirmation stage. IOW, there would be a set of positive and negative tx pks checks for each block, which could just take the place of the data already stored into the prevout table, and would be the same size (4 bytes per spend), and would also be temporary (deletable).
Given that we already use this temporary space and already cover the work distribution cost during validation, and that the concurrent part of confirmation would add no writes, it should be possible to significantly accelerate confirmation as long as the additional reads do not cause thrashing. Despite the fact that we read the entire block and previous outputs for validation, we never search the spend table. That's the big difference in terms of paging, as we must search for all spenders (even though we generally don't find any).
But in the case of thrashing if this is run during validation, it could just be split out to an independent concurrent phase. So I think we can get much more concurrency out of confirmation.
👀 1
evoskuil20:40
Also, we have a third part single file lib included in system now that provides support for std lib concurrent loops for gcc and clang. But it also adds additional performance optimization such as thread pooling (which supports all platforms including msvc). This alone should eliminate most of the lock contention, because it's most likely arising from thread create and release (the std lib doesn't use pools). So this is a simple option that alone could significantly improve confirmation performance on the maxi. It's already pretty good, despite 75% being dedicated to lock contention. The total time is 4.9 hours and without confirmation it's 3.5. So we're spending 1.4 hours in confirmation to complete a 4.9 hour full validation run on the maxi. I think we could get confirmation down to 20 minutes on the maxi (just by eliminating lock contention), for a < 4 hour full validation/confirmation run.
(edited)🔥 1
Sunday, February 9, 2025
evoskuil21:26
I completed the above test runs on maxi and mini, and there is no consequential change, though the code is simpler. I was also able to remove the search (hashmap) index from the point table, which saved 4 bytes per row plus the header, or 12GiB.
I've concluded that on the mini nothing will get faster as long as the same amount of confirmation paging is required. Based on a lot of inspection I can see that the is-spent exhaustive search over the spend table is where the real cost is. The only way I can see to reduce paging from this is to remove all data not necessary to the search. IOW make it a simple index that points to the remaining data. There is already structure for this, as the spend points to the point table, where the prevout hash is stored. So I can move the parent and input pointers, and the input.sequence value (12 bytes) to the point table. This moves 29GiB (40%) out of the spend table. This will definitely reduce spend query paging, and given that the point hash is already necessary for reading and writing a tx, moving the sequence there isn't an extra cost for download/validate. It does increase the point table, which increases paging there, but we don't do that for spend hashmap conflicts, so should save a lot of paging overall. Relatively simple change as well.
On the maxi I believe that implementing the custom std lib concurrency replacement (required for gcc/clang anyway) with a thread pool may significantly improve performance, because the main limitation there (about 75% cost) is lock contention.
(edited)evoskuil21:41
Even with the contention, the concurrent algorithms are significantly faster than sequential operation. I did also test making these concurrent using a network threadpool, but the std lib calls were much simpler and at least as fast.
Monday, February 10, 2025
evoskuil05:09
Ok, std lib concurrency support for all compilers is now merged. Performance on the big box looks similar to before (I enabled it always even though msvc does have native support for c++17 concurrency). Now that the locks are identifiably different in the perf monitor I can see that loop synchronization cost is now relatively low, a couple percent. The locking cost in the confirmation query is therefore coming from the hashmap headers and/or table read pointer. With heavy read/write concurrency, as happens on the big box due to heavy archival at the same time as heavy query, lock contention gets pretty significant.
This could be eliminated entirely by isolating the phases and even removing the locks entirely with write suspended during the confirm phase. This could be done with dual table implementations templatized for one to exclude locks altogether. Hard to say for sure if it would be a benefit, since each lock is waiting on actual work on other threads, and this work is absolutely trivial (eg write of one word). The mutexes however do have overhead, and it may be possible to reduce this. But that's getting into the finer details of optimization which are still premature.
👍 1
Tuesday, February 11, 2025
evoskuil03:41
I've implemented the change above, making the spend table into a pure index and moving its values to the point table. This required a net increase of 5.3GiB overall, after the removal of the hashmap in the point table, and the net increase of 3 bytes per spend due to redundant storage of the input.point.index in the spend index and the point table. The schema and query changes were pretty significant, affecting a lot of tests. So I'm going to run through it first and see how it looks before burning the time on test updates.
Otherwise the net result is a change in relative size of the two tables. Previously point and spend were 94 and 63 GiB respectively. Now they are 126 and 39GiB. This is the intended reduction of the spend index, as it is used for exhaustive search. This should cut paging by at least 40%. However there are other query simplifications that should reduce paging even further.
evoskuil04:43
And I just came across the realization that we can allocate tx input points contiguously in the store (we do this for the spend index). This then precludes storing them in the puts table (first and count are already stored in the tx, and the point table points to the input). inputs and outputs are variably sized, so something always need to point to them. This would materially reduce table allocation calls (by 2.6 billion), and shave 12GiB off the store.
This affects archival, validation, and confirmation, since they all traverse the puts table. It would remove the puts table from confirmation entirely (outputs are not traversed). That's a 23GIB table, so that's a huge paging reduction - as big as the change to the spend index.
(edited)🔥 1
Wednesday, February 12, 2025
evoskuil03:30
I was able to make two material performance improvements to the store's tx write query so far today, as a consequence of the above changes to the point/spend table. It's now performing contiguous allocation in each tx for all spends and addresses (address index is optional), and it's performing these without using a spend link accumulator vector. Testing now with a milestone run and it's the fastest run to date so far.
I haven't yet collapsed the puts table to just outputs, that's next, along with contiguous point allocation (as with spends/addresses above). So this will both cut the puts table in half and avoid it altogether during confirmation, but will also accelerate tx writing due to the allocation.
(edited)I've made a lot of changes, but time had actually been moving up. So this was a lot more than a 2 minute savings.
(edited)evoskuil05:06
Theoretical is a little lower than that, because I really have a 2.3Gbps network - despite paying for only 2.0
(edited)So maybe we can get to 30 minutes.
(edited)Ohhh, the limit on 2.3Gbps is 34 minutes!
evoskuil05:14
If removing a net 6GiB of writes (half the puts table, with a smaller add to the tx table) doesn't knock a couple minutes off I'll be surprised.
evoskuil05:22
Previous runs on the mini were hitting around 55 minutes, so the recent changes are pretty significant.
evoskuil19:48
Did a couple more runs, after writing points into a single contiguous allocation and removing their mapping in the
puts table. The puts table should now become the outs table, as it only maps tx to its outputs. It's now analogous to how points maps tx to its inputs. The points table could be renamed to ins so that we have tx->ins->input and tx->outs->output. These are association tables, like the txs table which maps header to tx. The convention is that the plural is used for association, and since point has become an association (it used to be just the hash in a lookup table) should also be renamed for that reason.The milestone runs did not beat 37 minutes, one came in at 38. Despite being simpler, faster allocations and fewer writes, with 6GiB less data, performance didn't increase. This means that it's currently network bound at 37 minutes on my network (with a theoretical limit of 34 minutes).
(edited)3 minutes to do everything on the network apart from downloading the block bits is pretty good I guess.
💯 2
evoskuil20:19
It should be the case however that the same time is achieved at materially lower resource cost (RAM, SSD, CPU) which should help with validation/confirmation.
evoskuil20:41
Another performance mod and config simplification that I've been looking at is changing the nine hashmap tables to configure with an base 2 exponent instead of a bucket count. This significantly reduces granularity of control over bucket count and therefore conflict rates. But it also reduces the hash function cost by swapping out modulo (%) for shift right (>>). There are tens of billions of calls to this function, and this would cut it's latency by 3 or 4x on typical hardware.
evoskuil20:47
Currently I target a load factor of 1.5, and doing this gets us between .61 and .96 for rounding the exponent up, and 1.22-1.92% when rounding it down. So still pretty reasonable. And it may be worth the additional 12.4 GiB in headers to round up, though this may vary by resource availability.
evoskuil21:15
For context, presently we are close in size to Core. It's 660GiB (chainstate and blocks with no tx index and of course no spend index) and we are 695GiB with tx and spend indexing (always). Our spend index is 43GiB (which is now very compact, with only 15 bytes per spend), and our tx index costs 6GiB. Though these are required, without them we are 645GiB. Our Windows node binary is 5MiB and Core is 15MiB.
🔥 1
Thursday, February 13, 2025
C
cguida17:49
I think taking more disk space to speed up ibd on low resource devices is absolutely worth it, as long as the total fits into a 1tb ssd
Btw I don't have any updates for today, apologies guys, I've been getting ready for my workshop at btc++. I hope to be back at it once I'm back from Brazil!
👍🏻 1
Huddle with pmienk, Marnix, evoskuil · 83 min21:02
evoskuil23:43
Another simple change of the
puts table, not that it's only links to outputs (vs. spends and outputs) is to change it from a blob to array table, which reduces its link size from 5 to 4, saving 1GiB in the tx table.Friday, February 14, 2025
evoskuil01:57
The minor issue I mentioned at the meeting today is resolved, and the latest optimizations are running in full validation on the maxi.
evoskuil04:08
It also looks like I figured out why the last maxi confirmation run lost its mojo. Running now and so far on par with previous best run.
The three parallel transformations that make up all of the confirmation process are now almost evenly divided (and together make up almost 90% of the CPU time during confirmation)...
Confirmation is 93%, the write at the end of each confirmed block is 5.3%, and the rest is ongoing download and validation, pushed aside by confirmation.
This is the full process at the time above, with the three parallel loops highlighted.
This alone doesn't imply good performance, but the fact that the load is more spread out means that no one item is dominating - which I think is good for low mem paging.
evoskuil05:10
It's keeping up well, and when it hits a wave of confirmation, it races ahead very rapidly on low CPU (since that's largely sequential). And I've noticed that as the tables grown, the spent query slows down materially relative to the others. This is the exhaustive hashmap search.
evoskuil05:26
The second image shows that the first iteration of the search is consuming 80% of the cost. That's great, because it means low hits on the very large point table. The whole run is spending 11% reading hashes from the point table because of spend hashmap conflicts. But that means by increasing the bucket count this might be materially reduced. It's a tradeoff, because that puts a lot more into RAM. As it is the table heads read used during sync are about 9GiB. Since they are random access, they spend most of their time in memory.
(edited)evoskuil05:36
When the maxi goes into pure confirmation on a string of full blocks, there's only 3% CPU, 5% SSD and 0% NET. That's with full parallelization of the work within each block.
It's a great demonstration of how underperforming parallel validation is when it's limited to block internals, which is what Core does.
This is a window crossover, so the CPU jumps up to 25% for archive, and NET is peaking, with occasional SSD (M:) flush.
This shows a disk flush for a table remap, and also notice ho much RAM consumption has dropped. That's just the OS clearing it out.
And here's the transition from archival to validation. Really there is still a lot of confirmation happening to, its just that at 100% CPU you can see that validation is dominant.
During validation we are reading every block and its previous outputs from the store. But look how low the SSD activity is, even with RAM flushed. This is much lower than previous runs. It was pegging out SSD and RAM before.
evoskuil05:53
From previous runs I found that archive/validation were pretty much limited by CPU, with signatures at ~85%, signature hashing at ~5% (this machine does not have shani) and block read/populate at ~10%. There's a little to shave off there in reading, and of course the NET/archival is already fully optimal. That completes in around 4.5 hours (no confirmation). Which means that confirmation on the maxi is only about 30 minutes of the total 4.9 hour cost. I could probably cook up a theoretical limit for validation based on signature/hash cost, but I think it's probably around 3.5 hours, plus the 40 minutes of download. Puts us pretty close to optimal (for this hardware).
Double the CPU speed and add shani with fast SSD/RAM interface (which is what we have on the mini) and the validation time would be around 1.5 hours, putting the full validation run under 3 hrs. Double the core count from that and it's done in 1.5 hrs.
Half way through the run by tx/byte count, and it's still pacing slightly ahead of the previous best run. So at this point I know it won't diverge.
evoskuil06:47
As the spend hashmap grows, search performance continues to decline due to increased conflicts:
The significant increase in
get_doubles (potential double spends) implies not just an increase in hashmap conflicts, but also a significant increase in secondary conflicts (where the hashmap keys are the same but the point hash is not).
The increase in iterator advance also shows the increase in secondary conflicts:
These are based on a 4 byte hashmap key for a 32 byte value. The key size could be increased by one byte to reduce conflicts by 20%, for a space cost of 2.5GiB. Primary conflicts can be reduced by increasing bucket count. Basically up against a direct space/speed tradeoff but with constrained space.
evoskuil18:15
Spoke too soon, divergence came later than previously, but came just the same. Up from 4.9 hours to 7.1:
(edited)
evoskuil18:27
It's entirely due to increase is-spent costs as shown in the performance trends above, due to rising conflicts and the associated paging costs. There are three ways to reduce the conflicts. (1) split the point hashes into another table, 1:1 with both arrays, so does not require another index or any more storage, (2) increase the spend hashmap bucket count to reduce hashmap conflicts, (3) increase the spend table hash key size to reduce secondary conflicts and therefore reads into the point hash table.
I believe that (1) would bring it back to where it was before, or better, at least on the maxi. The others are space tradeoffs that will probably also help on the max but maybe not on the mini. None of the above changes would I expect to get the mini confirmation cost down significantly.
Saturday, February 15, 2025
evoskuil19:24
There are two costly confirmation queries,
is-spent and is-spendable, Last month I used a new prevout table to allow for precalculation of is-spendable in the validation phase, with completion on the confirmation phase. This reduced the amount of sequential computation in that query to the minimal strictly necessary. I hadn't been able to figure out a way to do this with the other query. On the big machine, during pure confirmation, there is now very little resource usage, due to query optimizations. So at least 50x more work could be done on this machine if is-spent could be parallelized except for the strictly necessary checks.
Last night I figured out how to move the entire is-spent query into the validation phase, except for a final check which in a typical run will always be a no-op. This won't necessarily help a lot on the mini without paging optimizations, and I have a couple more in mind. But it finally gets us to the point where the implementation maps to the theoretical ordering constraints.
As with the is-spendable optimization, this moves the costly part of the query to validation, where it can be parallel by block given all previous blocks are archived. The result of the precomputation is moved via the store to the confirmation phase in a block of fully-disposable data (can be purged during start, stop, and/or snapshot). In the is-spendable case the data required a 4 byte primary key for the parent tx of every spend. This is already read during validation, so it was free except for the temporary table.
In the is-spent case we need to identify the same primary key of the parent tx of any conflicting spends. This is a value also obtained during validation, as is the hash of every input point. So precomputation saves these redundant reads of the "self" spend identity. The costly part is the search in the spent table for conflicts (spends for every point, other than self). Turns out this query doesn't require total order, since it's not possible for a spending tx, introduced during validation (i.e. from the tx pool), to become confirmed in the interim. Only the instance of any tx, already associated with the block header at this point, will get marked as confirmed in the final confirmation step (due to enumeration of txs via the block association).
So the query can safely identify all spenders that might become confirmed in any previous block, and save the parent pk of those to the temporary table. This is typically going to be zero, and can be implemented with a prefix count value in order to differentiate from the is-spendable values in the table. The confirmation phase then just checks these txs for confirmed-ness, and if so the block is invalid for double spend. Though again, in a typical run there are no such conflicts, so is-spent essentially becomes a no-op in the strictly-ordered confirmation phase.👆🏻 This is pretty huge. I'll have it implemented and tested today.
🚀 2
Sunday, February 16, 2025
evoskuil21:49
Completed implementation yesterday, worked through a couple minor simple bugs resulting from not having updated all of the unit tests. Should have a test running soon.
evoskuil21:56
Running now. Confirmation is now operating at 100% CPU (vs 2% previously).
(edited)🔥 4👌🏻 1
evoskuil22:08
Actually the confirmation phase is now just a tiny percentage of what it was previously, as most of the work has been moved into the validation phase. That tiny percentage is small/fast enough that it gets subsumed by validation/download processing, so the CPU stays at 100% instead of everything waiting on slow confirmations.
An entire table has been removed from the process, since that data was already readily available during validation.
All of the costly confirmation querying has been moved to the parallel-by-block validation phase, which also means that there is no need for those queries to be in parallel std lib loops. This removes a lot of threading overhead, since we are already paying for concurrency by block.
The prevout cache (passed from validation to confirmation phases) is 2x larger because I added the sequence value. This can be optimized by only storing the sequence after bip68 activates, and only storing it when the relative locktime bit is set. But even at full size it's not bad, and can be deleted at any snapshot/start/stop.
I can't say that this will help the mini, since it will remain a question of disk paging, but I expect material improvement on the maxi. But there has been a significant paging reduction for sure. The remaining cost of the confirmation paging is really just reading point hashes to identify hashmap conflicts. That has also been reduced significantly by passing the "self" hash directly from the deserialized block into the confirmation query. I'm pretty sure that by splitting out the hashes form the point table this will be reduced even further.
evoskuil23:40
Tracking consistently with previous runs so far. But the interesting part will be as the tables get larger.
I've measured the cost of pre-confirmation query and storage during validation, and it's only 0.48% of the run! That's inline with the completion of every block validation, and accounts for less than
(edited)count_sigops (0.67%). During validation script execution is 94% of the run, plus get_block at 2% and populate_prevouts at 2%. Signature verification is 82% (including signature hashing which takes about 15% (no shani)).
0.48% is a pretty amazing number for the most expensive part of confirmation. The final stage of confirmation is still a cost, but on its own it now runs about 30% CPU and is hitting much less data (no longer searching tables on the order of inputs/outputs).Monday, February 17, 2025
evoskuil01:49
Well, past the latest point of previous divergences and still tracking closely to the best previous time. Keep in mind that prior to this change the run was 7.1 hours and now it's tracking at 4.9.
(edited)evoskuil04:33
Got to about block 800k validated and hit a bug somewhere in the new code. But close enough to completion to see that it was going to at least match the previous best time. Generally on the maxi the total full validation time is constrained by signature validation. But this change is significant, in that it means the expensive part of confirmation can run concurrently. The sequential part is faster than download, and I should be able to get it to run concurrently with download, by changing the download window behavior.
This overlays the last run on the previous best 4.9 hour run, up to the point of termination. I noticed that the way I'm issuing validate reporting events is causing them to be mostly created at the completion of the window. That's why the orange and light blue lines appear as steps are very large. I've corrected this.
evoskuil04:51
Notice that the new validate was tracking above (slower) then the previous. That's because most of the cost of confirm has been moved to validate. This of course delays confirmations, but they are much faster as a result, so they catch up, and the result is about the same. What the design change means is that with more resources, confirmation can scale almost as well as validation.
The point table is currently 136GiB, due to consolidation with the parts of the spend table removed from the spend index - to speed up the spender queries. I believe that was the main reason that the entire thing got about 50% slower. However this change has restored the original best performance, which basically tracks signature verifications. Once I split the point table out again it will be reduced to 86GiB, which should further reduce confirmation paging and therefore query cost. Hopefully that will help with the mini.
evoskuil05:04
Notice in the image the the purple (baseline confirmation) progress segments (non-vertical portions) are MUCH steeper than the green (new confirmation). This shows how much faster is the totally-ordered portion of the confirmation. You can see that its slope is slightly shallower than archive, even toward the end. Archive doesn't perform any query, so is largely independent of table size, but confirmation is driven by query, so this is great.
💥 1
Also, this run was with normal validation/confirmation thread priority, while I was working on the same machine. The baseline was performed at high priority, which does make somewhat of a difference.
evoskuil06:39
Figured out what the fault was. I overflowed the table link domain when I added the sequence value to the table. Just need to make the row size equal to the two values as a pair, and it will restore the link to 61% utilized vs. 122%. Otherwise it's a header increase of 850,000 bytes total by changing to a 5 byte link.
(edited)evoskuil18:51
It's amazing what the right small change can do when you're working on the order of billions of events. I reduced the frequency of
(edited)get_context() calls during final confirmation (to get prevout block context for the purpose of evaluating relative locktime - which can be avoided if the relative locktime disabled bit is set in the sequence), which after 1 hour has yielded at 12% performance improvement.evoskuil21:47
Running 15 minutes ahead of best full validation (4.9 hrs), with 60% complete.
(edited)evoskuil22:30
18.5 minutes ahead now at 75% complete. Looks like we will hit a new full validation record of 4.5 hours, without even shani.
🚀 3
evoskuil22:42
For reference, this machine takes Core 24.5 hours, with no signature validation or txindex.
1 reply · pmienk · last Feb 17, 2025
evoskuil23:56
Soon, I don't think this will put a big enough dent in the mini run. I'm splitting out the points and inputs association to two tables, similar to where I got the original record run. This run was with those still combined. I don't think it can get too much faster on the maxi, just due to hashing and signatures, but on the mini that should help, because it will reduce paging.
(edited)If this machine had shani and processors and bus as fast as the mini it would be 3 hours or less.
(edited)Tuesday, February 18, 2025
evoskuil00:04
I think I'll also look at different maxi memory configurations to see if there's an inflection point.
pmienk00:30
I can't remember your hardware. No processor upgrade to shani that fits the socket/board?
I think my machine was the 1151, and last I checked I didn't see a supported sha chip
Dunno about the xeon stuff
evoskuil00:36
If I was going to do it I'd do the whole box. It's dual 16 dual core CPUs (64), bus is PCIe3 I think.
evoskuil02:59
Just completed the split of the point table into
point + ins. Later I'll rename the puts table, which is not just output associations, to outs and we'll have three association tables with consistent naming (also txs).
The change used a new tactic, where I keep the tables 1:1 and keep the indexes in sync. I had to add a new allocation method from the map to memory manager to memory map, but it was simple due to similar code. This allows the tables to be independent from a threading standpoint, as required by the whole store, but to also ensure that entries with the same array offset are paired across the tables. This saves a lot of space and preserves the value of splitting the tables - by not linking them, which would require a read of both.Running a test now. I'd expect this to be slower when the store is small, but to pay big dividends in paging when the stored number of inputs gets into the billions.
The theory is that the combined table is not required for the expansive confirmation query, only the point table (just point hashes) and the spend index (7 byte "short" points) are required. So this removes 16 bytes per row, from a table with 2.6 billion rows. The table gets hit hard during confirmation, and this cuts it from around 126GiB to around 84GiB.
evoskuil04:14
Ok, running that mini test now.
Ran latest on the maxi and it was doing well after about 1.5 hrs, tracing a couple minutes behind the previous record run.
evoskuil04:42
The likely reason for the slight loss is that I added a final confirmation height index write for each block.
evoskuil04:53
The last few minutes of validate/confirm was disrupted by some diagnostics I was running on the same machine.
The graph is significantly smoothed out be (1) parallelizing most of the confirmation work into the "validate" phase, which makes "confirm" fast enough to hug tightly to validate, (2) Decoupling confirm from archive/validate, so that validate controls the download window and confirm completes when able. This allows confirm to fill in CPU gaps during download and validate. Formerly it fell far behind because it was slow, creating data locality issues, and (3) validate completion is now performed without returning to the validator strand, so completions are logged in a more timely matter. This is mostly a reporting change, but there is also perf savings in not returning to the strand.
As you can plainly see, per block performance does not slow due to the accumulation of data in the store, only because of the increase in average block size. This is really better than I had even hoped for, as I never anticipated being able to parallelize most of confirmation.
(edited)evoskuil05:06
I'll let the mini run to completion and will plot out the result. It's set to a 5k (vs. 50k) window for validation, to improve data locality. I'm not sure if time will be materially improved, but I'll be happy if the plot is similar to above - in that it remains linear for a given average block size and steps smoothly. Excess paging would throw it off that line as the store increases in size.
evoskuil05:49
This is the exact same graph overlaid with Bitcoin Core ASSUME VALID times sampled at 50k intervals, from the same machine.
7 replies · eynhaender, evoskuil · last Feb 18, 2025
evoskuil13:31
I’ve already posted similar data. Lots of denial about whether we are actually validating. Lots of claims they will improve. They accepted a bunch of perf pull requests but really achieved nothing.
eynhaender15:43
libbitcoin is for core what starship is for saturn V. My current build doesn´t fire up though.
😂 1
evoskuil15:52
Yeah, the seed nodes are not active. You can set any number of seeds by:
[network]
seed = ip or dns host [:port]
seed = …
evoskuil05:50
If it's not clear, the BC times are non-linear with relation to the store size. In other words, the longer the chain gets, the slower each block gets. And again, this is without even validating.
😱 1
evoskuil14:03
Here's the mini, at 55% completion by tx count. Almost 10 hours and slowing, so tracking about the same as previous runs. Constant max SSD read rate from validation with insufficient RAM to prevent page swapping from being the bottleneck. At this point I don't think that will change, so I'm going to look for a sufficient RAM target for linear mini validation.
🫤 1
7 replies · cguida, evoskuil · last Feb 26, 2025
C
cguida00:32
i see that in your next message you said 15 hours with full validation in BC, will libbitcoin exceed that?
evoskuil01:46
The mini has never beat Core on full validation. Download and validate is faster, at 7.8 hours, but confirmation puts it into thrashing.
C
cguida17:27
Hmm ok cool. Curious what the lower bound is to avoid thrashing! If it's 64GB or 128 that's still a lot better than 256 haha
evoskuil14:09
Still faster than assume valid BC on the maxi 😉 But BC does better with shani on the mini, about 14 hours assume valid and 15 with full validation. On linux that's about 10/11. On the monster mac (768GB, 16 dual cores, no shani) it's also around 24 hours.
👀 1
M
Marnix20:05
Built libbitcoin-node using install-cmake.sh (as per GH actions) -> bn is succesfully built, however when executing bn it gives this error?:
./bn: error while loading shared libraries: libbitcoin-database.so: cannot open shared object file: No such file or directory
This does not happen when using autotools, install.sh
13 replies · pmienk, Marnix · last Feb 18, 2025
M
Marnix20:08
./install-cmake.sh --build-dir=/home/user/libbitcoin-node/ --prefix=/home/user/libbitcoin-node/prefix --disable-static --enable-ndebug -Denable-ndebug=yes --build-boost --build-secp256k1
pmienk20:10
- do you have some other installation?
- is libbitcoin-database.so located within the prefix directory?
M
Marnix20:15
specifically what are /home/user/libbitcoin-node/prefix/lib contents
~/libbitcoin-node/prefix/lib$ ls
cmake libboost_program_options.so
libbitcoin-database.so libboost_program_options.so.1.78.0
libbitcoin-network.so libboost_regex.so
libbitcoin-node.so libboost_regex.so.1.78.0
libbitcoin-system.so libboost_system.so
libboost_chrono.so libboost_system.so.1.78.0
libboost_chrono.so.1.78.0 libboost_test_exec_monitor.a
libboost_container.so libboost_thread.so
libboost_container.so.1.78.0 libboost_thread.so.1.78.0
libboost_iostreams.so libboost_unit_test_framework.so
libboost_iostreams.so.1.78.0 libboost_unit_test_framework.so.1.78.0
libboost_json.so libsecp256k1.la
libboost_json.so.1.78.0 libsecp256k1.so
libboost_locale.so libsecp256k1.so.2
libboost_locale.so.1.78.0 libsecp256k1.so.2.2.1
libboost_prg_exec_monitor.so pkgconfig
libboost_prg_exec_monitor.so.1.78.0
pmienk20:17
off the top of my head it looks like the binary somehow lacks reference to the prefix directory. off the top of my head I can't remember how to inspect the binary for this. however, if this were the case, the CI system should be failing which it isn't.
pmienk20:18
however, shared library builds are not supported after a certain repository (can't remember, might be database) and aren't supported on windows at all.
Thursday, February 20, 2025
Saturday, February 22, 2025
evoskuil03:36
Was on the road for a few days. Back now, running some more performance tests. It looks like the movement of most of the confirmation query to the validation (partial order) phase significantly increased paging in that phase, resulting in an increase in full validation cost on the mini. In the same build it significantly increases performance on the maxi, where paging is not a material issue. The record run on the maxi (non-shani) is 4.5 hours, but since this is now running most of confirmation in parallel, it can scale much better. More/faster threads/network will proportionally reduce sync time.
Tuesday, February 25, 2025
evoskuil07:28
Today I changed the store hashmaps to use a power of 2 bucket configuration, which allowed for a simpler configuration and swap of modulus (%) for bitwise and (&) in the bucket selector. There are tens of billions of these calls and this reduces the latency of each from 3 or 4 to 1.
Tomorrow I'm going to look at memory fence replacements of the high frequency mutex locks in the store to see if that makes any difference. Simple change, worth checking.
🚀 1
darkturo13:13
Hello :-)
I don't mean to disturb, I was just looking around, I was just curious about libbitcoin and how to use it for some of my projects, and ended up here by accident perhaps.
Yeah, just thought in saying something in case you see me here lurking.
👋 1
1 reply · neillm · last Feb 25, 2025
neillm15:08
Greetings! I recommend looking at the various repo wikis, depending on your interest. For example: https://github.com/libbitcoin/libbitcoin-explorer/wiki
🙏 1
Stack overflow is dead.
👀 1
Wednesday, February 26, 2025
We are so cooked.
🤕 1
Absolutely shocking. Very limited scope, but JFC... we're gonna need a bigger boat.
This was a genuine interaction, asking it things I wanted to understand and using real code from our database primitive hashmap header.
🔥 2
evoskuil04:12
With a lot of great help from Grok3 I completed the lock-free implementation (sans test changes) of the hashmap (and arraymap) heads. Test is running now on maxi.
(edited)evoskuil04:23
So far it's running very slightly behind the previous run. However that may be because blocks are smaller at this point and therefore contention is lower. Also during full validation, validation is going to totally dominate. At the same time the head files are slightly larger as a result of the change, which is an extra cost. So I'm switching it now to a milestone run.
evoskuil04:28
The technique requires that the table head buckets are aligned, on 8 or 4 byte boundaries (rounded up link sizes). This is where the extra size comes from. I templatized the tables so that word alignment could be optionally enabled, and implemented the padding based on that. It doesn't affect link size (i.e. in the table bodies), just pads the head. The head cells must be atomically read/written, and ordered between write and read. This was previously achieved with a read-writer lock pattern using
std::shared_mutex. This is still in place in the case where the template is not parameterized for alignment. This is done on the big hashmaps with low perf requirement (like address). The others are aligned and therefore use the memory fencing instead.The resulting bucket read method looks like this:
TEMPLATE
inline Link CLASS::top(const Link& index) const NOEXCEPT
{
using namespace system;
const auto raw = file_.get_raw(link_to_position(index));
if (is_null(raw))
return {};
if constexpr (Align)
{
std::atomic_ref<integer> head(unsafe_byte_cast<integer>(raw));
return head.load(std::memory_order_acquire);
}
else
{
const auto& head = to_array<size_>(raw);
mutex_.lock_shared();
const auto top = head;
mutex_.unlock_shared();
return top;
}
}And the write like this:
TEMPLATE
inline bool CLASS::push(const Link& current, bytes& next,
const Link& index) NOEXCEPT
{
using namespace system;
const auto raw = file_.get_raw(link_to_position(index));
if (is_null(raw))
return false;
if constexpr (Align)
{
std::atomic_ref<integer> head(unsafe_byte_cast<integer>(raw));
next = Link(head.exchange(current, std::memory_order_acq_rel));
}
else
{
auto& head = to_array<size_>(raw);
mutex_.lock();
next = head;
head = current;
mutex_.unlock();
}
return true;
}As you can see we can take an atomic reference, which is awesome. The memory must be aligned, and the atomic treats the referenced memory as an atomic variable.
atomic.exchange() is an optimal swap. But it's std::memory_order_acquire and std::memory_order_acq_rel that are doing the heavy lifting. These result in zero CPU instructions. Instead they just place an ordering constraint on the compiler. And since most CPUs (all we support) implement atomicity for aligned word read/write, we get the atomicity for free also (apart from the head padding). So basically, totally lock and delay free (i.e. no spinlocks, etc) magic.🔥 1
I had planned on this for some time, and decided to tackle it today. Grok3 really made it easy to go through all of the possibilities. That's how I landed on the tradeoff above, which should be optimal.
evoskuil04:42
That fact that I got through about a third of the chain on the maxi, reading and writing blocks concurrently, validating and confirming, and had no problems, means that it's very likely properly guarded, as the node is so heavily parallel that it's very unforgiving of any mistake.
evoskuil05:37
Looks like a full sync is almost exactly the same as with
std::mutex. This is good to see, as it implies very low lock contention, probably also aided by std::mutex avoiding any lock unless there is a material delay.
@jungly60 seconds faster in a one hours maxi sync! 🙂
evoskuil07:56
39 minutes on the mini milestone run, close to record time. hacked around the lack of Apple support for
std::atomic_ref<T> using std::atomic<T>. All checked in... totally lock-free thread-safe hashmaps!🎉 3
In other news... Grok3 and I have decided that the Linux underperformance (and possible macOS) is a consequence of Linux (and possible macOS) being stupid.
👀 1
This is a sort-of related link: https://lwn.net/Articles/572911/
18 replies · pmienk, evoskuil, parsevalbtc · last Feb 27, 2025
evoskuil00:31
sudo sysctl -w vm.dirty_ratio=80 sudo sysctl -w vm.dirty_background_ratio=40 sudo sysctl -w vm.dirty_expire_centisecs=6000
evoskuil00:32
Something like that. I'm having a long chat with Grok32 about it and related questions.
pmienk00:33
huh, those numbers are much larger than my defaults of 20 and 10. My reading was that moving down the ratio or absolute bytes would more frequently but more quickly flush.
evoskuil00:37
More frequent flushing isn't good. The node is constantly writing to disk whereas on Windows on the same hardware writes are infrequent.
pmienk00:39
ahh, well fingers crossed that makes the difference and it's a system configuration issue
evoskuil00:41
I got on this path after watching the performance monitor, and very little resources being used, like nothing. Except it was constantly writing to disk.
evoskuil00:45
From Grok:
Thanks for sharing the full map::remap_(...) implementation—it's a beautifully ugly cross-platform beast
evoskuil01:41
Here's an interesting related statement:
lower caps (Linux/macOS) flush earlier, slowing effective write rate (19MB/s vs. ~333MB/s)
evoskuil01:44
That's a 17.5x advantage for Windows on the same box over Linux. And the difference in times I see is very roughly about 37 minutes to 10 hours (16x).
evoskuil01:51
You're absolutely right to catch that nuance—my phrasing in "Why This Matters" might've implied that remaps directly slow the effective write rate, which isn't precise. Let me clarify: the remaps themselves aren't slowing the write rate directly, and they're not tied to the dirty page cap in terms of their frequency (they're driven by your 5%/15% growth logic). However, the interplay between remaps, the lower dirty page caps on macOS/Linux (6-12GiB/6.4GiB vs. Windows' 32GB/256GB), and synchronous I/O operations in your map::remap_(...) creates a bottleneck that throttles the effective write rate to 19MB/s (10 hours) versus Windows' 333MB/s (37 minutes). Let's disentangle this and explain how caps and remaps interact to cause your 17.5x slowdown, ensuring we align with your setup (690GiB, 32 files, 16 threads).
P
parsevalbtc10:06
Awesome, in the linked chat there is also a mention to journaling in ext4 filesystem that can have a role in the slow down.
Thursday, February 27, 2025
evoskuil01:53
Grok3:
You're absolutely right to catch that nuance—my phrasing in "Why This Matters" might've implied that remaps directly slow the effective write rate, which isn't precise. Let me clarify: the remaps themselves aren't slowing the write rate directly, and they're not tied to the dirty page cap in terms of their frequency (they're driven by your 5%/15% growth logic). However, the interplay between remaps, the lower dirty page caps on macOS/Linux (6-12GiB/6.4GiB vs. Windows' 32GB/256GB), and synchronous I/O operations in your map::remap_(...) creates a bottleneck that throttles the effective write rate to 19MB/s (10 hours) versus Windows' 333MB/s (37 minutes). Let's disentangle this and explain how caps and remaps interact to cause your 17.5x slowdown, ensuring we align with your setup (690GiB, 32 files, 16 threads).
evoskuil01:59
You're welcome—glad we're honing in on this! Your addition about threads blocking during remap is a critical insight, and it amplifies the impact of our earlier findings. The fact that a single table's remap not only blocks writes (and reads) to itself but also suspends all threads—due to the relational nature of your data where each logical write spans smaller writes across all 32 tables—explains why the 17.5x slowdown (10 hours on Linux/macOS vs. ~37 minutes on Windows) is so severe. This relational dependency turns a single remap into a system-wide bottleneck, especially with 16 threads and frequent remaps driven by your 5%/15% growth logic. Let's weave this into our analysis of your map::remap_(...), connect it to dirty page caps and syncs, and pinpoint why Linux/macOS lag so dramatically while Windows cruises.
8 replies · pmienk, evoskuil, jungly · last Feb 27, 2025
evoskuil02:53
Well, back story is that I referred to the function that I posted as ugly. So it was trying to make me feel good.
🤣 1
jungly05:03
I have been using cursor for two months now . If you know what you are doing and ask it precise small things to do, your productivity goes up dramatically.
The chat conversations you had with grok, you can have in the IDE and it generates patches for you to review and apply.
I was very skeptical of LLMs for coding in the beginning. But cursor totally converted me.
Also, cursor let's you choose one of the many models out there, so it's model agnostic.
👀 1👍🏻 1
evoskuil20:07
I think it's thinking through every table reallocation, including remap lock contention.
Huddle with pmienk, evoskuil, Marnix, cguida, noah.3 · 115 min20:54
