Bitcoin full node and query server
July 2026
Archive of the libbitcoin Slack #general channel.
Wednesday, July 1, 2026
Vano Chkheidze00:05
Eric, we looked at the horizontal-table -> vertical-chunk rewrite question.
Short version: I don’t think we should pipeline the rewrite through the GPU as a standalone step. The direct C++ engine surface is already column-native, so no engine/API change is needed. If libbitcoin keeps horizontal mmap storage, the rotation is a storage-layer staging cost, not an engine integration issue.
Local microbench numbers on this machine:
- ECDSA CPU full AoS->SoA rewrite: ~22.9 GB/s, about 39s projected over 3.5B tuples.
- Best CPU chunk size was ~1024 rows: ~29.1 GB/s, about 31s projected over 3.5B tuples.
- Schnorr was similar: ~30s projected at best chunk size.
- Multithreading did not help; throughput stayed around ~20-23 GB/s, so one core is basically saturating the write path.
- GPU staging on RTX 5060 Ti was PCIe-bound: pinned H2D ~19.9 GB/s, and H2D + deinterleave kernel projected around ~40s.
So my recommendation is:
1. Keep the direct column API as the engine boundary.
2. If storage stays horizontal, do CPU chunk rotation, preferably fused with the existing read/staging path and overlapped with verification.
3. Don’t offload only the row->column rewrite to GPU; the PCIe transfer cost removes the benefit.
4. Treat vertical mmap aggregation as an end-to-end libbitcoin storage experiment. It may still win if the mmap aggregation change eliminates the contention you’re seeing, but the engine does not need a different interface for that.
No C ABI, shim, or bridge is needed for this path.
evoskuil01:19
The times you are showing above would eliminate the need for the additional table complexity. However I'm not sure that we will see the same type of behavior when at memory limits.
evoskuil03:24
Major database
mmap refactor incoming. Our mmap wrapper (map) is now a variadic template (more compile time yay), factored from one into four source files...
github.com/libbitcoin/libbitcoin-database/pull/847/changes
The rather large interface is unchanged, not test impact. There is just a template parameter for table widths, with the default being one column of width 1 (map<1>). This reduces to the identical compiled code as the previous map class. But now we can define an map as map<4, 32, 33, 64> (our ecdsa signature table). Under the covers there are four mmaps of specified columns and an aligned number of rows. The single map has all of the same logical and physical size values and methods, aggregating the tables.
This is similar to what I did previously, but that maintained the full map object for each table. So that meant all guards applied to each table independently. All allocator locks and remaps were independent. This reduces that to the one set of controls just spread over the independent mmap files. So the type changes trivially for a SoA table, the constructor accepts the corresponding number of file system paths, and divides up the numeric config across the columns.
And there is an accessor that allows the stream read/write in logical tables to obtain the offset pointer the specific columns. So that is buried in the existing logical table wrapper object for the vertical tables, so that the columns are sub-named. If this doesn't bring the perf back into line with horizontal rows, I'll be sad, because the impl is pretty slick - and there's really nothing else to optimize.evoskuil05:02
This implementation is actually faster for the base (non-batching) milestone run, presumably because I had to move the map into a template, which gives it all internal linkage. maxi
(edited)milestone run to 950,000 in 39.0 mins.🎉 1
josie11:30
nice work regarding the vert tables! was also starting to suspect it was increasing io. i went down a different route with my benching and tried to understand the ultrafast internals a bit better. i came up with a few patches ive been running that have shown measurable improvements. namely, i want to move anything that can be considered sig verification to the GPU, i.e., no CPU marshalling batches and instead columns are streamed directly into reusable GPU buffers, verified on device, and copied back. i dont think we can ever truly get zero copy since we have to cross a device memory boundary. but if that is possible and the goal, i think this at least is a good step toward it anyways. @Vano Chkheidze you can see the patches i am using here: github.com/josibake/…/patches
9 replies · josie, evoskuil, Vano Chkheidze · last Jul 01, 2026
josie15:18
might have a bug in one of my patches , but it raises another consideration: i think there should be a clear error code returned from ultrafast that indicates "I could not complete the operation / internal error," separate from consensus invalid. right now, if there is an issue inside the engine, it is treated as a consensus failure , which is the wrong treatment imo. I'm adding a patch now for that and running again, but wanted to mention here as a consideration
Vano Chkheidze18:04
we are moving to direct integration without bridge or shim i also build direct integration part so i will take useful part from your patches and will integrate there
josie18:11
What would we do with an error from a math function?
if a function with preconditions satisfied cannot satisfy its postconditions, it should throw an exception. since we can't throw an exception across a library boundary, it should translate to an error code. libbitcoin can then decide what to do, which in this case id argue execute a clean shutdown. it should not treat it as a consensus failure , which is not an error but a conclusion. in the case here, i was trying to get dynamic chunking working, ultrafast failed to acquire the memory it needed on the gpu and that was interpreted as the batch being consensus invalid by libbitcoin due to an array with all zeros
(edited)Vano Chkheidze18:17
Agreed. We should keep consensus verdicts and engine/operational failures strictly separate.
If verification completes, then a row can be valid or consensus-invalid. But if the engine cannot complete the operation due to GPU allocation, launch/copy failure, unsupported backend, cancellation, or an internal invariant failure, that must be reported as an operational error, not as “all signatures invalid”.
For the direct libbitcoin integration we are moving toward a C++ status/result contract for the GPU/batch paths, so libbitcoin can decide whether to retry on CPU, abort cleanly, or shut down. The direct path will not go through the C ABI/bridge, but the same distinction will exist internally through GpuError/status propagation.
evoskuil18:32
The design objective was of course not to pass GPU failures as invalids. And we could of course abort if the math cannot be completed. There is really no rational execution path at that point. If we cannot do the math, the process needs to terminate, just as I would expect it to do if we cannot do simple addition.
The secp library isn't a GPU library, it's a math library. The GPU is one of the tools that it has available. If no GPU can be used, it should fall back to CPU verification. Because if it doesn't that is specifically what the caller is going to have to do. When we call libsecp256k1 we don't have any failure mode, this should not be any different.
evoskuil18:36
This is also why we don't want the caller to have to chunk up the batch for GPUs, and why we do not want a distinct call for CPU or GPU. We arrange the data in an optimal form for either, and expect the call to always complete. There is not even any memory allocation required to execute the call (we don't allocate any when we invoke on the secp loop), so not even OOM should be a possible failure condition.
Vano Chkheidze18:42
Agreed. We’ll align to that contract: libbitcoin should keep one secp-style verification call, with no CPU/GPU split and no caller-managed chunking or GPU recovery path. GPU will be treated as an internal engine accelerator. If it is unavailable or unsupported, the engine falls back to CPU. If the math cannot be completed, that is a fatal engine failure, not consensus invalid data.
👍 1👍🏻 1
josie11:30
im happy to open these as PRs but figured it would be best if you took a look first to see if these are in the right direction
evoskuil16:42
nice work regarding the vert tables
I still have to put this into the table object, presently its just a variadic mmap that's being used for single mmaps. Should have that done today.
evoskuil16:54
I completed a full validation run against the variadic mmap and at
5.09 hrs its fully consistent with the previous implementation, so I'll update the vertical tables and see how that goes.
Also I integrated signature table pruning into store.prune(), so now when the store becomes current and coalesced it drops the ecdsa and schnorr table set down to their logical size (which is zero at that point), which reclaims disk space back to the configured default initial size (and zero when stopped, as before). So if you configure with default (1 byte), they grow to GB size and then shrink to 1 byte in reservation as well as on disk. Previously they would retain the largest reservation until shutdown (the expected behavior for append-only tables).2026-07-01T16:54:55Z.1 Capture ecdsa.... (40858454 / 41171928) = 99.2386% 2026-07-01T16:54:55Z.1 Capture multisig. (3659847 / 3762952) = 97.2600% 2026-07-01T16:54:55Z.1 Capture schnorr.. (3459475 / 3506957) = 98.6461% 2026-07-01T16:54:55Z.1 Capture threshold (258785 / 258785) = 100.0000%
Cumulative signature capture rates from block 950,001 to block 956,240 (current).
(edited)Interesting that
(edited)op_checkmultisig alone is producing as many individual signatures as all of schnorr, and ecdsa is almost 12x schnorr. This is just the raw sigs encountered, the pairings that must be executing are even higher due to op_checkmultisig combinatorics.Vano Chkheidze17:50
with gpu full zero copy will not be possible because data will be transfered to gpu buffer but we will avoid redundant copy and also calculations if there will be any other operations that can be moved to gpu side we can move them too because as in past fore example on gpu y decompression is free instead of cpu
👍 1
Thursday, July 2, 2026
neillm03:01
Build issues with ultrafast. Any suggestions?
LOG
~/libbitcoin/master/source ~/libbitcoin ********************** Configuration ********************** BUILD_boost : yes BUILD_secp256k1 : BUILD_ultrafast : yes BUILD_SRC_DIR : /home/neillm/libbitcoin/master/source BUILD_OBJ_DIR : /home/neillm/libbitcoin/master/source/obj BUILD_OBJ_DIR_RELATIVE : BUILD_CONFIG : release BUILD_LINK : static BUILD_FULL_REPOSITORIES : BUILD_POST_INSTALL_CLEAN : BUILD_USE_LOCAL_SRC : yes BUILD_SKIP_TESTS : yes PARALLEL : 96 PREFIX : /home/neillm/libbitcoin/master/install DISPLAY_VERBOSE : CONFIGURE_OPTIONS : --with-ultrafast --without-tests --prefix=/home/neillm/libbitcoin/master/install --enable-ndebug ********************** Toolchain Configuration Parameters ********************** CC : gcc CFLAGS : -O3 CXX : g++ CXXFLAGS : -O3 LD_RUN_PATH : /home/neillm/libbitcoin/master/install/lib LD_LIBRARY_PATH : :/usr/local/cuda-13.3/lib64:/usr/local/cuda-13.3/lib64:/home/neillm/libbitcoin/master/install/lib PKG_CONFIG_PATH : /home/neillm/libbitcoin/master/install/lib/pkgconfig:/home/neillm/libbitcoin/master/install/lib/pkgconfig LDLIBS : BOOST_ROOT : /home/neillm/libbitcoin/master/install CMAKE_PREFIX_PATH : CMAKE_INSTALL_PREFIX : /home/neillm/libbitcoin/master/install CMAKE_INCLUDE_PATH : CMAKE_LIBRARY_PATH : ********************** Preparing to acquire boost ********************** [0;93mReusing existing 'boost' directory...[0m ********************** Preparing to build boost **********************full file
18 replies · pmienk, neillm, evoskuil · last Jul 06, 2026
pmienk03:24
The last time I tested the installation routine, shrec/UltrafastSecp256k1/main was 4.1.0.7 and our codebase wouldn't accept compilation without bitcoin-core/secp256k1/0.7.0.
There may be drift in parameterization or change in behavior since then. As your log ends during Ultrafast compilation, I'd guess cuda libraries are missing during linking. it would be a behavior change since last I looked where there was fail-over into silent disabling of cuda. I'd prefer it to be a configuration error if a necessary dependency for configuration isn't found.
pmienk03:25
Currently working through autotools changes to support ultrafast. Will likely loop back soon after, so hopefully in the next week I have a better story for you.
🙏 2
neillm12:49
neillm@dev:~/libbitcoin$ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2026 NVIDIA Corporation Built on Tue_Jun_09_02:43:40_PM_PDT_2026 Cuda compilation tools, release 13.3, V13.3.73 Build cuda_13.3.r13.3/compiler.38244171_0 neillm@dev:~/libbitcoin$ nvidia-smi Thu Jul 2 08:49:02 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 610.43.02 KMD Version: 610.43.02 CUDA UMD Version: 13.3 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 Quadro RTX 6000 On | 00000000:21:00.0 Off | Off | | 33% 28C P8 9W / 260W | 1MiB / 24576MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+
neillm17:56
Any updates? This is the actual error that happens:
In file included from /home/neillm/libbitcoin/master/source/libbitcoin-server/builds/gnu/UltrafastSecp256k1/src/opencl/tests/test_opencl.cpp:9:
/home/neillm/libbitcoin/master/source/libbitcoin-server/builds/gnu/UltrafastSecp256k1/src/opencl/include/secp256k1_opencl.hpp:18:10: fatal error: secp256k1/types.hpp: No such file or directory
18 | #include "secp256k1/types.hpp"
| ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [src/opencl/CMakeFiles/opencl_test.dir/build.make:79: src/opencl/CMakeFiles/opencl_test.dir/tests/test_opencl.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1010: src/opencl/CMakeFiles/opencl_test.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....pmienk17:58
I don't have the hardware to test any build options other than CPU. So no, I've got nothing new.
✅ 1
neillm22:45
Disabling OpenCL in the build options seems to work, but not sure if that's a valid configuration (i.e. cuda on, opencl off)
pmienk22:46
OpenCL needing to be disabled means it tried to build with OpenCL, and couldn't instead of detecting that it wasn't supported by your environment and autodisabling.
pmienk22:47
but if you have Cuda support, then you've got something that should work on your hardware
neillm22:50
Yeah, it finds both cuda and opencl on my machine (because they're both there and working), but it fails on that missing header error above
neillm22:54
Hrm, fun:
bs: /home/neillm/libbitcoin/master/install/include/bitcoin/database/tables/caches/ecdsa.hpp:272: bool libbitcoin::database::table::ecdsa_correlate::allocate::to_data(libbitcoin::database::flipper&) const: Assertion `!sink || sink.get_write_position() == minrow' failed. Aborted ~/libbitcoin/master/install/bin/bs -c bs.cfg.v4.testnet4
Huddle with pmienk, eynhaender, Marnix, evoskuil, Eric Chennells, josie · 185 min15:54
1 reply · evoskuil · last Jul 02, 2026
pmienk22:38
https://github.com/libbitcoin/libbitcoin-network - so nice to not see any of that autotools cruft in the root anymore
❤️ 1
1 reply · neillm · last Jul 03, 2026
neillm03:08
WTF? I love that autotools cruft 🤣 [Actually, I have to see how this builds now ... 😞 ]
🤓 1
Saturday, July 4, 2026
Vano Chkheidze08:27
Main docs
- Canonical bridge-free integration: github.com/shrec/…/blob/…/LIBBITCOIN_INTEGRATION.md#…
- GPU opt-in contract: github.com/shrec/…/blob/…/LIBBITCOIN_INTEGRATION.md#…
- libbitcoin batch/column mapping example: github.com/shrec/…/blob/…/LIBBITCOIN_INTEGRATION.md#…
- Validation checklist: github.com/shrec/…/blob/…/LIBBITCOIN_INTEGRATION.md#…
Integration code/examples
- Direct C++ header entrypoint: github.com/shrec/…/blob/…/libbitcoin.hpp#…
- Direct verify API functions: github.com/shrec/…/blob/…/libbitcoin.hpp#…
- CMake target + GPU opt-in flag: github.com/shrec/…/blob/…/CMakeLists.txt#…
- Direct GPU hook smoke wiring: github.com/shrec/…/blob/…/CMakeLists.txt#…
- Correctness test example: github.com/shrec/…/blob/…/test_direct_verify.cpp#…
- GPU hook smoke test: github.com/shrec/…/blob/…/test_direct_gpu_columns_hook.cpp#…
- Direct benchmark example: github.com/shrec/…/blob/…/bench_direct_batch.cpp#…
Here are the updated bridge-free libbitcoin integration docs and examples. The canonical path is header-only C++ via <ufsecp/libbitcoin.hpp> / ufsecp::lbtc::*, with one CMake target and no shim, no C ABI, and no bridge. GPU column acceleration is opt-in and internal to the engine; the libbitcoin caller API stays unchanged and falls back transparently to CPU.
Monday, July 6, 2026
Vano Chkheidze22:29
We added a bridge-free libbitcoin direct benchmark and report for hash256_var_batch, targeting variable-length Bitcoin HASH256 workloads such as txid/wtxid preimages after libbitcoin performs CPU-side serialization.
The direct surface stays C++ only: no C ABI, no shim, no bridge. The benchmark JSON records whether the production GPU hook is installed and accepted, so CPU fallback results cannot be mistaken for GPU throughput.
Report:
github.com/shrec/…/blob/…/…
👍🏻 1
2 replies · evoskuil, Vano Chkheidze · last Jul 07, 2026
evoskuil02:16
I've been sidetracked with the holiday and working out the optimized vertical table implementation. I'll get to this as soon as I can.
Vano Chkheidze09:06
i build full birdge less integration for you and also i extended gpu functions for you to move as much operations on gpu as possible in batches to free cpu
Tuesday, July 7, 2026
Vano Chkheidze10:57
Hi all, We finished the current libbitcoin direct-integration benchmark/example package for `shrec/UltrafastSecp256k1`. The goal of this integration path is: - Direct C++ integration via `<ufsecp/libbitcoin.hpp>` - No C ABI dependency for libbitcoin integration - No bridge layer - No shim layer - Build-flag controlled inclusion - Only the libbitcoin-relevant public-data entrypoints exposed - CPU fallback kept available, GPU hooks used when enabled/available ## Main links Integration guide: https://github.com/shrec/UltrafastSecp256k1/blob/dev/docs/LIBBITCOIN_INTEGRATION.md Public operations benchmark report: https://github.com/shrec/UltrafastSecp256k1/blob/dev/docs/LIBBITCOIN_PUBLIC_OPS_BENCHMARKS.md Public operations example: https://github.com/shrec/UltrafastSecp256k1/blob/dev/compat/libbitcoin_direct/examples/public_ops_example.cpp Public operations benchmark source: https://github.com/shrec/UltrafastSecp256k1/blob/dev/compat/libbitcoin_direct/bench/bench_public_ops.cpp Hash256 var GPU/direct report: https://github.com/shrec/UltrafastSecp256k1/blob/dev/docs/LIBBITCOIN_HASH256_VAR_GPU_REPORT.md Libbitcoin direct README: https://github.com/shrec/UltrafastSecp256k1/blob/dev/compat/libbitcoin_direct/README.md Current integration commit: https://github.com/shrec/UltrafastSecp256k1/commit/72b300d0561c3a2ccf371565e54630060b633f95 ## Covered libbitcoin-facing entrypoints The public direct C++ header now covers benchmark/example paths for: - `xonly_validate_batch` - `pubkey_validate_batch` - `taproot_commitment_verify_batch` - `tagged_hash_batch` - `tagged_hash_batch` tag-string overload - `tagged_hash_var_batch` - `hash256_batch` - `hash256_var_batch` The example target calls all of these through `<ufsecp/libbitcoin.hpp>`. ## Build flags For the direct libbitcoin package: ```bash cmake -S . -B out/libbitcoin-public-ops-bench -G Ninja \ -DCMAKE_BUILD_TYPE=Release \ -DSECP256K1_BUILD_LIBBITCOIN=ON \ -DSECP256K1_BUILD_LIBBITCOIN_TESTS=ON \ -DSECP256K1_BUILD_LIBBITCOIN_BENCH=ON \ -DSECP256K1_BUILD_LIBBITCOIN_EXAMPLES=ON
Then build:
cmake --build out/libbitcoin-public-ops-bench --target \ example_lbtc_public_ops \ bench_lbtc_public_ops \ bench_lbtc_hash256_var \ bench_lbtc_direct_batch \ test_lbtc_direct_verify \ test_lbtc_direct_operations -j
Run tests:
ctest --test-dir out/libbitcoin-public-ops-bench -R lbtc_direct --output-on-failure
Run example:
out/libbitcoin-public-ops-bench/compat/libbitcoin_direct/example_lbtc_public_ops
Expected example result:
example_lbtc_public_ops: PASS (direct C++, no C ABI/shim/bridge)
Run full public-ops benchmark:
out/libbitcoin-public-ops-bench/compat/libbitcoin_direct/bench_lbtc_public_ops \ 8192 3 80 512 80 512 \ --json out/libbitcoin-public-ops-bench/lbtc_public_ops.json
Run hash256-var benchmark:
out/libbitcoin-public-ops-bench/compat/libbitcoin_direct/bench_lbtc_hash256_var \ 262144 5 512 80 512 \ --json out/libbitcoin-public-ops-bench/lbtc_hash256_var.json
Local baseline results
These numbers are from a local CPU/direct baseline run with GPU disabled, so they should be treated as correctness and direct-call overhead evidence, not as GPU acceleration claims.
bench_lbtc_public_ops, production direct path:
operationthroughputxonly_validate0.15 M rows/spubkey_validate0.29 M rows/scommitment_verify0.04 M rows/stagged_hash9.39 M rows/stagged_hash_tag_overload9.42 M rows/stagged_hash_var4.46 M rows/shash2566.33 M rows/shash256_var4.11 M rows/s
bench_lbtc_hash256_var local baseline:
modethroughputpayload bandwidthns/rowserial reference4.60 M rows/s1296.68 MiB/s217.3direct CPU forced4.67 M rows/s1316.40 MiB/s214.0direct production4.64 M rows/s1307.02 MiB/s215.5
Validation completed locally
The following passed locally:
- libbitcoin direct tests
- public operations example
- public operations benchmark JSON generation
- hash256-var benchmark JSON generation
- source graph quality check
- backend assurance validation
- doc drift check
- fast local gates
Notes
The direct integration is intentionally limited to libbitcoin-relevant public-data operations. Secret-bearing signing paths remain outside this libbitcoin direct public-ops package and continue to follow the project’s CT/security rules.
GPU-enabled benchmark numbers should be generated separately on the target CUDA/OpenCL/Metal hardware with the GPU build flags enabled. The current published local numbers are CPU/direct baselines.
Wednesday, July 8, 2026
Thursday, July 9, 2026
Huddle with pmienk, neillm, evoskuil, Eric Chennells · 162 min15:54
6 replies · neillm, pmienk, evoskuil · last Jul 09, 2026
evoskuil16:41
2026-07-09T07:07:38Z.1 Batch verify rate ecdsa.... (69257 / 0) = 69257 sps 2026-07-09T07:07:38Z.1 Batch verify rate schnorr.. (2669 / 0) = 2669 sps 2026-07-09T07:07:38Z.1 Capture ecdsa.... (2653358659 / 2666696373) = 99.4998% 2026-07-09T07:07:38Z.1 Capture multisig. (840674831 / 844696245) = 99.5239% 2026-07-09T07:07:38Z.1 Capture schnorr.. (289214946 / 300248374) = 96.3252% 2026-07-09T07:07:38Z.1 Capture threshold (1613084 / 1613084) = 100.0000%
evoskuil16:43
Line 3099543: ecdsa_secs.......... 0 208 Line 3111469: ecdsa_secs.......... 0 218 Line 3136271: ecdsa_secs.......... 0 227 Line 3140665: ecdsa_secs.......... 0 231 Line 3143011: ecdsa_secs.......... 0 233 Line 3145955: ecdsa_secs.......... 0 235 Line 3154158: ecdsa_secs.......... 0 243 Line 3156735: ecdsa_secs.......... 0 245 Line 3159492: ecdsa_secs.......... 0 247 Line 3161866: ecdsa_secs.......... 0 249 Line 3163448: ecdsa_secs.......... 0 250 Line 3167300: ecdsa_secs.......... 0 296 Line 3167592: ecdsa_secs.......... 0 297 Line 3170877: ecdsa_secs.......... 0 461 Line 3184876: ecdsa_secs.......... 0 463 Line 3225266: ecdsa_secs.......... 0 467 Line 3231002: ecdsa_secs.......... 0 471 Line 3238837: ecdsa_secs.......... 0 477 Line 3246859: ecdsa_secs.......... 0 483 Line 3255101: ecdsa_secs.......... 0 489 Line 3263837: ecdsa_secs.......... 0 496 Line 3271855: ecdsa_secs.......... 0 502 Line 3280828: ecdsa_secs.......... 0 509 Line 3289028: ecdsa_secs.......... 0 516 Line 3294738: ecdsa_secs.......... 0 521 Line 3296660: ecdsa_secs.......... 0 523 Line 3325749: ecdsa_secs.......... 0 549 Line 3331839: ecdsa_secs.......... 0 555 Line 3340202: ecdsa_secs.......... 0 564 Line 3346980: ecdsa_secs.......... 0 571 Line 3352350: ecdsa_secs.......... 0 575 Line 3355181: ecdsa_secs.......... 0 578 Line 3372246: ecdsa_secs.......... 0 633 Line 3408260: ecdsa_secs.......... 0 940 Line 3421641: ecdsa_secs.......... 0 947 Line 3459612: ecdsa_secs.......... 0 949 Line 3477812: ecdsa_secs.......... 0 969 Line 3481143: ecdsa_secs.......... 0 973 Line 3485790: ecdsa_secs.......... 0 979 Line 3488567: ecdsa_secs.......... 0 982 Line 3522305: ecdsa_secs.......... 0 1037 Line 3578790: ecdsa_secs.......... 0 1206 Line 3617440: ecdsa_secs.......... 0 1458 Line 3637257: ecdsa_secs.......... 0 1463 Line 3662814: ecdsa_secs.......... 0 1464 Line 3665707: ecdsa_secs.......... 0 1468 Line 3668539: ecdsa_secs.......... 0 1473 Line 3671028: ecdsa_secs.......... 0 1477 Line 3674221: ecdsa_secs.......... 0 1483 Line 3677101: ecdsa_secs.......... 0 1487 Line 3679711: ecdsa_secs.......... 0 1492 Line 3683398: ecdsa_secs.......... 0 1498 Line 3692836: ecdsa_secs.......... 0 1519 Line 3694575: ecdsa_secs.......... 0 1522 Line 3696403: ecdsa_secs.......... 0 1525 Line 3702838: ecdsa_secs.......... 0 1536 Line 3706342: ecdsa_secs.......... 0 1544 Line 3706590: ecdsa_secs.......... 0 1545 Line 3708940: ecdsa_secs.......... 0 1549 Line 3710846: ecdsa_secs.......... 0 1552 Line 3712771: ecdsa_secs.......... 0 1555 Line 3714972: ecdsa_secs.......... 0 1559 Line 3716732: ecdsa_secs.......... 0 1562 Line 3718385: ecdsa_secs.......... 0 1564 Line 3723164: ecdsa_secs.......... 0 1575 Line 3723806: ecdsa_secs.......... 0 1577 Line 3724829: ecdsa_secs.......... 0 1577 Line 3725422: ecdsa_secs.......... 0 1579 Line 3820504: ecdsa_secs.......... 0 2120 Line 3861430: ecdsa_secs.......... 0 2727 Line 3902983: ecdsa_secs.......... 0 2728 Line 3986869: ecdsa_secs.......... 0 2828 Line 3996712: ecdsa_secs.......... 0 2859 Line 4005465: ecdsa_secs.......... 0 2860 Line 4008193: ecdsa_secs.......... 0 2861 Line 4008960: ecdsa_secs.......... 0 2863 Line 4080652: ecdsa_secs.......... 0 3085 Line 4210039: ecdsa_secs.......... 0 3479 Line 4226403: ecdsa_secs.......... 0 3481 Line 4256084: ecdsa_secs.......... 0 3482 Line 4274785: ecdsa_secs.......... 0 3528 Line 4277364: ecdsa_secs.......... 0 3534 Line 4281666: ecdsa_secs.......... 0 3545 Line 4281896: ecdsa_secs.......... 0 3545 Line 4284034: ecdsa_secs.......... 0 3551 Line 4287942: ecdsa_secs.......... 0 3559 Line 4293153: ecdsa_secs.......... 0 3574 Line 4293240: ecdsa_secs.......... 0 3574 Line 4295677: ecdsa_secs.......... 0 3579 Line 4297631: ecdsa_secs.......... 0 3584 Line 4300973: ecdsa_secs.......... 0 3592 Line 4305877: ecdsa_secs.......... 0 3604 Line 4307911: ecdsa_secs.......... 0 3608 Line 4311861: ecdsa_secs.......... 0 3621 Line 4311944: ecdsa_secs.......... 0 3622 Line 4320246: ecdsa_secs.......... 0 3642 Line 4321420: ecdsa_secs.......... 0 3645 Line 4322567: ecdsa_secs.......... 0 3647 Line 4323445: ecdsa_secs.......... 0 3649 Line 4324391: ecdsa_secs.......... 0 3651 Line 4380313: ecdsa_secs.......... 0 3827 Line 4380354: ecdsa_secs.......... 0 3828 Line 4422843: ecdsa_secs.......... 0 4405 Line 4443872: ecdsa_secs.......... 0 4436 Line 4621926: ecdsa_secs.......... 0 5367 Line 4697071: ecdsa_secs.......... 0 6726 Line 4697072: schnorr_secs........ 0 6726 Line 4745191: ecdsa_secs.......... 0 6728 Line 4745192: schnorr_secs........ 0 6728 Line 4752320: ecdsa_secs.......... 0 6729 Line 4752321: schnorr_secs........ 0 6729 Line 4895217: ecdsa_secs.......... 0 9750 Line 4895222: schnorr_secs........ 0 9750 Line 4922642: ecdsa_secs.......... 0 9752 Line 4922643: schnorr_secs........ 0 9752 Line 4975868: ecdsa_secs.......... 0 9768 Line 4975869: schnorr_secs........ 0 9768 Line 5154365: ecdsa_secs.......... 0 13481 Line 5154366: schnorr_secs........ 0 13481 Line 5154797: ecdsa_secs.......... 0 13484 Line 5154798: schnorr_secs........ 0 13484 Line 5170531: ecdsa_secs.......... 0 13486 Line 5170532: schnorr_secs........ 0 13486 Line 5195071: ecdsa_secs.......... 0 13576 Line 5195072: schnorr_secs........ 0 13576 Line 5206153: ecdsa_secs.......... 0 13577 Line 5206154: schnorr_secs........ 0 13577 Line 5213469: ecdsa_secs.......... 0 13628 Line 5213470: schnorr_secs........ 0 13629 Line 5279596: ecdsa_secs.......... 0 14874 Line 5279597: schnorr_secs........ 0 14874 Line 5280768: ecdsa_secs.......... 0 14880 Line 5280769: schnorr_secs........ 0 14880 Line 5379668: ecdsa_secs.......... 0 16326 Line 5379669: schnorr_secs........ 0 16326 Line 5392669: ecdsa_secs.......... 0 16329 Line 5392670: schnorr_secs........ 0 16329 Line 5405492: ecdsa_secs.......... 0 16332 Line 5405493: schnorr_secs........ 0 16332 Line 5411567: ecdsa_secs.......... 0 16334 Line 5411568: schnorr_secs........ 0 16334 Line 5416461: ecdsa_secs.......... 0 16335 Line 5416462: schnorr_secs........ 0 16335 Line 5433876: ecdsa_secs.......... 0 16463 Line 5433877: schnorr_secs........ 0 16463 Line 5439143: ecdsa_secs.......... 0 16552 Line 5439144: schnorr_secs........ 0 16552 Line 5441714: ecdsa_secs.......... 0 16686 Line 5441715: schnorr_secs........ 0 16686 Line 5442130: ecdsa_secs.......... 0 16693 Line 5442131: schnorr_secs........ 0 16693 Line 5445356: ecdsa_secs.......... 0 16738 Line 5445357: schnorr_secs........ 0 16738 Line 5448103: ecdsa_secs.......... 0 16798 Line 5448104: schnorr_secs........ 0 16798 Line 5450293: ecdsa_secs.......... 0 16930 Line 5450294: schnorr_secs........ 0 16930 Line 5450365: ecdsa_secs.......... 0 16931 Line 5450366: schnorr_secs........ 0 16931 Line 5453790: ecdsa_secs.......... 0 16985 Line 5453791: schnorr_secs........ 0 16985 Line 5461475: ecdsa_secs.......... 0 17474 Line 5461476: schnorr_secs........ 0 17474 Line 5461545: ecdsa_secs.......... 0 17482 Line 5461546: schnorr_secs........ 0 17482 Line 5465487: ecdsa_secs.......... 0 17756 Line 5465488: schnorr_secs........ 0 17756 Line 5471006: ecdsa_secs.......... 0 17760 Line 5471007: schnorr_secs........ 0 17760 Line 5474932: ecdsa_secs.......... 0 18057 Line 5474933: schnorr_secs........ 0 18057 Line 5481748: ecdsa_secs.......... 0 18066 Line 5481749: schnorr_secs........ 0 18066 Line 5490495: ecdsa_secs.......... 0 18710 Line 5490496: schnorr_secs........ 0 18710 Line 5499164: ecdsa_secs.......... 0 18711 Line 5499165: schnorr_secs........ 0 18711
Sunday, July 12, 2026
evoskuil16:01
Completed a maxi full validation capture in 3:18 last night (full validation but sig batches unverified), thought this is leaking 24% of the signatures into uncaptured validation (due to the avoidance of locking). So that's an extra 43 minutes of signature verification that can be removed. That would bring the capture down to about 2:35. CPU verify on maxi is about 3:00, and full validation is about 5:00 so that means the capture overhead is about 35 minutes (we would expect capture/verify on CPU to be 5:35), which is pretty tight/reasonable for the amount of additional capture work. Verifying these batches on a modern GPU at 5M sps takes 20 mins, so that gets the full validation with GPU down to ~3:00.
(edited)👌🏻 1👌 1
evoskuil16:40
After initially optimizing the full vertical table implementation for minimal locking (1 vs 4 per composite 4 column table), and eliminating all allocations during multisig capture, I found that there was still a significant performance issue. Eventually I was able to determine that this was chaotic behavior not caused by any single inefficiency (like table allocator
(edited)locks or copy and malloc excess) but instead by insufficient control over batch sizes.
The config sets a minimum batch size, and then the node cuts off the capture in the validation message pipeline. However because a full window worth of block validations (e.g. 50k blocks) can pass that gate almost immediately, being queued up for concurrent validate, the result is a pressure wave that eventually cuts out all batch draining until the window closes. This dramatically oversized the batch tables creating a large amount of I/O back pressure.
This was exacerbated by an oversight in the calculation of the window itself when batching, as using the prevalid state (validated and batch accumulated, but sigs not yet verified) and the valid state (post-verify) led to a doubling of the intended window size. After resolving this so that it maintains the window only on the valid state, performance was materially improved. But the major issue of excessive table sizing remained.
Prior to optimizing above, the full validation run bypassing sig verify took about 6:00. After the above changes that was down to around 4:00. Then made the change to increase control over table excesses, but moving the batch cutoff out of the ASIO message pump and into the executing validations. At the same time I removed the reader-writer mutex lock that was set up to guard the batch tables during verify, replaced with an atomic turnstile. This reduced the above total time to 3:18 as it brought batch tables into a very well-controlled range (1.0 - 1.2x of target).
However the turnstile was set up to allow blocks to continue to validate (unbatched, with inline sig verify) when the batch cutoff was signaled. This allowed about 24% of all batchable signatures to bypass capture. That's about 43 minutes of pure sig verify cost. This is far greater than optimal, so I've modified the turnstile to allow blocks to write into the batch cutoff, while still allowing them to validate inline during the batch verify. It may turn out that validating during GPU verify is also suboptimal, as pre and post-processing the GPU may be slowed by it, in which case that can be adjusted as well.Vano Chkheidze16:48
i'm developing more gpu functions for you to speedup more part from cpu to gpu
👍🏻 1
Monday, July 13, 2026
evoskuil00:48
Well it doesn't look like I'm gonna capture that additional time. We are starting to hit up against sequential confirmation cost, where when we remove validation below a certain point confirmation becomes the long pole. Though that's on top of not having shani, and unoptimized store deserialization. But the design and behavior are now solid, and a big win given a decent GPU. At this point it's just tuning and config.
evoskuil00:54
We're already at 1:50 on a high end machine with GPU batching, before these last fixes. They might not be as significant on a machine without I/O pressure. But not a bad starting point, since it's about 5x faster than anything else out there.
🎉 2
evoskuil01:14
So that's over 5 weeks on batching. I'm moving back to the admin interface on server. The embedded (and fallback to file system) html site was already done, and I just completed the initial event and log message http/ws subscription and ws push notification implementation. Weill finish up the testing on that next and @Rowan Voskuil should be able to start plotting.
👍 1
Tuesday, July 14, 2026
evoskuil17:09
I completed the log and event subscribers to the admin website, which run through the same pipeline as the embedded or external pages for the site. So we can now start plotting.
evoskuil17:17
Also due to some suggestions by @josie the maxi batched validation (without batch verify) is now reduced from
3:18 to 3:00. That implies a full validation run with CPU batching should be around 5:15, noting that full unbatched validation run on maxi is 5:00.evoskuil17:29
Admin interface is now pumping messages for
log and event capture. notifications are via ws, subscription via http or ws (same model as native). These are the subscribers, responses are all json. embedded (or file system) site pages are exposed just as with native.
/v1/log/subscribe?filter=1024 /v1/event/subscribe?filter=8
evoskuil19:42
filter is a bitmask of log or event types, where it currently maps each bit to the corresponding enum ordinal position. A filter value of zero drops the subscription. The server only holds the mask and one closure for each, where the closure captures message traffic and filters by the mask, sending a single ws message for each match.1 reply · evoskuil · last Jul 15, 2026
evoskuil03:54
Missing filter is treated as zero value (unsubscribe). Filters are also not cumulative spanning calls, each new call replaces the full set.
(edited)Wednesday, July 15, 2026
Performance for the ws log and events looks excellent. Block explorer doesn't get subscription notifications until the node is current. No limit on that for the admin site.
(edited)evoskuil06:02
This maxi milestone run completed to block
(edited)950,000 in 39 mins and was emitting events and log messages over websockets almost the entire time.You can see here that there is almost no lag in the websocket logging via the browser vs. the console.
evoskuil06:34
This is merged, the simple admin console is embedded.
[admin] bind = 127.0.0.1:81 connections = 100
1 reply · neillm · last Jul 15, 2026
neillm14:03
Awesome, looking forward to playing with it! [On travel this week, so hopefully around the weekend]
👍🏻 1
evoskuil14:23
New maxi milestone record of
37.7 minutes to 950,000. Constant emission of logs and events via websockets on the previous run was just under 39 mins, within normal variance.🫠 1
Thursday, July 16, 2026
evoskuil02:18
Just getting started on the new box ("Shelly"). Builds much faster, shani/avx512, all tests pass, store build in 4 secs and closed in 2 secs, seeded in 3 secs, headers synced in 13 secs....
(edited)🎉 2
evoskuil03:09
shelly first milestone sync comes in at 34.5 mins, on the same 2.5 Gbps network as maxi but still managing to shave 3.3 mins off the record. I believe this ties the record set by @josie on his 10 Gbps network. That was with default config, so I'll vary that a bit, but time now for some batching and validation.😮 1👌🏻 1❤️ 1
2 replies · josie, evoskuil · last Jul 16, 2026
josie07:25
nice! i think that does tie the milestone run i did, which tells me its not network bound, or hetzner is charging me 45€ a month for "10gbps" but is actually giving me something closer to 2.5 🤔
evoskuil15:29
I think both are possible. Also I may be getting a little more than 2.5, since this machine is handling my dual NICs a little better.
evoskuil06:56
shelly first signature batch (batched sigs unverified) sync comes in at 1.93 hrs, just a few minutes shy of @josie batch with verify. This was also my default config from maxi (batching only 1m at a time). With full verify on this machine, full validation should come in around 2.0 hrs.🔥 1💪 1
3 replies · josie, evoskuil · last Jul 17, 2026
josie07:31
another awesome datapoint. what i infer from this is the large GPU im using isnt being fully utilised, so you can get more or less the same speedup on a much less powerful machine
evoskuil15:31
Note that above doesn't verify the sigs, so to compare it to yours would assume zero GPU verify time. This machine has a very beefy GPU, I just haven't tried it yet. But it's for sure GPUs aren't being fully utilized as most of the run they will sit idle. The real cost is in the collection, not the verification, which is why I've been focused on this verification free run.
👍 1
evoskuil15:12
Even with updated config, this number doesn't budge, similar to the milestone time. Both are hitting up against table write allocator lock contention.
evoskuil15:36
And for the big news...
shelly just hit 1:40 full validation (with no batching).
That's down from 4.85 on maxi and handily beats @josie's former non-batching record (though I forget the number) and even ekes out a win over his fully-batched 1:50 former record!🔥 1
8 replies · josie, evoskuil, Vano Chkheidze · last Jul 16, 2026
josie15:46
with no batching, does this mean batch tables but CPU executor ? or previous architecture, without batch tables and plain ol secp?
amazing numbers 😍
evoskuil15:55
The previous run was with a 1M batch size, but with verify commented out. There’s not much point in measuring CPU batch and I don't yet have a GPU build.
evoskuil19:49
I just got my GPU machine and will be running some performance tests soon. Nothing else at the moment. @josie might have something.
evoskuil15:42
Note that this run can't be directly compared to the prior batching run, since in this run I bumped the config to better match the target, so I suspect that the batch run will drop proportionally. The main differences were that the batched thread counts were constrained (node at 64 vs. 128, network at 16 vs. 32) and peer count was 100 vs. 150.
👍 1
Huddle with pmienk, Marnix, evoskuil, Rowan Voskuil · 166 min15:54
evoskuil19:01
I assume that by "IBD" the plot is referring to assume valid, and the numbers track pretty closely to what @eynhaender obtained above (at 300Mbps, to 900k vs. ~950k). But IIRC that was ~3.0 hrs for bs on the 1Gbps connection.
evoskuil19:19
Notice that Core is much slower over the faster network at block 900k, due to the additional indexing (which is always on for us).
Friday, July 17, 2026
evoskuil16:32
At
(edited)44.8 mins Shelly crushed the milestone with address index (Electrum) record. IIRC it was previously ~90 mins on maxi. Of course this also includes full tx and spender indexes, which are always available in libbitcoin.🎉 1
1 reply · evoskuil · last Jul 17, 2026
Saturday, July 18, 2026
evoskuil16:43
Shelly just crushed another 950k block milestone run, at
27.91 mins. The only change was a reduction in the network thread count from 16 to 12.evoskuil16:56
Counterintuitively, the much faster machine with twice the thread count requires fewer network threads for optimal sync. This is a consequence of reduced write contention, while the threads being individually much faster (including shani) compensates for the narrower pipeline.
(edited)evoskuil17:14
Node threads (validation) experience almost no write contention, as blocks are validated on independent threads and there is only one row per block written upon completion. Block magnitude I/O (10^6) is essentially free, while tx/put magnitude (10^9) gets expensive.
This is an important observation, because we always write blocks one tx at a time. Within the tx allocations are singular for each table, so if there are 1000 output rows, that's one row allocation. But there is no good reason to not do this at block level. That would reduce write contention by 3 orders of magnitude, as there are thousands (10^3) of txs in a block. This will then allow the thread count to go up, possibly to the physical limit. And I think at this point we become network limited in milestone potentially up to 10Gbps.
evoskuil17:26
When validating we are reading at high concurrency, but this imposes no contention, and the downloader quickly becomes cut off by the validation window (the phases are designed to alternate). But adding capture to validation introduces write concurrency, which is a material part of its cost. I measured that difference today by cutting out the write call, and it's 32% of the capture (without verify) total runtime (1.31 vs. 1.93 hrs).
This is the same issue as above. It can be mitigated using thread-static batch tables, which is exactly how our linear memory allocator worked - we just allocate against an array buffer where the array index is an ordinal bound to the thread id. This would just mean that each thread's batch would be either evaluated independently or copied into a single buffer. Not clear which would be optimal. But this would eliminate capture contention.
(edited)At 8 network threads, shelly hit
28.2 mins milestone (essentially the same as 12).Consider what this implies - we can hit a milestone of under
(edited)30 minutes on 8 fast threads and lots of RAM. But given that the run is almost entirely append, the RAM is not as significant as it might seem. The flush to disk just has to keep up with the network, which is very reasonable.evoskuil18:08
At 4 network threads, shelly hit
(edited)30.3 mins milestone, so the bottom is somewhere between 8 and 16. But 30 minutes, on 4 threads!evoskuil18:59
This supposed consensus bug in BIP110 is a case where someone activates the fork after the fork is active, after validating blocks under the previous rules. So the user is changing the rules out from under the store. That's not a consensus bug. The node will sync just fine from genesis and all who do so will end always up on the same chain. Changing rules that affect already-validated blocks is user error and affects only that user, it a way unique to that user's situation. While segwit added a guard for this, other soft forks and hard forks have not done so. x.com/DathonPwn/status/2078514978773192779
As a general with with libbitcoin, if you change any forks you need to resync the store. If you add a rule that is not yet active, then you can get away with the rebuild/reboot over the same store, and we don't prohibit that. there are far too many rules and interactions to try and figure out what to revalidate. Core just hardwires all forks, so from that perspective there is only the next one to consider.
evoskuil19:38
This includes changes to checkpoints (adds are a soft fork, removals are a hard fork), and changes to configured milestone. The presumption being that the previously-known valid block is no longer so.
evoskuil19:55
Breaking changes to table configuration (bucket count) is guarded against, as is a body size that disagrees with its header's recorded body size. But changes to the above fork configuration is not guarded. We would not implement a feature to "reindex" the chain, since it's generally faster to download it. But also as each header is archived with the fork flags that are active for it. These don't change in the past - ever, so retroactively changing consensus rules is just user error. And knowing what is "the past" for a given consensus rule can require evaluation of the chain from genesis. So the rule is, either set the rule before it's active on your chain, or resync your chain.
We don't have a general cache of non-table data, like for initial config values. However I devised a mechanism for this due to the necessity of knowing the configured Electrum full chain Merkle root optimization interval. That's one byte stored in the genesis block's txs association. Since that's blob storage and only set on store creation, it's the ideal location. I'll eventually expand this to retain the fork flags (32 bits), which will allow startup to warn against changes to rule configuration. However this would never go away, since it's not modifiable. So you would perpetually get warned that you modified your fork state and that consensus reliability is based on your expertise if you modify it. And then of course if you de-modify it the warning goes away. Which you would of course be initially warned about 🙂.
evoskuil21:53
Dropped this in just now, so the server can populate the flags on store create (via
(edited)database::settings) and read them out at startup via dedicated query. When they differ it can issue a startup warning. Just 4 bytes added to the end of the first txs (slab hashmap) table blob (genesis tx association), next to the 1 byte for the Merkle root cache depth. github.com/libbitcoin/libbitcoin-database/pull/859
Sunday, July 19, 2026
evoskuil01:00
A unlimited concurrency window (0) with 12 network threads resulted in a milestone time of
(edited)29.6 mins, so it's actually faster to use the window (which is the cause of the stepping above). The above numbers of confirmation, not download. Those are loosely coupled, but because of the window closures, confirmation shares CPU times at the end of each window, and closes out conformation state. Otherwise there is a very long tail at the end of the run where all of the confirmations live, and at that point the data is no longer proximal, so a little more expensive.👌🏻 1👌 1
evoskuil16:25
Shelly
1.7 hrs full validation on 12 network threads, now isolating optimal node (validator) thread count.👀 1
evoskuil20:49
Shelly crushes another record -> full validation in
1.47 hrs with 12 network threads, but increased from 64 to 128 validation threads (matching physical).evoskuil21:02
I think that with a simple change to the store block write query, milestone can be cut in half and full validation can be reduced by the same absolute amount (~15 minutes). This could however end up becoming network bound. But it looks like my dual homed configuration might keep up without a load balancer. If I'm right this would put the milestone record at
(edited)00:15 and validation at 1:15.evoskuil22:04
The estimate of lock reduction for writing a single full block is 70k lock acquisitions across 7 tables down to 7 allocator and 7 remap (14 total). You read that right.
evoskuil22:16
I previously applied this technique on a per-tx basis, but there are up to 5k txs per block, so applying it on a block basis is again a significant reduction.
Monday, July 20, 2026
evoskuil00:16
I've consolidated the
ins and points tables into an aggregate hashmap, same idea as the batch/silent tables but for different reason. This means these settings are no longer a valid:
[database] point_buckets = point_size = point_rate =
And this is new (formerly
point_buckets):
[database] ins_buckets =
Default value for
ins_size changes to 34250000000, which is the combined size defaults for the former two tables, now two columns.evoskuil01:06
These tables are logically one in a relational model, but are implemented independently due to very different access patterns. The huge size of each drags down the other in unnecessary paging costs. The new SoA design for GPU optimization was based on this implementation, but because there were so many columns, it had to be generalized. So now the generalization has been backported to the original concept. And for this it gets a decent reduction in lock/read as well, not huge but probably measurable.
evoskuil02:01
The above is in, first run at 12 threads was unchanged. But 32 threads just hit a new record of
23.5 mins. It's possible thread count can go all the way to 128 given this change, but I'll prob hit bandwidth limits before that.evoskuil02:33
Checking the flame graph with 64 threads now,
write contention has entirely disappeared, where before the above change it was dominant and milestone CPU has dropped to almost nothing, averaging <15%, saying the same thing (contention burns CPU). And we certainly are not writing at bus/RAM speed (no disk I/O at all given the RAM size). So this is a picture of being network limited.evoskuil03:18
I should say that CPU is about 4% for most of the run, and only picks up toward the end due to duplicate detection (bloom filter saturation). I need to resize the hashmap heads to alleviate this pressure.
evoskuil03:27
Raw bandwidth calculations against my dual 2.5Gbps NICs on measured 2.3Gbps delivered Internet show the theoretical limit to transfer the first 950k blocks is
22.5 mins (including 30 secs for header sync, which is purposely not peer-optimized) and I am hitting 23.5 mins. So just 60 secs below the absolute limit, if all we did was transfer the full chain as a file in a single stream.evoskuil04:20
It's hard to overstate how significant was the change above. I'm now truly up against the network, so it doesn't show in the times. But lock contention went from limiting this machine to 25 minutes, to being
unlimited on this machine by any factors other than network, until 100Gbps. And at 100Gbps (>20x my current speed), a milestone run to 950k takes 60 seconds. And at that point it's up against nothing but CPU and RAM speed. Milestone is fully scalable with hardware at this point. The only exceptions are the header sync and the confirmation chase (which can easily be made fully parallel in milestone).🚀 2
It didn't move the needle much (
(edited)-30 secs) in full validation either, for the same reason. When it's downloading it's already up against the limit. But that will now compress all the way to ~zero given sufficient bandwidth.
evoskuil05:54
Shaved off another 30 secs by tuning the hashmap load factors, new
milestone record 23.0 mins.🚀 2
evoskuil08:57
Mine are libsecp. @josie is batching on ultrafast. Just hit a revord of
59 minutes full validation with GPU verify.Vano Chkheidze09:06
i'm doing few more optimizations will release 4.6 soon i made few invariants that can be constants and will not be calculated all time let see how much speedup we will get
🔥 2
evoskuil09:55
And @josie also just hit
14 mins milestone to 950k, owing to his 10 Gbps network. Now I've gotta optimize milestone confirmation, because it looks like that cost him an extra 5 mins which we can parallelize out.Ima need a bigger pipe! I've got the load balancing figured out. I could bump up my service to 10Gbps with two additional 2.5Gbps lines.
(edited)🚚 1
I did like 10 full sync runs today.
evoskuil19:12
The reduction in configured load factor also brought full CPU validation down to a new record of
1:21.
Transitioning to a lotta Internet
RAM draining
Completely network bound
Tuesday, July 21, 2026
evoskuil00:15
In order to get GPU past CPU, the write contention of signature capture needs to be resolved. This is similar to the problem of block archival write contention resolved yesterday, but this occurs during validation. Turns out that this can be resolved in a way that also eliminates nearly all of the capture leakage (24% of all signatures), and without any additional complexity or much disruption. At that point capture cost should become almost free aside from potential paging impact - though that won't affect a big machine at all.
evoskuil06:05
The capture and dump time is now
1.13 hrs where it was formerly 1.95 hrs. Write contention on the capture path is now gone, as is capture leakage. So on a lower CPU machine with a good GPU (like @josie 's) the improvement should be significant.evoskuil09:42
GPU verify now running on shelly, which is first Windows GPU run (cuda). First run full validation complete in 1.06 hrs. CPU was idle most of the run, sig verify cost is zero after capture (8x faster than capture and running concurrent with it).
Wednesday, July 22, 2026
evoskuil03:27
Completed a couple full validation runs using
(edited)Ultrafast as a drop-in secp256k1 replacement (no GPU linkage). The straight inline validation run came it at +1 second vs. the baseline of 1.35 hrs, so these are equivalent. The batched CPU run came in at 1.65 hrs which I assume will also match the a baseline from secp256k1 (running now).
While it would be nice if we got a CPU bump from UF, it's sufficient/excellent that we are seeing parity. This allows it to be set up as a dependency swap, as opposed to an additional dependency. And the GPU functionality can be incorporated into the statically-linked builds for use where GPU is available, with the engine dropping into CPU where not.
However since full validation CPU batching is always slower than inline, this will remain an important configuration knob. The user or outer controller will have to termine whether it's worth turning on batching. If it's turned on and the GPU is slow, unsupported, or missing - UF will drop into CPU verification. But at least (in this worst case) the difference (20%) between batched and inline CPU is not huge.I also completed a couple of modest optimizations, one in the batch post-processing and one to make
set_strong(...) parallel within a block, which should speed up confirmations, especially at the end of a milestone run. Maybe get us close to 10 mins (from 14 presently).
Each spike is
1M signatures being evaluated in a parallel loop over libsecp256k1.It requires about
3,800 of those spikes to validate mainnet.evoskuil04:03
It looks like my projection from CPU inline to batched for UF was incorrect. It appears that its
(edited)1.65 hrs is going to be significantly faster than libsecp256k1 iteration. However, at least on this machine, inline CPU verification is even faster, at 1.35 hrs.👍 2
2 replies · josie, Vano Chkheidze · last Jul 22, 2026
josie19:37
i saw something about pippenger for ECDSA, which I did not think was possible. does it work in your engine?
Vano Chkheidze19:46
We do have Pippenger MSM in the engine, on CPU and CUDA, but it is not used for standard ECDSA verification. Schnorr batches and generic MSM can use Pippenger. Our ECDSA batch path uses Montgomery batch inversion plus per-signature GLV Strauss dual-scalar multiplication, because standard ECDSA still requires an individual
R.x mod n == r check and does not provide the recovery parity needed to reconstruct R unambiguously. So any wording suggesting “Pippenger ECDSA” is misleading unless it refers to generic MSM or a recoverable/explicit-R variant.evoskuil18:32
Its much faster when batching, but unfortunately there isn't a scenario where CPU batching makes sense for a node.
Still better to have that as a fallback though.
Really what we need now is an engine function that will tell us whether GPU batching will be optimal.
(edited)And I think almost any supportable GPU might be net faster, since it is completely offloaded from the CPU, as long as it can complete a batch before the CPU has generated the next one.
Vano Chkheidze18:36
with opencl you can use any gpu but this need much better batch management main thing is what you need to speedup CPU batch generation
evoskuil18:39
batch generation has certain invariants that don't speed up. We need to populate the batch flat and normalized in RAM, and we need to evaluate scripts to get at the sigs. Anything we do to speed the evaluation up will also accrue as a benefit to inline CPU validation. CPU batching will always be an additive cost to CPU batching. So the optimization of the batch (e.g. pippinger) has to overcome that extra cost just break even. That seems very unlikely.
(edited)GPU on the other hand is an unused resource, and therefore is not additive. Its evaluation cost is therefore zero. That's how it can overcome the relatively high cost of generating the batch.
but this need much better batch management
How so?
Vano Chkheidze18:45
yes much smaller but all depends hardware if gpu and cpu sharing same memory in this scenario many things changes because there is no pcie upload and so on but i want tell that with intel and other gpu you can accelerate other things also not only sync
evoskuil21:08
Acceleration requires batching, and the batch capture will likely remain more expensive than not batching. Merkle hashing is already batched per block - vectorized in sse4/avx2/512. Batching over a hash accumulator (i.e. groups of txs in a block and signature hashing) won't be an optimization over shani.
yes much smaller but all depends hardware
These are the things that make the library really useful - sorting out all of these platform-specific intricacies. It should be simple to tell the caller whether a batch is going to go down a CPU path or a GPU path. If we have that call, we can override batch config at startup.
Signature verification is costly, deferrable, and operates over relatively small fixed width data. That makes the accumulation of it over large numbers of blocks feasible. Batching within a single block would be a loss.
Identity (block/tx/witness) hashing is not deferrable, needs to happen on a per block basis. Signature hashing operates over large amounts of conditionally-traversed data, so also cannot be accumulated into a batch.
(edited)Thursday, July 23, 2026
Huddle with evoskuil, Marnix · 35 min16:01
evoskuil17:23
I've generalized electrum service version negotiation so that it can handle version ranges that are beyond those defined in the build, such as a v1.8 where the build is limited to v1.7. Also added tests for the version range enabled in config.
evoskuil17:47
Claude summarizing why POSIX operating systems are fundamentally broken, tracing it back to a consequence of process forking...
"
- fork() is the root cause.Forking a process nominally requires duplicating its entire address space. COW makes that affordable only if the kernelpromisesmemory it doesn't have — strict accounting would make fork of any large process fail outright (a 400GB process would need 400GB of headroom to fork a shell). So POSIX institutionalized overcommit to keep its own primitive viable.
- Overcommit converts allocation failure into asynchronous process termination.The kernel's promises come due at page-touch time, when there's no error path to return — so it kills processes (OOM killer, jetsam). Windows, having never carried fork, does strict commit accounting against RAM + pagefile: allocation fails at a decidable point with an error you can handle, and committed memory you actually use just pages — degradation, not death.
- The contract violation.An OS multiplexes resources: excess CPU demand yields slowness, never termination. Strict-commit RAM behaves the same way — more demand, more paging, slower. Killing a process because the kernel promised what it didn't have is the OS defaulting on its own accounting and executing the creditor. We agreed fork itself is archaic — posix_spawn and threads cover its legitimate uses — but the overcommit regime it justified persists.
- The mmap consequence — the part that bites libbitcoin.Because an overcommitted kernel is perpetually at risk of needing pages it promised elsewhere, it must keep reclaim viable — and dirty file-backed pages can't be reclaimed until cleaned. So POSIX kernels aggressively bound the dirty pool and force writeback early. Linux at least exposes the bound (
vm.dirty_ratioet al., system-wide); macOS's UBC hardcodes it with no interface at all. Net result: on a 796GB machine with the RAM bought and paid for, macOS permits roughly 20GB of accumulated dirty mmap before forced writeback throttles the writers — while Windows lets the accumulate-then-flush design work exactly as intended, by default. And the whole regime is futile anyway: the overcommit it protects still breaks eventually; it just chooses when to break someone else's process.
"
(edited)4 replies · neillm, evoskuil · last Jul 31, 2026
neillm13:00
v3 server just got backstabbed 😞
[4354091.298471] bs invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0 [4354091.298521] oom_kill_process.cold+0xa/0xb7 [4354091.298886] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name [4354091.299165] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/user.slice/user-1000.slice/session-45.scope,task=bs,pid=4257,uid=1000 [4354091.299406] Out of memory: Killed process 4257 (bs) total-vm:1772734212kB, anon-rss:39880944kB, file-rss:8kB, shmem-rss:0kB, UID:1000 pgtables:1801732kB oom_score_adj:0 [4354093.703073] oom_reaper: reaped process 4257 (bs), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
evoskuil17:49
I asked how and when it started, and the lineage is damning...
"About 63 years. The lineage:
- 1962–63— Melvin Conway (of Conway's Law) described the fork–join concept in his paper "A Multiprocessor System Design," as a notation for splitting computation into parallel paths. That's the origin of the idea and the name.
- ~1965–66— Project Genie's Berkeley Timesharing System on the SDS 940 implemented fork as an actual system call creating a process copy — Butler Lampson and colleagues. This is the direct ancestor of the Unix semantics.
- 1969— Thompson carried it into the first Unix on the PDP-7, reportedly in part because copying the process image was one of the easiest ways to implement process creation on that machine: the original fork fit in roughly 27 lines of PDP-7 assembly. No virtual memory, no COW — it literally swapped the process image out and kept both copies.
So the design your 796GB machine is being throttled by was devised for a machine with about 8K words of memory, and chosen substantially because it was easy to implement on 1969 hardware — the address-space-duplication semantics weren't a considered design goal so much as an artifact of swapping. Every subsequent complication — COW, overcommit, the OOM killer, untunable dirty-page bounds — is scaffolding erected to keep that 27-line expedient viable as memory grew six orders of magnitude around it.
It's also worth noting the industry has quietly conceded the point: POSIX itself added
(edited)posix_spawn (1999), Apple's own frameworks discourage fork-without-exec (Objective-C/Swift runtimes are fork-unsafe by design), and fork doesn't exist on Windows NT, nor in any modern OS research design — Fuchsia's Zircon, for instance, deliberately omitted it. The 2019 HotOS paper "A fork() in the road" (Microsoft Research/BU) makes essentially your argument academically: fork is a terrible abstraction whose costs — overcommit among them — are systemic. The technique outlived its usefulness decades ago; the memory-accounting regime it necessitated is the part that refuses to die."So when a process gets killed by the OS for OOM, just make a mental note that your operating system has a catastrophic bug, because doing it right was harder.
(edited)evoskuil18:00
And as a result we will have to work around this crapshow in macOS, and Linux requires OS level paging config, and both of course can still OOM kill.
But what I find interesting is that Linux can be configured to behave honestly via
vm.overcommit_memory = 2 (strict, meaning correct, accounting). So when a JVM reserves a huge heap, and then forks it because muh, never heard of threads, it will get a failure if there is insufficient actual reservable virtual memory to accommodate what has been requested.By setting this the actual offending process will get a handleable failure, so the OS doesn't have to go hunting for arbitrary unrelated non-offending processes to kill. And a process like libbitcoin, that uses file-backed mmap, can never fail to allocate the map unless there is insufficient reservable disk space at the time of the map/remap.
evoskuil18:09
Of all of the stupid things an operating system can do, this is the worst category. The purpose of the OS is to manage shared resources. Returning success on a malloc that cannot be satisfied (overcommit) and then killing (possibly other!) processes when that malloc'd address is actually touched (too late to return an error), is just unforgivable. It would be exactly like killing the process because another process wants to use the network, the CPU, or the keyboard for that matter.
(edited)So if you want an honest Linux, set
vm.overcommit_memory = 2 and watch applications fail because they assume large overcommit, and watch libbitcoin keep going because it does not. All of our mmap is file-backed, and when there is insufficient space, we handle the disk full condition gracefully.If you want an honest macOS, you are out of luck. As one of us has said many times, macOS is not a real OS. And so I'll end up writing a full blown virtual memory system for it like it's 1980. 🙄
(edited)
Remco18:22
I think I found a tiny bug: github.com/libbitcoin/libbitcoin-node/blob/…/full_node.cpp#…
should this be return is_current_chain(true); ?
with 'delay_inbound = true' (the default), even fully synced, none of the endpoints work (logged as 'disabled')
👍 1
evoskuil18:26
Yes, that's regression from recent work in which I renamed
(edited)is_current(...) overloads to eliminate ambiguities, and this one got the wrong name. PR welcome.👍 2
Friday, July 24, 2026
evoskuil15:25
On my
Mac Pro (which now needs a name), milestone to 950,000 completed in 101 mins over my (single) 2.5 (2.3) Gbps Internet. This is the fastest Mac sync ever by a wide margin, based on a prototype mmap replacement that controls write flushing rationally. It's a 5 year old machine with 796GB RAM, 32 threads and without shani. So it should be network limited but is currently about twice that. But now that's just optimization, not the broken OS. This can also be applied to Linux with no changes, which would avoid the need for OS paging config.🚀 1
2 replies · parsevalbtc · last Jul 25, 2026
evoskuil15:33
The prototype is simplified, just to prove the concept. Write flushing was limited to two fixed points, at ~500GB and at completion. Flush rate (RAM to SSD) exceeded 1.9 GB/sec, so this machine will be easily network limited at 2.3 Gbps. That first leg hit
722,933 confirmed in ~30 minutes.evoskuil18:22
The 101 mins reported above was a reporting misinterpretation. Actual time was
(edited)56 mins, but 4 mins was spent in a shutdown/restart to force the flush at 500GB. So with managed dirty flushing the big mac will be network limited, just as maxi. And that also implies that our mac mini should also be network limited, at about 50 mins. So once the details are worked out, macOS will be a first class citizen. And instead of prompting for OS config, we can poorly-configured (default) Linux on the same path.evoskuil20:09
FYI, recent Autotools builds (including with
--build-config=release) have been defaulting to !NDEBUG (debug build), which is a dramatic performance hit. This can be avoided by explicitly setting --enable-ndebug until the default is updated. CMake and MSVC unaffected.
Kevin Ravensberg23:13
Small trial test with GPU verification on my 5080, will do a full IBD overnight and report back.
👍🏻 1
Saturday, July 25, 2026
evoskuil05:51
@josie has shattered the full validation record (of 55 mins IIRC) after some more performance work in the block read and database config.
New
full validation record time is 26 mins, and milestone is 14 mins.
We got an offer today to perform benchmarking on a 100 Gbps network. If we remain network limited that would reduce milestone to less than 2 mins (dunno the specs available for validation).Kevin Ravensberg06:56
No records broken, but it did not freeze up or crash this time!
~3.9h IBD time, storage speed bound by PCIe gen4 drive.
❤️ 1
evoskuil07:02
Thanks for posting. Can you include specs, build command line, and runtime config settings? Otherwise it's hard to say if this is great or problematic.
Kevin Ravensberg07:07
*Hardware*
- AMD Ryzen 9 5950X, 16C/32T
- 64 GB RAM
- Samsung 990 PRO 4TB NVMe (chain on internal NVMe, not USB)
- NVIDIA RTX 5080 16 GB, driver 610.62
- Windows 11 Home 26200
*Build*
libbitcoin
master, 2026-07-21/23: system 8f0c082, database 6cf5992, network 12ff6dc, node 5ae5d93, server 6828fdd.
Toolchain: VS 2026 (18.8.12021.73), toolset *v145*, vs2026 solutions. Dependency: *UltrafastSecp256k1-vc145 4.5.0.0*.
v145 is required, not preferred — the vc143 package references __std_rotate, which exists only in the v145 STL, so it will not link against any v143 toolset.
msbuild libbitcoin-<repo>.sln /p:Configuration=StaticRelease /p:Platform=x64 /p:PreferredToolArchitecture=x64 /p:Option-sha=true /p:Option-ultrafast=true /p:Option-cuda=true /m:2 /p:MultiProcessorCompilation=false
Option-ultrafast defines WITH_ULTRAFAST → HAVE_ULTRAFAST and sets Linkage-ultrafast=static. Option-cuda links secp256k1_gpu_host + secp256k1_cuda + cudart_static and adds /INCLUDE:secp256k1_gpu_columns_provider_anchor. *Both default to false* — build without them and it compiles clean, links no GPU, and silently runs CPU-only. Verify with a 25.6 MB .nv_fatb section in the resulting binary. (/m:2 + MultiProcessorCompilation=false only works around C1060: out of heap space on bs.vcxproj.)
*Runtime config* (non-defaults only)
ini [bitcoin] milestone = 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f:0 checkpoint = 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f:0 [node] batch_signatures = 100000 maximum_concurrency = 4000 delay_inbound = false thread_priority = true memory_priority = true
Both milestone and checkpoint pinned to *genesis*. This is the critical setting: stock
milestone is height 950,000 and bypass = is_under_checkpoint(height) || is_milestone(link), so defaults skip script/signature validation for ~99.3% of the chain. Pinned to genesis, nothing is bypassed.
batch_signatures is the only GPU-related knob in libbitcoin — upstream default is now 0 (inline CPU verify). Non-zero routes through the ufsecp bridge, which binds CUDA via UFSECP_LBTC_AUTO. No device selector, no env var. Process ran under a Job Object at 75% CPU cap.
*Results — 959,516 blocks, all signature-verified*
- 22,450 s elapsed; tip reached at ~3 h, remainder idle at tip
- signature verification: *219 s total* — ecdsa_secs 11 s over 27,835 batches, schnorr_secs 208 s over 9,951 batches
- 27,834 of 27,835 ECDSA batches completed under the log's 1 s resolution; the single 11 s outlier is first-batch CUDA context/module load
- observed per-batch rates 4–8M sigs/sec, consistent with the aggregate
*What limited it — storage, per the run's own disk sampler (33,501 samples)*
Reads climb from 27 MB/s to a 3,309 MB/s 15-minute average, peaking at 5,166 MB/s. Writes stay bursty and modest. That is prevout resolution: random lookups into the archive tables, growing with inputs-per-block and table size.
archived − validated gap: p50 1,337, p90 3,954, max *4,000* — exactly maximum_concurrency. Download was pinned at its ceiling waiting on validation to drain, so this was validation-limited, not bandwidth-limited. Raising the window would not help.evoskuil15:50
Thanks!
The batch is too small and the window is way too small. Go with the default window of 50,000 and a batch of 1,000,000.
Download will always end up waiting on validation even when you are bandwidth limited, as validation always takes longer, so that conclusion is in erroneous. I would expect 4,000 to significantly slow the run.
No reason to change delay inbound, leave that true.
The node priorities are already true by default.
There is no good reason to remove checkpoints, as doing so is a hard fork. But they only take a couple of minutes to doubly-validate, so it's not significant.
The build is missing sse4/avx2 hardware optimization. Not significant given shani, but would still be net positive.
Only question... what's the measured bandwidth?
Kevin Ravensberg22:54
bandwidth: ~62 MB/s sustained ≈ 500 Mbit/s average over the full download.
build settings:
avx2.... platform:1 compiled:1
sse41... platform:1 compiled:1
shani... platform:1 compiled:1
avx512.. platform:0 compiled:0
👍🏻 1
So 1/5th of the available bandwidth for this machine (2.5GB/s network adapter)
33 replies · Kevin Ravensberg, evoskuil · last Jul 28, 2026
evoskuil01:21
About the same as mine. Can you run and report again with the following?
[bitcoin] # only known block to this user is genesis. milestone = 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f:0 [node] # default maximum_concurrency is 50,000, so can just leave out. maximum_concurrency = 50,000 # clean stop point for benchmark comparisons. maximum_height = 950,000 # defaults to 0 (disabled) and this is a good typical size. batch_signatures = 1,000,000 [database] # the batch tables are disabled by default, so set initial capacity. ecdsa_size = 100,000,000 schnorr_size = 10,000,000 prevalid_size = 1,000,000(edited)
evoskuil01:23
Note that in new builds the _size settings will change to:
[table.ecdsa] size = 100,000,000 [table.schnorr] size = 10,000,000 [table.prevalid] size = 1,000,000
Kevin Ravensberg23:05
It's running tonight, I'll report back tomorrow morning with the (improved) results!
❤️ 1
Kevin Ravensberg04:24
Seems to be stuck a few hundred blocks from 950000 now...
With this looping in the log:
2026-07-28T04:22:15Z.2 Accepted peer connection [xxx:34328] on binding [0.0.0.0:8333].
2026-07-28T04:22:15Z.5 Invalid heading magic (0x57cf7924) from [xxx:34328] possibly encrypted connection to clear endpoint.
2026-07-28T04:22:15Z.2 Inbound peer channel stop [xxx:34328] invalid message heading magic
evoskuil04:31
"Accepted peer connection" means that peers are connecting to you, which means the node is "recent" ("current" or at the max block) and has therefore enabled inbound connections.
"Invalid heading magic" means that the peer communication is invalid (the peer message header "magic" value is incorrect) and is therefore being dropped.
Seems to be stuck a few hundred blocks from 950000
I don't think so. It appears to have completed at around 3:20.
The plot's diagnostics are contradictory. If you set a max block of 950,000 then that's where it stopped. That's indicated in all three "frontiers" except "validated". The "target tip" will never be reached.
Check into your diagnostics regarding why validated is incorrect.
(edited)
evoskuil04:43
Thanks!
CPU should be the limiting factor on this machine/network, hitting 100% during most of validation. If it's not then the most likely culprit is I/O, so take a look at your SSD read speed during validation.
(edited)
evoskuil04:47
Schnorr is faster per sig than ecdsa. Certainly if it's falling into CPU validation within the batch verify it will be very slow. But the schnorr batches are always much smaller than the ecdsa batches. They are run at the same time based on the first to cross the target (always ecdsa). Also ultrafast batches up the jobs for the GPU, it shouldn't matter how many are fed to it, unless ultrafast has a problem. It's possible that 1M can be slower for these specs, we are still tuning these numbers. But it would be at least as bad or worse for ecdsa than schnorr. Also the lower schnorr batch sizes may give you a lower sps rate.
Kevin Ravensberg04:49
Yeah, Claude thinks it has something to do with extra 15GB RAM for the preallocated tables
evoskuil04:49
It's not the pre-allocation unless you over-allocate. Since they will expand anyway.
(edited)
Kevin Ravensberg04:54
Two of his points are checkable, and one of them cuts against his own numbers.
My 15 GB claim does not survive contact with the disk. Retracting it — evoskuil is right.
batch_ecdsa_* 152 MB total (identity 7, key 37, message 36, signature 72)
batch_schnorr_* 130 MB total
~283 MB, not sparse (apparent == allocated)batch_ecdsa_signature.data at 74,573,056 bytes ÷ 64 B/sig ≈ 1.16M rows — i.e. the table settled at roughly the 1M batch target, not the 100M I configured. So ecdsa_size = 100000000 did not translate into 100M rows of resident allocation, and the memory-pressure-from-preallocation theory has no support. The RAM pressure is real (mean working set 39.9 GB, free RAM <2 GB for 48.7% of the run) but it comes from the ~890 GB archive store being mapped, not from the batch tables.
Caveat: those are post-purge sizes, so they bound the resting state, not necessarily the peak.
Re-plotting at 4× finer resolution since he flagged the sampling rate:
Correction to my own earlier figures — they were contaminated by the post-completion idle tail:
Sending the 15s plot, but the numbers matter more — my earlier CPU figures were wrong.
I was wrong about CPU, and evoskuil is closer to right
My earlier stats included 5,115 s of post-completion idle — the node hit maximum_height at 15,406 s but the sampler ran to 20,528 s. Restricted to the actual run:
contaminated run window only
bs CPU mean 35.4% 46.9%
median — 54.1%
p90 — 73.7%
max 76.2% 76.2%
at/near 75% cap 6.6% 8.7%
below 15% 37.6% 17.2%p90 is 73.7% against a 75% cap. The top decile is pressed flat against the Job Object ceiling, and the median is 54%. So "CPU should be the limiting factor" is much better supported than my earlier reading suggested — and it cannot reach 100% because the launcher caps it at 75%.
My "raising
CpuPercent would buy you very little" was wrong. It is now the single most obvious thing to test.
What is left after that
stall episodes 51 median 30s p90 135s max 165s total stalled 2,520s = 16.4% of the run free RAM < 2 GB 64.6% of samples
During those stalls reads drop to 266 MB/s while CPU sits under 15% — both quiet, so neither is starving the other. And memory pressure is worse than I reported: nearly two-thirds of the run had under 2 GB free.
Answering his other points
Table pre-allocation — retracted. He is right. On disk the batch tables total ~283 MB, with
(edited)batch_ecdsa_signature.data at ~1.16M rows, i.e. about the 1M batch target rather than the 100M configured. No 15 GB, no support for my memory-pressure theory. Worth noting his own suggested ecdsa_size = 100000000 against batch_signatures = 1000000 is 100× the target, so "size in relation to the batch target" argues for ~1M there.
Schnorr still looks anomalous. His model says schnorr is faster per signature and always the smaller batch, so it should be a rounding error. Measured: schnorr 1,740 s over 1,137 batches versus ecdsa 2 s over 3,111 batches. Smaller batches, faster per-sig algorithm, 870× the total time. If ultrafast were merely slower at 1M it should hurt ecdsa at least as much — it did not, ecdsa actually improved from 11 s to 2 s. That points at the schnorr path specifically rather than at batch size.
...evoskuil04:56
ecdsa_size = 100000000 did not translate into 100M rowsTable sizes are in bytes, not rows.
😅 1
evoskuil05:00
And memory pressure is worse than I reported: nearly two-thirds of the run had under 2 GB free.
That's not worse, that's better 🙂.
💯 1
evoskuil05:01
Measured: schnorr 1,740 s over 1,137 batches versus ecdsa 2 s over 3,111 batches.
How is this measured?
evoskuil05:02
During those stalls reads drop to 266 MB/s while CPU sits under 15%
I think this is referring to downloads, not stalls.
evoskuil05:04
The plots aren't very useful due to the sampling rate. This is what it really looks like.
evoskuil05:23
The above plot is milestone, but at the tail it's full validation (100% CPU). This move in 50k (by default) waves.
(edited)🚀 1
Sunday, July 26, 2026
evoskuil02:05
Also, database config when batching (initial capacity):
[database] ecdsa_size = 100,000,000 schnorr_size = 10,000,000 prevalid_size = 1,000,000
evoskuil06:44
Another full milestone run compared with the previous. The second one adds throttling (slowing writes when memory pressure increases, in order to create drain time). "Validation" isn't full validation, both are milestone.
(edited)
Coming in at
(edited)1hr on the Big Mac is an insane win. This machine took at least 10 hours last time I checked.Yesterday's first run was just a proof of concept. This is the full deal. Still requires some verification under RAM pressure, disk full, and full validation - but it's feature complete. We'll also be testing this against Linux, which gives us an OS configuration-free option.
🚀 2💪 1🔥 1
eynhaender13:18
Here are the first numbers of our own memory management in comparison to what we did before - Dirty Paging on Linux.
9 minutes less to
(edited)950k (72min -> 63min), no tampering with any kernel parameters, no need for sudo.eynhaender13:28
The system profile of this sync. Milestone run on 2.5GBit network with our own memory management (blue line).
And the system profile of the sync managed by the linux kernel (orange line). Same machine/network/settings.
(edited)
1 reply · evoskuil · last Jul 27, 2026
evoskuil15:02
posix memory management replacement is now merged. It's the only option (no option) on macOS, and can be optionally compiled (as shown immediately above) on Linux. The Linux preprocessor flag that currently enables it on Linux is MANAGE_STAGING, which I do not recommend setting on Windows builds 🙂. I'll bet exposing that more cleanly and adding some config to allow tuning of the memory manager's constants.1 reply · evoskuil · last Jul 26, 2026
evoskuil23:55
It's great to finally have this under our control, already tuning it up for even better performance.
🚀 4
Monday, July 27, 2026
evoskuil07:04
database
table config has been moved into individual sections per table:
[table.header] buckets = rate = size = [table.txs] buckets = rate = size = [table.tx] buckets = rate = size = [table.ins] buckets = rate = size = [table.outs] rate = size = [table.input] rate = size = [table.output] rate = size = [table.candidate] rate = size = [table.confirmed] rate = size = [table.strong_tx] rate = size = [table.duplicate] buckets = rate = size = [table.prevout] buckets = rate = size = [table.validated_bk] buckets = rate = size = [table.validated_tx] buckets = rate = size = [table.ecdsa] rate = size = [table.schnorr] rate = size = [table.silent] rate = size = [table.address] buckets = rate = size = [table.filter_tx] buckets = rate = size = [table.filter_bk] buckets = rate = size =
evoskuil21:41
More config changes:
[bitcoind] credential = user:pass
Formerly:
[bitcoind] username = user password = pass
This allows for multiple credentials (WIP on method authorizations).
[bitcoind] credential = user1:pass1 credential = user2:pass2(edited)
Tuesday, July 28, 2026
evoskuil09:05
Service starts up, creates store, syncs chain, provides block explorer and service endpoints, survives reboots while running, no problem.
Wednesday, July 29, 2026
evoskuil21:02
Full validation run on the Big Mac completed in
(edited)6:19, which is proportional to its CPU power. Once the core problem was figured out the resolution was relatively straightforward. Basically macOS runs a linear search across all memory pages every 30 seconds, to identify evictable pages. As size grows the search cost exceeds 30 seconds, and the machine essentially locks itself up in a vain attempt to preserve memory, despite having a huge excess of memory. Easy for us to work around. Hard to take this operating system seriously.🤣 2🫠 3
evoskuil23:39
All three operating systems now Network bound on milestone and CPU bound with validation...
2 replies · parsevalbtc, evoskuil · last Jul 30, 2026
evoskuil23:58
These aren't the fastest runs to date, those are
14 mins (milestone) and 26 mins (GPU validate) from @josie on his 10Gbps network.🚀 1
1 reply · Rowan Voskuil · last Jul 30, 2026
Thursday, July 30, 2026
2 replies · evoskuil · last Jul 30, 2026
evoskuil07:15
Linux/macOS
(edited)daemon and Windows service install/uninstall with automatic start controlled shutdown and are now implemented. Installation and uninstallation are both via a new command line switch (--daemon or -d). There is an optional Boolean parameter, defaults to true (false for uninstall). This works in conjunction with the existing --config (or -c) parameter, which takes the path to the config file. The process is loaded from the existing path to the binary that executed the command, and the config file parameter is resolved to an explicit path. There is also a new --user (or -u) parameter that accepts a username:password for the process.evoskuil08:04
Also completed verification of
user logoff and other process termination suspension. The hard part was Windows, but that's been done for some time. And it turns out that the signal handling already in place satisfies macOS and Linux.
1 reply · evoskuil · last Jul 30, 2026
evoskuil15:51
Revised mini validation estimates. Stork ran to 900k before disk full, extrapolated from there.
Huddle with pmienk, evoskuil, eynhaender, Rowan Voskuil · 191 min15:51
