Bitcoin full node and query server
June 2026
Archive of the libbitcoin Slack #general channel.
Monday, June 1, 2026
josie15:39
hey all, ive got a working prototype for a silent payments scan electrum endpoint. ive got the libbitcoin-system PR ready (https://github.com/libbitcoin/libbitcoin-system/pull/1856), if people want to look. im cleaning up the database PR next, which is where i think ill want the most review, hoping to throw it up today and then the remaining ones tomorrow
👌🏻 1👍🏻 1🎉 1👌 1
for the linux people, im also maintaining a nix module and pkg for libbitcoin: https://github.com/josibake/nix-libbitcoin . one thing im missing is knowing what defaults to recommend for linux. i also like to include a few recipes when making nix modules like this , e.g., like running libbitcoin on a raspi. for the people who have been running on linux and smaller harder: id very much appreciate if you could share the configs and ill update my nix repo defaults and include some overlays
👍🏻 1
12 replies · Slackbot, eynhaender, josie, evoskuil · last Jun 02, 2026
eynhaender16:30
This is the config I use on the PI5. The most crucial part here is the bucket sizes of the store, which reduces memory requirement and therefore thrashing. Eric might want to help us to boil it down to the minimum entries.
[log] # --------------------------------------------------------------------------------------------- Logging Settings path = /home/eynhaender/test/log maximum_size = 10000000 application = true news = true session = true protocol = false quitting = false proxy = false remote = true fault = true verbose = false [database] # ---------------------------------------------------------------------------------------- Database Settings path = /mnt/ssd/bitcoin turbo = true point_buckets = 682988568 # statt 1365977136 → spart 5,09 GB tx_buckets = 234611262 # statt 469222525 → spart 0,87 GB strong_tx_buckets = 234611262 # statt 469222525 → spart 0,87 GB interval_depth = 11 [bitcoin] # ----------------------------------------------------------------------------------------- Bitcoin Settings milestone=000000000000000000010b93c9ea1c29fea277383f0f7d1f26de8b5802e885ff:950000 [node] # -------------------------------------------------------------------------------------------- Node Settings # Pruning snapshot occurs at first coalesced, otherwise manual snapshot only. threads = 1 thread_priority = true memory_priority = true delay_inbound = true allocation_multiple = 1 allowed_deviation = 1.5 sample_period_seconds = 10 currency_window_minutes = 5000 maximum_concurrency = 50000 minimum_fee_rate = 0.00001 maximum_height = 950000 [network] # ----------------------------------------------------------------------------------------- General Network Settings threads = 6 retry_timeout_seconds = 5 # 1s, 2s, 7s f/m/s loopback, 5s for large connect_timeout_seconds = 5 # 1s, 3s, 5s f/m/s loopback, 10s for large [peer] # -------------------------------------------------------------------------------------------- General P2P Settings path = eynhaender/development/libbitcoin/test/network # cashed hosts file goes here user_agent = "/libbitcoin4:0.14.2/" enable_alert = false # send/log alert messages (no send option) enable_reject = false # send/log reject messages enable_address = true # in/out address messages enable_address_v2 = false # in/out address v2 messages enable_witness_tx = false # in/out wtxid for tx announcements enable_compact = false # in/out compact block announcements enable_relay = false # in/out transaction announcements validate_checksum = false # validate inbound message checksum invalid_services = 176 # shitcoins services_minimum = 9 # witness node (required of peer to outbound) services_maximum = 9 # witness node (provided to peer of non-seed) [+64 for filters] protocol_maximum = 70015 # current default (send_headers). [70015 for filters, else 70012] protocol_minimum = 31800 # Satoshi:22.0.0 minimum (headers). handshake_timeout_seconds = 5 # time to complete handshake. channel_heartbeat_minutes = 5 # ping maximum_skew_minutes = 5000 [outbound] # ---------------------------------------------------------------------------------------- P2P Outbound Settings connections = 100 host_pool_capacity = 10000 connect_batch_size = 5 use_ipv6 = true seeding_timeout_seconds = 42
eynhaender16:33
The kernel settings for dirty paging on linux is crucial too: https://github.com/libbitcoin/libbitcoin-server/blob/master/docs/theory-of-operation.md#linux-dirty-page-writeback-tuning-notes
👍 1
evoskuil18:33
bucket sizing is a tradeoff, as it reduces head size but proportionally increases hashmap paging. But when we aren't searching the hashmaps (as in milestone) it's a good one.
evoskuil18:34
Yeah, there's a lot of defaults in that config.
node.threads = 1
This will give you one thread for block validation. Prob only ideal on a 1 thread machine.
(edited)evoskuil18:37
Otherwise apart from node and network threads, and database tables, this is basically defaults.
This will stop the sync at block 950,000, mostly for benchmarking and testin.
maximum_height = 950000
This depends on your situation, it's bumped up above:
seeding_timeout_seconds = 42
josie17:12
here is the database PR for the silent payments prevout_summaries table: https://github.com/libbitcoin/libbitcoin-database/pull/804, ill open the node and server PRs tomorrow.
👍🏻 1
evoskuil20:16
I've posted reviews of both PRs. Code looks solid, we'll have SP indexation for sure.
🔥 1
Wednesday, June 3, 2026
josie12:13
I've refactored the database and system PRs based on @evoskuil’s review and thrown up the node PRs:
- https://github.com/libbitcoin/libbitcoin-node/pull/1025 , not related to silent payments but a warning i spotted in clang
- https://github.com/libbitcoin/libbitcoin-node/pull/1026 not related to the silent payments index but needed for broadcasting from sparrow
- https://github.com/libbitcoin/libbitcoin-node/pull/1027 silent payments index PR for node
im still working on the server PR, mostly just cleaning it up and refactoring after the database and system changes, aiming for later this week. overall, coming together nicely!
🎉 1💪🏻 1👍🏻 1💪 1
Thursday, June 4, 2026
1 reply · evoskuil · last Jun 04, 2026
evoskuil13:36
Hi Vano, it's going well. We should have first full execution today or tomorrow. We have a working NuGet package for the vc145 builds, are working on full build system integration, and have most of the program logic complete. Initial tests against mainnet show that we can capture about 94% of all signatures - a higher rate than expected.
🔥 2
evoskuil13:36
replied to a thread: view
Hi Vano, it's going well. We should have first full execution today or tomorrow. We have a working NuGet package for the vc145 builds, are working on full build system integration, and have most of the program logic complete. Initial tests against mainnet show that we can capture about 94% of all signatures - a higher rate than expected.
🔥 2
Vano Chkheidze13:46
i released 4.1 on main branch with last fixes so you can move to main release branch
🍻 1👊🏻 1
1 reply · eynhaender · last Jun 05, 2026
eynhaender18:57
I have switched the nuget package to the main branch. This is the first build with a new version numbering. The packaging gets his own version number so this release is version 4.1.1.1.
https://www.nuget.org/packages/UltrafastSecp256k1-vc145/
(edited)nuget.org
Native Windows NuGet package for UltrafastSecp256k1 — ultra high-performance secp256k1 elliptic curve cryptography. Built with MSVC vc145 (Visual Studio 2026), x64, Release + Debug, static (default) + shared. Includes the libbitcoin bridge (ufsecp_lbtc_bridge).
👍 1
Huddle with pmienk, evoskuil, josie, Eric Chennells, neillm, eynhaender, Rowan Voskuil, Enligths, Roberto · 155 min15:55
7 replies · neillm, pmienk, Roberto, eynhaender +2 · last Jun 05, 2026
Roberto18:01
got get the kids at school. i like the format of the call, will try to come more often. thank you all
❤️ 1
pmienk18:28
# Build options
option(SECP256K1_BUILD_CPU "Build CPU implementation" ON)
option(SECP256K1_BUILD_CUDA "Build CUDA GPU support" OFF)
option(SECP256K1_BUILD_ROCM "Build ROCm/HIP GPU support (AMD)" OFF)
option(SECP256K1_BUILD_OPENCL "Build OpenCL support" OFF)
option(SECP256K1_BUILD_METAL "Build Apple Metal GPU support" OFF)
# ── Independent GPU feature modules ───────────────────────────────────────────
# Toggle individual GPU operations on/off SEPARATELY from the CPU feature
# modules. The GPU op surface is independent: a consumer may want, for example,
# BIP-324 compiled on the CPU but stripped from the GPU build (or vice-versa).
# Default ON (full GPU op surface, no behaviour change). Turning one OFF removes
# its kernels + backend dispatch from every GPU backend (CUDA/OpenCL/Metal); the
# stable C ABI entry point remains and returns UFSECP_ERR_GPU_UNSUPPORTED.
option(SECP256K1_GPU_BUILD_ZK "GPU ZK ops (knowledge / DLEQ / bulletproof / SNARK witness)" ON)
option(SECP256K1_GPU_BUILD_BIP324 "GPU BIP-324 AEAD encrypt/decrypt ops" ON)
option(SECP256K1_GPU_BUILD_FROST "GPU FROST partial-signature verify op" ON)
option(SECP256K1_GPU_BUILD_BIP352 "GPU BIP-352 silent-payment scan op" ON)
option(SECP256K1_GPU_BUILD_ECDH "GPU ECDH batch op (secret-bearing)" ON)
option(SECP256K1_GPU_BUILD_MSM "GPU multi-scalar multiplication op" ON)
option(SECP256K1_GPU_BUILD_HASH160 "GPU Hash160(pubkey) batch op" ON)
option(SECP256K1_GPU_BUILD_ECRECOVER "GPU ECDSA public-key recovery batch op"
🙏 1
Friday, June 5, 2026
evoskuil23:49
Multisig table is now integrated from system->database->node->server. I'm about to do a run to collect and measure. Should be batching soon.
👍 1🎉 1
Saturday, June 6, 2026
evoskuil00:20
Because of the growing table count, I've renamed some tables (filesystem names only) - so update if you have any automation about it.
namespace libbitcoin {
namespace database {
namespace schema {
/// These should be ASCII only.
namespace dir
{
constexpr auto heads = "heads";
constexpr auto primary = "primary";
constexpr auto secondary = "secondary";
constexpr auto temporary = "temporary";
}
namespace archive
{
constexpr auto header = "archive_header";
constexpr auto input = "archive_input";
constexpr auto output = "archive_output";
constexpr auto point = "archive_point";
constexpr auto ins = "archive_ins";
constexpr auto outs = "archive_outs";
constexpr auto spend = "archive_spend";
constexpr auto tx = "archive_tx";
constexpr auto txs = "archive_txs";
}
namespace indexes
{
constexpr auto candidate = "index_candidate";
constexpr auto confirmed = "index_confirmed";
constexpr auto strong_tx = "index_strong";
}
namespace caches
{
constexpr auto ecdsa = "batch_ecdsa";
constexpr auto schnorr = "batch_schnorr";
constexpr auto multisig = "batch_multisig";
constexpr auto duplicate = "cache_duplicate";
constexpr auto prevout = "cache_prevout";
constexpr auto validated_bk = "validated_bk";
constexpr auto validated_tx = "validated_tx";
}
namespace optionals
{
constexpr auto address = "option_address";
constexpr auto filter_bk = "option_filter_bk";
constexpr auto filter_tx = "option_filter_tx";
}
namespace locks
{
constexpr auto flush = "flush";
constexpr auto process = "process";
}
namespace ext
{
constexpr auto head = ".head";
constexpr auto data = ".data";
constexpr auto lock = ".lock";
}
} // namespace schema
} // namespace database
} // namespace libbitcoinevoskuil03:11
Running now with all three tables accumulating signature tuples:
schnorr: [hash(32)][xkey(32)][sig(64)][block-fk(3)] ecdsa: [hash(32)][cpk(33) ][sig(64)][block-fk(3)] multisig: [hash(32)][cpk(33) ][sig(64)][m|n(1)][group(2)][blockid-fk(3)]
evoskuil03:21
I'm now executing a batching run, just colleting the data into three tables (no periodic purging to simulate batch completions, just saving all of it). The most direct comparison to this is the client filter (neutrino) runs, as these perform the exact same block and prevouts read in order to populate the filter data, and then write the data for each block. Sigs is an order of magnitude more data being written, and we are also fully executing validation, excluding ~94% of sigs (99.99% until taproot). However filter computation is non-trivial.
A milestone neutrino run on this same machine completes to block 850k in 1.91 hrs (that's the last run I have, a year old, missing some big optimizations. The current signature batching run is tracking very closely against the neutrino milestone benchmark. Passing block 500k organized and about 110GiB of signature data between ecdsa single and multisig, with 99.99% and 99.95% capture rate respectively.
(edited)I expect that on a modern machine the total run with batch evaluation will not add any material cost, since we will purge the tables after evaluation, which will probably save more time in paging than the added cost of the signature evaluations.
Passing block
(edited)500,000 downloaded/validated/confirmed/organized at 33 minutes. That's about a 2.5 hour pace to complete full validation over the Internet on a 10 year old machine without shani. Adding full Electrum address and neutrino indexing to this would bring it closer to 3 hours.evoskuil04:48
Passing block
(edited)800k now at 2.0 hrs with 400GiB of signature tuples stored, schnorr starting to pick up. 99% signature capture rate.evoskuil06:39
We that ended up at about
3.5 hrs full validation baseline to 950k. The three signature batches accumulated 486GiB of signatures, and toward the end write time was most of the CPU time, resulting from flushing. This should mostly go away once the tables are purged in each window. Still, even 3.5 hrs isn't bad at all for full validation cost on this machine.evoskuil07:08
Also the total schnorr signature capture ratio for schnorr is only 60% at the end. This is because I'm not batching from tapscript - and the 40% of those that we have to validate are a big chunk of the cost increase in the later blocks, even though it's only about 6% of the total chain sigs. I've found that I can actually capture most of these as well, using the same technique, just implies a 4th batch table to be optimal (ecdsa/schnorr * single/multiple).
ecdsa: [hash(32)][cpk(33)][sig(64)][block-fk(3)] multisig: [hash(32)][cpk(33)][sig(64)][m|n(1)][group(2)][blockid-fk(3)] schnorr: [hash(32)][xpk(32)][sig(64)][block-fk(3)] threshold: [hash(32)][xpk(32)][sig(64)][m|n(1)][group(2)][blockid-fk(3)]
Vano Chkheidze09:22
UltrafastSecp256k1 ⇄ libbitcoin — 4-table batch verify: throughput + usage
The bridge now supports your full 4-table batching model first-class. At the verify boundary the four tables collapse to two verify kinds: a
multisig row is one ECDSA signature, a threshold (tapscript CHECKSIG / CHECKSIGADD) row is one Schnorr signature. The m|n + group bytes are accounting metadata carried in the opaque tail and are never interpreted by the bridge (m-of-n satisfaction stays your job, downstream of the per-row verdict).
Throughput (measured)
bench_lbtc_4table, measured on this machine — Intel i5-14400F + NVIDIA RTX 5060 Ti, performance governor, best-of-4, 1,000,000 signatures per table, end-to-end (host de-interleave + H2D transfer + kernel + per-row result copy — what a node actually pays, not kernel-only).
tablerecord / tailGPU (CUDA)CPU (1 core, reference)ecdsaEcdsaRecord(129) + 33.84 M/s0.034 M/s (~29.5 s / 1M)multisigEcdsaRecord(129) + 63.84 M/s0.034 M/s (~29.3 s / 1M)schnorrSchnorrRecord(128) + 34.90 M/s0.074 M/s (~13.5 s / 1M)thresholdSchnorrRecord(128) + 64.88 M/s0.074 M/s (~13.7 s / 1M)
Key result — the m-of-n tail is free. multisig/threshold (6-byte m|n+group+block-fk tail) verify at the same rate as the single ecdsa/schnorr tables (3-byte tail), on both backends. The verify core reads only the leading record and walks stride = record + key_size, so the accounting tail costs nothing — measured, not inferred.
Notes:
- CPU is thesingle-core mandatory reference(the consensus anchor, always available). A node parallelizes it across cores; the figure above is per-core.
- GPU figures are end-to-end for a saturating 1M batch; smaller batches pay relatively more transfer/launch overhead.
- Numbers are valid only for this machine; re-measure elsewhere.
Usage
#include "ufsecp_libbitcoin.h"
using namespace ufsecp::lbtc;
Controller ctrl; // AUTO: GPU if present, else CPU reference
if (!ctrl.ok()) { /* no usable backend — the only failure is reported here */ }1. Single ECDSA / Schnorr table — results[]
// rows: n × (EcdsaRecord(129) + block-fk(3)); key_size = 3 (the opaque tail) std::vector<uint8_t> results(n); ctrl.verify_ecdsa(rows, n, /*key_size=*/3, results.data()); // results[i] == 1 valid / 0 invalid ctrl.verify_schnorr(srows, n, /*key_size=*/3, results.data());
2. Multisig / threshold table — typed-span, zero-copy
// Your packed table forwards directly; key_size = sizeof(Row) - record is recovered from the type. std::span<const MultisigRow> ms = ...; // EcdsaRecord + m|n + group + block-fk (135B) std::span<const ThresholdRow> th = ...; // SchnorrRecord + m|n + group + block-fk (134B) ctrl.verify_multisig (ms, r1.data()); // == ECDSA path, key_size 6 ctrl.verify_threshold(th, r2.data()); // == Schnorr path, key_size 6
3. Collect — rejected-id list (no results[] array)
// In-place: VALID rows get their key tail zeroed; INVALID rows keep it intact.
std::span<MultisigRow> batch = ...; // MUTABLE
ctrl.collect_multisig(batch);
for (auto& row : batch) // survivors = the rejected set (m|n+group+block-fk intact)
if (tail_nonzero(row)) rejected.push_back(row);4. Pure C ABI (no C++) — the 4 tables map onto 2 calls
// multisig record IS the ECDSA record; threshold record IS the Schnorr record. // m|n+group+block-fk is the opaque key tail (key_size = 6). ufsecp_lbtc_verify_ecdsa (ctrl, ms_rows, n, /*key_size=*/6, results); // multisig table ufsecp_lbtc_verify_schnorr(ctrl, th_rows, n, /*key_size=*/6, results); // threshold table ufsecp_lbtc_verify_ecdsa (ctrl, e_rows, n, /*key_size=*/3, results); // ecdsa table ufsecp_lbtc_verify_schnorr(ctrl, s_rows, n, /*key_size=*/3, results); // schnorr table
Canonical packed rows (on-wire source of truth,
#pragma pack(1)):
struct MultisigRow { EcdsaRecord record; uint8_t mn; uint8_t group[2]; uint8_t block_fk[3]; }; // 135
struct ThresholdRow { SchnorrRecord record; uint8_t mn; uint8_t group[2]; uint8_t block_fk[3]; }; // 134
tes from dev branch
evoskuil18:10
I designed multisig in both cases to operate over the base ecdsa and schnorr "single" calls. The only difference from the UF lib perspective is the increased key size required for the group correlations. The ecdsa multisig correlations are much more complex than the schnorr threshold correlations. I've implemented the former here.
As these correlations are the result of inter-row dependency I assume they cannot be processed effectively with concurrency. The schnorr threshold correlations are the same idea but just a simple computation over the set.
evoskuil18:18
Also, the list above refers to "no copy" a couple times. We are collecting huge amount of signatures, and that will increase some once I add threshold. Over 99% of the whole chain, which is about .5TB of data when multisig is flattened for batch. So avoiding copy seems significant. The node is currently implemented such that there is no copy cost to produce the table, and the table itself is structed to be fed directly into the engine.
The tables are necessarily structured "horizontally" (each row contains a tuple). However the engine seems to require a "vertical" structure - where there are three (four with keys) columns of data: sig-hashes, public-keys, signatures, correlation-keys. The bridge is copying the entire buffer in order to produce this transformation. So over the course of a single full chain validation we will stream .5TB of signature data, and that will be copied into another .5TB (in chunks) to create this transformation.
Can you confirm that this is correct, and whether is absolutely necessary?
As for the multisig correlation computation, I don't expect this to be significant, since the algorithm is O(n) in the batch size and performs no allocation.
evoskuil18:30
If the above transformation is indeed necessary, then it would be possible for us to produce the signature batches in this vertical format, however it requires either (1) making an up-front assumption about the chunk size (i.e. how many signatures we expect to batch in one go), or (2) constructing 4 independent mmaps per logical table (which would result in 16 tables total). Both of these are feasible. It should be that they are close to equivalent from an I/O cost standpoint. #2 is more flexible though it requires a little more setup overhead and some amount of increased contention.
evoskuil18:39
Also if we did make this change it would allow us to collapse the single-sig and muti-sig calls.
Vano Chkheidze18:39
i will extend now bridge and you will have both version vertical and horizontal and use any as you need
evoskuil18:42
Thanks, but that's not what I'm asking. The bridge already performs this transformation. I'm asking if the transformation (from row to column) is necessary.
Because whether it is done in the bridge or in our code the cost remains the same. I'm trying to determine if the cost can be eliminated.
Vano Chkheidze18:46
Correct. With the current UF GPU verification ABI, the GPU-facing input is columnar: message hashes, public keys, and signatures are passed as separate contiguous buffers. So if the source table is stored as horizontal rows, some row-to-column transformation is currently required before calling the existing GPU path.
What I meant by “no copy” was only that the bridge does not interpret or copy the correlation tail into the verification core. That was too broad as written. The packed-row path still de-interleaves the verified record bytes into the existing columnar GPU ABI.
The clean way to avoid that bridge-side transformation is to expose both formats:
1. keep the current packed-row API for callers that naturally produce `[record | key]` rows;
2. add a columnar API for callers that can store/produce `hashes[]`, `pubkeys[]`, `sigs[]`, and optional `keys[]` directly.
Then libbitcoin can choose the storage layout. If you keep the row table, the transform is necessary for the current GPU ABI. If you produce vertical columns, the bridge can forward them directly to UF’s existing GPU calls and skip the row-to-column copy.
I agree that the multisig correlation pass itself should not be the bottleneck if it is O(n), allocation-free, and runs after per-row signature verdicts are known. The expensive part to avoid is streaming the signature records through an extra host-side de-interleave pass.
evoskuil18:50
Ok, that's helpful info. After reviewing the bridge implementation I don't believe that we will end up needing this. It's a nice to have for now, but ultimately we don't want the custom code or transformation code. But for now if you want to provide both vertical and horizontal interfaces that's helpful. Also it looks like there's no need for any custom support for multisig.
I'll look into vertical storage. In that case we would not need to pass the extra key field, as it would just be an independent column we would retain for correlation.
The one thing that would be really helpful right now is a cost analysis on the transformation cost (from horizontal to vertical) over a run of 3.5 billion tuples.
Vano Chkheidze18:55
For a 3.5B-tuple run, the transform cost is dominated by memory traffic, not CPU arithmetic.
With the current row path, each tuple has to be read once from the horizontal table and written once into vertical buffers before UF can call the existing columnar GPU ABI.
Approximate payload per tuple:
- ECDSA: 32-byte hash + 33-byte compressed pubkey + 64-byte signature = 129 bytes.
- Schnorr: 32-byte hash + 32-byte x-only pubkey + 64-byte signature = 128 bytes.
- If the correlation key is part of the transformed bridge record, add 6 bytes, but for vertical storage it can remain an independent libbitcoin-side column and does not need to be passed to UF verification.
So for 3.5B tuples:
- ECDSA verify payload: 3.5B * 129 = ~451.5 GB decimal, ~420.5 GiB.
- Schnorr verify payload: 3.5B * 128 = ~448.0 GB decimal, ~417.2 GiB.
- With a 6-byte key included: add ~21.0 GB decimal, ~19.6 GiB.
A row-to-column transform touches at least read + write, so the extra memory traffic is roughly:
- ECDSA: ~903 GB decimal / ~841 GiB.
- Schnorr: ~896 GB decimal / ~834 GiB.
- With 6-byte key copied too: add another ~42 GB decimal / ~39 GiB traffic.
Wall-clock estimate depends on sustained host memory bandwidth:
- 50 GB/s effective: ~18 seconds for ~900 GB.
- 100 GB/s effective: ~9 seconds.
- 150 GB/s effective: ~6 seconds.
- 200 GB/s effective: ~4.5 seconds.
In practice it may be worse if this is NUMA-unfriendly, competes with mmap/page-cache traffic, or runs while GPU transfers are also consuming bandwidth. It may also be partially hidden if transform is pipelined while previous chunks are on the GPU.
The main point: the transform is not asymptotically expensive, but at 3.5B tuples it is a real full-chain memory-bandwidth tax. Vertical storage avoids one full read+write pass over roughly 0.45-0.47 TB of data per validation run. If the correlation key remains an independent libbitcoin column, UF does not need to receive or copy it for pure signature verification.
Vano Chkheidze19:06
Concrete example:
Horizontal storage today:
row[i] = hash32 | pubkey33 | sig64 | correlation_key6
For ECDSA that is 129 bytes of verification data plus the 6-byte key. The bridge then has to de-interleave this into:
hashes[n][32]
pubkeys[n][33]
sigs[n][64]
before calling the existing UF GPU verification ABI.
Vertical storage would instead keep the data as:
hashes[] = mmap column, n * 32 bytes
pubkeys[] = mmap column, n * 33 bytes for ECDSA, or n * 32 for Schnorr
sigs[] = mmap column, n * 64 bytes
keys[] = independent libbitcoin correlation column, not needed by UF verify
Then the bridge can call the column API directly:
ufsecp_lbtc_verify_ecdsa_columns(
ctrl,
hashes,
pubkeys,
sigs,
count,
results);
or, if you want the collect-style behavior:
ufsecp_lbtc_verify_ecdsa_columns_collect(
ctrl,
hashes,
pubkeys,
sigs,
count,
keys,
6);
In the pure verify case UF does not need the key column at all. The key can remain fully owned by libbitcoin and used later for correlation.
Cost example for 3.5B ECDSA tuples:
3.5B * (32 + 33 + 64) = 451.5 GB of verification payload.
A row-to-column transform must at least read that payload once and write the column buffers once:
~451.5 GB read + ~451.5 GB write = ~903 GB memory traffic.
At 100 GB/s sustained host memory bandwidth, that is roughly 9 seconds of extra memory-bandwidth work. At 50 GB/s it is roughly 18 seconds. If the 6-byte key is also copied, add about 42 GB more traffic, but with vertical storage the key does not need to be passed to UF verification.
So the transform is not algorithmically complex, but over a full-chain-sized 3.5B tuple run it is a real memory-bandwidth tax. If libbitcoin can produce vertical columns directly, UF can skip that pass and use the existing GPU-facing layout directly.
check dev branch
evoskuil19:07
Ok, that's helpful. Given these numbers it's inconsequential, and the real issue will be contention with mmap. So it's not worth the complexity of optimizing it until we have measured numbers under the current design - which is certainly simpler and already implemented. So I'll proceed now with adding the threshold signature support (with table), starting with a measurement of the capture rate. That's quite simple now that I've implemented the more complex ecdsa multisig caching, so should take no more than 1 day.
I'll also be working on getting some real batches flowing through the engine. The question there will be chunk size.
I'd like to be able to send a chunk of any size to the bridge and have it internally chunk up the work as optimal for the hardware/scenario. This would simplify the calling a lot.
(edited)Vano Chkheidze19:08
show me exact format you want i will adopt not problem check dev branch and return me feedback
I am assuming that the calls will fail if the row count is too high, but maybe the internal chunking is already handled. If that's the case then it's fine as is. I'll just send a billion rows or so.
Vano Chkheidze19:16
The current bridge methods already accept an arbitrary row count and internally split the work into hardware-sized chunks.
So from the caller side the format remains unchanged:
rows = [record | optional correlation key] * n
For example:
ufsecp_lbtc_verify_ecdsa(ctrl, rows, row_count, key_size, results);
or for collect mode:
ufsecp_lbtc_verify_ecdsa_collect(ctrl, rows, row_count, key_size);
You do not need to pre-split the batch unless you want to for your own mmap / scheduling reasons. The bridge walks the input in internal chunks and dispatches each chunk to GPU or CPU fallback.
So sending a very large row count is logically fine, including “a billion rows”, assuming:
- the row buffer is addressable and remains valid for the duration of the call;
- `results` has `row_count` bytes for the results API;
- `row_count * stride` fits the platform address space / `size_t`;
- the mmap/page-cache behavior is acceptable for that one long call.
The bridge should not fail merely because the count is larger than the device batch size. Device-sized batching is handled inside the bridge. The main practical limits are memory mapping, addressability, result-buffer size, and how much latency you want in a single call before control returns to the caller.
evoskuil19:18
Yes, that's perfect. And the mmap observation is good. The major cost here will be paging, but that's unavoidable.
We will control the logical chunks so as to prevent the mmap tables from getting too big. With enough RAM we want to never hit the SSD. Of course the row transformation doubles the memory cost, so at some point vertically striping the storage will probably make sense.
evoskuil19:58
I've determined that the multiple table approach to creating the vertical layout is the only reasonable approach (both paging optimal and consistent with out table fault tolerance constraints). But that would imply 16 physical tables for 4 logical tables. By collapsing to 2 logical tables (ecdsa and schnorr) we simplify quite a bit, and optimize the calls. This requires unused correlation data of an extra 3 bytes per row on the single sig rows. That amounts to almost 8GB over the full run, out of 493GiB total, so only about 1.6% wasted. There's no material cost increase to correlation, we just combine the two methods in both ecdsa/multisig and schnorr/threshold.
(edited)So the current design had us on course to add 4 logical/physical tables. This would double that to 8 physical tables, to create a logical structure of two (ecdsa and schnorr). It would eliminate the .5TB of row transformation/allocation over the full run, satisfying the original intent of just passing the mmap buffer directly to the GPU. It also removes passing any correlation data (now 6 bytes per row) to the engine.
(edited)The database already fully supports multi-table striping in this manner, so won't require any additional work. We use this to independently store points from inputs so as to avoid the huge paging cost of points when we aren't attempting to retrive them.
So I'm going to take this on sooner than later. I would however like to get at least a proof of concept run through the engine today, and to measure the capture rate for threshold, so I'll tackle those first.
Vano Chkheidze20:05
That sounds like the right tradeoff.
If the vertical layout is produced directly from mmap-backed physical tables, then collapsing to two logical tables (ECDSA and Schnorr) and keeping the correlation bytes in the stored row format is preferable to paying the horizontal-to-vertical transform cost. The extra ~3 unused bytes on single-signature rows is a small fixed storage overhead (~1.6% by your estimate), while avoiding the extra ~0.5TB copy/allocation over a full run is a much cleaner win.
From the engine/bridge side I would keep the interface at the column level:
- ECDSA: hashes, public keys, signatures, optional/retained correlation column
- Schnorr: hashes, public keys, signatures, optional/retained correlation column
The correlation column does not need to be passed into the core verify engine unless that specific call needs it. For normal single-sig rows it can just remain part of the caller-owned table layout and be ignored. For multisig/threshold correlation, the bridge can use the same physical format and select the correlation behavior internally.
So I agree: two logical tables backed by vertical physical tables is simpler, avoids the row transform, reduces call complexity, and keeps the wasted bytes small enough that it is not worth optimizing separately until measured otherwise.
👍🏻 1
Monday, June 8, 2026
all of this batched functions are avalible now in libbtc shim for you please test them and give me feedback which one gives real gains in real pipeline
🎉 1
Tuesday, June 9, 2026
evoskuil00:38
If you log the script of a large percentage of outputs, the log will lag a lot. And when you hit close… it can be a long wait. This is a tapscript spam wave, going on for about an hour now.
😲 1
The same key repeated endlessly 58,883 at a time in threshold scripts. This is what I was missing in the schnorr sig capture.
We run a sig validation on each key. And that has been included in all of my numbers so far.
(edited)The current trick to the issue above is to disable that log level before hitting close. That dumps the log almost instantly.
evoskuil03:06
Made it to inscriptions. Those only have one signature, but trivially captured so patterned that too.
evoskuil14:29
Hey Vano, most of the work in the last couple of days has been isolating the various popular restricted script patterns in tapscript so that we can aggregate them. It's a game of whack-a-mole, but it works because this is for the history/IBD, new stuff doesn't have much impact, that those patterns can be added later. I'm going to cut that work off today and try and get a pass running. Last I tried we still hand a linkage issue with our NuGet package.
Vano Chkheidze14:33
last time i made nuget static package but why not include engine directly in code as part of code instead of lib build flags give you this ability also to compile engine as part of your source
4 replies · evoskuil, Vano Chkheidze · last Jun 10, 2026
Vano Chkheidze06:31
no with build flag you can change type and it will not build as lib will build like it's part of your source code
Vano Chkheidze06:33
SECP256K1_BUILD_AS_OBJEC Build fastsecp256k1 as OBJECT library (no .a — objects go directly into parent link step)evoskuil06:49
Oh, I see - effectively embedding the cpp files. That's a CMake feature, which leaves some significant gaps for build support.
clone repo as submodule configure all build flags turn off all futures you dont need and compile it as part of your source it have zero dependencies so will build without problem
3 replies · pmienk, Vano Chkheidze, evoskuil · last Jun 09, 2026
pmienk16:45
The model we've used for all other dependencies is to perform a configured installation and then relied on either cmake targets of pkg-config for linking while on windows/vs using nuget. I'm currently looking into your use of cmake as you install targets and a pc file (which is great for us supporting autotools and cmake), however you don't appear to install the generated library. I'm working though your targets to determine the best placement for this kind of change and making sure I understand what's going on within your recursive CMakeFile.txt files as I would not want to cause you any undue maintenance burden
👍 1
Vano Chkheidze16:47
if will be needed to fix something in cmake ping me any time or just send fix pr and will merge to repo for future use i'm open for any colaborations
👍 1
evoskuil17:52
We use these terms (sort of informally): Unit Test - smallest unit possible/practical (e.g. lowest level method/function). Component Test - aggregate of units (e.g. class or aggregating function/method). Functional Test - system function (e.g. endpoint communication over a socket, public API). Acceptance Test - performed over the compiled executable (e.g. customer acceptance). We target full Unit Test coverage and Component and/or Functional as necessary to ensure expected aggregate behavior. No external dependencies (e.g. Python or other tools), just Boost Test in our pattern. Can be data-driven (which is when we allow loops in a test). Acceptance is outside of the build (against the executable or compiled lib).(edited)
Vano Chkheidze14:36
binary will became very small nuget package is full source and you dont need so much dead code in your project
🔥 1
Wednesday, June 10, 2026
evoskuil03:45
replied to a thread: view
Do you mean that the implementation can be included as a headers-only library?
evoskuil06:49
replied to a thread: view
Oh, I see - effectively embedding the cpp files. That's a CMake feature, which leaves some significant gaps for build support.
it available on dev and main branches now
👍🏻 1
evoskuil23:04
If anyone is syncing with latest there is a logging bug. You can disable to log torrent by disabling application level logging.
[log] application = false(edited)
🙏 2
Thursday, June 11, 2026
evoskuil14:27
Still, even 3.5 hrs isn't bad at all for full validation cost on this machine.
After increasing the schnorr batch somewhat and bypassing the .5TB buffering, the total maxi time to validate to 950k without sigs is 3.0 hrs (vs. 5.0 with sigs). There's still a lot of schnorr to be gained through increased pattern matching, but I don't want to burn more time on that until we have the full measure. I suspect that's about 30 minutes more of savings. I'm going to plot the comparison against a with sigs run and see if there's an inflection point at schnorr. If so that will give a better estimate of the likely outcome.
👍 1
evoskuil14:53
This shows the total of all signature operation invocations after 295k (last checkpoint) to 950k (benchmark). I counted all misses but captures as signature count, so I bumped the op_checkmultisig miss rate by x3 as a better estimate. We could increase this rate with a little more effort in the extractor. The schnorr threshold miss rate is zero because all of the misses are driven individually into schnorr single sig. So schnorr is low and threshold is high. But as you can see, between them it's still a good rate. Like multisig, threshold also counts the total signatures (for its positive matches). The true threshold rate can come up a bunch more, driving down the schnorr miss rate and maybe savings us 30 mins of CPU time.
Vano Chkheidze15:30
if you need any kind of acceleration from my side if you have any idea we can discuss and i will extend engine
❤️ 1
Huddle with pmienk, eynhaender, Eric Chennells, evoskuil, Rowan Voskuil, Marnix, Enligths · 177 min15:55
3 replies · eynhaender, evoskuil · last Jun 11, 2026
eynhaender17:04
Current capture rates @ lower 600k:
2026-06-11T17:00:45Z.0 Efficiency multisig. 99.9764% = 386926003/(386926003+91486) 2026-06-11T17:00:45Z.0 Efficiency ecdsa.... 99.9918% = 932347684/(932347684+76931) 2026-06-11T17:00:45Z.0 Efficiency multisig. 99.9764% = 386926187/(386926187+91486) 2026-06-11T17:00:45Z.0 Efficiency ecdsa.... 99.9918% = 932348544/(932348544+76931) 2026-06-11T17:00:45Z.0 Efficiency multisig. 99.9764% = 386926948/(386926948+91486) 2026-06-11T17:00:45Z.0 Efficiency ecdsa.... 99.9918% = 932352956/(932352956+76931) 2026-06-11T17:00:45Z.0 Efficiency multisig. 99.9764% = 386928147/(386928147+91486) 2026-06-11T17:00:45Z.0 Efficiency ecdsa.... 99.9917% = 932357743/(932357743+76931) 2026-06-11T17:00:45Z.0 Efficiency multisig. 99.9764% = 386929215/(386929215+91486) 2026-06-11T17:00:45Z.0 Efficiency ecdsa.... 99.9918% = 932358136/(932358136+76931) 2026-06-11T17:00:45Z.0 Efficiency multisig. 99.9764% = 386929302/(386929302+91486) 2026-06-11T17:00:45Z.0 Efficiency ecdsa.... 99.9917% = 932360468/(932360468+76932) 2026-06-11T17:00:45Z.0 Efficiency multisig. 99.9764% = 386929510/(386929510+91486) 2026-06-11T17:00:45Z.0 Efficiency ecdsa.... 99.9918% = 932361084/(932361084+76932) 2026-06-11T17:00:45Z.0 Efficiency multisig. 99.9764% = 386929564/(386929564+91486) 2026-06-11T17:00:45Z.0 Efficiency ecdsa.... 99.9918% = 932362252/(932362252+76932) 2026-06-11T17:00:45Z.0 Efficiency multisig. 99.9764% = 386929687/(386929687+91486) 2026-06-11T17:00:45Z.0 Efficiency ecdsa.... 99.9917% = 932364194/(932364194+76933) 2026-06-11T17:00:45Z.0 Efficiency multisig. 99.9764% = 386929818/(386929818+91487) 2026-06-11T17:00:45Z.0 Efficiency ecdsa.... 99.9918% = 932365434/(932365434+76933) 2026-06-11T17:00:45Z.0 Efficiency multisig. 99.9764% = 386930045/(386930045+91487) 2026-06-11T17:00:45Z.0 Efficiency ecdsa.... 99.9918% = 932368141/(932368141+76933) 2026-06-11T17:00:45Z.0 Efficiency multisig. 99.9764% = 386930076/(386930076+91487) 2026-06-11T17:00:45Z.0 Efficiency ecdsa.... 99.9917% = 932369056/(932369056+76933) 2026-06-11T17:00:45Z.0 Efficiency multisig. 99.9764% = 386930082/(386930082+91487)(edited)
eynhaender18:44
2026-06-11T18:44:08Z.5 Insufficient services (1032) by [66.85.157.56:8333] showing (3085) /Satoshi:25.0.0/ 2026-06-11T18:44:09Z.0 Efficiency ecdsa.... 99.6359% = 1895305241/(1895305241+6926814) 2026-06-11T18:44:09Z.0 Efficiency multisig. 99.9601% = 725149057/(725149057+289162) 2026-06-11T18:44:09Z.0 Efficiency schnorr.. 74.0383% = 23630642/(23630642+8286152) 2026-06-11T18:44:09Z.0 Efficiency threshold 100.0000% = 185585/(185585+0)
evoskuil23:43
@Vano Chkheidze’s
(edited)Ultrafast built and linked with no libsecp256k1, using @eynhaender's NuGet package, system tests all passing. Building now for a baseline full validation run tonight.🔥 2😎 1💪 1
The unit tests are sufficient to show basic batch correctness, as these were previously written against the fallback implementation, which uses a c++17 parallel loop over the batch. That's now running against Ultrafast (based on the changed NuGet dependency selection via the vs2026
.props integration).
// batch_verify
// ----------------------------------------------------------------------------
#if defined(HAVE_ULTRAFAST)
template <typename Batch>
data_chunk batch_verify(const std::span<Batch>& batch, bool) NOEXCEPT
{
// OOM is unrecoverable.
static thread_local ufsecp::lbtc::Controller context{ UFSECP_LBTC_AUTO };
if (!context.ok()) std::abort();
const auto count = batch.size();
data_chunk results(count);
const auto out = results.data();
const auto in = pointer_cast<const uint8_t>(batch.data());
if constexpr (is_same_type<Batch, schnorr::batch>)
{
constexpr auto extra_size = sizeof(Batch) - (sizeof(hash_digest) +
sizeof(ec_xonly) + sizeof(ec_signature));
ufsecp_lbtc_verify_schnorr(context.get(), in, count, extra_size, out);
}
else
{
constexpr auto extra_size = sizeof(Batch) - (sizeof(hash_digest) +
sizeof(ec_compressed) + sizeof(ec_signature));
ufsecp_lbtc_verify_ecdsa(context.get(), in, count, extra_size, out);
}
return results;
}
#else
template <typename Batch>
data_chunk batch_verify(const std::span<Batch>& batch, bool turbo) NOEXCEPT
{
// par_if doesn't throw.
BC_PUSH_WARNING(NO_THROW_IN_NOEXCEPT)
const auto policy = poolstl::execution::par_if(turbo);
BC_POP_WARNING()
// Used only to produce order for concurrency.
std::vector<size_t> it(batch.size());
std::iota(it.begin(), it.end(), zero);
// Collect signature validation results as corresponding integer booleans.
data_chunk results(batch.size());
std::for_each(policy, it.cbegin(), it.cend(), [&](size_t row) NOEXCEPT
{
results.at(row) = to_int<uint8_t>(verify_signature(batch[row]));
});
return results;
}
#endif (edited)Friday, June 12, 2026
Vano Chkheidze00:30
great waiting final results 🙂 please update to latest dev branch before i release v4.2.2
Saturday, June 13, 2026
evoskuil23:20
Completed the first full validation run with
ufsecp. There was one issue that took some time to track down, just a difference in expectations regarding signature encoding. This was using @eynhaender’s NuGet package with the Windows build as a drop-in replacement for secp256k1.evoskuil23:25
I didn't get a complete measure of performance, since the run was interrupted at around 700k for the above issue, and then I completed from that point. Up to that point the run as around 150% of the
secp256k1 time, which is not surprising given that @Vano Chkheidze has said the Windows builds aren't optimized. And the NuGet package may not be optimal either. But it at least gets the toolchain moving for signature batching and silent payment search.evoskuil23:33
I spent most of yesterday isolating the above issue, and most of the day today working off some code debt in
libbitcoin-system. Just factoring the 1 secp256k1 wrapper .cpp file into 7 (it's grown a lot over the years, with bip66, taproot, and now batching). Much easier to work with now. So now back to batching. The open work is to coordinate invocation and summation in the validation chaser. Once we get a good run in the current table format, I'll rotate from horizontal to vertical format (2 tables -> 2 sets of 4 tables), so that we get clean no-copy GPU processing over the mmaps.🎉 3
Sunday, June 14, 2026
evoskuil13:34
You can rely on Intel sse4 and Arm neon, which are available on nearly all production machines.
Vano Chkheidze13:36
i optimized msvc pepline fully works very fast now i dont have arm machine to test
if we want windows arm optimization i will focus on this too
CLang works very fast evrywhere
but as you want MSVC i will extend MSVC support on supported platforms just i cant test it on arm
👍🏻 1
and mac
evoskuil13:38
I ran a did a maxi full validation run last night (libsecp256k1 unbatched), just to verify performance impact due to several low level changes.
4.85 hrs to 950k is a new record, taking back the mantle from Brunella.👌🏻 1
arm also but for linux arm
theoretically your node will work great on jeston devices also
and for nodes jeston devices are great because they have gpu cpu both and are cheap
on jeston batch will work very efectivly because cpu and gpu have unified memory
Vano Chkheidze14:59
@evoskuil do you have arm windows device? can you benchmark on arm windows and send me results?
evoskuil15:20
I don't. I cross compile and sometimes we get a CI machine, but that's no good for benchmark.
Tuesday, June 16, 2026
evoskuil03:57
Today I tackled the concurrency design for integrating signature batch collection and verification into the node's validation chaser. This was a challenging puzzle, like many before. But it appears to have folded in nicely. With just a little bit of glue we should be batching tomorrow.
🎉 3
Wednesday, June 17, 2026
evoskuil02:29
Batching now, single-sig only, ecdsa and schnorr, using the parallel loop over
secp batching in concurrency window default (50k blocks), on maxi. Concurrency model seems to be working nicely, mostly 100% CPU since startup. Averaging right around 317,000 signatures per second in each block. The maxi has 64 x 2.1 GHz threads, and note that we are not pausing other operations during batch processing, the node continues to download and confirm - only validation is suspended during the periods of batch processing.
I expect and it appears that this baseline will be slower than our traditional runs. The batching incurs overhead in buffering the signatures to mmap and in post-processing the results, in CPU, RAM and I/O. However GPU processing this same work will free most of its CPU cost and complete in fraction of the time. At block 450k the signature verification cost is about 8 minutes. That should come down to 30 secs on a typical modern GPU (not maxi), and just a few seconds on a high end GPU system.👍 1
evoskuil02:37
The profiler shows me that during the concentrated batch processing, the verify loop is consuming
70% of CPU. 20% is going to download, parsing, hashing, and storing. and about 5% confirmation queries, and about 5% to reading.Interestingly total time is tracking only
(edited)2.5 mins behind the record maxi full validation run. But because I wanted to see the behavior from genesis, this run also includes the 295k checkpointed blocks that aren't in the benchmark. That normally takes about 8 mins additional time. IOW this run is a little faster so far than the best full validation benchmark, and all we've really done is organize the signatures to be validated close together. So it does appear that the cache coherence is offsetting the marshalling overhead.👍 1
evoskuil03:02
And now the batch job is passing the record run, at around block 550k, which is only about 20% of the chain. So I expect this will set a new record without even GPU offloading. But I'm getting impatient to see what this will loop like on ultrafast, with my 10 year old GPU, so I'm going to shut down, rebuild, and give it a go.
👍 1🎉 1
Also, given that these CPU batch runs can last quite a long time, I'll have to inject cancellability into them. Not sure how that would go with GPU processing, or whether it would be necessary.
on gpu also not problme gpu batch is fast and before start new batch there can be checked token
evoskuil13:34
We use a c++
std::atomic<bool>& for cancellations. The cancellable function just checks the value in the loop. We should add this to the batch APIs.evoskuil14:10
We could instead use
(edited)std::stop_token here (which wraps std::atomic<bool>) as it's helpful to be more explicit for the public interface.evoskuil16:29
Update: we are compiling and executing batch over
ufsecp now, but maxi needs a GPU driver update (drivers from 2018) and really a new GPU (2015). So I'll be moving the trial runs over to a mini build. That will also give us avx512 and sha-ni. At the same time I'm working on several cleanup items pertaining to batching, including:
(1) passing cancellation.
(2) managing ufsecp configuration.
(3) fine control over batch sizing (configurable sigs count threshold).
(4) generalizing multisig for threshold formulations.
(5) removing dead code (multisig table, turbo parameter).
(6) vertical table storage (4 physical tables per logical table).
These are all straightforward changes, hope to be done today.Vano Chkheidze16:30
i removed config ini form code and made path setter function and default is same dir
👍🏻 1
now building building standart for you without bridge
Thursday, June 18, 2026
Huddle with pmienk, eynhaender, Marnix, josie, evoskuil, Eric Chennells, Rowan Voskuil · 172 min15:55
3 replies · Rowan Voskuil, evoskuil · last Jun 18, 2026
evoskuil19:25
It was a disk full condition at ~750k.
2026-06-18T15:58:58Z.6 Require [5553809024] bytes and [4394893312] are free.
evoskuil22:12
I've finally got the batching concurrency window under control. The
maxi is very consistently batching (using secp parallel loop) 360,000 sigs/sec at all block heights, and entirely independent of the download window.Vano Chkheidze22:17
hello review and if any changes are needed give me feedback i also added cuda runtime static build in cmake
and also made one header for easy integrations
evoskuil22:28
Integration looks fine, though I will be working toward the vertical tables and opaque ecdsa signature format only. Currently we are using the horizontal tables and opaque sigs. @pmienk should review the build profile (first link).
(edited)I would suggest that you default the
UFSECP_LBTC_AUTO parameter, since that's probably all we'll ever use.Vano Chkheidze22:36
Good point. The C++ wrapper already defaults to UFSECP_LBTC_AUTO, and I updated the libbitcoin docs/examples to use that form:
static thread_local ufsecp::lbtc::Controller context;
So libbitcoin does not need to spell the backend enum at normal call sites. The raw C ABI remains explicit. @pmienk can review the build profile here:
https://github.com/shrec/UltrafastSecp256k1/blob/dev/docs/LIBBITCOIN_INTEGRATION.md#build-profile
👍🏻 1
i optimized all Y decompress paths and in some cases gives improvements on cpu side too and memory footprint stays low
👍🏻 1
Note that the target window size is 1M sigs in all cases. However we do not enforce this, we just start the capture when it's crossed. Yet the signature capture is so fast that by the time we cross that one line of code into the unique lock, and the existing writers complete, we can capture well over 100M more sigs (160M peak so far). But we will never see < 1M until the chain is current.
(edited)The initial ecdsa table allocation is 25GiB, and that hasn't been crossed (19.4GiB was the closest), so we're just reusing that buffer.
evoskuil23:24
Since we aren't yet capturing multisig/threshold, this is not yet efficient in total time. We are processing about 1/3-1/2 of all ecdsa sigs (
op_checkmultisig) in script. The use of a 1M sig threshold is pretty arbitrary, bigger may be better or worse. This is running behind the last trial I ran where the threshold was tied to the currency window. The triggering cost is def lower in that approach.- Total Signatures:
626,321,786 - Total Verification Time:
1,808 secs - Overall Average Throughput:
~346,400 sps
evoskuil23:36
Give
(edited)3,784,861,520 capturable sigs and this rate of 346,000 sigs/sec, that's 3.0 hrs of signature validation. This machine downloads and fully validates the chain in 4.9 hrs, and performs a milestone sync in 0.9 hrs. That means we are net burning 1.0 hrs reading blocks and prevouts from the store for validation, signature hashing, and setting states, and confirming.When batch validating we are consuming
95.5% of all CPU in the validation. So offloading this entirely to another processor (GPU) implies that on this old machine we should complete validation in ~2.0 hrs.And now this run has overtaken the record full validation time, at around block 600k.
(edited)Friday, June 19, 2026
evoskuil04:27
(1) passing cancellation. [done]
(2) managing ufsecp configuration. [done]
(3) fine control over batch sizing (configurable sigs count threshold). [done]
(4) generalizing multisig for threshold formulations.
(5) removing dead code (multisig table, turbo parameter). [turbo done]
(6) vertical table storage (4 physical tables per logical table).
Getting there, going to work next on the multisig formulation and then the table rotations.
evoskuil17:03
Batched (CPU on
(edited)secp) full validation run on mini was significantly faster than previous up to block 700k, but that could be due to previous optimizations, so I'm running again without batching.1 reply · evoskuil · last Jun 20, 2026
evoskuil16:47
Similar times here, but hard to compare due to the current batching implementation. The normal
node.maximum_height setting doesn't yet finish off the residual batch, which also prevents confirmations from completing. So I've done some work to ensure that a capped run completes all batching and confirmations as in unbatched implementations.evoskuil18:03
No, everything we've discussed so far sounds good. I'll try out some of the CPU/Windows performance improvements after soon.
evoskuil18:15
Merged the
(edited)bitcoind json-rpc and REST interfaces last night, thanks @Djuri !❤️ 1🎉 1🔥 2
1 reply · josie · last Jun 19, 2026
Sunday, June 21, 2026
evoskuil20:45
(4) generalizing multisig for threshold formulations. [done]
Now just this:
(6) vertical table storage (4 physical tables per logical table).
Monday, June 22, 2026
evoskuil02:51
Got the latest
ultrafast and did a non-batching run to 250k without checkpoints. So this is just a basic substitution of secp256k1, which is now configurable in the vs2006 IDE against @eynhaender’s published NuGet packages. Run time (Windows maxi) to 250k confirmed was 435 secs vs. the benchmark run of 410 secs. This is live Internet and a small number of blocks/sigs, so subject to some variation. But a respectable time at +6.1%. I'll do a full run validation tonight to get a solid comparison.
WRT batching, I'm running that in CPU mode as well, but as missing something in my config, so still getting only 1 out of 64 threads in ultrafast. The fallback batching, targeting chunks of 1 million sigs is slightly faster than basic algorithm (full validation parallel by block), which is nice to see. No loss at all in the marshaling, which is non-trivial, even some slight net gain.
Still running on horizontally-aligned tables, which is the last bug piece of work. Will be swapping that out for 4 tables for ecdsa and 4 tables for schnorr. Same storage cost, just more bookkeeping. But eliminates rewriting the entire table as it streams for batching, which at .5TB is significant.
Have also integrated cancellation, so that now works on ultrafast batching as well as our fallback batch implementation over secp256k1.
I completed folding the ecdsa single and multi-sig implementations into the same table, so we're capturing over 99% of all ecdsa sigs. More significantly I also completed the implementation of schnorr threshold signature scripts, which are much more complex to capture. I don't yet have a full capture rate total yet for schnorr, But I'd estimate around 90% based on previous scans, bringing the combined ecdsa/schnorr capture rate to 98% or so.These are the patterns we are capturing:
is_ecdsa_batchable --------------------------------------------------- chain::script::is_pay_public_key_pattern(ops) chain::script::is_pay_key_hash_pattern(ops) chain::script::is_pay_witness_key_hash_pattern(ops) chain::script::is_pay_multisig_standard_pattern(ops) is_schnorr_batchable --------------------------------------------------- chain::script::is_pay_taproot_key_path_pattern(ops) chain::script::is_pay_tapscript_single_pattern(ops) chain::script::is_pay_tapscript_timelock_pattern(ops) chain::script::is_pay_tapscript_inscription_pattern(ops) chain::script::is_pay_tapscript_multisig_pattern(ops) chain::script::is_pay_tapscript_threshold_pattern(ops)
ECDSA Capture
constexpr bool script::is_pay_public_key_pattern(
const operations& ops) NOEXCEPT
{
return ops.size() == 2
&& is_public_key(ops[0].data())
&& ops[1].code() == opcode::checksig;
}
constexpr bool script::is_pay_key_hash_pattern(const operations& ops) NOEXCEPT
{
return ops.size() == 5
&& ops[0].code() == opcode::dup
&& ops[1].code() == opcode::hash160
&& ops[2].data().size() == short_hash_size
&& ops[3].code() == opcode::equalverify
&& ops[4].code() == opcode::checksig;
}
constexpr bool script::is_pay_witness_key_hash_pattern(
const operations& ops) NOEXCEPT
{
return ops.size() == 2
&& ops[0].code() == opcode::push_size_0
&& ops[1].code() == opcode::push_size_20;
}
constexpr bool script::is_pay_multisig_standard_pattern(
const operations& ops) NOEXCEPT
{
if (ops.size() < 4u ||
ops.back().code() != opcode::checkmultisig)
return false;
const auto& m_op = ops.front();
const auto& n_op = ops[ops.size() - 2u];
// Standard form is restricted to minimal encoding.
if (!m_op.is_positive() || !n_op.is_positive())
return false;
const auto m = operation::opcode_to_positive(m_op.code());
const auto n = operation::opcode_to_positive(n_op.code());
if (is_zero(m) || is_zero(n) || (m > n) || (n > 16u) ||
(n != ops.size() - 3u))
return false;
// Standard multisig requires valid public key forms.
for (auto op = std::next(ops.begin()); op != std::prev(ops.end(), 2); ++op)
if (!is_public_key(op->data()))
return false;
return true;
}Schnorr Capture
// This script type is fabricated in witness::extract_taproot.
constexpr bool script::is_pay_taproot_key_path_pattern(
const operations& ops) NOEXCEPT
{
return ops.size() == 1
&& ops[0].code() == opcode::checksig;
}
constexpr bool script::is_pay_tapscript_single_pattern(
const operations& ops) NOEXCEPT
{
return ops.size() == 2
&& ops[0].data().size() == ec_xonly_size
&& ops[1].code() == opcode::checksig;
}
constexpr bool script::is_pay_tapscript_timelock_pattern(
const operations& ops) NOEXCEPT
{
return ops.size() == 5u
&& ops[0].is_nonnegative()
&& ops[1].is_timelock()
&& ops[2].code() == opcode::drop
&& ops[3].data().size() == ec_xonly_size
&& ops[4].code() == opcode::checksig;
}
constexpr bool script::is_pay_tapscript_inscription_pattern(
const operations& ops) NOEXCEPT
{
return ops.size() > 4u
&& ops[0].data().size() == ec_xonly_size
&& ops[1].code() == opcode::checksig
&& ops[2].code() == opcode::push_size_0
&& ops[3].code() == opcode::if_
&& ops.back().code() == opcode::endif;
}
constexpr bool script::is_pay_tapscript_threshold_pattern(
const operations& ops) NOEXCEPT
{
const auto size = ops.size();
if (is_zero(size))
return false;
// opcode::within requires an additional number in the script.
const auto within = (ops.back().code() == opcode::within);
if ((is_even(size) == within) ||
(size < (4u + to_int<uint8_t>(within))))
return false;
auto op = ops.begin();
if ((op++)->data().size() != ec_xonly_size)
return false;
if ((op++)->code() != opcode::checksig)
return false;
while (op != std::prev(ops.end(), 2 + to_int(within)))
{
if (((op++)->data().size() != ec_xonly_size) ||
((op++)->code() != opcode::checksigadd))
return false;
}
if (!(op++)->is_unsigned32() || (within &&
!(op++)->is_unsigned32()))
return false;
return op->is_threshold();
}
constexpr bool script::is_pay_tapscript_multisig_pattern(
const operations& ops) NOEXCEPT
{
if (ops.size() < 2u || !is_even(ops.size()))
return false;
auto op = ops.begin();
while (op != std::prev(ops.end(), 2))
{
if (((op++)->data().size() != ec_xonly_size) ||
((op++)->code() != opcode::checksigverify))
return false;
}
if (((op++)->data().size() != ec_xonly_size) ||
((op++)->code() != opcode::checksig))
return false;
return true;
}So right now all we are missing in terms of being able to test GPU batching against 98% of all signatures on the chain (with a fully configurable target batch window), is a machine to test it on. The horizontal->vertical table work is just an optimization for that scenario.
(edited)3 replies · josie, evoskuil · last Jun 22, 2026
josie07:35
I've got a modern GPU machine (linux) that I can use to test on. Just need the config to run. Its on 1GBPS hetzner. I also reached out to support to see if I could get a 10GBPS link, which I think would be interesting to test the theory of how offloading to a GPU frees up CPU
👍🏻 1
evoskuil14:30
I'm not sure where we are with the linux build integration, but otherwise it's ready to go.
josie15:24
heh, considering im building the full stack to test the updated silent payments PR on gpu, im sure ill figure out really quickly where its at
evoskuil14:38
Total run time as
secp1 replacement is 8.25 hrs vs. my benchmark of 4.85 hrs. So 194% of our normal runtime when not batching (just using the drop-in secp256k1 replacement). A short run was only 106.1% behind, but as blocks increase in size signature density increases and more significantly dominates the run.Tuesday, June 23, 2026
josie14:55
ive got a build running on my gpu box to test the silent payment index + scanning on gpu, i naively rebased my branches on master, and kicked off a sync but noticed its not using gpu. not that surprising, is the gpu batching only on your fork @evoskuil? or is this a config issue on my end?
(edited)if its on your fork i can rebase my branches there
1 million is a good target.
👍 1
Apart from that you need to have
WITH_ULTRAFAST defined (at least in -system build), include ufsecp headers vs secp, and link bs to ufsecp vs secp.In that case ufsecp both substitutes all secp and performs batching using its best available option.
These are independent behaviors but coupled for build simplification.
With that alone you should see small batches periodically varying wildly but clustered above 1M. This will reflect in both the default log/console and in the events file. The log will show capture rate and verification time and redundantly the events will show verification time.
Once the node is current, validation transitions to sequential by block. A last couple small batch(es) will close out the chain top, and if there is a maximum height configured the same will close that out.
This has been fully verified in all scenarios for a secp build. Ufsecp has some outstanding windows build issues, so while it works I don't have good numbers yet. Batching has significant issues that may just be perf, secp replacement for non-batched completes a full chain validation but is just about 2x the time.
In secp builds the batching operates over the table using a parallel loop, and is slightly faster than unbatched.
The batch tables are purged after every verification, and if non-zero are always coupled. The default table sizes should be configured, as otherwise they start at 1 byte and expand (mremap) at a default 5% rate from there. But once expanded they don't contract in allocation until node stop, so it's pretty efficient. I initialize to 10MB each. That are empty at shutdown after a complete sync. However they always start back up as configured, which could be optimized. Really we want them to physically shrink to 1 byte each after the node is current/recent, and stay that way at startup.
Stopping during a validation run will leave the tables populated, and they will be purged at startup. This means the cache is reaccumated for those blocks. This can be optimized to retain ecdsa (easily). Some more work for schnorr, so for now this is the simple solution.
Based on my test runs I would expect this to reduce sig validation on a 64 thread 2.1 GHz machine from 3 hrs to < 30 mins on a modest modern GPU.
At that point signature hashing is the next largest cost, which if you have shani, is 1/5 of the cost otherwise. Then the cost moves to I/O, which if you have enough RAM moves to confirmation, which is inherently sequential, so requires a fast CPU or more query optimization.
But on a 10Gbps network, with shani, a fast CPU, modern GPU, and enough RAM, full Internet sync validation should be well under 1 hr.
josie15:35
up and running, might need to do some param tuning but so far seems to be chugging through. this is 1gbps , but im requesting a 10 gbps link for this box
214 replies · evoskuil, josie, Eric · last Jun 24, 2026
evoskuil15:36
What sustained speeds do you actually measure? I read you get about .5 on this setup.
(edited)evoskuil15:38
You should see a milestone sync running at full bandwidth limit once it hits block 300k or so. Validation runs cycle between Network and CPU limits.
josie15:44
still working my way around parsing the logs, but:
15:28:32 5,366,900 sigs 3s 1.79M sigs/s 15:30:45 1,071,533 sigs 0s ~1.07M sigs/s 15:30:55 2,315,127 sigs 0s ~2.32M sigs/s 15:31:06 4,558,730 sigs 1s 4.56M sigs/s 15:32:16 29,354,060 sigs 8s 3.67M sigs/s 15:33:31 36,497,618 sigs 10s 3.65M sigs/s 15:37:20 1,468,875 sigs 0s ~1.47M sigs/s 15:38:18 5,883,209 sigs 1s 5.88M sigs/s
🔥 1
josie16:08
heh yeah, just shut down and im trying to optimise my network config a bit and also give a bigger batch
evoskuil16:08
The secp batch rate on my machine is 360K sigs/sec, and you are almost 10x that at 3.33M sigs/sec.
evoskuil16:09
Great to see that it actually works. Now I have some motivation to work on it today, and to buy my new machine.
josie16:11
i dont really understand all the nobs in the network config, do you have a recommended one youve been using for your fast benchmarks ?
evoskuil16:12
In frigate are you passing the data to the GPU in horizontal rows, and are you using the 8 byte prefix and compressed keys?
evoskuil16:12
The defaults are pretty good for network. Give me the high level specs and I'll make some suggestions.
josie16:13
frigate uses duckdb as the embedded database. duckdb supports GPU acceleration and writing custom in database extensions
josie16:15
here is the relevant config:
[network] threads = 24 [node] threads = 24 maximum_concurrency = 300000 batch_signatures = 10000000 warn_dirty_background_ratio = 90 warn_dirty_ratio = 90
josie16:18
outbound:
[outbound] connections = 1500 connect_batch_size = 1 host_pool_capacity = 200000 seeding_timeout_seconds = 420
evoskuil16:18
I'd leave maximum_concurrency at the default of 500k. batch to 5M, ratios are just startup warnings... you need to make sure those settings are in effect on linux. matching threads to physical for node is a good start, it's for full validation and performs the block/prevout reads in parallel. With sigs no longer hitting the CPU this might go higher.
evoskuil16:19
batch size is too low, leave default of 5, since only 1 in 5 addresses is valid on average. otherwise you delay your reconnection time significantly, which increases download delays.
(edited)👍 1
evoskuil16:20
Host pool probably doesn't need to be increased, but also prob won't matter much. Just uses more ram and increases runtime circular buffer cost.
evoskuil16:21
If you aren't getting seeded in time, this is the right number to change. It won't actually delay startup, since once you have enough addresses the node starts while you finish seeding. It just delays stop if seeding fails. So if you need to increase this it's harmless.
evoskuil16:22
I leave network threads at 16 even on my 64 thread machine with 256GB RAM on my 2.5Gbps network. I did a bunch of trail runs and this was a sweet spot.
evoskuil16:25
node.delay_inbound = false native.bind = 127.0.0.1:8080 database.ecdsa_size = 10000000000 database.schnorr_size = 10000000000(edited)
evoskuil16:27
I also use this on big machines:
[database] header_rate = 15 txs_rate = 15 tx_rate = 15 point_rate = 15 ins_rate = 15 outs_rate = 15 input_rate = 15 output_rate = 15 candidate_rate = 15 confirmed_rate = 15 duplicate_rate = 15 prevout_rate = 15 strong_tx_rate = 15 validated_bk_rate = 15 validated_tx_rate = 15 filter_tx_rate = 15 filter_bk_rate = 15 address_rate = 15 ecdsa_rate = 15 schnorr_rate = 15
evoskuil16:27
The percentage table growth rates. Requires more disk overall due to up to 15% pad when synced.
(edited)evoskuil16:43
It polls but once the node is current it transitions to websocket push updates.
(edited)josie16:45
but in theory, unless i fucked up the code, you should be able to point sparrow at this API when finished and get a silent payments gpu scan
josie16:47
yeah it was faster on a previous run. im wondering if the batch size being so large is stalling out since early blocks dont have many sigs?
josie16:54
yeah was just looking into it looks like zfs was grinding to a halt for a second, not sure why
josie16:56
this box was formly a prod box for frigate + fulcrum, where we wanted to be able to quickly recover index data without resync, but for a bench test, ZFS is overkill and a slowdown. tomorrow ill rebuild the box with just good ol nvme ext4
josie16:58
page settings:
vm.dirty_ratio = 95 vm.dirty_background_ratio = 95 vm.dirty_bytes = 0 vm.dirty_background_bytes = 0 vm.dirty_expire_centisecs = 12000 vm.dirty_writeback_centisecs = 12000 SwapTotal = 0 SwapFree = 0
josie16:59
yeah i really just think that was a ZFS fluke , which should be solved by taking that out. zfs is adding CoW, compression, etc
josie17:02
[bitcoin] milestone = 0000000000000000000000000000000000000000000000000000000000000000:0 [database] address_rate = 15 candidate_rate = 15 confirmed_rate = 15 duplicate_rate = 15 ecdsa_rate = 15 ecdsa_size = 10000000000 filter_bk_rate = 15 filter_tx_rate = 15 header_rate = 15 input_rate = 15 ins_rate = 15 output_rate = 15 outs_rate = 15 path = /var/lib/libbitcoin-server/blockchain point_rate = 15 prevout_buckets = 1000000 prevout_rate = 15 prevout_size = 4000000000 schnorr_rate = 15 schnorr_size = 10000000000 silent_buckets = 1 silent_rate = 50 silent_size = 4000000000 silent_start_height = 709632 strong_tx_rate = 15 turbo = true tx_rate = 15 txs_rate = 15 validated_bk_rate = 15 validated_tx_rate = 15 [electrum] bind = 0.0.0.0:50001 maximum_history = 1000000 maximum_subscriptions = 1000000 self_bind = 136.243.9.246:50001 server_name = libbitcoin-gpu-test silent_payment_threads = 20 [inbound] bind = 0.0.0.0:8333 [log] path = /var/log/libbitcoin-server [native] bind = 136.243.9.246:8180 connections = 16 [network] threads = 20 [node] batch_signatures = 5000000 delay_inbound = false maximum_concurrency = 500000 threads = 14 warn_dirty_background_ratio = 90 warn_dirty_ratio = 90 [outbound] connect_batch_size = 5 connections = 100 host_pool_capacity = 10000 seeding_timeout_seconds = 420 [peer] enable_relay = false
josie17:06
i strongly suspect this is related to the higher GPU batch size , when i had it 1M, it validated from 0-400k in 30 mins
evoskuil17:08
I generally set milestone to genesis, so that it is found. I doubt this makes a difference however. Though otherwise it keeps looking for it.
evoskuil17:09
There is no batching at all in the first 295k blocks, so it wouldn't make any difference in that range.
evoskuil17:10
You have a 50k window, so it would still be looking at the same range regardless of the batch size.
evoskuil17:11
Confirmation not proceeding makes sense, as this is not coupled to other modes. If you got 295k fast and only waiting on confirmation, that's normal. I'm not used to watching this in the browser.
josie17:13
here is what im seeing in the logs that makes me think theres a race:
block_confirmed max=295001
block_organized max=295001
block_validated max=363037
but 295002 is archived, from the logs
josie17:15
because the GPU tables drained, but confirmation is still stuck at 2950001, despite us having 295002, the node kept downloading and eventually bricked
evoskuil17:19
because the GPU tables drained, but confirmation is still stuck at 2950001
confirmation is loosely coupled to download/validation, it does not delay them, and if CPU is at 100% (common in full validation) then it will be deferred. Also during high CPU logging will be deferred.
(edited)evoskuil17:19
despite us having 295002, the node kept downloading
So this is perfectly normal.
(edited)josie17:24
youre right, the server getting killed seems unrelated, systemd sigkilled it after telling it to stop, which was likely initiated by another nix service on the box for mem usage
👍🏻 1
evoskuil17:33
I haven't implemented handling for all stop signals on linux/mac, just <ctrl-c>. logoff/shutdown signal handling for windows is implemented, but service (daemon) registration isn't. These are some release punchlist items.
evoskuil17:36
So if it was killed on this run but not the previous runs, it likely means it was running quite a bit faster. The OS panics when the memory acquisition rate is high.
(edited)josie17:42
i think i may have found the issue. perhaps race condition isnt the best way to frame it but it does look a deadlock can happen. https://github.com/libbitcoin/libbitcoin-node/blob/75fd039ba9a51951e1542ef78a97c26167ae63bd/src/chasers/chaser_validate_batch.cpp#L52 the comment says batched_ is only a sort optimisation, but process_valids() uses batched_ afaict as the only source of blocks to mark valid and then clears it
josie17:43
it seems a block whose rows were purged but whose link missed process_valids() remains unvalidated, so get_validated_fork() stops at that height and the confirm chaser never advances beyond it, which would match what i saw with the stuck block at 295002
evoskuil17:44
It's redundant with the results from the batching output, but if we use the batch output results for positive and negative, we need to merge the results from the ecdsa and schnorr tables. On order to prevent duplicated validated events and store updates, that requires a merge/sort of the two result sets.
evoskuil17:44
So we reduce the results to negative only, and use this for the success set (which is always 100% of the results).
evoskuil17:45
evoskuil17:46
This is all done in the batch verification critical section:
https://github.com/libbitcoin/libbitcoin-node/blob/75fd039ba9a51951e1542ef78a97c26167ae63bd/src/chasers/chaser_validate_batch.cpp#L86
which prevents any batch accumulation.
evoskuil17:47
I've run this quite a few times over the whole chain and haven't seen any issue with it.
evoskuil17:49
I think you are assuming that the confirmation chaser is hung, but there's really no reason to believe that's the case.
josie17:50
i feel like im not explaining myself well. the problem i suspect is this:
{
std::shared_lock lock{ mutex_, std::defer_lock };
if (capturing) lock.lock();
if ((ec = block.connect(ctx, capture)))
return ec;
batched = capture.batched;
faulted = capture.faulted;
}
this code acquires a lock but releases it before the block link is added in
batched_ . it seems that the chaser can acquire a lock after the rows are written but before the blocks push_batch() handler has ran. so it verifies and purges the rows but only links in batched_ are marked as validjosie17:51
that doesnt seem to match what the comment says and implies to me the signature row can be visible before batched_ is aware of it , but the final decision is made off of what batched_ is aware of
josie17:52
its still unclear to me why the batch size would make a difference here, because i definitely did not see this with 1M batch size
evoskuil17:53
batched_ is protected by the strand. Any pending update to batched is added after the critical section completes, because it is also stranded.josie17:53
tldr; process_batch() can verify and purge a block’s signature rows before that block’s delayed push_batch() adds its link to batched_, so process_valids() never marks the block valid.
josie17:54
this leaves the block stuck unvalidated with no batch rows left to reprocess, so confirmation stalls at that height even though later batches keep completing
evoskuil17:54
This of course means that batched_ can be updated after it's accumulated sigs are verified. This is fine.
evoskuil17:55
It may be a problem in the very last ("residual") batches of the run, but you didn't encounter that.
evoskuil17:56
The critical section prevents updates to the tables during evaluation, since otherwise we would have to truncate the table above the writes.
evoskuil17:56
The batch is not coupled to the tables, it's a list of all blocks that have been accumulated. They don't have to process at the same time.
josie17:58
i think we are talking past each other, i still see an issue: what youre saying is only true if batched is not part of the commit part , but it that doesnt seem to be true
evoskuil17:58
It's only essential that the tables are processed before the batch records that accumulated them, so that invalidation preceeds validation.
josie17:59
// set_block_valid(batched_ excluding ecdsa/schnorr failures)
// ------------------------------------------------------------------------
if (!process_valids(residual))
{
fault(error::batch4);
return;
}evoskuil17:59
I understand what you are saying, and it's a good question. But I don't think you are following how message flow is is being processed in the strand.
josie17:59
and the callee:
for (const auto& link: batched_)
{
if (link == header_link::terminal)
continue;
size_t height{};
if (!query.get_height(height, link) ||
!query.set_block_valid(link))
return false;
notify_block(system::error::success, height, link, false);
}
batched_.clear();josie18:00
but at least from what i do understand, this looks like a smoking gun to me and would explain the stalled block i just saw
evoskuil18:01
The process was terminated, which means very high resource consumption. That means confirmations were not happening.
(edited)evoskuil18:02
Confirmation only proceeds in the CPU gaps, once validations are contiguous, and the queries are fairly expensive. So it is common for there to be significant delays.
(edited)evoskuil18:03
Also such activity depends on the proactor pump delivering messages, which affects logging, and eventing as well. So under heavy load logging can be significantly delayed.
josie18:04
i remain unconvinced. will try to reproduce with a test and either humbly admit defeat or gloat in victory 🍺
👍🏻 1
evoskuil18:15
Think of it this way... there is a continuous flow of values into a table, and into a vector.
The table writes are concurrent by block and at the granularity of tx outputs. The vector writes are sequential (stranded) by block (input would be way too much to message-buffer, blocks is easy/slow). The vector is only submitted to the message pump after the table writes for that block are committed.
The critical section only makes write and verify/purge mutually-exclusive (and since verify is extremely rare, there is no meaningful contention).
The strand will deliver batched messages to the vector at any time, but cannot do so during verify, because both are on the same strand. So batched cannot be increased during verify, and is purged at the end of verify.
So a latent block id making its way through the message pump may land on batched after the table(s) that contained the records for that same block are purged. This doesn't matter. The tables only carry verification failure, so if the block failed, it's a reset.
This latent block id will land once verification completes and has purged both the tables and the batched_ vector. This is fine. The block will get marked as valid on the next verification.
The final (residual) batches ensure that both the tables and the batched_ vector are processed in full before batching gives way to non-batching.
(edited)evoskuil18:20
The most challenging part of this was that last step... specifically because of the issue you are raising.
josie19:00
haha racing this race (but actually stepped away for the evening , dinner ain't gonna cook itself). but i'm going to pick this up first thing tomorrow will bring up the server without ZFS
👍🏻 1
evoskuil19:05
I updated the batch post-processing to handle the valids in a parallel loop, so should be a little faster.
E
Eric00:12
The FreeBSD Forums
I'm finally reading the Second Edition of "The Design and Implementation of the FreeBSD Operating System" and found this passsage:
> The ZFS filesystem integrated from OpenSolaris is the one exception to the integrated buffer cache. ZFS has its own set of memory that it manages by itself. Files...
😯 1😵💫 1
josie10:31
@evoskuil alright alright, gotta eat humble pie. after reading your explanation carefully and reading the code, i agree that a later batch can recover a link that was placed after its rows were purged. so this was not a stall, and was instead libbitcoin and ZFS fighting for memory. i also had a leftover watchdog on the box that would kill the service using a lot of memory (this was a leftover from when i was running fulcrum and frigate on the server and didnt want things to get locked up)
👍🏻 1
josie10:43
i am glad i chased it down, tho, because i do think i have something. i kept looking at the code and something didn't sit right. so here is what i believe can happen:
imagine block
a has a bad sig. when validating `a` the bad signature row is written into the batch table under the batch mutex. But a’s block id is not added to batched_ at that moment, rather it is posted later to the strand via push_batch().
before push_batch() runs, block b finishes and triggers batch verification. verifier sees a’s bad signature row, correctly marks a unconfirmable, and purges the table. but because a’s id is not yet in batched_, the invalid link cleanup cannot mark a’s entry as terminal.
now a’s delayed push_batch() finally runs and appends a to batched_. On the next batch, process_valids() treats every non-terminal link in batched_ as valid, so a can be marked valid even though it was already marked unconfirmable for an invalid signature.
relevant code paths (not using line links because i also have some local changes):
// chaser_validate_parallel.cpp
{
std::shared_lock lock{ mutex_, std::defer_lock };
if (capturing) lock.lock();
if ((ec = block.connect(ctx, capture)))
return ec;
batched = capture.batched;
faulted = capture.faulted;
}
^ signature rows are written during block.connect(), but the block link is not added to batched_ here.
// chaser_validate.cpp
if (batched)
{
++batch_backlog_;
POST(push_batch, link, height);
return;
}
^ link posted later
// chaser_validate_batch.cpp batched_.push_back(link); process_batch(is_maximum());
^ link can arrive after another batch processes rows
// chaser_validate_batch.cpp
for (const auto& link: invalids)
{
if (!query.get_height(height, link) ||
!query.set_block_unconfirmable(link))
return false;
notify_block(system::error::invalid_signature, height, link, false);
}
std::ranges::replace_if(batched_, [&](const auto& link) NOEXCEPT
{
return contains(invalids, link);
}, header_link::terminal);
^ this only removes invalid links that are already present in batched_
// chaser_validate_batch.cpp
for (const auto& link: batched_)
{
if (link == header_link::terminal)
continue;
if (!query.get_height(height, link) ||
!query.set_block_valid(link))
return false;
notify_block(system::error::success, height, link, false);
}
^ and lastly later marks every non terminal batched_ link valid, even if that link arrived late and the block had already been marked unconfirmable.
(edited)josie10:54
here is the test i used to try and reproduce this:
BOOST_AUTO_TEST_CASE(chaser_validate__process_batch__late_invalid_batched_link__does_not_become_valid)
{
configuration config{ chain::selection::mainnet };
config.database.path = TEST_DIRECTORY;
config.node.batch_signatures = 1;
const auto events = [](auto, auto) NOEXCEPT {};
store store_{ config.database };
query query_{ store_ };
BOOST_REQUIRE(test::clear(TEST_DIRECTORY));
BOOST_REQUIRE_EQUAL(store_.create(events), database::error::success);
BOOST_REQUIRE(query_.initialize(config.bitcoin.genesis_block));
const chain::block invalid_block{ block1_data, true };
const chain::block trigger_block{ block2_data, true };
const chain::block valid_block{ block3_data, true };
BOOST_REQUIRE(query_.set(invalid_block, database::context{ 0, 1, 0 }, false, false));
BOOST_REQUIRE(query_.set(trigger_block, database::context{ 0, 2, 0 }, false, false));
BOOST_REQUIRE(query_.set(valid_block, database::context{ 0, 3, 0 }, false, false));
const auto invalid_link = query_.to_header(invalid_block.hash());
const auto trigger_link = query_.to_header(trigger_block.hash());
const auto valid_link = query_.to_header(valid_block.hash());
BOOST_REQUIRE(query_.push_candidate(invalid_link));
BOOST_REQUIRE(query_.push_candidate(trigger_link));
BOOST_REQUIRE(query_.push_candidate(valid_link));
// invalid_link has a bad sig row, trigger_link has a good sig row.
BOOST_REQUIRE(query_.set_signature(sighash_bad, ecdsa_compressed,
ecdsa_signature, 0, invalid_link));
BOOST_REQUIRE(query_.set_signature(ecdsa_sighash, ecdsa_compressed,
ecdsa_signature, 0, trigger_link));
BOOST_REQUIRE_EQUAL(query_.ecdsa_records(), 2u);
code invalid_state{};
code trigger_state{};
code valid_state{};
{
const network::logger log{};
full_node node{ query_, config, log };
batch_test_chaser chaser{ node };
// trigger_link push_batch() processes both table rows before
// invalid_link delayed push_batch() has appended invalid_link to batched_
chaser.complete_batched(trigger_link, 2u);
chaser.drain();
BOOST_REQUIRE_EQUAL(query_.ecdsa_records(), 0u);
BOOST_REQUIRE_EQUAL(query_.get_block_state(invalid_link),
database::error::block_unconfirmable);
BOOST_REQUIRE_EQUAL(query_.get_block_state(trigger_link),
database::error::block_valid);
// invalid_link arrives late into batched_ after its bad row was purged
chaser.complete_batched(invalid_link, 1u);
chaser.drain();
// later good batch causes process_valids() to iterate the stale
// invalid_link and call set_block_valid() on it.
BOOST_REQUIRE(query_.set_signature(ecdsa_sighash, ecdsa_compressed,
ecdsa_signature, 0, valid_link));
chaser.complete_batched(valid_link, 3u);
chaser.drain();
invalid_state = query_.get_block_state(invalid_link);
trigger_state = query_.get_block_state(trigger_link);
valid_state = query_.get_block_state(valid_link);
}
BOOST_REQUIRE(!store_.close(events));
// this fails on upstream: invalid_state becomes block_valid.
BOOST_CHECK_EQUAL(invalid_state, database::error::block_unconfirmable);
BOOST_CHECK_EQUAL(trigger_state, database::error::block_valid);
BOOST_CHECK_EQUAL(valid_state, database::error::block_valid);
} (edited)josie10:56
seems the simple enough to fix, tho, i tried this:
diff --git a/src/chasers/chaser_validate_batch.cpp b/src/chasers/chaser_validate_batch.cpp
index a269929f..9b8cfda7 100644
--- a/src/chasers/chaser_validate_batch.cpp
+++ b/src/chasers/chaser_validate_batch.cpp
@@ -210,8 +210,13 @@ bool chaser_validate::process_valids(bool residual) NOEXCEPT
continue;
size_t height{};
- if (!query.get_height(height, link) ||
- !query.set_block_valid(link))
+ if (!query.get_height(height, link))
+ return false;
+
+ if (query.get_block_state(link) == database::error::block_unconfirmable)
+ continue;
+
+ if (!query.set_block_valid(link))
return false;
notify_block(system::error::success, height, link, false);josie11:03
could still be off base tho! the tricky thing about trying to repro behaviour like this in the test is its too easy to do things in a way that doesnt match how the code actually runs, so would appreciate eyes on this
josie11:47
another interesting insight: the GPU i have seems to degrade at higher batch sizes, e.g., 60-70M and it was 3.51M sigs/sec, but at 300M it was down to 2.3 - 2.5M sigs/sec. this seems dependent on the backlog of inflight blocks coming in, so not really something i can tune for. working on a patch that would make it much tighter to the actual GPU batch size set by the user , because sometimes too large of a batch size can make the GPU hang
josie12:20
i think i got a decent way to control batch size without locking everything up, im running that now and its looking like it will finish full validation in < 2.5 hours
evoskuil13:18
The engine should be controlling the batch size that the GPU sees. What we send gets chunked up internally. So it really should not matter if the batch we send is large.
(edited)josie13:19
that doesnt seem to be the case, or rather, the work that the engine needs to do is creating a slowdown.
josie13:19
once i got into the later blocks i was seeing slowdowns in gpu processing correlating with larger sig batches being sent in the logs
evoskuil13:22
That would be a bridge or engine problem. All we pass to the bridge is a pointer and row count.
evoskuil13:22
evoskuil13:23
The issue you described above is correct, I mentioned this yesterday and commented it here.
evoskuil13:25
The resolution you show above is correct but not ideal, as that creates a hashmap search for every confirming block.
evoskuil13:26
1 million hashmap queries on the block validated table is not catastrophic, but ideally we want to avoid that when we can.
evoskuil13:30
Tighter control over the sig accumulation would be nice, as long as it doesn't come at the cost of increased contention. Generally the size of the table shouldn't matter except as it may increase paging and mremap costs. If you are seeing significant perf reduction at larger batch sizes, in the verification, take a look at whether that's coming from paging. If it's not then it should be manageable in the bridge/engine.
josie15:07
ah nice! i missed the bug explanation in the convo yesterday, but admittedly wasn't reading as carefully as i could have been. sadly i was hoping to have a sip of the celebratory beer , but doesnt count if the bug is known 😕
evoskuil15:18
Well the discussion led to me seeing it, so partial credit. The state search isn't a good call in this situation mainly because it would impact the hot path whereas the problem it would resolve only hits the ice cold path (~never happens).
evoskuil15:20
i think i got a decent way to control batch size without locking everything up
Lemee know what the idea is and I'll give some feedback.
(edited)👍 1
evoskuil15:25
//...We also don't want to read block state before // every write of valid state.
This was a ref to your proposed solution. So good insight and valid solution, for a very complex interaction. 👍🏻
❤️ 1
josie15:28
to the preposed solution i had for batch sizing, (afk so keeping it short):
target reached -> seal -> drain admitted writers/links -> verify/purge -> reopen
josie15:29
which obviously comes with some performance hits, im running with it now to see if i noticed anything and will keep tinkering. i think it will be super small tho
josie15:33
altho i am noticing some IO contention / paging which makes me wonder if ive got the machine set up correctly / appropriately sized (only 62GB). will let this finish and run again tomorrow
evoskuil15:38
You will definitely have a significant amount of paging. The chain is 800GB and you have 62GB RAM. All of the blocks are written (as in milestone) and then read (with prevouts) for validation. With enough RAM this goes away entirely, but otherwise it's the tradeoff that allows for concurrent validation.
evoskuil15:39
But WRT batch verification the question is really just whether the large batch slowdown is the result of paging, or is it from the engine.
josie15:39
oh yeah, on that im not sure. now that i see the paging here, im guessing its probably the result of paging, and perhaps the engine is fine. as you said, you're just passing a pointer
evoskuil15:40
which obviously comes with some performance hits
Yes, blocking the validator is a significant performance hit. In fact I'm planning to remove the critical section once I've got a mmap top prune feature added to the tables.
evoskuil15:41
The batch tables can continue to be written while batch verifying, the only reason I lock during verify is so that we can purge the entire table.
evoskuil15:41
but by controlling the batch size, i am consistently seeing 5M/s which is what i wanted
Sure, that's easy to achieve, but the lock isn't good.
evoskuil15:42
mmap doesn't provide a simple way to purge the front of a map, which is what we need for this (sort of circular buffer).
(edited)evoskuil15:43
The size of the batch shouldn't be too important. The purpose of the configured value is to just set a minimum.
evoskuil15:44
If a lower average tends to outperform, then I would just reduce the configured minimum.
josie15:45
i had considered that , but i mostly wrote this as a quick patch to ensure something else wasnt going on with my gpu
evoskuil15:45
I work hard to keep the implementation lock-free, so even though this lock is cheap, it does suspend all validation while the batch verify is running, which given that verify runs on GPU and validation runs on CPU is not ideal.
(edited)evoskuil15:47
also i set batch_size, i want it to be exactly that haha
Eventual consistency is a necessary design principle for efficient concurrent code.
(edited)evoskuil15:49
WRT the hashmap query... you will notice that we use query quite liberally. Often I'll use a query as opposed to moving around a captured value that we already have...
evoskuil15:50
That's because most queries are direct navigation to a memory address that is 99.999% likely already paged in.
evoskuil15:51
So it's important to make this distinction WRT what is a hashmap query. If the query operates over a table with a large body then it can get very costly, as it potentially results in paging across the full body, as the conflict resolution linked lists are in the body.
(edited)evoskuil15:52
One of the big no-no's is reading before writing. This was initially evaluated, such as for txs. The cost is prohibitive.
josie15:54
oh looking at the the design i posted above again, the lock has nothing to do with batch table size control, rather that was for the wedged block scenario i saw in the first place. it was happening again, subsequent batches were not clearing it out. as i understand it, your argument is residuals would have eventually cleared it out, but then i started noticing io pressure (mmap), and thought id try this run and see if that fixed the problem
evoskuil15:54
We do however have a feature in the hashmaps that allows us to determine with very high probability if there is a conflict (potential existence), using a distributed bloom filter in the hashmap header. It's just integrated into the base hashmap heads, but isn't sufficiently allocated to be useful except for the points table.
(edited)josie15:54
however, (while not seeing the wedged block), i am still running into the io pressure which leads me to believe the machine is just under resourced and the wedged block wasnt really the issue
evoskuil15:55
"residuals" in the code refers only to the tail end (< configured size) of the entire run.
(edited)evoskuil16:00
The behavior associated with the valid-after-invalid issue would not cause a delay, it's no different than if that one block had been validated in the next batch. Since the batches aren't ordered, it's not consequential.
evoskuil16:02
The efficient resolution of that issue is to create a sidecar vector of invalids, as a member of the chaser. Then just push any invalid, strand protected, and check that list before setting valid. That makes the hot path check a free test against an empty vector.
evoskuil16:03
> oh i think they are separate things (valid after invalid vs delay)
They are, the delay is the natural course given how default priorities are set.
(edited)evoskuil16:04
That sidecar could be cleared at a coalescence point, but really not worth doing.
(edited)evoskuil16:07
Note that I see significant degradation in ultrafast CPU verification when the machine is stressed. It goes from 75% to 200% of secp256k1. This is not a paging issue, as it happens with individual sigs on otherwise identical runs and when there is over 100GB of free RAM. But I haven't measured GPU.
(edited)evoskuil16:09
I've assumed it's the Windows path, or it's limited to CPU, but maybe not. But it's not limited to the batching scneario.
(edited)josie16:10
secp256k1 does everything on the stack , no heap or memory allocation. i wonder if that could be a different with ultrafast and making a difference on a stressed machine ?
evoskuil16:11
It could be the batch horizontal-vertical rewrite getting disproportionately more costly, but we'll have that removed shortly.
(edited)evoskuil16:13
Yeah, in the unbatched run it can't be tied to batch size... I'm just referring to the direct drop-in replacement for
secp256k1.evoskuil16:14
It gets through millions of sigs and is beating secp by -25%, and then falls behind to complete at about +50-100%.
(edited)evoskuil16:18
bool chaser_validate::process_invalids(const header_links& invalids) NOEXCEPT
{
BC_ASSERT(stranded());
auto& query = archive();
for (const auto& link: invalids)
{
size_t height{};
if (!query.get_height(height, link) ||
!query.set_block_unconfirmable(link))
return false;
invalids_.push_back(link);
notify_block(system::error::invalid_signature, height, link, false);
}
// Set all invalid links in batched_ to terminal (to be skipped).
if (!invalids_.empty())
{
std::ranges::replace_if(batched_, [&](const auto& link) NOEXCEPT
{
return contains(invalids_, link);
}, header_link::terminal);
}
return true;
} (edited)evoskuil15:41
Forgot to mention that you may need to build ufsecp with PP defines for the target GPU libs.
👍 1
Thursday, June 25, 2026
Huddle with pmienk, eynhaender, Eric Chennells, Rowan Voskuil, evoskuil, josie, Enligths · 218 min15:53
6 replies · evoskuil, eynhaender, josie · last Jun 25, 2026
josie18:56
gotta run now, but here is the current full validation sync (it does have shani):
Current status: - Runtime: 1h51m - Latest completed boundary: 850000 - Current tip: 955376 - Remaining: 105k blocks - Progress by height: 89% Recent 50k windows: 700k -> 750k: 13m43s 750k -> 800k: 13m50s 800k -> 850k: 13m35s Estimated finish: 21:22-21:28 CEST. Estimated total runtime: about 2h20m-2h26m.
evoskuil19:04
2026-06-23T21:33:16Z.0 intel... platform:1. 2026-06-23T21:33:16Z.0 avx512.. platform:0 compiled:0. 2026-06-23T21:33:16Z.0 avx2.... platform:1 compiled:1. 2026-06-23T21:33:16Z.0 sse41... platform:1 compiled:1. 2026-06-23T21:33:16Z.0 shani... platform:0 compiled:0.
Friday, June 26, 2026
evoskuil16:39
https://uncensoredtech.substack.com/p/a-pruned-node-reorg-ub-in-core-310
So I guess we’re out of consensus again.
Saturday, June 27, 2026
evoskuil18:06
Completed a database generalization that allows a set of mmaps (files) to be defined and configured as one set, while retaining fluent names to navigate to the individual files. So the set abstraction can hide the distinction for nearly all of the overhead, while keeping the detail is conveniently accessible.
Did the same for tables that would use the multiple file mappings. A table combines a head and body. So for a four column SoA (vertical) table there are 8 files (mmaps) as 4 heads and 4 bodies, with corresponding programmatic names and file names. The table abstraction similarly allows this to be defined and configured as one table, while retaining fluent names to navigate to the individual columns.
These changes allow 11 tables, with 22 files to be represented as 3 tables with 6 files. And if we want to split out multisig from ecdsa and schnorr, that becomes 5 tables over 10 files, vs. 19 tables over 38 files. The actual tables and files are still there, just a lot less code overhead, thanks to the magic of metaprogramming.
(edited)I've also updated the store implementation of silent payment as vertical tables and integrated that back to libbitcoin-system, where the integration with secp exists. So that's all ready for the
ufsecp and secp scan integration.Also completed the vertical table defines for
ecdsa and schnorr. Just need to update tests and adjust libbitcoin-system secp to accept the vertical layout (and pass it to ufsecp), and we'll soon be batching without .5TB of transformation.🚀 3
Sunday, June 28, 2026
evoskuil03:07
I'm merging updates to system and database incrementally, so signature batching will be broken for a bit (default is disabled).
Vano Chkheidze12:01
i will update you soon i was very bussy this 2-3 days will finish my part and will ping you
👍🏻 1
evoskuil14:55
Did a maxi milestone run last night, just to keep tabs on it. New record of
39.4 mins.🚀 1
evoskuil19:47
This morning I did the work to incorporate signatures in batch tables on startup, including following a snapshot recovery.
Kevin Ravensberg20:20
I've tasked Claude to build libbitcoin for me on an external SSD, let's see how far I get first time 😄
🔥 1
Or the top of the log if it's running:
2026-06-28T20:13:35Z.0 Version Information... libbitcoin-server: 4.0.0 libbitcoin-node: 4.0.0 libbitcoin-network: 4.0.0 libbitcoin-database: 4.0.0 libbitcoin-system: 4.0.0 2026-06-28T20:13:35Z.0 intel... platform:1. 2026-06-28T20:13:35Z.0 avx512.. platform:0 compiled:0. 2026-06-28T20:13:35Z.0 avx2.... platform:1 compiled:1. 2026-06-28T20:13:35Z.0 sse41... platform:1 compiled:1. 2026-06-28T20:13:35Z.0 shani... platform:0 compiled:0.
Kevin Ravensberg20:34
(base) PS D:\libbitcoin\node> bs -d
intel... platform:1.
avx512.. platform:0 compiled:0.
avx2.... platform:1 compiled:1.
sse41... platform:1 compiled:1.
shani... platform:1 compiled:1.
👍🏻 1
I want to have the native websocket active, so claude is now restarting libbitcoin and failing xD
evoskuil20:36
I think default is 80. if you change it to 8080 you may be ok without a firewall change.
some example config...
[native] # ============================================================================= # 80/443 (use default) # native RESTful interface (http/s, stateless html/json) bind = 127.0.0.1:80 bind = [::1]:80 safe = 127.0.0.1:443 safe = [::1]:443 key_pass = libbitcoin key_path = m:/certs/server/server-ecc384-key-enc.pem cert_path = m:/certs/server/server-ecc384-cert.pem connections = 10
There should be an error in the log at startup for the port being restricted if it was failing at startup when you enabled the interface.
on an external SSD
external not generally good.
Also Claude managed to crash itself hehe, by libbitcoin taking all the free system resources
42 replies · evoskuil, Kevin Ravensberg · last Jun 29, 2026
evoskuil00:25
Taking "free" resources can't hurt another application, because those resources are free.
evoskuil00:30
Clod must have some assumptions about how quickly it can get things, since those things are always available, apart from a disk being full. That implies a race condition.
Kevin Ravensberg00:43
I know that forcing the other apps into swap file on disc will hurt their performance. So it is good to leave a little of it (1~2%) free.
evoskuil00:45
That of course is the fundamental job of the operating system, and it does this. I am performing a full validation sync right now on this machine. Shows 100% CPU and RAM is 116GB used and climbing. It will hit almost 256GB (the physical RAM) and the machine will continue to be responsive - less so than normal, but that's intended.
(edited)evoskuil00:46
I'm on Windows. It's far better than macos in this regard, and much better than Linux, as it properly dynamically manages these resources.
(edited)evoskuil00:48
No, I don't know what's the problem. I don't have much info. But I have < 1hr syncs on 32GB minis on Windows.
evoskuil00:52
Normal when you are trying to do something as fast as possible with a scalable impl is 100%
evoskuil00:55
The blocks are buffered in RAM and the oldest accessed pages of that RAM are written in bulk to the disk by the OS as necessary.
(edited)evoskuil00:56
With a slow (external) SSD, it will take more time to move block data to the disk to free up memory for other things. The disk becomes a bottleneck.
evoskuil00:57
It's not the amount of RAM being used, it's the rate at which blocks are coming in relative to the rate at which they can go out. Your network is much faster than your disk. It's a flow problem.
evoskuil00:58
You can reduce the flow coming in by just reducing the number of peers or the number of network threads.
evoskuil01:03
will make a difference 🙂. It's a simple idea. To store the chain coming in at 2.5Gbps you have to be able to write to RAM at 2.5Gbps and when RAM is full you have to be able to write to disk at 2.5Gbps. Otherwise RAM will remain full and the backpressure hits the network.
evoskuil01:04
Of course CPU has to also be able to parse and shuffle data round fast enough to maintain those rates as well, or you become CPU bound.
evoskuil01:59
Yep, so is your network. There's always one bottleneck, and quite clearly that's presently your disk.
evoskuil02:00
That's a solid machine. With a fast internal SSD and 2.5Gbps you'll milestone sync in < 1hr.
(edited)
153 replies · pmienk, evoskuil, Kevin Ravensberg · last Jul 01, 2026
evoskuil21:40
Yes, and even on fast internal I'd expect to see that periodically on lower RAM machines.
Kevin Ravensberg21:43
Yeah, I resorted to unplugging the drive as there was nothing happening anymore and even explorer.exe was struggling
Kevin Ravensberg21:44
Maybe I'll grab another internal nvme, or hopefully I get the chance to buy the Steam Machine and do it on there 😄
👍🏻 1
evoskuil21:47
A thumb drive is plenty to run Core. If you want that speed you can just set one network thread.
😅 1
Kevin Ravensberg21:55
I don't think it was logged anywhere, now trying again with some memory cap on the process and logging with resource monitor
Kevin Ravensberg21:56
Well, after unplugging the SSD and stopping bs process I could restart explorer.exe and continue
evoskuil22:00
Yep, you can't halt the process and reuse the store, unless you have taken a snapshot to recover to. That happens automatically once the node is current, and you can command it at any time.
evoskuil22:00
That doesn't mean you synced 178GB, that's just the initial file allocations at the time of halt.
Kevin Ravensberg22:05
Yeah, is there a part that I can split off to the faster OS disc that is not the entire database?
evoskuil22:06
The store doesn't have configuration for independent file paths. Nothing too difficult about it tho.
evoskuil22:07
I considered putting the headers on their own path, they need high speed random access and the bodies need high speed write append (for milestone). Ahd the headers are much smaller.
(edited)Kevin Ravensberg22:08
Yeah, maybe machines with a very fast nvme OS-disc can benefit if the headers live there whilst the block body data goes to a slower larger drive.
Kevin Ravensberg22:11
The DDR4 memory kit I currently have costs 1000eu now new, bought it for 400 or so. So upgrading to 128GB is quite expensive.
👍🏻 1
Kevin Ravensberg22:15
I do have 2.5GB ethernet on this machine, so networkwise I might set a record
🔥 1
evoskuil22:16
I hit
39.4 mins on my 2.5Gbps Internet for milestone sync to 950k last night. Tough to beat, almost the theoretical limit.Kevin Ravensberg22:17
I need to buy an extra 10G network pcie card, because I have 8GB/s available on my fiber
evoskuil22:19
Looks like we're getting down closer to 2 hrs for full validation sync... we'll see soon.
evoskuil00:14
- confirmed and validated are two distinct states.
- archived and validated are not ordered.
- confirmed can have long delays and then race ahead (not constant rate at all).
👍 1
evoskuil00:19
If you give me some info I can let you know what's going on, but this doesn't tell me much.
evoskuil05:02
Overcommit is the operating system making promises it knows it cannot keep. OOM kill when swap is available is fundamental unreliability. That unreliability is a consequence of an ancient primitive that long ago lost its usefulness, and has corrupted every flavor of Unix that followed. It's so pervasive that people have entirely forgotten what is the fundamental contract of an operating system.
👀 1
Kevin Ravensberg19:53
Update: My PC 'froze' overnight, but this time after a successful snapshot, so it's continuing IBD now 😛
*froze as in the password field to login to winblows had disappeared and I could not force open a terminal window like I normally could.
Kevin Ravensberg17:55
I told it to do half way or so, but turns out it's better not to snapshot half way IBD.
Kevin Ravensberg18:39
Well, partially unresponsive. The terminal with libbitcoin is unresponsive and new terminals likewise. The snapshot tool worked, other running apps worked. Restarting seems to not work either in this state.
Kevin Ravensberg18:42
My personal log file of a monitoring script that periodically looked at the progress
evoskuil18:43
That's not my question. I'm just trying to understand from where you are getting the progress indications for the node.
(edited)Kevin Ravensberg18:44
I'll look it up in the script, don’t remember if it is checking the localhost webinterface or terminal output
evoskuil18:46
the web interface is disabled by default until current, and the console does not emit block status by default, and emitting it would be far too verbose for IBD.
evoskuil18:46
It may be monitoring the event log file, but logging is also low thread priority and buffered.
Kevin Ravensberg18:48
2. Sampler:
monitor-ibd.ps1
A PowerShell loop (meant to run continuously in the background) that every 60s:
- Reads the tail of
events.log - Takes themax height seenfor each of the 4 stage names (events log out of order under concurrency, so max-over-tail is the safe aggregation)
- Appends one row to
D:\libbitcoin\node\logs\ibd-progress.csvwith awall-clock UTC timestamp— this is what makes it survive node restarts, sinceevents.log's own elapsed-seconds column resets each time the node restarts.
evoskuil18:48
Furthermore confirmations will always back up against validation, so if you are monitoring the log for confirmations, there can be long delays. If you're in a situation where the node is barely making progress due to resource constraints, then there may be no such progress indications.
💯 1
Kevin Ravensberg18:48
These four are tracked:
archived block_archived - downloaded + structural/PoW check
validated block_validated - full script/signature validation (heavy)
confirmed block_confirmed - validated AND confirmable
organized block_organized - pushed onto the confirmed chain (height)
Kevin Ravensberg18:49
I limited it to only use 24GB of RAM, as a mitigation of my PC getting sluggish
evoskuil18:49
Sure, that's a reasonable approach. The console log doesn't flush until its buffer is full, so the event log is the right place.
evoskuil18:51
Yes. I was just trying to isolate the question of process termination or hang, vs. expected behavior.
(edited)evoskuil18:51
This is expected behavior. You need to give it at least 32GB for validation, and a fast disk.
Kevin Ravensberg18:53
Working on that! And I probably should build from source once more to make use of the GPU validation?
evoskuil18:54
The process is not synchronous. It's highly concurrent and asynchronous. Things do not execute in lock step as you might intuit after using a single-threaded validator like Core. For example, a huge amount can be happening and nothing in the log.
evoskuil18:55
GPU validation is still a bit experimental, and ufsecp is not currently useful on Windows.
evoskuil18:56
When you say IBD are you including full validation? At 24GB that is not an accurate expectation.
evoskuil18:58
milestone sync (which is currently the default, at block 950k) with 32GB on 2.5 Gbps and 16 dual core decent threads takes 39 minutes.
(edited)Kevin Ravensberg18:59
I've got 32 threads / 16 cores, should be able to get all withing 1 hour too then
Kevin Ravensberg19:00
Yeah, the Samsung 990 Pro is one of the fastest PCIe Gen4 (which I'm getting a second one off for libbitcoin storage)
evoskuil19:01
Then it's not clear to me what's happening on your machine. With those specs and a milestone run, even 24GB should be sufficient.
Kevin Ravensberg19:02
I think it will get better once I actually use an internal NVMe drive instead of the external USB-C drive
evoskuil19:04
It is likely that the node is simply trying to flush 20+ GB of data to the drive, and everything is just waiting on that.
(edited)
evoskuil19:11
Actually I think it's just that it's external (which we have always told people is not supported).
(edited)evoskuil19:12
mmap writes dirty pages back as many tiny (~4 KB), scattered I/Os. Internal SSD doesn't care — low latency, deep queues, so it absorbs the pattern invisibly. USB external has high per-I/O latency and shallow queuing, so that same drizzle of small writes grinds along at single-digit MB/s, turning a ~1-minute transfer into hours.(edited)
evoskuil19:15
With enough RAM, we don't use the SSD at all, just at startup and shutdown for persistence.
(edited)
evoskuil19:17
Eventually you 'll be able to sync and validate the whole chain over the network and store it in RAM with no persistence at all, just initialize it at every startup from the network - like many apps do today that never could have even a decade ago.
Kevin Ravensberg19:18
Also the direct memory access links between servers is crazy, goes over an optical fiber
evoskuil19:18
I worked on 300 baud (bits per second) networks. With that kind of perspective you can see what's coming.
🤯 1
evoskuil19:20
Even at 14,400 baud ("fourteen four") loading a single lo res image required a coffee break (up to 4 mins). That was only 35 years ago. 😅
(edited)🙌 1
Kevin Ravensberg19:25
I only experienced the dial up internet over phone lines as a kid, that was 4x faster already
👍🏻 1
evoskuil19:29
Now we can sync and index the entire bitcoin blockchain over home Internet in the time it took to transfer a single VGA image in 1984.
🚀 1
Monday, June 29, 2026
evoskuil01:16
The conversion to vertical (SoA - Structure of Arrays) tables is complete. I just tested with ultrafast using both batched and unbatched CPU processing. The CPU batched (on Windows at least) still has some significant problems, but it runs very nicely in our native batch processing in a parallel loop over secp256k1. That same unbatched CPU run with ultrafast (on Windows at least) slows significantly over time, to about 1/2 the secp256k1 rate.
So despite the table rotation from AoS to SoA, which means we eliminated a significant copy/alloc cost in the bridge, the same sort of issues remain. The silent payments SoA tables are also complete, just waiting on the integration from @josie.
I'll knock out a full validation run tonight using our native CPU batching to get a benchmark following the table rotations. That change doesn't help the CPU batching, and comes at some small cost, so this will allow us to measure that cost - which is for the benefit of GPU batching. The SoA tables require no conversion into the GPU pipeline.
(edited)evoskuil01:25
Overall it's excellent progress. The final table structure is in place, 95+% capture rate for all signatures, handling for snapshots and restarts, CPU vs. GPU, batched vs. unbatched, libsecp vs. ultrafast, NuGet builds... and it's all compiling and working properly. Performance is not degraded at all in signature collection and CPU processing in batch (slightly better actually). So we are ideally situated to spend time now working through the details of optimal signature batch processing over batches of any size.
Initial tests on a Linux box with modern GPU showed us processing 5M sigs/sec, where the CPU processing on our fastest benchmark machine to date gives us 360K sigs/sec in the same configuration. That's almost 14x faster, which cuts the signature verification time on our fastest runs from 2.7 hrs to 19 minutes, making a 4.85 hr run into a 2.5 hr run.
evoskuil01:56
We are exactly 4 weeks from the initial work on signature batching overall, and this includes some amount of time spent on silent payment integration along the same lines. Tomorrow I'm moving back to the release punch list, starting with getting the admin interface fully implemented.
👌🏻 1👌 1
I'll continue to work with @Vano Chkheidze on ufsecp integration, and with @josie on that and silent payments integration. And my machine should be here in a couple weeks, so I'll be able to do some real GPU batch testing.
🔥 3
Tuesday, June 30, 2026
evoskuil15:47
2026-06-30T14:39:51Z.1 Batch verify rate ecdsa.... (287081769 / 820) = 350099 sps 2026-06-30T14:45:01Z.1 Batch verify rate schnorr.. (107437982 / 309) = 347695 sps 2026-06-30T14:45:02Z.1 Capture ecdsa.... (2138408877 / 2147002284) = 99.5997% 2026-06-30T14:45:02Z.1 Capture multisig. (770292387 / 771820889) = 99.8020% 2026-06-30T14:45:02Z.1 Capture schnorr.. (134916587 / 143926332) = 93.7400% 2026-06-30T14:45:02Z.1 Capture threshold (453526 / 453526) = 100.0000%
schnorr accounts for all missed schnorr threshold. there are still about .5B sigs to go, and the capture rate should continue to climb. this is maxi, averages about 350k sps.
evoskuil15:55
2026-06-30T15:50:00Z.1 Batch verify rate ecdsa.... (332294470 / 964) = 344703 sps 2026-06-30T15:54:25Z.1 Batch verify rate schnorr.. (90764789 / 265) = 342508 sps 2026-06-30T15:54:26Z.1 Capture ecdsa.... (2414732212 / 2425290707) = 99.5647% 2026-06-30T15:54:26Z.1 Capture multisig. (813212242 / 815289836) = 99.7452% 2026-06-30T15:54:26Z.1 Capture schnorr.. (225181854 / 235398785) = 95.6597% 2026-06-30T15:54:26Z.1 Capture threshold (953052 / 953052) = 100.0000%
Hitting the 95% target in all categories!
During batch verify we hit
99.11% CPU utilization within ecdsa::signature_verify() and 99.90% within schnorr::signature_verify() (these are executed in independent batches non-concurrently).Yet during this operation, at 100% CPU and 100% RAM utilization on the PC, the network, console, and block explorer remain fully responsive.
(edited)👊🏻 1👊 2
evoskuil16:07
Love how we can see the top confirmed block, and yet also query by its hash a (most recent) header that's not yet confirmed.
(edited)
evoskuil16:30
08:02:45Z.1 Batch verify rate ecdsa.... (5347951 / 14) = 381996 sps 08:03:03Z.1 Batch verify rate ecdsa.... (1009305 / 3) = 336435 sps 08:03:17Z.1 Batch verify rate ecdsa.... (1000770 / 3) = 333590 sps 08:03:48Z.1 Batch verify rate ecdsa.... (1073483 / 3) = 357827 sps 08:04:26Z.1 Batch verify rate ecdsa.... (7842509 / 21) = 373452 sps 08:09:22Z.1 Batch verify rate ecdsa.... (62846069 / 170) = 369682 sps 08:10:16Z.1 Batch verify rate ecdsa.... (1004741 / 7) = 143534 sps 08:10:38Z.1 Batch verify rate ecdsa.... (2059874 / 6) = 343312 sps 08:10:46Z.1 Batch verify rate ecdsa.... (1101780 / 3) = 367260 sps 08:11:01Z.1 Batch verify rate ecdsa.... (2643441 / 8) = 330430 sps 08:11:20Z.1 Batch verify rate ecdsa.... (2335074 / 7) = 333582 sps 08:11:58Z.1 Batch verify rate ecdsa.... (7356151 / 21) = 350292 sps 08:13:43Z.1 Batch verify rate ecdsa.... (17513874 / 53) = 330450 sps 08:23:39Z.1 Batch verify rate ecdsa.... (121999975 / 345) = 353623 sps 08:43:07Z.1 Batch verify rate ecdsa.... (260099703 / 718) = 362255 sps 09:07:55Z.1 Batch verify rate ecdsa.... (317242859 / 930) = 341121 sps 09:30:45Z.1 Batch verify rate ecdsa.... (283132935 / 863) = 328079 sps 10:00:05Z.1 Batch verify rate ecdsa.... (328328857 / 1117) = 293938 sps 10:30:58Z.1 Batch verify rate ecdsa.... (347752711 / 1100) = 316138 sps 11:17:41Z.1 Batch verify rate ecdsa.... (378092789 / 1426) = 265142 sps 12:19:19Z.1 Batch verify rate ecdsa.... (356852026 / 1006) = 354723 sps 12:19:21Z.1 Batch verify rate schnorr.. (452495 / 1) = 452495 sps 13:27:13Z.1 Batch verify rate ecdsa.... (332047984 / 946) = 351002 sps 13:28:32Z.1 Batch verify rate schnorr.. (27479633 / 78) = 352302 sps 14:39:51Z.1 Batch verify rate ecdsa.... (287081769 / 820) = 350099 sps 14:45:01Z.1 Batch verify rate schnorr.. (107437982 / 309) = 347695 sps 15:50:00Z.1 Batch verify rate ecdsa.... (332294470 / 964) = 344703 sps 15:54:25Z.1 Batch verify rate schnorr.. (90764789 / 265) = 342508 sps
evoskuil16:55
2026-06-30T16:46:57Z.1 Batch verify rate ecdsa.... (274333559 / 770) = 356277 sps 2026-06-30T16:50:13Z.1 Batch verify rate schnorr.. (64691586 / 195) = 331751 sps 2026-06-30T16:50:14Z.1 Capture ecdsa.... (2653331301 / 2666668842) = 99.4998% 2026-06-30T16:50:14Z.1 Capture multisig. (840673646 / 843602106) = 99.6529% 2026-06-30T16:50:14Z.1 Capture schnorr.. (289213432 / 300246826) = 96.3252% 2026-06-30T16:50:14Z.1 Capture threshold (1613053 / 1613053) = 100.0000% 2026-06-30T16:50:14Z.1 Batch verify rate ecdsa.... (28919 / 0) = 28919 sps 2026-06-30T16:50:14Z.1 Batch verify rate schnorr.. (1545 / 0) = 1545 sps 2026-06-30T16:50:15Z.1 Capture ecdsa.... (2653358659 / 2666696373) = 99.4998% 2026-06-30T16:50:15Z.1 Capture multisig. (840674831 / 843603291) = 99.6529% 2026-06-30T16:50:15Z.1 Capture schnorr.. (289214946 / 300248374) = 96.3252% 2026-06-30T16:50:15Z.1 Capture threshold (1613084 / 1613084) = 100.0000%
That's the last full batch, and then the final "residual" batch. Despite being rate inefficient, the residual takes less than 1 second.
3,784,861,520 total capture 3,812,161,122 total signatures from 295,001 to 950,000 ------------------------------------------------------ 99.28% total signature capture rate!!
That's really astonishing, far better than I had hoped for.
There was a validation break in threshold yesterday, that both @josie and I hit. Simple inversion of the conditions for
<= and >= threshold, that wasn't hit until almost entirely complete. So we both now have complete batched runs, me with CPU on Windows against libsecp256k1 and him with GPU on Linux with ultrafast.🚀 2🔥 1🎉 1
1 reply · josie · last Jul 01, 2026
josie11:47
oops, forgot to open the fix PR for this: github.com/libbitcoin/libbitcoin-system/pull/1903
eynhaender20:08
Can't wait to get my hands on it. I will have brunella up and running by tomorrow. Dropped a RTX3060TI in there. Not much but I should be able to run the GPU validation too.
💯 1💪🏻 1
evoskuil20:48
The vertical batch tables change seems to have significantly increased the I/O cost relative to the horizontal. The horizontal tables produced a neutral if not slightly better outcome with batched verification, despite the significant increase in writes. The net verification cost against both table types inclusive of read and post-processing cost is about the same (350K sps). And the marshalling code is identical. And perf analysis shows a general increase in store contention/paging across the spectrum (all costs increasing as they all resolve to the store (read block, read prevouts, confirmation queries, signature writes).
Full CPU validation time when batching vertical is 2x batching horizontal (9 vs. 4.5 hrs), and the latter is slightly better than no batching at all. This is significant enough that we will need to measure the benefit of potentially-optimized GPU throughput against this increased marshalling cost. Since GPU can't make up more than 4.5 hrs on 3 hrs of GPU signing cost, the above is prohibitive even with an infinite sps GPU.
evoskuil20:58
I considered that this could be the consequence of 8 tables periodically remapping in contention with each other. But I determined that the performance hit is significant even well below the point at which any of the tables would require a remap. And the performance hit is also not an issue of being memory constrained and therefor causing a lot more paging, since this is easily observable with 200+GB of free RAM and no disk activity. Furthermore than are no reads into these tables while marshalling, and all of the verification reads are under the (unchanged) sps performance measure.
So somehow going from 2 tables to 8 tables, with plenty of RAM and the same exact marshalling code, dramatically increases the write cost, but it's not yet clear to me why that is.
(edited)evoskuil21:22
I will likely create a sxs implementation in both system (for verify) and database (for backing tables and queries) so that we can easily measure both scenarios, with and without the GPU. But I suspect that dynamic transformation of GPU chunks (only, not for CPU) may significantly outperform this.
evoskuil22:22
First I'm going to attempt a change to the mmap aggregation to make the vertical table aggregate a first class citizen in the map (mmap abstraction), so that there is only one allocator, set of guards, etc. over the set of files. This 4x increase of contention is really the only cause I can see for the perf hit, and it can be eliminated.
evoskuil22:27
I don't think so, I've got a clear roadmap for this next step. This is entirely on our end. If this doesn't resolve it we just end up using the horizontal interface and pay the cost of the chunk rotations, which is only necessary for the GPU scenario.
(edited)One thing you might look at in that case is what is the most efficient way to rewrite the horizontal tables into vertical chunks. Is that not a step that can be pipelined through the GPU, or is it not worth it since it's really just memcopy.