Bitcoin full node and query server
September 2026
Archive of the libbitcoin Slack #general channel.
Tuesday, September 1, 2026
evoskuil21:12
I've been working my way through the
(edited)bitcoind endpoint surface. It's sprawling, but our frameworks have been accommodating it nicely.
Last week I added a high performance scan to produce unspent and unspendable totals at a given block height. That was the only thing that the utxo accumulator model provided that we don't already index for. But since this is really only an administrative function (satisfying curiosity), there's no reason to cache it.
We won't be supporting any wallet methods that require the presence of the bitcoind wallet, or that require submission of a private key to the server. There is actually a set of RPC functions for performing pure computation for PSBT, and I've added those. But have rejected the two of those that performing signing and therefore require private key transmission. All of these can be one locally by a client/wallet - in no way requiring a node.evoskuil21:21
We also won't be supporting any method that is a test hook, and don't support banning (DoS vector, dumb), assumeutxo blob generation/import, dynamic pruning, or stopping the node via RPC (19 test hooks, 58 wallet only methods, 3 sign methods, 3 ban methods, 2 assumeutxo methods, 1 prune method, 1 stop method).
All methods that require a tx pool or mining interface are deferred to v5. Some methods are partial support due to mixed confirmed and unconfirmed results. Otherwise I expect to be complete today or tomorrow with the full interface.
Thursday, September 3, 2026
evoskuil01:09
In tomorrow's meeting I'd like to discuss closing out
version3 support (builds, patches and repos not used by v4: blockchain, client, protocol, consensus. At the same time we could close out the very dated contrib project repos: pylibbitcoin, python-libbitcoin, and libbitcoin-c, and drop the original electrum fork (created from our original spesmilo repo when libbitcoin split off from that) and the secp256k1 fork (used to maintain version control in v3 before that library was versioned).
If any v3 code is in use it's likely system and possibly explorer. Given that explorer depends on client and protocol (as well as system), this would end access to that as well. Before we release v4 we should bring bx up to at least its v3 feature set, updated for v4 server. I don't expect that to be more than three days work.
I've also reviewed all open issues on the 5 bs repos and closed out those that are not relevant to v4. There were a few that are basically just indicating that there is no tx pool and no mining interface, which really don't need tracking issues. I've also added the major issues open on my v4 punch list, which is getting short.👍 3
evoskuil05:25
libbitcoin
A Bitcoin full node and query server. One node that speaks the protocols your software already uses.
Just the remaining network management methods now, and a bit about bitcoind tx/script json serialization.
Huddle with pmienk, evoskuil, Marnix, Rowan Voskuil, neillm · 246 min15:51
2 replies · neillm · last Sep 03, 2026
Vano Chkheidze19:01
operation UltrafastSecp libsecp v0.8.0 ratio
--------------------------------------------------------------
FIELD
mul 20.18 23.60 1.17x
sqr 18.81 19.27 1.02x
add 7.62 12.14 1.59x
negate 10.51 11.75 1.12x
normalize 5.74 13.73 2.39x
from_bytes 5.12 13.00 2.54x
inverse (var) 1,502.74 1,565.12 1.04x ← იყო 0.86x
SCALAR
mul 48.26 54.40 1.13x
add 9.27 9.73 1.05x
negate 10.50 12.95 1.23x
from_bytes 8.75 9.29 1.06x ← იყო 0.71x
inverse (var) 1,573.29 1,577.07 1.00x
inverse (CT) 2,518.37 2,632.50 1.05x
GROUP LAW
double 111.77 127.89 1.14x
add (mixed J+A) 225.75 216.72 0.96x ← ერთადერთი
pubkey combine 1,754.53 3,244.23 1.85x
SCALAR MULT
ecmult a*P+b*G 35,069.80 34,724.28 0.99x
ecmult_gen k*G 8,800.88 16,888.72 1.92x
scalar_mul k*P 31,475.22 32,441.95 1.03x
pubkey_create 8,800.88 19,767.52 2.25x
SERIALIZATION
compressed 33B 15.44 25.69 1.66x
uncompressed 65B 13.23 28.77 2.17x
SIGNING (CT vs CT)
ECDSA sign 19,447.31 28,368.92 1.46x
Schnorr sign 15,371.24 21,390.94 1.39x
Schnorr keypair 14,684.50 19,762.92 1.35x
VERIFICATION
ECDSA verify 37,523.71 37,516.00 1.00x
Schnorr verify 37,738.10 37,841.44 1.00x
--------------------------------------------------------------evoskuil22:29
I'm seeing these types of messages now:
2026-09-03T22:00:45Z.5 Invalid headers payload from [74.138.71.14:8333] with bytes (000dfdd0070000032018fb3ddd3ac22973b6b9bb02e04323f15fc610f155da000000000000000000003a1d4f36eb09fe7700d59b90c0851e479fe7986c5f59d1cb6a63dd21adc72e0b514c746ad43a0217823085fa000020b426bec7d77c250698748e1e3e6f7d790941a2407ab58ef101000000000000000000a16b2dbc21347f5e6424c746d0b73c97a46f0d547ac2944315c9da209d22afddc34c746ad43a02178ccd50c10000c0a623fdb0cc955f1b0ab6d839a78f065c8299a19765fefe28000000000000000000004ae1b5ca5a763bd538d781ad0e51cf59f96b9ce97875a636d6fa42a259955313854d746ad43a02174298d2e50000c01429c9b776...)
This is another shitcoin operating on the bitcoin network. Fortunately it's identifying itself, so can be masked out using our shitcoin flags:
00010000000000000000000010110000
[peer] invalid_services = 268435632 # 4 shitcoins(edited)
Friday, September 4, 2026
evoskuil00:09
One think I failed to mention in the meeting today.
(edited)btcd extends bitcoind jrpc interface, so it reimplements it and then adds methods to the same endpoint. We are doing the same, so there's a btcd combined interface, which is just derived from bitcoind, and an independent bitcoind interface.
However, the btcd implementation of bitcoind is not only ancient, it's drifted. I made the decision to not attempt to follow btcd down that road, and instead we serve up actual/current bitcoind under btcd. It's a compat break, but I think a pretty reasonable one.Also
btcd implements json-rpc-v2, which includes batching and a version property (and some other semantics). However it does not implement named parameters, which is an incompatibility with jrpcv2. We do support named params for btcd, which is a compatible extension. I derived the jrpc parameter names from the Go source code.evoskuil03:35
Auto-config is now merged, as is the address index on. At/above 32GB, defaults set a load factor of 2.5 on all hashmaps except outs (address), which is set to LF5.0, at block 950k.
(edited)
evoskuil03:40
This is the result of default configuration on an 8GB (or less) Linux or Windows or 10GB (or less) macOS machine.
(edited)
Load factors scale linearly between these upper and lower bounds. If a table's buckets are configured explicitly, then that value is honored instead. So a zero
outs bucket count disables the address index, and a higher bucket count reduces its load factor (e.g. for faster search on bigger machines).Milestone time on
(edited)shelly at ~5Gbps (closer to 4.6) is 23.8 mins, pruned and with the full address index (all LF2.5). That's fully network limited.evoskuil04:31
The minimum recommended RAM is
8GB for milestone and 16GB for full validation, +2GB each on macOS. Below those levels heads cannot remain mostly RAM-resident, and the node will thrash during sync, and Linux/macOS may OOM kill the process.🚀 1
evoskuil06:13
The optimized query to return the total coin, unspendable amount, and total count of unspendable outputs now completes in 40 seconds on shelly, 80 seconds for hash accumulation. Keep in mind that we have no utxo accumulator, so this is a full store scan. Since this is really only an administrative query, that's plenty fast. And of course it will be much slower if turbo is disabled (single threaded query) and/or on a weaker machine (shelly has 128 threads).
🔥 1
evoskuil23:03
Above has been optimized to
(edited)22 secs no hash, 36 secs muhash, and 83 secs serialized hash. These are 3 different options for the utxo accumulation state in the bitcoind rpc call. Peak private memory during the scan has been reduced to 1.5 GB (2.3 GB serialized). These are insanely fast numbers given that this is scan of all outputs, and their spenders, in the confirmed chain of 950k blocks, correlated to detect spentness, and the hashing includes the hash accumulation of the entire set of unspent output scripts and values. Nothing is cached or indexed, just a very parallelized total scan.evoskuil23:10
One of the most costly aspects initially was the bitcoind "unspendable" distinction (arbitrary, basically just script too long and first opcode is op_return), required hitting the output table body for every single output during the scan. I was able to bypass that entirely by setting that bit into the outs association table when each output is written (essentially free), and was able to merge this bit into an existing oversized link (still > 10 years of run time) to avoid any store size increase. We have a far more expansive concept of unspendable which is set during script parse, allowing script validation to be bypassed. But that's not cached and wouldn't help here anyway. IOW, there are a lot of provably unspendable scripts that are considered unspent (spendable) by Core, and those get counted in this accounting.
evoskuil23:43
The near-daily WTF discovery today is this...
In the
scantxoutset method Core returns the unspent outputs of an address. Of course it has no address index, so it just searches the entire utxo index. So just for fun it returns in the response, not just the relevant unspents, but the count of unspents that it searched (i.e. the size of the utxo index). This number isn't useful for anything, but they just have it so why not send it.
So we have an implementation of the method that does the same - a full scan of the chain, about 25 sec, and returns the nonsense number. But with the address index enabled, we can return the answer immediately - but without the utxo set size. I've implemented it such that when there is an address index, the set size just returns the found size, and without it we do the full scan (just for this one method).
But today there are two fun warts. http is half duplex, so you can't make a second call with a first call pending. But these calls are slow. So what Core does to get progress or cancellation for these calls is to accept those from an independent call. A second call asks if there are other calls outstanding on another channel, and allows a second call to cancel them (instead of the first caller just dropping the connection, which is what we do). And of course if the call hasn't returned yet, you know it's in progress still.
So why do this? Because they don't monitor the socket for drop when there work in progress. As a result, when the first caller drops, there's no association to the work in progress. So the long-running query just keeps running! So a second caller can say hey - stop that other call that I dropped. And you might be wondering, can the orphaned calls accumulate? No, because they allow one channel scan to block scans on all other channels. Seriously, it's this stupid.😅 1
Saturday, September 5, 2026
evoskuil00:01
So we always return 0 for scans in progress, because if you are calling, you aren't scanning, and we aren't going to share state across channels! And so there is nothing to cancel, and the call is basically a no-op. There can be many scans running at the same time, but each scan can be limited to one thread by disabling turbo in config.
evoskuil04:24
Busy day, closed out all of the open bug issues, and some of the feature issues. Issue boards are current.
evoskuil06:20
ZeroMQ socket is in and tested, now working on CurveZMQ (CurveCP for ZeroMQ). We had this in v3, but with the full ZeroMQ dependency (what would also now be a Libsodium dependency given that ZMQ dropped Tweet NaCl). This is so much simpler, since all we implement is ZMQ framing as an ASIO stream, and Curve as a privacy stream (same as the Boost Beast TLS stream, and our custom bip324 P2P privacy stream).evoskuil17:53
CurveZMQ is integrated, including config and utilities for keygen. We have zmq keygen commands in bx, and the preexisting conventional base85 key encoding, so these will drop right in for bx support as well. We can add self-signed tls certgen there also, once WolfSSL is factored into a base lib, since bx doesn't take the network dependency.evoskuil19:41
Default thread count selections (when not set or set to zero) is now implemented for network and node threads. Network defaults to hardware, capped at 32, node defaults to hardware. This and zmq are not merged yet. Note that default bucket counts based on RAM were previously implemented as well.
(edited)Sunday, September 6, 2026
evoskuil05:35
The full
ZeroMQ interface is merged, including CurveZMQ with both server auth and mutual auth (neither available in bitcoind natively). Config settings:
[bitcoind_zmq] # IP address to bind, multiple allowed, defaults to empty (disabled). bind = <value> # The Z85 encoded CurveZMQ public key of an authorized client, multiple allowed, defaults to empty (any client). cert = <value> # The required maximum number of connections, defaults to '0'. connections = <value> # The maximum connection duration, defaults to '60'. expiration_minutes = <value> # The idle timeout, defaults to '10'. inactivity_minutes = <value> # The Z85 encoded CurveZMQ server secret key, defaults to none (unencrypted). key = <value> # The maximum allowed request size, defaults to '4000000'. maximum_request = <value> # The maximum topic subscriptions per connection, defaults to '100'. maximum_subscriptions = <value> # The minimum retained read buffer size, defaults to '4000000'. minimum_buffer = <value> # The send rate limit in bytes per second, defaults to '0' (unlimited). rate_limit = <value> # IP address to secure (CurveZMQ) bind, multiple allowed, defaults to empty (disabled). safe = <value>
evoskuil18:50
evoskuil23:12
I'm out of release preview punch list items, apart from unit test expansion (which is ongoing). So I've taken on two items on the v4 release list.
- Generalize addressing for bip155 support and implement Tor, I2P and cjdns addressing.
- Implement outbound connection binding, and load balance outbound connections.
I've decided against support for network bridging, so Tor, I2p, and IP will apply to the full node, not on a per-channel basis. Not only is this dramatically simpler, it also forecloses a Core privacy leak. Obviously the networks need to be bridged, but this is not the way to do it. A bridge is simply two nodes on distinct networks sharing a private manual IP connection.
Monday, September 7, 2026
josie15:10
got my old silent payments branches rebased and working on all the recent changes. got a first pass query implemented for scanning and ill be doing some e2e tests with sparrow same as before. if that ill goes well ill start throwing up PRs against the relevant repos
🙌 1👍🏻 1🎉 1
pmienk15:35
During last weeks meeting, it was proposed to move our meeting time to Wednesdays, 9:00 pst, 12:00 est
3 replies · pmienk, eynhaender, evoskuil · last Sep 08, 2026
evoskuil23:40
I've added
sha3 based on our sha::algorithm<> template. And also base_32 based on our radix library patterns. The previous base_32 ("bech32") has been renamed to base_32b as it's not the RFC compliant one. These are complete and tested in system, and will provide the necessary support for the bip155 addressing enhancements listed above.Also made some fixes and improvements to the
zeromq socket and notification service.Tuesday, September 8, 2026
evoskuil04:32
Cranking out unit tests across
system::chain.
Also working through a generalization of network::channel_rpc to allow the jrpc services (electrum/sparrow, bitcoind/btcd) to operate over tcp/s, http/s, ws/s over the same socket, based on the connection handshake (and ws upgrade) alone - controlled by caller. This will also simplify some code.Still performance tuning on the posix
mmap implementations. Presently working on getting it to flush and settle RAM as the node becomes fully synced. This is automatic on Windows, but we have to aggressively hold head RAM in posix, which means we need to let it go as well. Working well.👍 1
evoskuil22:04
VS Community Edition now includes the formerly Enterprise c++ coverage tool, so I can easily pull out local coverage metrics from debug builds (unoptimized builds don't miss lines). Formerly we had only Linux coverage (in CI), so this gives a more complete picture, due to some conditional compilation. Also makes it fast/easy for me to see where the gaps are. Our CI bar has long been set to 80%, which is the level where unit testing provides a material code quality benefit. Getting the last 20% is a lot more work, because you always get the easy stuff first (which is also why it's not worth as much). We'll get to 80%+ on all repos to hit the release gate, and on network/server protocols much of that will be hit by functional tests. And I'll keep working toward that 100%.
Wednesday, September 9, 2026
evoskuil04:25
Universal json-rpc is now fully tested and merged. So electrum, bitcoind, and btcd are each served as json-rpc over tcp/s, http/s, ws/s on the same binding.
💪 1
7 replies · josie, evoskuil · last Sep 09, 2026
josie15:17
this stuff is awesome. made adding the sparrow/silent payments specific end point stupid simple. was able to throw a bunch of code
👍🏻 1
josie15:19
ive got an end to end im building and letting sync, going to be running some tests tomorrow but so far looking good.
evoskuil15:21
Cool. My server changes should calm down now. I'm going to do some work on electrum v7 (which keeps changing) and remaining open items having to do with the handshake and a couple older version changes that don't fit our dispatch model. That will wrap electrum/sparrow until v7 is finalized.
👍 1
josie15:17
replied to a thread: view
this stuff is awesome. made adding the sparrow/silent payments specific end point stupid simple. was able to throw a bunch of code
👍🏻 1
evoskuil15:17
The universal rpc refactor above allowed the new
[sparrow] service to be integrated as a derivation from [electrum] in the same manner as [btcd] derives from [bitcoind]. Sparrow contains two extension methods from frigate and another unique to sparrow, and since electrum doesn't look to be taking this in the immediate future, I just dropped it in as an additional endpoint to maintain proper version control. The btcd->bitcoind and sparrow->electrum are not pass-thru, and are not doubly-implemented, they are simply class derivations.Huddle with pmienk, neillm, evoskuil, Eric Chennells, eynhaender, Marnix · 201 min15:55
Thursday, September 10, 2026
evoskuil02:24
constexpr code (that evaluates at compile time) is not counted in coverage stats, either in the numerator or denominator, it's like code that's missing. We test this code with static_assert, and there's a LOT of that. So that's tacked on in the yellow bars to give some sense of scope.
evoskuil17:23
The only source changes made are to disable includes for headers not referenced in the build, and this:
+/* LIBBITCOIN: fix warning when WIN32_LEAN_AND_MEAN is already defined. */
+#ifndef WIN32_LEAN_AND_MEAN
#define WIN32_LEAN_AND_MEAN
+#endifThe in-place change allows for and easy source diff against the previous version. However wolf did refactor some large files into smaller ones, so there are a number of additions (no removals).
Rowan Voskuil19:39
Since I missed the meeting yesterday(forgot we moved day) ill just put my weekly update here.
This week I worked on implementing a feature that shows a output's status if its spent or not when viewing a transaction.
The second thing I worked on that I still just have on my fork is a new section when viewing a block that shows the active/inactive consensus rule flags with BIP references.
👍 1👍🏻 1
1 reply · evoskuil · last Sep 11, 2026
Friday, September 11, 2026
evoskuil05:20
replied to a thread: view
There's a lot of new cool awesome stuff in the admin interface!
evoskuil06:35
FYI, there was a regression in the POSIX mmap optimization work that caused store corruption due to lack of null fill for arraymap header expansion. The is now resolved.
evoskuil17:44
Cleaned up the layout a bit, moved bitcoind rest to its own section, despite it being co-located on the same endpoint.
evoskuil23:46
bx is now back in the fold, only dependency is system, 28 methods stubbed, to be replaced over bs native interface. lots to do still, but was a big chunk of work just to get building and tests passing.evoskuil23:51
As we discussed, I am now going to hide the full set of non-v4 repositories.
- libbitcoin-blockchain (v3)
- libbitcoin-client (v3)
- libbitcoin-consensus (v3)
- libbitcoin-protocol (v3)
- pylibbitcoin (2019)
- python-libbitcoin (2018)
- libbitcoin-c (2017)
- secp256k1 fork (support of v3 due to lack of versioning)
- electrum (we formerly owned this and the spesmilo repo)
That will take us down from 17 to 8 repos.
👌🏻 1
Saturday, September 12, 2026
evoskuil00:39
secp256k1 and electrum will remain for a while as archived, since those are forks and the clearance is different. otherwise all done
evoskuil02:47
Wallet 100%, overall 96%, thought it's actually higher due to static_assert testing of constexpr functions (some of which are very large, like hashing).
Sunday, September 13, 2026
evoskuil20:46
Complete:
[x] warn on RAM low (hard/soft) [x] warn on CPU build if not optimal [x] warn on GPU build/config if not optimal [x] commandline option to skip warnings [x] chain/machine/wallet unit test coverage [x] add esplora interface [x] wolfssl upgrade
Working on basic tx pooling to satisfy client-server interface requirements, generalizing the address pool for Tor/I2P gossip, and NIC load balancing.
evoskuil20:53
The above warnings will fire as a group on every startup if the build is not optimal for the given hardware. This includes
CPU (simd/sha/neon/crypto), min RAM, store on internal SSD, file system space, and GPU. The GPU detection uses the same criteria as ultrafast, but doesn't require it in the build. So it works like the CPU features, where we can tell the user that the build is not optimal, without the dependency in place. With the dependency in place and usable, we can also automatically enable it - making it configurable but on by default if available.evoskuil20:58
The new
--accept (or -a) command line option disables this prompt and the milestone prompt, and both prompts are disabled at daemon startup.is significant improovments on cpu?
evoskuil22:45
no, haven't done any benchmarking for a while. mostly working on test, auto-config, mmap, and service endpoints.
Monday, September 14, 2026
2 replies · neillm, evoskuil · last Sep 14, 2026
evoskuil15:37
With the dependency in place and usable, we can also automatically enable it - making it configurable but on by default if available.
GPU detection now warns when not compiled and when batching is disabled if compiled, but defaults to enabled.
👌🏻 1
evoskuil22:19
Updated our coverage badges, so those should start reflecting more accurately. The way it was set up they were caching forever.
Tuesday, September 15, 2026
evoskuil01:54
I've just completed the work to accept txs and packages, in the case where they validate against confirmed txs (or self + confirmed in the case of packages), into the store. This is not a full tx pool as I have
not enabled incoming p2p tx or package acceptance or relay, and am not accepting conflicting txs, and am not indexing unconfirmed txs. But this will accept/reject from client/server submissions, based on own node minimum fee rate, and relay those to peers based on their minimum fee rate. I still have a bit to do in terms fee rate, and package forwarding.
The significant aspects are really in the store, whereas the store subsumes unconfirmed txs into blocks (not compact yet, just full blocks). I've enabled this once the node becomes current, and it remains enabled until shutdown, even if the node falls out of currency. This is necessary to ensure that txs get absorbed once they are accepted. It's a race condition, and duplicates are of course allowed, and we already had unconfirmed txs (as a consequence of reorgs). The absorption is costly and unnecessary during IBD, which is why we keep it disabled until current. This coincides with the node rejecting submitted txs/packages when it is not current.So with the store work done, and this light implementation in place, we are going to be exercising important pool code (transaction chaser) in v4. The main limits are that (1) wallets can't submit a tx or package until predecessors are confirmed, and (2) wallets won't see unconfirmed txs relayed via p2p. This latter limitation could be opened up pretty easily, limited to accepting only those that spend confirmed outputs and those that are not conflicting (first seen). To handle conflicts and spending of unconfirmed, we need an index, which is the hard point for v5 (mempool vs. tx pool).
👌🏻 1👌 1👍 1
POSIX mmap bug has been isolated and verification in progress. Should have a patch soon. It's possible you won't see it on a typical run. It's a scenario in which null fill doesn't reach expanded heads.
(edited)🎉 1
1 reply · evoskuil · last Sep 15, 2026
1 reply · evoskuil · last Sep 15, 2026
Wednesday, September 16, 2026
evoskuil00:05
I'm just about done with the implementation of incoming tx relay and all that entails, including fee-filter, not-found, relay-flag, unrequested relay/not-found, unresponsive peer, validation, announcement, config, etc. most of the framework was already in place, so just putting it all together. with relay enabled, once the node is current it will start accepting, validating, storing, and announcing unconfirmed txs up to its configured fee-rate.
This is the v3 tx pool, on steroids. Not a mempool (no way to get the set of unconfirmed txs without enumerating the full store). No acceptance of unconfirmed double spends (first seen policy, like the old days and v3). Pool txs are associated to their incoming blocks (and still no compact blocks), so space is preserved. Txs already validated via the pool are still revalidated when the block arrives (this can be optimized out for both full blocks and compact blocks). And of course there is no mining template generation. Also no package relay (not sure we'd ever implement that as specified).
(edited)evoskuil00:54
Also I've been chipping away at the
address pool/relay generalization. That's done now and I'll be moving on to integrating that with Tor/I2P connections. Settings changes: these are added (peer.gossip_ipv6 replaces old outbound.use_ipv6).
[peer] gossip_ipv4 = true gossip_ipv6 = true gossip_tor = true gossip_i2p = true
If you don't enable tor or i2p gossip then the network falls back to address v1 so that it doesn't get that traffic. Otherwise it will gossip these types and save them into the address file, and the network(s) that it connects to are what its allowed to gossip from these settings (and from the configured seed/self/manual settings).
This isn't consistent with the design I mentioned in the last meeting, but it's actually simpler and easier and gives people the option to do what other nodes do, which isn't a great idea but easily disabled now.
evoskuil01:11
In order to further generalize the settings envelope within the store, so that it can become mutable and remain part of the snapshot, I'm moving it from a sidecar in the genesis block txs slab entry to a first class citizen, head only slab table called
(edited)envelope.head.
This will allow the store to carry the one-way latch into pooling, since once pooling starts it really should remain on even through a restart, but presently there's no way to save or detect that state. It's a general issue that will keep coming up, so better to resolve it.evoskuil05:59
Added a scheme version to the source and settings envelope, as first setting read, validated on load, logged. This will allow the code to detect and adapt to schema changes.
Huddle with pmienk, neillm, eynhaender, Eric Chennells, evoskuil, Rowan Voskuil · 207 min15:52
3 replies · evoskuil, eynhaender, neillm · last Sep 16, 2026
Thursday, September 17, 2026
evoskuil01:15
peer.enable_privacy is being replaced by node.provide_privacy for consistency, as the latter is how we set all service bits (advertisements).evoskuil02:32
node.limited_blocks is now part of the settings stored in the envelope (one way latch at store create). This ensures pruning stays consistent (checkpoint and milestone blocks are pruned). However since the milestone drives pruning, it gets stored and therefore becomes immutable as well.evoskuil02:43
To clarify a previous question, the envelope is a one-way latch, mostly at store creation. A config setting that drifts on a later startup is presently just ignored.
evoskuil04:52
Address gossip with all of the
address v2 complexities of including tor/i2p. All complete now in seed/outbound/manual/inbound, with the ability to enable disable/gossip of IPv4/IPv6/Tor/I2P. The only limitation is that we don't have support for incoming I2P connections. This enables completion of some of the WIP bitcoind/btcd methods around connection admin as well.evoskuil04:59
I've modified the
--settings option so that it now emits all config setting values, and annotates each non-defaulted value with # defined.evoskuil07:06
I'm implementing the
I2P Simple Anonymous Messaging (SAM) proxy for I2P inbound. Interestingly this is an inbound proxy analog to our outbound only socks5 proxy. This will derive from our acceptor class in the same manner that the socks proxy derives from our connector class. Config almost the same as socks, the rest already implemented. This is the last piece of full support for both I2P and Tor.evoskuil18:29
Just merged server change to bump the default
peer.protocol_maximum = 70016 so that address v2 gossip is available by default.evoskuil18:34
I recommend full stack rebuild, also I didn't bump the store schema version, but changed the schema, so rebuild store.
(edited)Also config settings that carry secrets are now annotated in the parser and masked (empty) on the
(edited)--settings dump. But when configured will still show as configured using the non-default comment: outbound.password = # configuredevoskuil18:47
sampling of
hosts.cache with address v2.
[peer] gossip_ipv4 = true # defaults true gossip_ipv6 = true # defaults false gossip_tor = true # defaults false gossip_i2p = true # defaults false enable_address_v2 = true # defaults true protocol_maximum = 70016 # defaults 70016 (send_address_v2)
66ohefkl3ny7mzw4rhzvql4eat3iooesbp7u4mg7nuanzel5dcvfwuqd.onion:8333/1789617292/3081 41.90.136.50:8333/1789646788/3081 emusbuwtrttrwf56twzwqiy65kfepwrdvjqlbhxrxxn3spseiha4ytad.onion:8333/1788270203/201329677 60.249.8.215:8333/1789648459/1033 209.222.97.190:8333/1789639930/3145 31.21.184.247:8333/1789529806/3081 89.0.148.160:8333/1787192611/3145 2enmmjciteoq4223fhu3zjd5abtkmg6svkac7e3q2fjpk3vdmp4hwsid.onion:8333/1789597157/3081 100.40.127.65:8333/1789395506/3081 95.168.169.227:8333/1789617869/134220877 149.102.228.62:8333/1787543186/67112009 86.49.255.26:8333/1789637844/1101 dktamk3jkks2wgjg6lrfbaya7nmqioadhppa6oauckilncwitqi2qxyd.onion:8333/1789593913/3145 tzzqhl4op7i2qncxxtwl7mlvluxelxgsrdeuqnfyvxvxmzcyho6votyd.onion:8333/1789636464/3081 viize3apqmtbzoixt3kozr7ktwc5nm5peuwtuvsvjtvpyhodxz72kgid.onion:8333/1789646121/67109901 89.35.197.146:8333/1789642995/3145 tvrb4r2cyf3rzrrpymfsx7ooikixam3k2obygm3akd6kg2ypmiba.b32.i2p/1789632435/3145 e2xlezwhgifo3sek4s7r2guepb4cuqr6rdyarv45nvrz63rerawypbad.onion:8333/1789617464/3081 98.10.145.173:8333/1789535259/201329673 85.185.32.154:8333/1789477190/3081 m54twflw5jblnrax2kivvtcyx27epxhjeyidvbkukgmydv5slxx5rjid.onion:8333/1787117698/1037 93.88.126.109:8333/1789561467/3145(edited)
evoskuil21:59
I've updated the
tx_archived event (set for validated and archived unconfirmed tx - i.e. arrived individually or via package, over p2p or client-server interface). It was logging the surrogate key, now it's logging the fee rate (directly comparable to the config setting of min fee rate).Friday, September 18, 2026
evoskuil02:21
We were dropping peers to aggressively for tx missing prevouts and locktime violations. That will get cleaned up with proper p2p relay, but until then I've relaxed the criteria.
neillm15:20
Is
gsl (github.com/imatix/gsl) maintained? I'm unable to build it because it requires pcre3, which apparently was deprecated and replaced with pcre2 😅evoskuil15:21
Not at Imatix. when Pieter Hintjens died that account went with him. But it's cloned over at ZeroMQ.
(edited)1 reply · neillm · last Sep 18, 2026
2 replies · neillm, pmienk · last Sep 18, 2026
neillm15:41
There's a line in CI that says 'Acquire gsl' -- what does that do, and is it a prebuilt binary?
pmienk15:42
I was wrong. we do build it there, but the checkout was what I saw and was thinking erroneous zeromq/zmq dependency.
neillm15:38
Looks like ubuntu may provide the pcre3 package now that I'm looking, but it's long gone on debian
I'd consider going with the docker image they maintain, but there are pathing breaks
1 reply · pmienk · last Sep 18, 2026
pmienk15:44
specifically, docker filesystem pseudo-isolation means directory scanning functions for files takes a hit unless we refactor - unless someone else has encountered this and has a better workaround
neillm15:51
I was able to download all of the required packages from an oldstable version of debian to get past this for now (packages.debian.org/bookworm/libpcre3-dev)
(edited)packages.debian.org
Old Perl 5 Compatible Regular Expression Library - development files
evoskuil20:08
I'm working through the last few open bitcoind/btcd methods, pertaining to network management and diagnostics. @josie has the three Sparrow SP methods, and @eynhaender is enabling some partial mempool method support for client compat. These will truly close out the last WIP client-server methods for v4. Everything else is either rejected (not much of this), partial/pending v5 mempool, or complete.
🎉 1🚀 2
evoskuil21:00
In reviewing the bitcoind endpoints I found that Core has a "feature" that allows the node to be configured to stop providing blocks once the byte upload has exceeded a threshold in a 24hr period. It drops peers that make block requests after this limit is reached. The node keeps advertising NODE_NETWORK, which makes this a bug. We implement DoS protection via a continuously imposed rate limit on outbound traffic. We throttle the next request reader based on the previous response size, so as to not exceed the outbound rate (except instantaneously). Since we also manage peer count, the combination allows the admin to set an overall rate limit. This prevents one channel from eating the bandwidth limit for all others.
4 replies · neillm, evoskuil · last Sep 18, 2026
evoskuil21:08
These are the other fields that they have to put out so that callers can manage this "feature".
evoskuil21:09
IOW, they are really trying to implement rate limit, but just don't seem to know how. The difference in behavior is that you can pull down a lot fast on one channel, and then nobody gets anything for the reset of the day - as opposed to every channel getting a maximum continuous rate. But it's not clear to me why you want rando internet peers to choke each other out and to provide a sawtooth pattern of service unreliability.
(edited)evoskuil21:13
- correction, not "callers" managing, it's the admin of the node that sees the client-server report on the status of the feature.
Saturday, September 19, 2026
neillm02:39
@pmienk How do I make a buildable PR these days? github.com/libbitcoin/libbitcoin-explorer/pull/757
27 replies · pmienk, neillm · last Sep 19, 2026
pmienk03:51
You're getting an error produced by two different issues colliding. (1) you used a feature branch name (2) action/checkout appears to leave a directory where git clone performing the same action does not. (2) is attempting to compensate for (1). I can revert (2) but we'll still have an issue or I can look into whether I can detect the failure to execute the step correctly and perform filesystem cleanup
🙏 1
neillm03:54
I made a PR from my own origin to a feature branch as usual, so I'd like to figure out how to improve that process -- thanks for the quick response, and can address in the coming days!
pmienk03:55
yeah, a decision a few months ago that would disable the use of feature branches was implemented with error and that error only fixed recently.
pmienk03:56
upon fixing that, I worked out a way to reinstate them without significant failure to the CI process. put that in earlier today, and we're hitting an edge case I was looking to test.
neillm03:59
Fair enough, just let me know what I need to do at some point in the coming days -- no rush 🙏
pmienk04:26
you can try the head of pmienk/libbitcoin-build - gotta validate the path's right. otherwise, i'll sync your changes for my own testbed
pmienk04:32
got it going here: https://github.com/pmienk/libbitcoin-explorer/actions/runs/35421536174
pmienk04:32
well, queued. fingers crossed the removal cleanup zeros the directory and the script performs proper clone
neillm04:38
I think I failed already? github.com/libbitcoin/libbitcoin-explorer/…/105840586822?pr=757
pmienk04:40
thanks, yep - pathing issue - will fix. think i can just strip working-directory as I fully qualified the removal path
🙏 1
pmienk04:49
github.com/pmienk/libbitcoin-explorer/…/105841490320
Looks like it got past -system clone
pmienk05:02
should be noted - using a feature branch <feature name> will build off thecodefactory/<repo>/<feature name> when available but fallback to libbitcoin/<repo>/master instead of thecodefactory/<repo>/master
4 replies · neillm, evoskuil · last Sep 19, 2026
evoskuil16:28
All of the various network management methods are now implemented in bitcoind and btcd. I only rejected one, and none are partial. The rejection is
getblockfrompeer, as I can find no scenario where it makes sense. It implies we have a header, and want a block, and aren't getting it so it's holding us up, and there's a given peer that we are already connected to that we know has it so we'll make that request and stuff it into the store if it succeeds (otherwise nothing). We already make requests to all the peers we are connected to (and can get connected to) to find a block that isn't being found. This seems to be some kind of test hack and maybe an assumeutxo hack and maybe a hack to unstick a stalled sync (even though it will eventually move to other nodes). One of the dumbest I've found, but in this case there's also no way to make sense of it given our architecture.1 reply · evoskuil · last Sep 19, 2026
evoskuil17:55
Some of this network management stuff is a little light on test and could use some live verification. It's all merged now.
(edited)pmienk16:40
Ci behavior for source sync has changed. When executing in repos with owner <owner> and libbitcoin dependency project <dep> and associated reference branch <branch>
- If <owner>/<dep>/<branch> exists and no error occurs on action/checkout, that combination is used
- Otherwise, the installation script is authoritative and will fulfill <dep> - this is defaulted to libbitcoin/<dep>/master
This allows for a maximum flexibility in developer workflow, enabling working in individual repositories using same named branches for all dependencies as well as feature branch naming relying on properly scoped work. If anyone wants to discuss this approach, changes, how it's accomplished, let me know or bring it up in the weekly.
🎉 1
Also, concurrency restrictions have been leveed upon CI builds.
Currently in libbitcoin at most one action per pull request is queued - the last build is active, others are canceled upon last build creation.
I've additionally limited personal repositories to a single active action per branch, with the same cancelation behavior
5 replies · pmienk, evoskuil · last Sep 19, 2026
evoskuil17:50
I've already found this very helpful. Just have to be careful to delay push if you want to differentiate potentially failing builds, but generally it's better to just keep queuing up the commits and the build stays at the top, instead of grinding through what is then a stack of irrelevance.
pmienk17:51
Yeah, and if you ever want to backtrack to locate a failure introduction you can always stop the top and manually trigger the rebuild.
👍🏻 1
pmienk18:13
BTW, merge does cancel pull request CI now, but it does it by generating a noop action. I'll look into making that one appear as canceled sometime as to not he misleading.
evoskuil17:55
replied to a thread: view
Some of this network management stuff is a little light on test and could use some live verification. It's all merged now.
(edited)evoskuil18:28
The custom POSIX mmap implementations are updated for head settling. Numerous milestone and validation IBD tests on both macOS and Linux have shown good results at no adverse performance impact. But this stuff is a bit complex and needs more interactive review. A full sync to the top (not just 950k) should allow memory to drain without a restart, and resume at full speed after a long stretch of not syncing (e.g. falls behind because of network failure).
👌🏻 1
evoskuil19:24
There is quite a bit of new functionality driven from the network management method implementations. For example, there has been no way to enumerate running channels, as that starts breaking the lock-free network design. When Core audits the channel set for diagnostics, it takes a lock of the entire network - all channels. We send a message to all channel subscribers and they write a response, and we collect the responses when they are all handled, or collect the first responder as the case may be.
So I generalized this approach to enumerate channels, to get their state and statistics, etc. I also added a broadcast to stop a channel, including the option to stop and drop a manual channel from retry. These are targetable by unresolved and resolved (for non-proxied IP addresses) address. I also added per-channel-per-p2p-message-type send and receive byte counters, and total send/receive byte counters for the process lifetime. These then report back to the caller and summarize as necessary once collected.
In order to avoid contention on lots of channels updated lots of atomics to obtain the process totals, I implemented a mechanism whereby the process totals are only updated when a channel stops, running channel totals are swept dynamically, and the two are reconciled by the caller to get the running totals. This is guarded by the set of running channel IDs being maintained at the point of channel drop, atomically with the total update. So the sweep can be subtracted from totals if double-counted. So a consistent snapshot in time with no locking, and the collection cost moved to the administrative caller.
(edited)Sunday, September 20, 2026
evoskuil18:02
I've closed one of the two remaining open DoS protection features.
We pace all responses to their requests, and treat jrpc batched requests as individual cycles, reading only one at a time, making it inherently impossible to DoS the node by request flooding. One request, response completed, next request. We also gate the query of a next item on the completed send of its predecessor (e.g. block requests), flattening the peak cost of any single request down to its individual parts - even if those parts are a single logical message response. The caller is forced to read each portion before we query the next.
However, with full duplex we can send notifications, which are "responses" without a corresponding request. Even though we control the size of each response, we can't control the rate at which they are read. So notifications can backlog indefinitely, just du to slow reading. Some notifications can be dropped, but in the supported interfaces this really only applies to the bitcoind zeromq interface, since zmq adds a sequence identifier to broadcasts, allowing messages to be dropped when the high water mark (HWM) is reached. For others, correctness dictates that the channel must be dropped.
This implies some rational measure of backlog, which isn't always as straightforward as it may seem. For performance reasons we don't serialize objects to a buffer in order to send them. Sometimes the serialization happens directly from the source to the socket, as chunks are read by the recipient. And the backlogged closures have their own cost, which can be more than the message itself. But all backlog now includes size approximations sufficient for the DoS guard. So while a full-duplex channel is at its HWN it either starts dropping messages (zeromq) or it drops the channel (p2p, tcp-jrpc, ws).
The other structural DoS protection feature that's not implemented is in the header tree.
Monday, September 21, 2026
evoskuil17:05
Closing a DoS that I had missed - lack of a handshake timeout on connections with a TLS or comparable (like P2PS) handshake. Otherwise the other end can hang the socket.
evoskuil17:45
We pace all responses to their requests
Found some open issues in the P2P implementation here, so ended up fully generalizing the design pattern across all channel types. Client-server interfaces have a pretty clear request-response vs. event-notify pattern, and r/r is guarded by not reading the next off the socket until the previous is fully sent. The e/n patter requires cost accounting, as otherwise a slow counterparty can force buffer overflow.
On p2p there wasn't a clear distinction between r/r and e/n, so I created one, which is controlled by the protocol invoking notify(message) (vs send(message)) in the cases where the message originates from an event, not a request. P2P is more complex than client/server given that multiple protocols may subscribe to and handle the same request on a given channel. This is because p2p mixes inventory types (e.g. tx and block) in one message, and similar issues, which makes it harder to isolate protocol behavior. So the request is only complete after all subscribers have released the message.
So both the notification distinction and the concurrent handling are now generalized to p2p, which closes the DoS gap there. This also coincides with the low level network code subsuming the monitor() function into this framework. A monitor is created on a channel when the message handler leaves the strand (e.g. to do potentially parallel and long-running tasks). This arms the socket to monitor for a counterparty drop, which then propagates cancellation. So this is now automatic in the r/r cycle. When a method departs the strand it explicitly requests a handle, which arms the monitor, and is automatically disarmed when the handle destructs.
So overall less code, simpler to get right, even less to manage in the protocol, correct/consistent across interfaces, and flexible enough to handle complex scenarios.
Tuesday, September 22, 2026
evoskuil00:09
This is now all done and merged. Lots of refactoring in network, some in node and server.
evoskuil03:52
I've finally implemented
outbound address binding. This allows you to control what NICs your outbound traffic is carried on. Inbound traffic is already bound, independently for each service. Outbound by default choses bindings sort of randomly. This prevents useful load balancing in most scenarios. On Linux you may have to modify your local routing tables to get efficient balancing, but otherwise it should balance fairly across networks of comparable available bandwidth. I haven't incorporated anything to manage networks of different quality, except that this allows you to avoid certain addresses (e.g. your Wi-Fi NIC).evoskuil04:55
Simple load balancing is working great. And to put double the channels on one NIC vs. the other, you can just bind the same NIC twice (or 3, 4... times).
evoskuil13:32
Shelly
24.7 mins milestone after all above changes (record is 23 mins), so still going strong.
evoskuil13:50
Real
libbitcoin-system coverage is about 98.17% after factoring in the thousands of static_assert(...) tests and the behavior of INLINE on the coverage metrics.Surviving mutants in the
machine:: namespace should be zero at this point, which is a higher bar.🎉 3
Wednesday, September 23, 2026
evoskuil14:13
I've modified the header organizer to close the last structural DoS gap. The header sync is a little more costly. I've measured 35 seconds vs 27 in the original, at 150 peers. So well worth the benefit. I've also dropped the organizer chaser template in favor of a header chaser, dropping the block chaser. This kills blocks-first for now, but that wasn't really functional yet anyway, and is largely academic.
Huddle with pmienk, evoskuil, eynhaender, Eric Chennells, Rowan Voskuil, neillm, Marnix · 205 min15:59
2 replies · evoskuil, neillm · last Sep 23, 2026
1 reply · evoskuil · last Sep 23, 2026
neillm23:53
Any pointers on getting Nvidia GPU support built on Linux? I tend to see this (on a machine where
nvidia-smi shows proper support with processes using the GPU as expected):
2026-09-23T23:50:16Z.0 Hardware configuration... 2026-09-23T23:50:16Z.0 intel... platform:1. 2026-09-23T23:50:16Z.0 avx512.. platform:0 compiled:0. 2026-09-23T23:50:16Z.0 avx2.... platform:1 compiled:1. 2026-09-23T23:50:16Z.0 sse41... platform:1 compiled:1. 2026-09-23T23:50:16Z.0 shani... platform:1 compiled:1. 2026-09-23T23:50:16Z.0 gpu..... platform:1 compiled:0.
13 replies · evoskuil, eynhaender, neillm, josie · last Sep 24, 2026
eynhaender01:57
I asked him about the build too today. He said he'd have to check what build command nix actually uses. Expecting an answer by tomorrow.
josie14:27
github.com/josibake/nix-libbitcoin , this is everything i use currently for building, and up to date. to make it a bit easier, i had claude dump all of the build steps into a stand alone script that doesnt need nix. disclaimer, i haven't testd the standalone script, but it should at least help with picking out the individual build commands being used.
josie14:29
note, i need to apply this patch when using a newer version of ultrafast: github.com/josibake/nix-libbitcoin/blob/…/libbitcoin-system-use-ultrafast-libbitcoin…
josie14:30
and this patch is need when using a specific CUDA (been meaning to upstream this to vano): github.com/josibake/nix-libbitcoin/blob/…/ultrafast-cuda-field-inv-launch…
josie14:32
haha take a look at the shell script i linked! that does everything i just mentioned, (applying patches, etc), and doesnt need nix
eynhaender14:34
Yea, absolutely. Just checked out the script, guess that's even more than I need. Thanks.
Thursday, September 24, 2026
evoskuil04:33
I've verified the bitcoind interface against the v32rc2 source (announced today), all good.
(edited)evoskuil05:48
Version Information... libbitcoin-system..... 4.0.9791 24165d6b-dirty libbitcoin-database... 4.0.4407 3d19705c-dirty libbitcoin-network.... 4.0.7077 5cab58c1-dirty libbitcoin-node....... 4.0.4002 726f5981-dirty libbitcoin-server..... 4.0.2979 355cd0c0-dirty compiled schema....... 4.0 database schema....... 4.0
Versioned Windows builds.
👍 2🤗 1
evoskuil06:26
Syncing on RPi5 (jennings) now, 4 threads, 16GB, sha, fast SSD, 2.5Gbps. Full build took 62 minutes (no tests).
evoskuil14:55
Jennings milestone to 950k in
1.45 hrs. Validation from 950k to top took 2.6 hrs, since there are just 4 slow threads running at 90-100%.evoskuil15:04
Looks like there has been a bug in the POSIX thread priority util, resulting in the priority threads (validation by default) becoming low priority. Probably not a huge difference, but will re-run the benchmarks to make sure. Found this on the Jennings run above.
move to this release
👍🏻 1
evoskuil17:34
Building a new NuGet package for ultrafast 4.6. Will do some benchmarking on full validation without GPU, and reverify GPU.
Friday, September 25, 2026
evoskuil00:11
Running full validation on jennings. That will take a while, as it's entirely CPU limited on 4 slow threads, even with sha/neon. But good to get our first full ARM full validation run. Our macOS CI is ARM, but no sync on them. And we don't have a Windows ARM box for testing yet. Also running another bm full validation run at the same time.
evoskuil01:46
My punch list is empty apart from long-running jobs, so the
gold-plating shall continue.evoskuil01:49
Hardware options for Windows Intel and ARM generally should be complete, so getting close. github.com/libbitcoin/libbitcoin-build/pull/464
evoskuil02:09
I've taken on two new chunks of work. First is to take advantage of to pooling to
bypass block.tx validations for pooled txs. Pooled txs are validated and archived, so there is no reason to do either again (except in the very rare case where the validation context has changed, which we manage by revalidating in the block's context).
First we added pooling to make wallets happy. Then we added block consumption of pooled txs to save space and prevent the unnecessary archival step. That's been running all week. But when we validate we still pull the whole block and prevouts, and validate the whole thing. That's very wasteful, since most of the txs have already been validated.
So now we are writing the validation context, along with fees, sigops, and all prevout parent tx links, to the validated_tx table for each tx. This now happens at every tx/package acceptance, in the new tx chaser. These values are all that is required to validate and confirm the tx for the block, without ever reading the tx. This part is done - table schema updated, queries updated, and unconfirmed txs validated and archived with this state as they arrive. That part's done.
Nest step is to identify the state record for each tx in a block for validation. This is optimized by the fact that we can determine from the header->txs->tx table association values alone, whether the tx was pooled or arrived with the block (unpooled). Pooled tx links will always be lower in value than the coinbase link in the association set. So we query up the metadata for pooled txs, validate the other txs, use the combined metadata to validate the block, and set the state of the block.
The prevout parent tx links are used, along with those of the queried/unpooled txs, to populate the prevout table cache, for optimization of block confirmation. The prevout table has been used for a couple years now, and is purged once the node is fully synced (current-coalesced state). Signature batching tables are also dropped in the purge. And now the tx metadata (validated_tx) table will be as well. This last one we'll probably retain much longer for miners in v5.So now we will have both storage and validation optimized away as blocks arrive individually (once current), so basically all blocks are arriving just as compact blocks, apart from the download. I expect that the download of the full block in this implementation is actually much faster to validate than the compact block. The compact block is about bandwidth and latency.
evoskuil02:19
You might then wonder, can we do compact blocks without a mempool? That's the next step after saving the re-archive, the re-validation, .... save the re-download.
5 replies · neillm, evoskuil · last Sep 25, 2026
evoskuil05:53
There appears to be an intermittent performance-eating regression in the new headers sync work. Patch inbound.
2 replies · evoskuil · last Sep 25, 2026
evoskuil06:16
can we do compact blocks without a mempool
Of course we can! We can do it exceedingly well in fact. And we can do it in a way that sets us up for the tx graph. Aside from managing conflicting spends, there's really no need for an unconfirmed tx index unless you are mining.
I've worked out a table mod to the
validated_tx table as a SoA (multicolumn) that will allow us to scan for compact block identifiers using SIMD or GPU, though SIMD is plenty fast unless it's like a 100M txs in the pool.
The genius of compact blocks is that you have to perform a linear scan over every single unconfirmed tx for every single compact_block.tx. It cannot be indexed, because the key changes with every new compact block. It's really the same model as silent payments (stealth) scanning.
And in another stroke of batshit crazy, cb v2 operates over the wtxid, which wasn't necessary, isn't optimizing or DoS protecting, and there is no other compelling reason to store it. But it's 32 bytes per tx, which is huge. So finding a way to avoid this storage and the cost of sweeping it for every mined tx, and still validate the block in picoseconds is the challenge.
Bruno Garcia19:25
I'm (re-)running a mutation analysis for machine/interpreter.ipp - will have results soon!
👍🏻 2🎉 1
evoskuil23:26
I've renamed the
validated_bk table to state (this carries the state transitions of a block as it moves through check->accept->validate->confirm->organize (or unconfirmable). And the validated_tx table to pool, as this now holds something like a tx pool, not validation states.
Transaction metadata remains in this table after a tx is confirmed (append only), though the table is pruned once the node becomes current (along with the prevouts cache and the signature batching tables).evoskuil23:42
There was no schema change here, but there was one yesterday (schema version is now 4.0.1.0), so just rebuild tables.
Saturday, September 26, 2026
A few minor c/s protocol fixes along the way.
Bruno Garcia12:41
Can you take a look and see if this kind of mutation is valid? https://mutanthub.space/mutants/3014 basically the only one missing, otherwise 100%!
mutanthub.space
A collaborative platform for discovering, reproducing and investigating surviving software mutants. Use whichever mutation testing tool you want.
evoskuil16:01
Technically yes, the test isn't testing the batching optimization path. I'll fix it so that its covered. Thanks!
❤️ 1
Sunday, September 27, 2026
evoskuil01:59
The embedded
(edited)qrencode C lib has been replaced with our own native c++ implementation of just the parts we use. github.com/libbitcoin/libbitcoin-system/pull/2014👍 1👍🏻 1
if any isue send pr or issue on git
evoskuil18:41
System is much higher, as that's the only repo still reporting on external includes.
(edited)
Monday, September 28, 2026
evoskuil02:46
Hardware configuration... intel...... platform:1. shani...... platform:1 compiled:1. sse41...... platform:1 compiled:1. avx2....... platform:1 compiled:1. avx512..... platform:1 compiled:1. avx512ifma. platform:1 compiled:0. avxifma.... platform:0 compiled:0. aesni...... platform:1 compiled:0. vaes....... platform:1 compiled:0. gpu........ platform:0 compiled:0.
😬
6 replies · neillm, evoskuil · last Sep 28, 2026
evoskuil17:30
We're
(edited)10% faster at Taproot Tweak (part of script verification) because we compute the tweaked key as tweak·G + P with a single point addition. libsecp256k1 doesn't implement batch vectorization via SIMD, so we are are 34x faster at CPU batching on AVX512. Other configurations show 0-2% slower on scalar signature verify, but there's still a little room left to squeeze that out.
First validation on internal secp
1.41 hrs on shelly, with address index. Previous range of 1.34 hrs to 1.70 hrs tho those were without the address index.evoskuil17:56
github.com/libbitcoin/libbitcoin-system/pull/2016
Performance tuning the hash algorithm template.
evoskuil18:05
There's also shani for
(edited)sha512 on newer platforms, which I'm adding in now. Don't have a machine for it though.evoskuil20:01
Would love a review, and maybe there's some vectorization in there you can use. Getting the hardware targets right for the SIMD is the real annoyance.
(edited)3 replies · evoskuil, Vano Chkheidze · last Sep 28, 2026
Vano Chkheidze20:18
yes in batching algos are used acelarated sha algos and you can use them in other cases too
evoskuil20:19
We don't have a sha batching scenario except for Merkle root (we discussed a while back), and that's fully optimized across all available hardware.
evoskuil20:18
The internal lib allows us to be free of the dependency for the common scenario, but the support remains to pull in either
libsecp256k1 or UltrafastSecp256k1, through installer and visual studio integrated options, and currently ultrafast is the only GPU option. I haven't retested CPU batching in a while, I'll be doing that soon across all three (libsecp256k1 is scalar only, so it's the floor).Vano Chkheidze20:20
UltrafastSecp256k1 now fully dominates on libsecp256k1 in benchmarks some parity most dominatntyester day i made BCH PhotonToken miner 10x speedup from 100mh/s 5060ti moved to 1.2gh/s
based on uf cuda kernels
2 replies · neillm, evoskuil · last Sep 28, 2026
evoskuil20:50
I don't expect CPU batching will be feasible. The main reason for integration is to drop the dependency, as it's really just one more cryptographic lib among the many we require.
Library Notes ------------------------- --------------------------------------------- aes256 chacha20 chacha20_poly1305 fschacha20 fschacha20_poly1305 salsa20 xsalsa20_poly1305 poly1305 x25519 secp256k1 ecdsa, schnorr, ellswift, recover, keys, math, batch sha sha160, sha256, sha512 (all variants) sha3 keccak (k), sha3 (h224, h256, h384, h512) rmd rmd128, rmd160 hmac hkdf pbkd PBKDF scrypt siphash muhash
It was a big deal when we were able to drop OpenSSL for the smaller libsecp256k1. We were actually the first project to ship with it. But carrying that huge C lib just for this is really no longer worth it. The same issue holds for the TLS lib, it's 90% code we already carry, getting fully duplicated (400k lines of TLS C code in network, after we trimmed it back). In this age those aren't suportable.
Vano Chkheidze21:03
i' not talking about batching instead if secp256k1 why not use
UltrafastSecp256k1 uf can be without gpu side also if you dont need gpu part cpu not dependent on gpuevoskuil21:08
I was referring to CPU batching. For unbatched (scalar) signature verification I haven't seen much differentiation. Ours is already withing 0-2% of libsecp256k1, which isn't even a measurable difference over a full validation, maybe just a tiny bit more CPU usage.
But I don't control what dependencies people use. We'll definitely keep the support open. Some will want libsecp256k1 because they think it's more secure, and others will want ultrafast for better performance.
(edited)👍 1
evoskuil21:30
Version Information... boost................. 1.91.0 secp256k1............. internal openssl (tls)......... wolfssl 5.9.2 libbitcoin-system..... 4.0.9899 323645f2 libbitcoin-database... 4.0.4474 380acf4a libbitcoin-network.... 4.0.7103 752690d0 libbitcoin-node....... 4.0.4039 6cba6135 libbitcoin-server..... 4.0.3047 38fed506 compiled schema....... 4.0.1 database schema....... 4.0.1
evoskuil22:19
Hardware configuration (arm64)... crypto..... platform:1 compiled:1. sha3....... platform:1 compiled:1. neon....... platform:1 compiled:1. gpu........ platform:1 compiled:1.
evoskuil22:34
Hardware configuration (x64)... aesni....... platform:1 compiled:1. vaes........ platform:1 compiled:1. shani....... platform:1 compiled:1. sha512...... platform:0 compiled:0. sse41....... platform:1 compiled:1. avx2........ platform:1 compiled:1. avxifma..... platform:0 compiled:0. avx512...... platform:1 compiled:1. avx512ifma.. platform:1 compiled:1. gpu......... platform:1 compiled:0. Available hardware acceleration is not compiled into this build.
Note that we are detecting the
(edited)AVX512 platforms that cause downclocking (including bigmac) and reporting those as platform: 0 because the loss exceeds the gain (unfortunately, because the gain is otherwise good). But we retain the options on the newer platforms that don't downclock after its use. Core doesn't implement avx512 vectorization.
Tuesday, September 29, 2026
Pippenger is decent across the board, as it's not vectorized. But limited usefulness for batching as Schnorr is presently much lower volume than ECDSA, and it vectorizes much better with IFMA anyway. The clear target for CPU batch is
avx512ifma, as was predicted.evoskuil05:53
Because of the flexibility we have with the algorithm templates it's easy to try out different shapes and drop in intrinsics. Despite heavy platform dependency there's no
(edited)#ifdef noise in the algorithms, and it's a single for for scalar and vectorized at all three levels. We come out on top in taproot, sha1 and sha512 simply because we have the intrinsics implemented and brandx doesn't. This is all done in C++, not a single line of C or Assembly.1 reply · evoskuil · last Sep 29, 2026
evoskuil05:56
We gain most of our performance advantage from the asynchronous and concurrent network stack over the no-utxo architecture. But we aren't leaving anything on the table above either.
In other news Professor Jennings (
(edited)RPi5) is passing 900k blocks on its meager 4 threads, totally CPU bound, otherwise calm and steady. Despite growth, the straight-line completion projections are slightly declining on a per tx basis. This would have been impossible a year ago.Bruno Garcia13:16
Ran a mutation analysis for the
include/bitcoin/system/impl/chain/script_patterns.ipp file - The results are great overall - over 95% of mutation score. You can be seen the result at: mutanthub.space/mutants?project=libbitcoin%2Flibbitc…&file=script_patterns&…mutanthub.space
A collaborative platform for discovering, reproducing and investigating surviving software mutants. Use whichever mutation testing tool you want.
🔥 1
5 replies · evoskuil, Bruno Garcia · last Sep 29, 2026
Bruno Garcia17:22
After killing, you can claim it in mutanthub! You can open a mutant and tell which commit or PR kills it. Me and other people will be aware and we can verify!
1 reply · Bruno Garcia · last Sep 29, 2026
Bruno Garcia20:52
re-run the analysis, now there are only 7 surviving mutants at all! see: mutanthub.space/mutants?project=libbitcoin%2Flibbitc…&mutationStatus=SURVIVED&…
🕺🏻 1🔥 1
1 reply · pmienk · last Sep 29, 2026
Wednesday, September 30, 2026
Huddle with pmienk, eynhaender, evoskuil, Eric Chennells, Marnix, Rowan Voskuil · 329 min15:55
