Bitcoin full node and query server

May 2026

Archive of the libbitcoin Slack #general channel.

Friday, May 1, 2026

Bruno Garcia11:48
fyi https://github.com/bitcoinfuzz/bitcoinfuzz/pull/534 adds libbitcoin-system into bitcoinfuzz (differential fuzzing of Bitcoin libs/impls) in case any of you want to take a look, would be very valuable. Especially to see if the proper functions are being used.
👍🏻 1
2 replies · pmienk · last May 01, 2026
pmienk17:29
libbitcoin-system is the appropriate target for the library excluding examples and tests.
pmienk17:30
however, you may wish to consider fuzzing against other configurations, say enabling the various cpu optimizations
pmienk17:38
https://github.com/libbitcoin/libbitcoin-build/pull/414 - will add output to list the commit and origin url of github repositories used by the installer. additionally, safety net for anyone who regularly runs against local sources (me, kevin)
👍🏻 1
2 replies · pmienk · last May 01, 2026
pmienk17:38
pull requests pending against all repos in response.
pmienk17:39
oh, and will correct the test output upload issue on error - stray '"' in the generator
evoskuil19:49
Boost bug on linux.
The Linux-only test failure

(electrum__blockchain_outpoint_subscribe__two_spenders__one_notification) demonstrates that this ordering is not observed on Ubuntu 24.04 with the Boost.Asio version used in the CI jobs, even though the same code path produces the expected ordering on Windows and macOS. This can only be explained as a platform-specific behavioral difference (or bug) in Boost.Asio's epoll-based reactor and thread-pool executor implementation on Linux.
The isolated cause of the test failures is therefore the reliance, in the two-spender notification path of complete_outpoint_subscribe, on an ordering guarantee that Boost.Asio's Linux implementation does not reliably provide in this configuration.
😳 1
1 reply · evoskuil · last May 02, 2026
evoskuil01:10
Turns out this was not real, so boost is still good and I've still got one failing test. :?
evoskuil20:11
ASIO provides very specific limited ordering invariants in the use of a strand. The issue in this case was that two jobs are posted to the same strand object with the expectation that they are executed in the same order in which they are posted. ASIO provides this invariant in only in the case where the jobs are both posted or dispatched from the same strand, even when those operations originate from different, asynchronously chained handlers executing on that strand. In the failing test case this condition held, but the two jobs were posted from different handlers executing on the same strand. Given that the handlers were asynchronously chained, the call order was assured, and the conditions were therefore satisfied. However on Linux the invariant did not hold, and appears to require that the jobs be posted from the same handler on the strand (which means in the same thread).
1 reply · evoskuil · last May 02, 2026
evoskuil01:10
Turns out this was not real, so boost is still good and I've still got one failing test. :?
evoskuil20:14
There was another unrelated issue in the network lib, pertaining to the write buffering to preclude wire interleaving of concurrent sends. That affected only the new rpc notifications, not the preexisting p2p override, and has been independently resolved as well.
🔥 1

Saturday, May 2, 2026

evoskuil05:35
Still chasing down this single test bug. But in the mean time @eynhaender completed the NuGet packaging of boost-1.91.0 for vc145 (vs2022), and I was then able to complete manual setup of libbitcoin-system vs2026 builds using that and my previously published secp256k1-vc145. Clean builds and all tests passing, and that covers all dependencies for bs and bx. Over to @pmienk now to add to libbitcoin-build generation.
evoskuil05:43
I also completed a full generalization of json-rpc over websockets and raw tcp. Previously we had only support for rpc over tcp (Electrum). So with just a few lines of additional code either can be called by the channel with the exact same parameterization. This gives us the ability to derive a btcd channel in server over the same rpc channel as Electrum, but as websockets (or tcp), and also allows us to generalize Electrum to websockets. And since websockets sits on an http base channel (which sites on a tcp base channel), it means that these can both operate over http. And since bitcoind is http, it can be generalized to both tcp and websockets as well. The protocol implementations are transport independent and the channels manage socket configuration (which also already includes TLS over http, websockets, and tcp - and socks5, basic auth, and mutual auth for http).
(edited)
🎉 1
evoskuil05:51
My target with the websockets rpc has been to complete eventing via our REST interface, so that @Rowan Voskuil can integrate his previous plotting work. This gets me a little closer to that, but that won't be json-rpc over ws, it will be json/text/data over ws (analog to the REST interface queries).
evoskuil07:45
arXiv.org
The rise of AI agents is transforming how software can be built. The promise of agents is that developers might write code quicker, delegate multiple tasks to different agents, and even write a full piece of software purely out of natural language. In reality, what roles agents play in professional software development remains in question. This paper investigates how experienced developers use agents in building software, including their motivations, strategies, task suitability, and sentiments. Through field observations (N=13) and qualitative surveys (N=99), we find that while experienced de…
✅ 1

Sunday, May 3, 2026

pmienk05:18
Any opinions on CI matrix: msvc builds, with the introduction of vs2026, should we add 2 more builds to cover 2026? remove 2022?
(edited)
evoskuil13:17
I'd prefer the addition of a vs2026 build as vc145 (with boost-vc145-1.91.0) and retain vs2022 as vc143 (with boost-vc143 1.87.0, or boost-vc143 1.91.0 when available).
evoskuil13:24
I completed a full generalization of raw tcp and websockets yesterday. Formerly we had independent socket methods for read/write of a tcp socket (fixed size only, for p2p) and read/write of a ws socket (variable size, not yet used). Now there are the same read/write calls for the socket regardless of socket type for both fixed size and variable size messages (based on asio buffer type overrides). This also holds for json-rpc overrides (completed a couple days ago). This significantly reduces code and in the socket and simplifies channel implementations.
evoskuil13:30
All full duplex socket types now write through the write queue, ensuring overlap safety. These changes allow us to add websocket notifications to the existing native and admin endpoints. These can easily be tied to simple subscribe/unsubscribe ws methods, and can capture any server activity, such as logging, events, txs, blocks, etc. Those can be piped directly into the block explorer and admin console via the integrated web sites. All html endpoints are inherently http endpoints, and all http endpoints inherently support TLS and websockets. So this is just a matter of adding a websocket interface definition with channel dispatch and protocol for method subscription and implementation (all boilerplate). This also allows us to now add the simple btcd (json-rpc) websocket endpoint for lightning node support.
(edited)

Tuesday, May 5, 2026

evoskuil18:09
I've changed the behavior of the service endpoints so that they no longer are suspended during backup or delayed while the node is syncing. This just splits the behavior off into an independent behavior that can be tied to the other as we see fit. For now it's independent and since the endpoints do not presently store anything to the database, is perfectly safe to run backup and other times when p2p connections are suspended.
evoskuil18:17
One correction, the node.delay_inbound setting remains effective, it's only the suspensions that are avoided, such as h <enter> and during snapshot (automatic and <b enter>) and in the case of a detected server fault. This is intended to allow for hot snapshots and other p2p pause without service endpoint interruption.
(edited)
@Eric Chennells has joined the channel18:21
evoskuil18:25
That's the type of one hour feature addition that makes me feel productive again, after killing a couple of days on a network socket/proxy refactor. 😅
💪🏻 1
evoskuil18:43
So I believe the network work is done, and thanks to @eynhaender for running down the server/linux bug on one half of one test. Turned out to be a bug in the test harness, failing on rapid second response due to reset of the receive buffer. We are still experiencing build failures in g++ due to a compiler bug, but I think @pmienk has a handle on it. We are also now running vs2026 (native vc145) builds in CI thanks to @eynhaender’s production of boost-vc145-1.91.0 NuGet builds (the guy who produced them for 15 years retired) and @pmienk’s updated build automation. Now that we control boost packaging we should also be able to put out native ARM Windows builds as well.
4 replies · pmienk, evoskuil · last May 05, 2026
pmienk18:53
Looks like you were right, I've seen builds where we survive by supressing the warnings. Very much looks like the warning computation is causing segfault. Haven't been able to CI repro enough with g++-15/16 due to the canonical ddos to verify a fix there, however my local 15 was the first to fail about a month ago.
evoskuil18:54
Credit to Grok, I wouldn't have seen it.
pmienk18:55
I suggest we bake in the warning disable to get the CI gates working and I can spend some time trying to repro and file bugs when time allows.
👍🏻 2
pmienk18:56
huh, light at the end of the tunnel - looks like I have a g++-16 build that passed without warning suppression - so they may have fixed the issue without us
🔥 1
@LesLie has joined the channel18:51
evoskuil19:03
With the network generalizations complete we can support ws/s messaging on any http/s channel, so I'm going to now add subscription and notification to our native endpoint for emission of high speed node events:
/// Reporting events.
enum events : uint8_t
{
    /// Candidate chain.
    header_archived,     // header checked, accepted
    header_organized,    // header pushed (previously archived)
    header_reorganized,  // header popped

    /// Blocks.
    block_archived,      // block checked
    block_buffered,      // block buffered for validation
    block_validated,     // block checked, accepted, connected
    block_confirmed,     // block checked, accepted, connected, confirmable
    block_unconfirmable, // block invalid (after headers-first archive)
    validate_bypassed,   // block checked, accepted [assumed]
    confirm_bypassed,    // block checked, accepted, connected [assumed]

    /// Transactions.
    tx_archived,         // unassociated tx checked, accepted, connected
    tx_validated,        // associated tx checked, accepted, connected
    tx_invalidated,      // associated tx invalid (after headers-first archive)

    /// Confirmed chain.
    block_organized,     // block pushed (previously confirmable)
    block_reorganized,   // block popped

    /// Mining.
    template_issued,      // block template issued for mining

    /// Timespans.
    snapshot_secs,        // snapshot timespan in seconds.
    prune_msecs,          // prune timespan in milliseconds.
    reload_msecs,         // store reload timespan in milliseconds.
    block_usecs,          // getblock timespan in microseconds.
    ancestry_msecs,       // getancestry timespan in milliseconds.
    filter_msecs,         // getfilter timespan in milliseconds.
    filterhashes_msecs,   // getfilterhashes timespan in milliseconds.
    filterchecks_msecs    // getcfcheckpt timespan in milliseconds.
};
These are internally subscribable in the endpoint protocol class and can then be filtered for emission to the endpoint notification subscriber.
@Sur Yoyo has joined the channel19:23

Wednesday, May 6, 2026

@colourorange has joined the channel07:29

Thursday, May 7, 2026

E
Eric Chennells15:52
will there be a meeting today?
1 reply · pmienk · last May 07, 2026
pmienk15:58
Feel free to join anytmie
pmienk15:53
I usually start it in 3 minutes, start time is declared as 9:00am PST, 12:00pm EST
Huddle with pmienk, evoskuil, eynhaender, Eric Chennells, Rowan Voskuil, neillm, Marnix, parsevalbtc · 176 min15:53
6 replies · Rowan Voskuil, neillm, Eric Chennells, pmienk +1 · last May 07, 2026
Rowan Voskuil16:44
[database] turbo = true filter_bk_size = 14,000,000 filter_bk_buckets = 900,001 filter_tx_size = 2,700,000,000 filter_tx_buckets = 900,001 address_size = 30,000,000,000 address_buckets = 1,315,297,540
Rowan Voskuil16:44
[native] bind = 0.0.0.0:80 bind = [::]:80 connections = 100
neillm17:48
Gotta hop, thanks all
👍🏻 1
E
Eric Chennells18:01
Thanks again Eric for the overview and enjoyed listening getting an idea of the project. Looking forward to being useful around here.
👍🏻 1
pmienk18:37
D:/a/libbitcoin-server/libbitcoin-server/libbitcoin-server/test/protocols/native/native_setup_fixture.cpp(167): fatal error: in "native_tests/native__top__json__expected": critical check !ec has failed D:/a/libbitcoin-server/libbitcoin-server/libbitcoin-server/test/protocols/native/native_setup_fixture.cpp(125): fatal error: in "native_tests/native__top__text__expected": critical check !ec has failed D:/a/libbitcoin-server/libbitcoin-server/libbitcoin-server/test/protocols/native/native_setup_fixture.cpp(141): fatal error: in "native_tests/native__top__data__expected": critical check !ec has failed D:/a/libbitcoin-server/libbitcoin-server/libbitcoin-server/test/protocols/native/native_block.cpp(52): fatal error: in "native_tests/native__top__formal_xml__not_acceptable": critical check status == http::status::not_acceptable has failed [OK != Not Acceptable] *** 4 failures are detected in the test module "libbitcoin_server_test"
pmienk18:37
replied to a thread: view
D:/a/libbitcoin-server/libbitcoin-server/libbitcoin-server/test/protocols/native/native_setup_fixture.cpp(167): fatal error: in "native_tests/native__top__json__expected": critical check !ec has failed D:/a/libbitcoin-server/libbitcoin-server/libbitcoin-server/test/protocols/native/native_setup_fixture.cpp(125): fatal error: in "native_tests/native__top__text__expected": critical check !ec has failed D:/a/libbitcoin-server/libbitcoin-server/libbitcoin-server/test/protocols/native/native_setup_fixture.cpp(141): fatal error: in "native_tests/native__top__data__expected": critical check !ec has failed D:/a/libbitcoin-server/libbitcoin-server/libbitcoin-server/test/protocols/native/native_block.cpp(52): fatal error: in "native_tests/native__top__formal_xml__not_acceptable": critical check status == http::status::not_acceptable has failed [OK != Not Acceptable] *** 4 failures are detected in the test module "libbitcoin_server_test"
evoskuil19:49
replied to a thread: view
👍🏻 1
evoskuil21:36
Possibly unpopular opinion... defensive programming hides bugs.
1 reply · neillm · last May 07, 2026
neillm22:21
AI doesn't disagree 😅
Firefox_Screenshot_2026-05-07T22-19-34.757Z.png
buggy_function.py Python · 5.1 KB
import logging

# Setup basic logging to make the "defensive" aspect visible
logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s')

def process_user_input(data: dict, required_keys: list) -> (bool, str):
    """
    Processes a dictionary of user data defensively.
    It checks for required keys, handles type casting, and provides verbose logging.

    Intended to be robust against missing keys or wrong data types.
    """
    logging.info("--- Starting process_user_input ---")
    
    # 1. Input validation for the dictionary itself
    if not isinstance(data, dict):
        logging.error("Input data must be a dictionary.")
        return False, "Input data must be a dictionary."

    # 2. Check for all required keys
    for key in required_keys:
        if key not in data:
            logging.warning(f"Missing required key: '{key}'. Returning failure.")
            return False, f"Missing required key: {key}"

    # 3. Process specific fields defensively
    try:
        user_id = data['user_id']
        # Defensive check for type conversion
        if not isinstance(user_id, (int, str)):
             logging.warning("user_id is not an integer or string.")
             return False, "user_id must be string or int."
        
        # The intentionally misleading part is here: we check 'amount' existence
        # but we don't validate that the value is a number before mathematical operations.
        amount = data['amount']
        if amount is None:
             logging.warning("Amount found but is None.")
             return False, "Amount cannot be None."
full file
👍🏻 1
evoskuil23:13
Websocket upgrade request is now flowing through native endpoint via functional tests. I've added these methods to the native interface:
method<"top_subscribe", uint8_t, uint8_t>{ "version", "media" },

method<"block_subscribe", uint8_t, uint8_t>{ "version", "media" },
method<"tx_subscribe", uint8_t, uint8_t>{ "version", "media" },
method<"input_subscribe", uint8_t, uint8_t, system::hash_cptr, uint32_t>{ "version", "media", "hash", "index" },
method<"output_subscribe", uint8_t, uint8_t, system::hash_cptr, uint32_t>{ "version", "media", "hash", "index" },
method<"address_subscribe", uint8_t, uint8_t, system::hash_cptr, uint32_t>{ "version", "media", "hash", "turbo" },

method<"log_subscribe", uint8_t, uint8_t>{ "version", "media" },
method<"event_subscribe", uint8_t, uint8_t>{ "version", "media" }
🎉 1
evoskuil23:24
The last two will move to the admin interface (security), and will eventually be parameterized for filtering. Top is just height. block/tx are unparameterized. input is spender of a point, output is a point, address is a scripthash. All accept version and media type. All can be executed in the REST interface, though notifications will not flow until the socket is upgrade to ws. Initial subscriptions for top, input, output, address will provide the current result set and then notify going forward.
I'll probably just make the json request model closely mapped to a json-rpc model, and will just generalize it so that any REST request can also be made via the WS. So fully interchangeable, but subscriptions can only flow thru ws.
And lots more things for Kevin to test! 🤓
🫡 1
The above list essentially adds a subscription to each existing category of requests. So by adding a /subscribe to the path/parser, it's easy to follow and implement.

Friday, May 8, 2026

evoskuil17:29
Reminder that the task to implement full chain and machine unit testing is still open and a release dependency.
evoskuil18:42
Heads up nerds. If you haven't noticed critical Linux kernel vulnerabilities have exploded over the past couple years. This is not due to their reporting change, it's real. There is no question this is coming from AI. That has reduced the cost of the discovery, making Linux worthwhile to target. Windows and Mac critical vulns have remained flat. Linux is finally catching up. Be careful out there.
IMG_1501
👍 1
Rest of the same table.
IMG_1502
Sorry, macOS not really flat, significant growth too, despite a dramatically smaller base for both desktop and server.
@Diego has joined the channel22:19

Saturday, May 9, 2026

eynhaender03:52
sparrow wallet talking to libbitcoin v4. As an electrum server and as block explorer.
(edited)
😮 2🔥 1
33 replies · evoskuil, neillm, eynhaender · last May 09, 2026
evoskuil03:53
Sweeeet!
💯 1
evoskuil03:54
Any rough edges left?
neillm03:54
If you get a testnet version going and need further testing, let me know, I have plenty of testnet coins 🙂
neillm03:55
Hrm, don't know of Sparrow either, but will check it out
eynhaender03:56
Yea, I have 2 unknown errors on my list. But I will figure it out tomorrow.
👍🏻 1👍 1
evoskuil03:56
We haven't run test net in v4 yet, but should be just config as in v3.
eynhaender03:57
Does the electrum (protocol) guy have a twitter account?
evoskuil03:57
Testnetv3
evoskuil03:57
Dunno
neillm03:58
I was telling @eynhaender we need to have some stable (ship ready) configs for 'mainnet', 'testnet' and 'regtest' at some point. Can just sit as templates in the data dir IMO
evoskuil03:59
we used to have that. Gotta maintain them tho.
evoskuil03:59
The config is two forks and some different numeric values in the bitcoin section.
neillm04:00
Yeah, but it makes it really easy for new folks to get acquainted
evoskuil04:00
Also have to have it in the online documentation.
💯 1
neillm04:02
bs.cfg -> mainnet.cfg | testnet.cfg | regtest.cfg 🤔
evoskuil04:03
testnet3.cfg
neillm04:03
^Yes, that too
evoskuil04:03
They got testnet4 now, which is all horked up. We need to add that fork.
evoskuil04:04
And signet 🙄
neillm04:05
It's almost like we want to have configs for supported bitcoin networks 😛
evoskuil04:05
Also litecoin and its various test nets.
evoskuil04:05
Tho we haven't kept up with its forks.
neillm04:06
I like Litecoin personally, but libbitcoin is NOT liblitecoin
evoskuil04:06
You should check the book. It's not libbtc.
neillm04:07
Ah, ok, then add supported configs?! 🤔
👍🏻 1
evoskuil04:13
I didn't add the forks initially, but I've maintained them. If it's not a shit coin and doesn't deviate much, then it's easy.
evoskuil04:14
It was that work that initially came from Lucas Betschart when he was doing contract work on feathercoin.
evoskuil04:15
The work to generalize the configuration.
neillm04:15
^I recall, I was a fan of feathercoin too 😇
evoskuil04:15
So that's why we don't have to recompile to do testnet, etc.
evoskuil04:16
I'm surrounded by shitcoiners.
😅 2
neillm04:29
🤣
File deleted
neillm04:32
signal-2026-05-09-003200.mp4 MPEG 4 Video · 578.5 KB
I think I'm going to make native websockets queries use the exact query as the corresponding http interface on the same endpoint. This would mean the websocket request is a string value matching the corresponding uri and query string. So the server just passes this on to the same parser, and then dispatches the resulting internal rpc object to the existing dispatcher.
The avoids creating an independent query language for websockets vs REST and we can properly call both the REST interface. This would make them formally identical and allow them to share the same parser and unit tests.
👍 1
The only difference apart from message framing and lack of http headers is that only ws connections can receive notifications.
1 reply · neillm · last May 10, 2026
neillm04:04
I think that sounds reasonable -- I can always complain later 🤣
👍🏻 1

Tuesday, May 12, 2026

R
Raimo10:22
are these known areas of improvement? or is Claude making these up? https://claude.ai/share/92757401-d515-40bd-a837-c3739ae978e3
5 replies · evoskuil · last May 12, 2026
evoskuil13:18
No, surprisingly most of these are bogus. The rest are either not applicable or not significant.
evoskuil13:26
1-4 and 6 are bogus. 5 is correct but not relevant (lacks context). 7 is correct and has been investigated, but denormalizing and not significant. 8 is not significant. 9 has been done repeatedly over years and may not be strictly optimal but should be very close. It has hallucinated a std lib replacement for boost multiprecision.
evoskuil13:33
10 is a reference to what is tested in CI. Packaged LTCG builds exist for windows, and these produce dramatically slower runtime. The comment about hash CU call boundaries is incorrect, these are all implemented in headers. Linux LTO builds should be tested however.
evoskuil13:47
It seems that it may have consumed /version3 not /master. The reason I know these off the top of my head is that much if this is work that was done for v4.
evoskuil13:48
Otherwise I'd say these are seasonable areas to investigate, but it should be able to see they are bogus.
eynhaender11:18
There a various areas of improvement. Many of those are highlighted by GROK, some by Claude, some by local LLMs and most of them by our team that uses these AI tools. This brings up many things and many of them are substantial where others might be just "made up". Thanks for sharing your prompt 👍🏻
(edited)
evoskuil13:18
replied to a thread: view
No, surprisingly most of these are bogus. The rest are either not applicable or not significant.
evoskuil13:26
replied to a thread: view
1-4 and 6 are bogus. 5 is correct but not relevant (lacks context). 7 is correct and has been investigated, but denormalizing and not significant. 8 is not significant. 9 has been done repeatedly over years and may not be strictly optimal but should be very close. It has hallucinated a std lib replacement for boost multiprecision.
evoskuil13:33
replied to a thread: view
10 is a reference to what is tested in CI. Packaged LTCG builds exist for windows, and these produce dramatically slower runtime. The comment about hash CU call boundaries is incorrect, these are all implemented in headers. Linux LTO builds should be tested however.
evoskuil13:47
replied to a thread: view
It seems that it may have consumed /version3 not /master. The reason I know these off the top of my head is that much if this is work that was done for v4.
evoskuil13:48
replied to a thread: view
Otherwise I'd say these are seasonable areas to investigate, but it should be able to see they are bogus.
evoskuil22:55
X (formerly Twitter)
Libbitcoin is coming. 👊
🔥 3🎉 3

Thursday, May 14, 2026

evoskuil15:35
Wow, about to cross 950,000.
🎉 1
1 reply · neillm · last May 14, 2026
neillm15:38
Screenshot (2).png
🔥 1
evoskuil15:38
There's an optimization I was planning to implement to get our time back down under an hour for 950k milestone. Might have to do that soon. Skip p2p block deserialization under a milestone. - just extract the hashes from the stream (as we now do) and forward it straight to the store. I made this optimization for outbound blocks a few months ago, because it was a little simpler. But that doesn't help our sync time.
Actually, this is not even limited to milestone, since we write all blocks directly to the store. We only need the deserialization when reading them back out for validation and/or client filters population. So it also provides a significant benefit for validation, as it would consume a huge amount less memory.
(edited)
Huddle with pmienk, evoskuil, neillm, Eric Chennells, Rowan Voskuil, eynhaender, Marnix · 302 min15:52
3 replies · neillm, Eric Chennells, pmienk · last May 14, 2026
neillm17:07
Gotta hop, thanks all!
👍 1👍🏻 1
E
Eric Chennells17:42
"nothing from my end" as they say but was great listening and getting more context. Thanks everyone.
👍🏻 2
pmienk18:22
Gotta hop off, catch you all next week
👍🏻 1

Friday, May 15, 2026

Saturday, May 16, 2026

1 reply · neillm · last May 16, 2026
neillm02:04
Yeah, you're killing it 🤷‍♂️
pmienk03:28
phillip@vali:~/code/libbitcoin/libbitcoin-build$ ~/test/cmake/prefix/bin/bs --version
Version Information...
libbitcoin-server:     4.0.2099
libbitcoin-node:       4.0.3592
libbitcoin-network:    4.0.6683
libbitcoin-database:   4.0.3839
libbitcoin-system:     4.0.9128
🔥 3
1 reply · pmienk · last May 16, 2026
pmienk03:29
one down, 2 to go
evoskuil17:59
Websocket notifications over the native http/s, ws/s interface are now integrated for top height, organized block, and organized unconfirmed tx (though we don't yet organize unconfirmed txs). Subscription can be done over http or ws, where the http subscription is RESTful and the websocket subscription is the same but with the full path in the websocket text body. Current value is returned for top/block and empty for tx (there's no canonical tx). These are returned in the media type requested as always, and the websocket mediate type defaults to json and can otherwise be specified in the query string (as with http). The mediate type specified upon subscription is retained for the life of the subscription and all notifications for that subscription type are issue with that media type (json/text/data). Unsubscription is the same path with the ?stop=true parameter. The return values for the unsubscribe are currently the same as for subscribe (current top/block, empty for tx).
(edited)
🎉 1
evoskuil18:10
The subscribers for address and for spender are stubbed in but need a little more work to managed the statefulness (limited map of subscriptions to specific addresses). Also the subscribers for log messages and high speed events are implemented and should be complete shortly. These initially just subscribe to log messages and all events. I'll be adding a numeric flags parameter to the query string for both, where the numbers map to the desired log subscriptions and event subscriptions. Sending empty flags will therefore be equivalent to sending stop for unsubscribe. The response from registration is currently empty. These are not the same events that drive the chaser system (those carry context such as block header surrogate key). The chasers drive the above top/block/tx flow. The high speed events are much simpler and cover really anything we want, including internal diagnostics and timings. Since http is half duplex, subscribing will not result in any notification traffic until the http connection is "upgrade" to websocket. But the subscriptions remain active.
evoskuil18:19
The subscriptions are:
/v1/top/subscribe
/v1/block/subscribe
/v1/tx/subscribe

/v1/address/[scripthash]/subscribe
/v1/output/[txhash]/[index]/subscribe
address and output currently parse/dispatch but are not implemented. log/event subscriptions aren't currently parsed, and these need to move to the admin interface. Defaults example:
/v1/top/subscribe?format=json&stop=false
(edited)

Sunday, May 17, 2026

evoskuil04:33
The websocket notification work is done and functional tested in the server native interface for top/block/tx. That will get the block explorer hooked in without polling. The log/event notifications are also all ready to flow to the admin interface. Lots of cool presentation we can get out of that. Can add all sorts of statistical, diagnostic, and performance info, and can add a lot more to the native interface as well.
👍 2
I've moved the fee estimator from server to node, as it didn't really have a proper home in server. In node it just drops into a new chaser_estimate and sits with all of the others. I expect that to be all hooked up and flowing through the electrum endpoint by tomorrow.
(edited)

Tuesday, May 19, 2026

evoskuil03:45
The fee estimator class is integrated into a new node::chaser_estimate, initialized when the node becomes current, and maintained from that point forward until shutdown. It's maintains its stack-based accumulation over a moving window of 1008 blocks (max/default) even if the node falls out of currency, and traverses reorgs. Calls are asynchronous and passed from a protocol (e.g. electrum fee estimation method) to the base node::protocol, to the base node::channel, to the base node::session to the node::full_node to the node::chaser_estimate and finally to its contained estimator. This is a standard flow in the network/node architecture. The completion hander passed from the electrum method is invoked on the chaser strand, and then the electrum handler bounces it back to the electrum protocol strand for send of the tcp response. The estimator is modeled after bitcoind fee accumulation, as this is what Electrum expects. It uses only block fee state as a very good approximation of current fee rates in comparison to analysis over unconfirmed txs. The estimator is fully unit tested, and the electrum endpoint is fully tested with empty fee data. Building a fee history for more robust testing is now over to @eynhaender for acceptance testing.
🚀 1
evoskuil05:18
In honor of block 950,000, which dropped today, we'll be updating our baseline performance numbers and default milestone. This will likely put us over 1hr on the mini and around 5.0 hrs for full validation on the maxi. In an effort to keep the former under 1.0 hrs, I'm looking into the inbound analog to the optimization I made a few months ago, which skips object model construction when emitting full blocks or txs in wire or base16 serialization. This sped up block query a lot, but the inbound analog may significantly improve sync. Specifically, by not parsing a block into the full object model, we can save up to 95% the memory consumption per block, since by default blocks allocate 20x their wire size using the linear accumulator. This is a big performance boost over default allocation, however at a large memory impact. Also the deserialization itself has a modest CPU hit. Presently we obtain the txid and wtxid for each tx directly from the wire buffer. These are attached as cache within each tx object, which significantly optimizes several downstream operations, including witness and non-witness merkle root derivation. In block download we presently perform block::identity checks when under checkpoint/milestone, and full block::check() when validating. However identity is sufficient, and requires not much more from the parse than the tx hashes (also requires the coinbase witness commitment and reservation). By moving block::check to the validation phase (when full block parse is already performed for validation) we retain the simplicity of block check methods, and only have to perform block::identity using parsed hashes and coinbase data. So tonight I'm running a maxi full validation with block::check moved to the validator (full parsing on download still). This will likely not be much of a difference in time. The next step would be to implement the identity checks in network::messages::block (presently they are performed by chain::block methods, where the message object contains the block object). In this design the identity check can be implemented by the message or passed through to the block if its been parsed (and we can implement on-demand deferred parse). The message would carry the moved socket buffer, which is then used to directly populate the store, with zero intermediate allocations. That will require a new query to write the block directly from the wire buffer into the tables. This can be done very efficiently using our rewindable fast stream parser, which is also how we obtain hashes from the buffer presently. So there are a few steps to this. (1) parse the coinbase of witness txs to obtain the reservation and commitment hashes, and (2) move the socket buffer to the block message with a null contained block object, along with parsed tx hashes, (3) expose block identity methods on the block message for node verification, (4) implement direct buffer/hash deserialization to store serialization. But the biggest hurdle I faced in the past was dealing with block::check without a parsed block. The realization that this could move to the validator without a material performance hit makes this a lot easier.
🎉 2
evoskuil13:38
Full maxi validation to 900k in 4.55 hrs with block::check moved from download protocol to validation chaser. My previous run was 4.67 hrs so it's safe to say that this change is not a performance degradation.
(edited)
🎉 2

Wednesday, May 20, 2026

evoskuil14:40
Full maxi validation to 950k in 5.55 hrs with essentially same implementation above. That's looks like it might be a non-linear increase, which means it's also time to resize the hashmaps again. Aligning this with our benchmark/milestone bump seems to makes sense.
(edited)
evoskuil20:39
At the top of the list of things I did not think I would see today, is this post from sipa setting the stage for Bitcoin Core to become an SPV node. https://github.com/bitcoin/bitcoin/pull/35054#issuecomment-4501781606
(edited)
👀 1
evoskuil23:22
One year ago tomorrow @Rowan Voskuil joined the libbitcoin team!
🎉 8

Thursday, May 21, 2026

Huddle with pmienk, Marnix, neillm, evoskuil, Rowan Voskuil, Eric Chennells · 133 min15:54
1 reply · neillm · last May 21, 2026
neillm17:01
Gotta hop, thanks all!
👍 1👍🏻 1
evoskuil19:42
First AI patch... hope this gets better. https://github.com/libbitcoin/libbitcoin-network/pull/814
(edited)
😅 1

Friday, May 22, 2026

eynhaender16:25
More Electrum? Here it comes: https://github.com/sparrowwallet/frigate
Seems as if every wallet will eventually bring its own electrum indexer.
evoskuil19:29
It's just one more Electrum subscribe method and a simple single table indexed in the validation chaser (same as client filters). 🤷🏻‍♂️
And BTW it's basically stealth payments, which we had 15 years ago and I eventually dropped. All the math primitives are system.
12 replies · josie, evoskuil · last May 23, 2026
josie17:44
it's a little different and worth mentioning: silent payments doesn't use an op_return for the shared secret, it uses the input public keys. this means you can't know whether a transaction is a silent payment unless you are the recipient. excellent for privacy, which was the goal (im one of the authors of the protocol). the trade-off is recipients must check every transaction with taproot outputs with an elliptic curve multiplication. for a public server supporting multiple cliente, this can be a heavy load on a server. the main thing frigate is doing is using GPUs for checking multiple transactions at a time for clients. could also be done with just CPU, but it's gonna be slow.
(edited)
josie17:45
for someone using libbitcoin as a personal server, not at all a problem and way better than the current setup where someone would need to run frigate + bitcoind + an electrum server
evoskuil18:09
Sure, in our indexing design we chose an op_return padded length that conflicted with more common op_return uses for this reason. Moving it into keys is obviously better hiding. But the rest is the same, the recipient still has to scan a huge number of txs with ec mult cost. But that can now be vectorized in GPUs, which has made it reasonable again. I dropped this index in bs in part because of the O(N) cost of every query. I talked with Craig about it last night. I think we could pick op the additional dependency conditionally for public deployment, it's pretty clean. But yeah, for personal use it's a much easier performance problem.
(edited)
josie18:14
nice! if you're gonna go the GPU route, frigate is using ultrafastsecp. this means you could also do full verification on a gpu which i suspect would get your full validation closer to a milestone run in speed
evoskuil18:17
Yes, I looked at it. Dependency-free C++ MIT, we could probably take it. It may not help much for validation, have to look into that. But we hate dependencies, and this is a big one.
evoskuil18:17
Really you should be working on this with us.
josie18:19
would love to! if there's appetite from your team to review , id be happy to take a swing at it. full disclosure, im a bit overcommitted at the moment but should have more time in june
❤️ 1👍🏻 1
evoskuil18:22
Yes, I'd really love to add this to our electrum interface. It's just one method, and the table/index is trivial. We could have all of the plumbing integrated in 2-3 days. I believe have all of the CPU primitives, well integrated and tested in nice C++ abstractions in libbitcoin-system.
evoskuil18:24
I'd also like to have a realistic analysis of what this lib can do for signature validation. Like is it just a question of marshalling, or are their other issues.
evoskuil18:26
If it made sense we could marshal millions of signatures at a time. But that's some work and we need to know what we can realistically get from it first.
(edited)
evoskuil18:27
The other aspect which might be worth looking at first, is personal server indexation.
evoskuil18:29
Our priority has to be on getting v4 out, but would love to have this moving forward at the same time.
👍 1
evoskuil19:44
Table size using our indexation (41 bytes per row, estimated ~200m current rows), that's a 7.6GiB index. This is 41% of the table size of the sparrow server (100 bytes per row), since we can use surrogate tx keys and relational data retrieval.
Total time start to completion, 1 day for system math/test, 1 day for database archival/query/test, 1 day for Electrum subscriber method/test.
12 replies · pmienk, evoskuil, eynhaender · last May 23, 2026
pmienk19:51
so... they take 3 days to viably serve up to a wallet? fully validated?
evoskuil19:56
These are my estimated times to implement the additional electrum method in libbitcoin.
pmienk19:56
ahh, cool
pmienk19:57
time to eating their lunch
😅 1
evoskuil19:57
what was the other optional index were were talking about adding recently?
evoskuil19:58
If I was going to do one table addition I would do them both, wouldn't take much more time.
pmienk20:00
I don't recall, it was in the meeting yesterday?
evoskuil20:04
No, last couple weeks.
eynhaender20:24
inscriptions
💥 1
evoskuil20:31
How did I forget 🙄
🤣 1
eynhaender20:32
lol
evoskuil18:15
I looked into this indexation more closely, and the real cost is the computational cost of a linear scan of essentially the entire tx history (a subset of all taproot txs) for every query. This is the same essential problem as stealth payments had, but this is heavily parallelizable, and can complete the history scan in like 10 seconds currently. But that requires an external dependency for server. Clean C++ MIT dependency, but a dependency nonetheless. OTOH as a personal server the problem is entirely manageable on a CPU.
👍🏻 1

Saturday, May 23, 2026

evoskuil18:03
X (formerly Twitter)
@gladstein Stealth payments, based on the same principle, existed over 10 years ago. IIRC it was proposed by a @peterktodd. It was integrated into Dark Wallet by @lunardragon420 and indexed by Libbitcoin Server. There was a proposed BIP that was never completed due to lack of adoption.
evoskuil20:24
X (formerly Twitter)
@MrHodl See https://t.co/bUnPYG5SYw and https://t.co/K291fHYLHg (and many other small ones). If we can get enough reviewers these could probably get into v32.
evoskuil20:31
I lost track of how many times going back many years that were were told that bitcoind was maximally optimized and utterly flawless - not an exaggeration. It was only when libbitcoin v4 started showing performance numbers that any real effort went into IBD. It directly spawned "swift sync" and related projects, forced maintainers to start reviewing optimization PRs, and resulted in funding for both. Good thing there is some actual competition, as otherwise they would have already abandoned validation. We've stalled that plan.
(edited)
💯 3
evoskuil23:46
The block deserialize plan above is now implemented. came out cleaner than I anticipated. Each of the 4 steps above took a day, now I'm debugging through it. Once the node is running I'll implement the unit tests, prob another day. Looking forward to having it running so can first get a measure of the benefit. I think I will end up removing the linear allocator. This will definitely be more efficient than that at a minimum, simpler, and restores some fundamental code to a much simpler and more readable state.

Sunday, May 24, 2026

eynhaender02:47
I spent some tokens to improve the scripts I use to generate my benchmarks. The data, well, speak for themselves.
libbitcoinpi_compare.png
👊 2🔥 2🚀 3
evoskuil15:44
X (formerly Twitter)
You all know that Libbitcoin v4 syncs up an order of magnitude faster on bigger machines, but did you know that with a few settings changes it is also much faster on a 16GB Raspberry Pi?

Monday, May 25, 2026

neillm02:12
Well that's some good light-hearted reading! 😅
evoskuil02:23
So far the arguments are: (1) Libbitcoin sucks. (2) You can trust Core in place of validation (it's in the process). (3) There's no central authority in the BIP, but see #2. (4) Hosting our trust blobs is expensive, so you do it. (5) Sync is slow, and if you do it fast you're hurting the network. (6) Validation is slow, and we will solve that by dropping that feature. (7) Sipa wants it, that comment about not validating wasn't supposed to be released yet. (8) It's really SPV but we don't want to have to change our wallet, so take this. (9) Satoshi said only use his software.
😞 4😄 1

Tuesday, May 26, 2026

evoskuil13:29
The wire deserialization optimization is running. performance is definitely improved. This bypasses the linear memory allocator for block construction so that becomes dead code. I'll remove that from the block constructors and related code, which will be simplifying. Still haven't had a chance to complete performance testing on low memory, those numbers should be available in the next couple days. But on the maxi the first 500k blocks in a milestone run (1/4 of the 900k chain) complete at about 70% of the best previous times up to that point.
🎉 2
1 reply · evoskuil · last May 26, 2026
evoskuil23:09
This has been merged.
🎉 1
E
Enligths19:37
After building libbitcoin-server with MSVC v2022, I wanted to write a small program similar to the ones available on libbitcoin-system wiki. https://github.com/libbitcoin/libbitcoin-system/wiki/Examples-from-Serialised-Data This example was written with libbitoin v3 but this is not my current problem. First I tried to run a hello-world program where the following header is present: #include <bitcoin/system.hpp>
#include <iostream>
#include <bitcoin/system.hpp>
using namespace std;

int main()
{
    cout << "Hello, world, from Microsoft C++!" << endl;
}
I am using the <cl> compiler provided with Visual Studio through command line. I added to %LIB path all the lib files generated during the bild process and all the header folders in %INCLUDE path.
REM ============================================================================
set include=C:\Users\tanan\source\repos\libbitcoin-server\include;%include%
set lib=C:\Users\tanan\source\repos\libbitcoin-server\bin\x64\Debug\v143\static;%lib%

set include=C:\Users\tanan\source\repos\libbitcoin-node\include;%include%
set lib=C:\Users\tanan\source\repos\libbitcoin-node\bin\x64\Debug\v143\static;%lib%

set include=C:\Users\tanan\source\repos\libbitcoin-network\include;%include%
set lib=C:\Users\tanan\source\repos\libbitcoin-network\bin\x64\Debug\v143\static;%lib%

set include=C:\Users\tanan\source\repos\libbitcoin-database\include;%include%
set lib=C:\Users\tanan\source\repos\libbitcoin-database\bin\x64\Debug\v143\static;%lib%

set include=C:\Users\tanan\source\repos\libbitcoin-system\include;%include%
set lib=C:\Users\tanan\source\repos\libbitcoin-system\bin\x64\Debug\v143\static;%lib%
REM ============================================================================


REM ============================================================================
set include=C:\Users\tanan\source\repos\.nuget\packages\secp256k1_vc143.0.6.1\build\native\include;%include%
set lib=C:\Users\tanan\source\repos\.nuget\packages\secp256k1_vc143.0.6.1\build\native\bin;%lib%

set include=C:\Users\tanan\source\repos\.nuget\packages\boost.1.86.0\lib\native\include;%include%
set lib=C:\Users\tanan\source\repos\.nuget\packages\boost_iostreams-vc143.1.86.0\lib\native;%lib%
set lib=C:\Users\tanan\source\repos\.nuget\packages\boost_locale-vc143.1.86.0\lib\native;%lib%
set lib=C:\Users\tanan\source\repos\.nuget\packages\boost_program_options-vc143.1.86.0\lib\native;%lib%
set lib=C:\Users\tanan\source\repos\.nuget\packages\boost_thread-vc143.1.86.0\lib\native;%lib%
set lib=C:\Users\tanan\source\repos\.nuget\packages\boost_unit_test_framework-vc143.1.86.0\lib\native;%lib%
set lib=C:\Users\tanan\source\repos\.nuget\packages\boost_url-vc143.1.86.0\lib\native;%lib%
REM ============================================================================
I got the following error (partially in french). It seems to me the compiler managed to find the header but got stuck somewhere along the lines. Maybe something is missing the compiler command. Is there someone that can help me?
C:\Users\tanan\code\libbitcoin-example>cl /EHsc hello.cpp

Compilateur d'optimisation Microsoft (R) C/C++ version 19.44.35227 pour x64
Copyright (C) Microsoft Corporation. Tous droits réservés.

hello.cpp
C:\Users\tanan\source\repos\libbitcoin-system\include\bitcoin/system/preprocessor.hpp(185): error C2338: static_assert failed: 'C++20 minimum required.'
Please define _WIN32_WINNT or _WIN32_WINDOWS appropriately. For example:
- add -D_WIN32_WINNT=0x0601 to the compiler command line; or
- add _WIN32_WINNT=0x0601 to your project's Preprocessor Definitions.
Assuming _WIN32_WINNT=0x0601 (i.e. Windows 7 target).
C:\Program Files\Microsoft Visual Studio\18\Community\VC\Tools\MSVC\14.44.35207\include\bit(11): warning STL4038: The contents of <bit> are available only with C++20 or later.
C:\Program Files\Microsoft Visual Studio\18\Community\VC\Tools\MSVC\14.44.35207\include\variant(12): warning STL4038: The contents of <variant> are available only with C++17 or later.
C:\Users\tanan\source\repos\libbitcoin-system\include\bitcoin/system/types.hpp(189): error C3646: 'data_t' : spécificateur de substitution inconnu
C:\Users\tanan\source\repos\libbitcoin-system\include\bitcoin/system/types.hpp(189): note: le contexte d'instanciation du modèle (le plus ancien) est
C:\Users\tanan\source\repos\libbitcoin-system\include\bitcoin/system/types.hpp(186): note: lors de la compilation du modèle de classe 'libbitcoin::data_t'
C:\Users\tanan\source\repos\libbitcoin-system\include\bitcoin/system/types.hpp(189): error C2059: erreur de syntaxe : '('
C:\Users\tanan\source\repos\libbitcoin-system\include\bitcoin/system/types.hpp(190): error C2334: jetons inattendus avant ':' ; corps apparent de la fonction ignoré
C:\Users\tanan\source\repos\libbitcoin-system\include\bitcoin/system/types.hpp(27): fatal error C1075: '{' : jeton correspondant introuvable
(edited)
pmienk19:46
Seems to say pretty early your are targeting a version of c++ below the minimum
C++20 required.
E
Enligths20:19
I added /std:c++20 option but still I got this error:
C:\Users\tanan\code\libbitcoin-example>cl /EHsc /std:c++20 hello.cpp
Compilateur d'optimisation Microsoft (R) C/C++ version 19.44.35227 pour x64
Copyright (C) Microsoft Corporation. Tous droits réservés.

hello.cpp
C:\Users\tanan\source\repos\libbitcoin-system\include\bitcoin/system/preprocessor.hpp(185): error C2338: static_assert failed: 'C++20 minimum required.'
Please define _WIN32_WINNT or _WIN32_WINDOWS appropriately. For example:
- add -D_WIN32_WINNT=0x0601 to the compiler command line; or
- add _WIN32_WINNT=0x0601 to your project's Preprocessor Definitions.
Assuming _WIN32_WINNT=0x0601 (i.e. Windows 7 target).
I replaced the code inside hello.cpp with an example from cppreference using some c++20 features and it compiled with success. So I am missing something when trying to include bitcoin\system.hpp
pmienk20:52
Are you lacking the import files located in builds/msvc/vs2022?
evoskuil23:09
replied to a thread: view
This has been merged.
🎉 1

Wednesday, May 27, 2026

evoskuil07:38
#if __cplusplus >= 202002L
    #define HAVE_CPP20
#endif

#if !defined(HAVE_CPP20)
    static_assert(false, "C++20 minimum required.");
#endif
https://learn.microsoft.com/en-us/cpp/build/reference/zc-cplusplus This constant is supported by all compilers.
(edited)
evoskuil17:40
I've merged all of the changes for the allocator removal. Completed maxi full validation run to 950,000 in 5.0 hrs. Previous record was 5.5 hrs though I have only one baseline run for that. At the top end we are downloading and validating 100,000 blocks per hour on this 10 year old machine.
(edited)
🎉 3❤️ 2
evoskuil23:59
With nothing more than modern threads (e.g. 4.4 base vs. 2.1) and sha-ni this machine would be completing a full validation over the Internet to 950k in < 2.5 hrs. The last bitcoind run I did on this machine took 24 hours to validate to 850k. Because of the sequential nature of its processing, sha-ni and individual core speed have a much more significant effect, and the lack of a native Windows build is also crushing.
(edited)

Thursday, May 28, 2026

evoskuil00:05
Working off some technical debt now. This PR makes all machine protected and public methods virtual, which will facilitate unit testing and also simplify @Bruno Garcia’s fuzzing support. https://github.com/libbitcoin/libbitcoin-system/pull/1852/changes/cd44e87f84cdb350a09afa3e63950b5af0c3591c
1 reply · Bruno Garcia · last May 28, 2026
Bruno Garcia14:11
Cool, will take a look today.
evoskuil00:06
I'm not going to merge it until after verifying validation performance impact against the above fresh baseline, which will run tonight.
(edited)
evoskuil06:39
The maxi validation record to 950k has already been broken by the linux build (same hardware/network), at 4.86 hrs (vs. the previous windows run of 5.0 hrs above). The only change in the code is the above addition of virtual keywords to all public/protected methods in the script machine. So it's safe to say that wasn't a performance hit, and that code has been merged.
🎉 3
evoskuil15:22
Updated windows maxi validation run to 950k in 4.98 hrs. Don't believe anyone who tells you that virtual methods in C++ hurt performance.
(edited)
Huddle with pmienk, evoskuil, neillm, Rowan Voskuil, Eric Chennells, Marnix, Enligths · 146 min15:50
4 replies · neillm, pmienk, evoskuil · last May 28, 2026
neillm16:30
testnet3 top is 4966156
neillm17:00
Gotta run, thanks again!
👍🏻 1

Friday, May 29, 2026

Vano Chkheidze01:53
🖐️ hello evryone
👍 1🍻 1👋 1
2 replies · pmienk, Vano Chkheidze · last May 29, 2026
pmienk01:56
Got a nice looking library there, great work. Hope you don't mind if we try and use it a little
🙌 1👍 1
Vano Chkheidze02:22
will be happy to help in integration process
🎉 1
evoskuil19:11
Ok, so I promised some updated stats following the wire deserialize optimization. This was not a huge amount of work, and not terribly complex, but a lot of details to get right and tested. In all it took a calendar week. The optimization consists of a block_view that contains a tx_view for each tx, contained within the network block message type, in place of the chain::block. The block view contains the socket buffer, which is the wire block. The view provides quick access to data within the buffer required for hashes and identity validation (merkle root and witness commitment), as well as access for store serialization. The block message still provides access to the chain::block by on-demand deserialization from the buffer, though that's not currently used. This eliminates the population of a chain::block object for the purpose of reading from the wire and writing to the store. The chain objects are really for validation, and we only validate what is read from the store, not from the wire. From the wire we just need identity, which is much simpler and provided by the views. This is a big optimization in that it prevents the computation and memory allocation for construction and destruction of the block, which can have many tens of thousands of sub-objects. We previously optimized that using the linear memory allocator, reducing the construction down to one allocation. But this significantly over allocates the block on average, and the block itself averages 5-10x the memory of the buffer it is constructed from (in one case 20x).
So this change not only eliminated the block construction/allocation, but also the complexity of the custom allocator injections into all constructed types. The optimization is also the direct analog to the block out optimization from a few months ago. That eliminated the database->block->wire construction, and cut block out time in half (and significantly reduced memory utilization). This eliminates wire->block->database construction. So now we only construct blocks for in the validation chaser, along with prevouts, for full validation and/or client filter construction.
evoskuil19:21
Here's a peek at the chain::block_view and chain::transaction_view. The other big chunk of work was in the store, primarily in these two methods that replace the chain::block and chain::transaction writers with view writers.
(edited)
evoskuil19:27
In terms of what we expected, the idea is not necessarily less computation but less RAM (non-mmap) utilization during download for a given number of blocks. The idea was that this frees up mmap for table heads, which need to remain mostly RAM resident) and working set for paging to support indexation and validation. Given much higher RAM I didn't expect this to have a material impact on the maxi, more on the mini and pi. With more available mmap paging is reduced and we can avoid thrashing at a lower RAM level. At the same time we reset our benchmark baseline to 950,000 blocks. The pi is @eynhaender’s new toy, with just 16GB RAM and 4 slow threads (~2.1GHz, same as the maxi, but with sha-ni, unlike maxi).
evoskuil19:41
So for milestone on the maxi, 900k had been hovering around 1hr for a long time. For some reason I don't have a run to 950k before the optimization. But the two runs after the optimization came in at 56.2 mins and 62.3 mins. The best time to 900k without the optimization was 51.9 mins and the worst was 63.7 mins. So this change alone have given us 50,000 blocks for free at the higher end.
(edited)
🆓 2
evoskuil19:49
In terms of full validation, to 900k the old implementation consistently takes between 4.6 - 5.0 hrs, and the only run I have to 950k came in at 5.55 hrs. The two runs to 950k after the optimization both came in at 5.0 hrs. In other words, we also got almost 50k blocks validated for free at the higher end.
🆓 2
So the optimization achieved the initial objective, which was to get us back to 1 hr maxi milestone runs, and also had the unexpected benefit of keeping maxi validation to 5 hrs.
evoskuil19:59
@eynhaender is still crunching the numbers for his maxi, mini and pi, for both milestone and full validation, and with bitcoind comparables, on linux from my fast WAN (at 1Gbps and 2.5 Gbps). I won't steal his thunder in terms of the comparisons, and he's got some nice plots coming. But all are essentially network limited in milestone runs. The optimization was not enough to prevent non-linearity during full validation on the Pi or the mini, but the numbers are certainly improved. On the linux maxi the run was slightly better than Windows, but well within expected variance.
(edited)
evoskuil20:07
In other news, after a conversation with @josie last night I was convinced that stripping witnesses below the milestone/checkpoint would be a fair pruning option (i.e. for nodes that want to transfer a validated full node state to a machine that is not expected to serve blocks). I ran a test of this last night, and the difference in store size was 607GiB vs. 847GiB at block 950k and the milestone time dropped from 1hr to 45 mins. This doesn't require any additional config, as it can just rely on the peer.services_maximum flags. By setting bit 10 (pruned) peers are notified that this node doesn't serve below depth 288. If this is set the node will now not download witness blocks below a checkpoint or active milestone. [At some point I'll also update the client filters feature so that the database table enable and archival aspect is also determined by its service bit.]
/// Serves only the last 288 (2 day) blocks.
node_network_limited = system::bit_right<uint64_t>(10),
I'm almost done with the implementation. I've added a field to the native interface configuration method so that it emits the top stripped block height in the case where bip141 is active. This makes the block explorer and callers aware of the behavior, as it cannot be observed in the blocks/txs alone. The node will drop peers requesting witness blocks at/below the stripped height, and will request non-witness blocks for any milestone/checkpoint block when the service bit is set. It also skips the witness commitment identity check for these blocks. The only challenge is avoiding a small performance hit from checking this context for each hash in get_data requests.
(edited)

Saturday, May 30, 2026

evoskuil00:35
This is merged. Set node.services_maximum = 1033 (9 + 1024) to not download witnesses under checkpoint/milestone. I just completed another maxi sync with this... to 950k in 43 mins.
(edited)
evoskuil01:05
X (formerly Twitter)
Every 50k blocks (almost 1 year) the Libbitcoin team bumps up its sync benchmark. Running on the same (now 10 yo) hardware we work to keep the benchmark milestone sync under 1 hour. After spending the last week implementing a deferred optimization... 950,000 blocks in 56.2 mins!
🎉 4🚀 3
evoskuil15:24
X (formerly Twitter)
Pi5 at 1Gbps Internet
evoskuil20:11
X (formerly Twitter)
There has been a complaint that this comparison only configures the default number of peers for Bitcoin Core. This however is its optimal config. https://t.co/gr49sQ1BgC
X (formerly Twitter)
There has been a compliant that this comparison unfairly includes both of Bitcoin Core's optional indexes (tx and spender), despite the fact that Libbitcoin is also providing these indexes. But for the sake of satisfying curiosity, this is the difference: https://t.co/Us2NYjbYsx
X (formerly Twitter)
There has been a complaint that this chart shows Bitcoin Core validating blocks that Libbitcoin is not validating. This is not the case, both are performing the exact same level of validation on the same set of blocks. Core's "assume valid" point was set to 950k.
X (formerly Twitter)
There has been a complaint that Libbitcoin's "milestone" is not wasting as much CPU as Bitcoin Core's "assume valid". We tend to agree. > Core only skips script/signature checks before assumevalid, while libbitcoin skips nearly all block-level validation below the milestone.
X (formerly Twitter)
There has been a complaint that the comparison is unfair because Libbitcoin is not storing two copies of each unspent output. We tend to agree. > did you add a UTXO set to libbitcoin for symmetry? Most of the work Core is doing is building and maintaining the UTXO set.
All of these complaints came from a guy who is funded to work on optimizing Core's IBD. He blocked me and then started spamming my feed, straight up calling me a liar several times. I'm not terribly surprised that they are not making much progress.
👻 1
evoskuil23:25
DeepWiki
This document provides a technical overview of libbitcoin-system, the foundational C++ library for Bitcoin application development. It covers the library's architecture, major subsystems, and key code
evoskuil23:38
DeepWiki
libbitcoin-server is the high-performance Bitcoin full node server component of the libbitcoin v4 ecosystem $1. It is designed to integrate the foundational libbitcoin libraries—including `libbitcoi

Sunday, May 31, 2026

evoskuil19:26
4 am two nights ago, front yard, out into the street. Nor’easter
IMG_1648
😮 1
Bow saw, clipper, ax
(edited)
IMG_1652IMG_1651IMG_1655
Chain saw
IMG_1660IMG_1662
Was a nice tree, but it left me with a good workout and a year’s worth of firewood.
🔥 1🪵 1
5 replies · josie, evoskuil · last Jun 01, 2026
josie20:06
finished trimming ~300 olive trees this past week, chopping down into firewood and mulch
e99e040d-2fad-4236-afa9-a1e07cbc17f2
💪🏻 1
josie20:07
will also be chopping an old cherry tree in a few weeks before a storm makes it look like yours 😅
evoskuil20:08
Chopping, like with an ax!
evoskuil20:09
I did everything except the trunk and major limbs with the bow saw. Then my neighbor showed up with the chainsaw. Bow saw wasn't big enough to get through the rest. I had my axe out, but that prolly woulda killed me.
josie07:05
haha cherry tree is a bit too big for an axe and too close to the house, will likely have to enlist the professionals. olive trimmings we are mostly doing by hand, but gotta hatchet them up before they dry, otherwise its brutal. feels like chopping rock!