Bitcoin full node and query server
December 2025
Archive of the libbitcoin Slack #general channel.
Monday, December 1, 2025
evoskuil06:17
replied to a thread: view
The node is now dispatching explore requests via the generalized dispatcher, with serialization tied to http header accept type and query string format, and with the witness option as applicable. Just need to hook up the queries for the remaining methods now. These are complete:
/v1/block/hash/[hash] /v1/block/height/[height] /v1/block/hash/[hash]/header /v1/block/height/[height]/header /v1/transaction/[hash]
with (optional):
?format=json|text|data&witness=true|false
🎉 2
Wednesday, December 3, 2025
evoskuil08:31
Two queries need a little database work, but otherwise the explore http endpoint is fully functional for all queries and formats.
enum explore_targets
{
/// /v[]/block/hash/[bkhash] {1}
/// /v[]/block/height/[height] {1}
block,
/// /v[]/block/hash/[bkhash]/header {1}
/// /v[]/block/height/[height]/header {1}
header,
/// -----------------------------------------------------------------------
/// /v[]/block/hash/[bkhash]/transactions {all txs in the block}
/// /v[]/block/height/[height]/transactions {all txs in the block}
block_txs,
/// /v[]/block/hash/[bkhash]/transaction/[position] {1}
/// /v[]/block/height/[height]/transaction/[position] {1}
block_tx,
/// /v[]/transaction/[txhash] {1}
transaction,
/// /v[]/transaction/[txhash]/block {1 - confirmed}
tx_block,
/// -----------------------------------------------------------------------
/// /v[]/inputs/[txhash] {all inputs in the tx}
inputs,
/// /v[]/input/[txhash]/[index] {1}
input,
/// /v[]/input/[txhash]/[index]/script {1}
input_script,
/// /v[]/input/[txhash]/[index]/witness {1}
input_witness,
/// -----------------------------------------------------------------------
/// /v[]/outputs/[txhash] {all outputs in the tx}
outputs,
/// /v[]/output/[txhash]/[index] {1}
output,
/// /v[]/output/[txhash]/[index]/script {1}
output_script,
/// /v[]/output/[txhash]/[index]/spender {1 - confirmed}
output_spender,
/// /v[]/output/[txhash]/[index]/spenders {all}
output_spenders,
/// -----------------------------------------------------------------------
/// /v[]/address/[output-script-hash] {all}
address,
/// /v[]/block/hash/[bkhash]/filter/[type] {1}
/// /v[]/block/height/[height]/filter/[type] {1}
filter,
/// /v[]/block/hash/[bkhash]/filter/[type]/hash {1}
/// /v[]/block/height/[height]/filter/[type]/hash {1}
filter_hash,
/// /v[]/block/hash/[bkhash]/filter/[type]/header {1}
/// /v[]/block/height/[height]/filter/[type]/header {1}
filter_header
};🙌 1
Thursday, December 4, 2025
Huddle with pmienk, evoskuil, eynhaender, Rowan Voskuil · 123 min16:55
Friday, December 5, 2025
evoskuil02:28
I added block/fees and tx/fee, renamed transaction to tx, added address/unconfirmed, /confirmed, and /balance, and renamed /inputs and /outputs to /input and /output. That gives us something line this:
/v1/block/hash/[bkhash] {1}
/v1/block/height/[height] {1}
/v1/block/hash/[bkhash]/header {1}
/v1/block/height/[height]/header {1}
/v1/block/hash/[bkhash]/fees {1}
/v1/block/height/[height]/fees {1}
/v1/block/hash/[bkhash]/filter/[type] {1}
/v1/block/height/[height]/filter/[type] {1}
/v1/block/hash/[bkhash]/filter/[type]/hash {1}
/v1/block/height/[height]/filter/[type]/hash {1}
/v1/block/hash/[bkhash]/filter/[type]/header {1}
/v1/block/height/[height]/filter/[type]/header {1}
/v1/block/hash/[bkhash]/txs {all txs in the block}
/v1/block/height/[height]/txs {all txs in the block}
/v1/block/hash/[bkhash]/tx/[position] {1}
/v1/block/height/[height]/tx/[position] {1}
-----------------------------------------------------------------------
/v1/tx/[txhash] {1}
/v1/tx/[txhash]/block {1 - if confirmed}
/v1/tx/[txhash]/fee {1}
-----------------------------------------------------------------------
/v1/input/[txhash] {all inputs in the tx}
/v1/input/[txhash]/[index] {1}
/v1/input/[txhash]/[index]/script {1}
/v1/input/[txhash]/[index]/witness {1}
-----------------------------------------------------------------------
/v1/output/[txhash] {all outputs in the tx}
/v1/output/[txhash]/[index] {1}
/v1/output/[txhash]/[index]/script {1}
/v1/output/[txhash]/[index]/spender {1 - if confirmed}
/v1/output/[txhash]/[index]/spenders {all}
-----------------------------------------------------------------------
/v1/address/[output-script-hash] {all}
/v1/address/[output-script-hash]/unconfirmed {all unconfirmed}
/v1/address/[output-script-hash]/confirmed {all unconfirmed}
/v1/address/[output-script-hash]/balance {all confirmed}5 replies · neillm, evoskuil · last Dec 05, 2025
neillm18:45
This is amazing 😮 Just had a chance to play with some of it. What's the best way to reduce logging of the explore queries (i.e. channel start, request, response, channel stop, etc)?
neillm18:46
Unrelated, but I'm seeing a ton of lines like this (from all different IPs):
2025-12-05T18:46:15Z.5 Header [0000000000000000000178231b20e267f3e51399b0325f0e8b5f2abdec561c07:925565] from [45.23.183.89:8333] block unconfirmableneillm18:48
For the last one, it's possible I need to re-sync. I had been pulling and rebuilding the code and just restarting the server, so not sure of the exact state there
evoskuil19:21
If you are rebuilding from different points in the codebase you need to resync. The line you are seeing implies the header is invalid, and the fact that you are seeing it from many peers implies it is valid, so it's just cycling rejection of that block. But I can't say what's happening given the state of your code/node.
There is fine-grained control over logging levels at (1) compile, (2) config, and (3) console. You can set default initial log enable/disable in the config file, and you can vary that at the console dynamically.
[log] application = true news = true session = true protocol = false quitting = false proxy = false remote = true fault = true verbose = false #objects = false # debug builds only
To set at the console hit
m <enter> for menu options (with node running).🙏 1
evoskuil19:37
The service endpoints currently have their additional logging at the
application log level.evoskuil02:32
Also, /txs returns block tx hashes in order, not whole txs. If you want that just get the block. For binary and base16 encoding, this is just a contiguous set (divide by 32 bytes).
evoskuil09:10
All explore queries are now fully functional in all encodings, except for the last three address queries. Also need to just dedup and lexically sort the spenders and address queries.
🎉 1
1 reply · evoskuil · last Dec 05, 2025
evoskuil17:35
The address and spenders queries are the only ones that are not strictly bounded to a small size by consensus. However the spenders query is only pulling conflicts (double spends), so it's effectively limited by fee rate. The address query however can get very large results. The largest address on the chain is
(edited)1HckjUpRGcrrRAtFaaCAUaGjsPx9oYmLaZ, which returns 5,179,483 outputs. This is a 178MB result in raw form. On a warm node this takes 7 secs to generate, which includes enumeration via the address table multimap, and lookup of each tx's hash and the output offset, to generate the output point, and the full buffering of the result. On a cold node it takes 2.7 min (mostly SSD read). There's no caching involved in the warm server, so with the chain memory resident, the warm time is close to the initial time.evoskuil17:43
I'll be adding a cancelability to the address queries so that channel timeout or closeout will terminate the query, allowing immediate close. The other queries are so small/fast it's not worth the overhead.
Sunday, December 7, 2025
evoskuil00:10
Cancellability is now implemented in the address queries of the explore endpoint. I've got a little more organizing to do (push a couple calls into the database), but I consider this endpoint complete (for http, not yet https or ws/wss).
Without cancellation, a query initiated by a client (e.g. browser) would keep running until complete, even after the connection was closed, either by the caller disconnecting, a channel timeout, or a node stop. This is always the case, but with all queries in the few ms range it wasn't an issue. But the address queries are essentially unbounded, and can take minutes for the largest addresses. So now these queries are executed off of the channel strand, and are passed a reference to
std::atomic_bool protocol_explore::stopping_{} which is updated in the strand:
void protocol_explore::stopping(const code& ec) NOEXCEPT
{
BC_ASSERT(stranded());
stopping_.store(true);
dispatcher_.stop(ec);
protocol_html::stopping(ec);
}So now these query terminations are immediate for anything that cases a channel stop.
🎉 2
I noticed that queries are not stopped/suspended/delayed when a snapshot is underway. At first I thought it was an issue, but then remembered that snapshots don't block read access to the store... so these just run concurrently with no problems.
The 7,451 outputs address query, returning the following (every one of that many outputs associated with the address), takes .3ms (on my 2.1GHz machine, prob closer to .1ms on a modern machine).
[{"point":{"hash":"db9847575ef720b8b5582bac74bcedef1b6a08903ddc27638024b8d0d3ba0400","index":0},"value":1270486495},{"point":{"hash":"2f480f47702a6c099e4d8a53c68cf0e882524fb2d417b3fb8ce0420285420600","index":0},"value":1296713611},{"point":{"hash":"72007a9deb5c1832e095330bb30a8296e23acfc2b1e5994f24f9ef1bae7c0700","index":0},"value":1282065814},
...
] (edited)2 replies · evoskuil · last Dec 08, 2025
evoskuil00:18
[{"point":{"hash":"ec16c4bdaf9c9f62778868694ba6ff01feb6a2bc624fb6d87eb8d9b17c6c3f04","index":0},"value":77595},{"point":{"hash":"79ea12ee2bb1728f03a35eaf0ccd4f8ec87ce7382ee82ddffa65b901df801212","index":0},"value":53163},{"point":{"hash":"cadcefdd2ba9917490ad619b2a90a954e3e018b597b0a5a750eaa5b191518e2b","index":0},"value":22556},{"point":{"hash":"4909fae2720cdd9620ad27d1537dcb83c4ac30f097fddae9cdd001b6535fef3c","index":0},"value":74609},{"point":{"hash":"a0e3d5062fd654fabd3ad37a7a206ab36ae9b69ddfca9473afbc3567f002a543","index":0},"value":4873},{"point":{"hash":"b44fc0d27baf40d412c6402923df8fc2b87611ab659f3456251ce34b643b5352","index":0},"value":37400},{"point":{"hash":"ff8090e0d0ebf2f956d456efd1d3a0a9dcb3634a96af86efdef8d888a5343b6d","index":0},"value":150000},{"point":{"hash":"1d05d899ec4043838c6658a7ed416b8c842e087f4495cf5d7adba3d7dea87972","index":0},"value":109904},{"point":{"hash":"aa8d5245516b08133716c6b88a3f393eacd8878680c9a5a3c434f7c03a662878","index":0},"value":107954},{"point":{"hash":"fb7aed4a84e061524d01ca08328ccd8234339b448a979560ff16e89128010ab9","index":0},"value":374084},{"point":{"hash":"33cdcbd55d2a96a2d8eb1e1e368f84e1fedf9a181e30fa4a3e11aa81a43262c7","index":0},"value":340249},{"point":{"hash":"8e8ca91038965142170669175c0673b98c6de16e1cf28c54eb37dfb80f3bd6ca","index":0},"value":75116},{"point":{"hash":"bb16fa968ca18aea8f6b8be9ae574d39c81063ce2c30bd434ff0843ea1ea3cea","index":0},"value":73950},{"point":{"hash":"f1a7cdda517d1c7e030480121a3f5b57a6f7f2857c9610792a6f90f9fbfc66fa","index":0},"value":362631},{"point":{"hash":"1e1b7d5355b288d52f4d1b3a25f4a9d53749dd7a8828d3f747ff2c1a3df07b62","index":1},"value":2818071},{"point":{"hash":"c388319c5828c6240974d7936ccec68d713325b7b4bb1febb182dc4fc32f4884","index":1},"value":100000},{"point":{"hash":"91b9b31a676762f846628bc4dd6e89d9c9731de5fc1ed926b643ad405a504d7e","index":2},"value":462582},{"point":{"hash":"53df9a90a4aa6e1951aba0348e94633b8e756e28ba0ab167fea6a8bd77b0fdb1","index":46},"value":14755},{"point":{"hash":"19e862404694f8e04cedf15d5458bf969184b708a86a72fafd749fd0b97cc21f","index":53},"value":1033541},{"point":{"hash":"4e4775e68f43d12134cd8da13af79afb39d6c1d4747bfa4815f506fd20684b4d","index":65},"value":21611},{"point":{"hash":"68e6d45692cd1ed09d2e4f7ac6dda8e45a999bb21f9a69e95b391407249b5907","index":81},"value":21777}]
evoskuil00:39
At some point I'm going to better organize (and further optimize) the queries and response structure for address history. But this demonstrate all of the options.
🔥 1
Monday, December 8, 2025
evoskuil00:32
I did some optimizing on the address queries, and the cold query against the largest (
(edited)5.2 million outputs) bitcoin address (returning all outputs points and spend value) has been reduced from several minutes to 30 secs. Chrome remains unable to present the json (aw snap). Chrome renders the base16 text format in 3.6 mins, which is funny since we produce it (end-to-end, warm server) in < 20 secs. The raw data rendering, that Chrome sends straight to a file, takes 17.5 secs to generate, and 2.2 ms for Chrome to download and serialize to a file. The original largest address warm query only took 7 secs, so you may be wondering why it's now 17.5 secs, but that's because I'm now (1) sorting the outputs, and (2) querying the value for every output.evoskuil00:38
Sorting the result set is important in that it ensures the same result is provided by the query from any server (caching and fingerprinting affected). Also it's not just sorting but deduplication, which is important given the distinct possibility of duplicate txs.
evoskuil03:32
I just implemented vectorization over the most costly part of the address queries. This requires the extra cost of collecting the output links into a vector first, so it's less efficient for small queries, but it cuts the large query times in half. Considering that the link collection is sequential, as is the sort/de-dup/serialization, that's a pretty large reduction in the query cost. 5.2 million now in 15 seconds. With modern cores that will be under 10 seconds easy - for the largest query possible.
👍🏻 1
So I think I'll provide both implementations and allow the caller to optimize large query requests, and enable them via config (since they will consume 100% CPU while running - the 5.2 million query takes about 8 seconds of my total CPU).
evoskuil04:53
In the past hour I added this config setting:
[database] turbo = true
Parsed it in node config (defaulted false), passed it to database, exposed to queries, it via store, conditioned the address queries on it.
...
And added this query string option:
?turbo[=true]
Parsed it in node's explore query string, added it as an optional argument to the explore interface address methods (defaulted false), added it to each of the address interface handlers, passed it to the parallel invokers, and into the queries.
...
So the node can be configured to enable turbo, and if enabled a caller can request it via the query string in the expectation that the result set is large. Small queries will be faster without it, but probably not noticeably so.
...
An thus the
turbo button was reborn!evoskuil05:02
Just to add some context, this is the plot of two maximal (5.2M outputs) queries, the first without turbo and the second with turbo. There is no perceptible increase in CPU for the non-turbo query. The turbo query jumps to 100% CPU for about 7.5 seconds. The rest of the 15 second time is the sequential address table search (link collection), sort and de-duplication, and wire serialization.
evoskuil05:41
The cold iron maximal query dropped from 2.7 minutes to 1.7 minutes, which means that paging (where all of the cost is in a cold node query) is accelerated by the concurrent requests.
To put that in context, the cold iron turbo query for
(edited)~7,500 outputs (mentioned yesterday) takes 1.62 seconds and the same query warm takes ~20 milliseconds.evoskuil05:53
Interestingly enough, smaller queries seem to be performing better with turbo as well. This query produces just 51 outputs. It takes just
~2 ms without turbo, but with turbo consistently takes ~1.5 ms. So probably optimal for a caller to always use it, if it's enabled. So I'll change the query string default to true (but retain it for ongoing testing).
http://localhost:81/v1/address/50d8b0211a23e12edf9890c7e0a474f7a87a0a53c0e11285d577991ed3931abdTuesday, December 9, 2025
Wednesday, December 10, 2025
evoskuil19:16
Burned a day tracking down an issue in the
network::socket class, pertaining to closing websocket connections. Another day refactoring all server base classes from libbitcoin-network into libbitcoin-node. These all now are managed via node::session_server and can be moved up to libbitcoin-server at some point with little refactoring.evoskuil23:58
I've removed the
(edited)[socket] endpoint and inherited protocol_explore -> protocol_html -> protocol_ws -> protocol_http -> protocol. Websockets requires underlying http for the "upgrade" handshake, and while not all html sites may use websockets, placing it into the hierarchy simplifies configuration. So there's just a simple bool option to disable the upgrade if not used by the endpoint. So websockets is now enabled on the explore and web endpoints, disabled on web. This means websocket connections are on the same port as the http requests for a given endpoint. I haven't integrated ws dispatch for explore yet thought.Thursday, December 11, 2025
evoskuil00:04
Presently I'm working on
protocol_bitcoind -> protocol_http -> protocol. This simply overrides protocol_http::handle_receive_post() and dispatches the universal request object (which automatically detects and parses up the json body), via the dispatcher for the bitcoind interface. So just a matter of implementing each of the methods. Initially it's just the node methods, so something like this:
/// Blockchain methods.
method<"getbestblockhash">{},
method<"getblock", string_t, optional<0_u32>>{ "blockhash", "verbosity" },
method<"getblockchaininfo">{},
method<"getblockcount">{},
method<"getblockfilter", string_t, optional<"basic"_t>>{ "blockhash", "filtertype" },
method<"getblockhash", number_t>{ "height" },
method<"getblockheader", string_t, optional<true>>{ "blockhash", "verbose" },
method<"getblockstats", string_t, optional<empty::array>>{ "hash_or_height", "stats" },
method<"getchaintxstats", optional<-1_i32>, optional<""_t>>{ "nblocks", "blockhash" },
method<"getchainwork">{},
method<"gettxout", string_t, number_t, optional<true>>{ "txid", "n", "include_mempool" },
method<"gettxoutsetinfo">{},
method<"pruneblockchain", number_t>{ "height" },
method<"savemempool">{},
method<"scantxoutset", string_t, optional<empty::array>>{ "action", "scanobjects" },
method<"verifychain", optional<4_u32>, optional<288_u32>>{ "checklevel", "nblocks" },
method<"verifytxoutset", string_t>{ "input_verify_flag" },evoskuil07:18
@Rowan Voskuil has pushed a first functional embedded block explorer!
[node] # set this to enable inbound endpoints without waiting for full sync. delay_inbound = false [explore] bind = 0.0.0.0:80 connections = 100
🔥 2🎉 2
1 reply · neillm · last Dec 11, 2025
evoskuil08:42
I managed to get all of the
(edited)bitcoind node-based json-rpc handlers stubbed in, along with request deserialization and dispatch. Also created a test form to post rpc message to the http endpoint. This is held up by the need to handle the browser's OPTION messages, for cross-origin handling. The work for both handing option and cross-origin/CORS was done previously in the base protocol_http and protocol_html, just need to connect the dots. After that it's just hooking up the queries. So probably a couple days and this will be functional, and just for grins we can drop a quick functional test suite on an embedded page.Huddle with pmienk, evoskuil, eynhaender · 140 min16:54
evoskuil23:54
The migration from network to node broke explore endpoint address query cancellability. Should have that resolved soon. Also the address queries had an improper buffer size calculation, and were completing off strand. So there may have been some instability post execution of those. All three issues caught by assertions the first time I ran some queries in a debug build. These should be fully resolved now.
Friday, December 12, 2025
evoskuil08:48
Turns out that the cancellability wasn't a regression. There are two distinct scenarios, and only one was handled. The other appeared to be handled, but I misinterpreted cancellation. Even the longest address query (5.2M outputs) only runs for 6-7 seconds on my machine (the other 7-8 is serialization/routing), so it's easy to miss.
evoskuil08:54
The caller disconnect scenario was not handled, but the process close scenario was. This is because on a process close the socket is terminated by the stop handler, which is ultimately invoked by the
<ctrl-c> event handler. But the caller disconnect is a per-channel signal that comes from the socket, and is only picked up by an async read handler being invoked. But while we are handling a call, there is no read handler pending. We resubscribe to the channel upon call completion. So even though the work is off strand, there was nothing to capture the close until the call completed. I resolved this by creating an async_wait method on the socket, which will pick up termination. The challenge was cleanly registering and deregistering this before and after the potentially long-running calls. That's all done and working now.There's also the channel timeout scenario, which is handled same as the process stop, since there is an external (timer) event that signals the channel stop.
evoskuil21:36
Took some time this afternoon to optimize the point table duplicate detection. This should knock about 10% off of the sync time.
There are some material optimizations still remaining in milestone, full validation, and block/tx query, but this one was pretty simple. Back to work now on the bitcoind json-rpc interface.
evoskuil22:28
Finished up
(edited)53.5 minutes to block 900,000 on my windows dev box. That beats the 900k world record exactly 10%. For context, the $350 mini usually beats my dev box by a bit (because it hash shani and avx512). We'll keep optimizing to make sure we can still hit under an 1 hour on 2Gbps Internet at 1M blocks.Saturday, December 13, 2025
evoskuil01:24
I just ran the above optimization on the Windows mini pc (shani/avx512), shattering the above world record at
47.6 minutes to block 900,000. That puts the 1 million blocks in 60 minutes target well within reach.🚀 3🎉 3
Sunday, December 14, 2025
evoskuil00:15
I just discovered this...
https://github.com/bitcoin/bitcoin/blob/master/doc/REST-interface.md
Had no idea. So that'll be added to the bitcoind json-rpc endpoint, and is basically a clone of the existing explore interface, though much more limited. Since it's REST these are url GET queries, where the json-rpc is json POST queries. This will take an afternoon.
(edited)
We use the same exact code to process all endpoints, including p2p incoming, and outgoing, and I've verified those up to something like 10k active connections.
evoskuil09:04
After a long day of chasing http and browser idiosyncrasies... we now have a functional
bitcoind json-rpc endpoint. One method done just as the proof, others all stubbed in. The rest of it is downhill from here :).🎉 2💥 1
Monday, December 15, 2025
evoskuil03:25
bitcoind-rest (URL GET) compat interface, same endpoint as JSON-RPC (POST)
// github.com/bitcoin/bitcoin/blob/master/doc/REST-interface.md
static constexpr std::tuple methods
{
// blocks (block_part is bin|hex only)
method<"block", uint8_t, system::hash_cptr>{ "media", "hash" },
method<"block_hash", uint8_t, uint32_t>{ "media", "height" },
method<"block_txs", uint8_t, system::hash_cptr>{ "media", "hash" },
method<"block_headers", uint8_t, system::hash_cptr, uint32_t>{ "media", "hash", "count" },
method<"block_part", uint8_t, system::hash_cptr, uint32_t, uint32_t>{ "media", "hash", "offset", "size" },
method<"block_spent_tx_outputs", uint8_t, system::hash_cptr>{ "media", "hash" },
// client filters
method<"block_filter", uint8_t, system::hash_cptr, uint8_t>{ "media", "hash", "type" },
method<"block_filter_headers", uint8_t, system::hash_cptr, uint8_t>{ "media", "hash", "type" },
// unspent outputs
method<"get_utxos", uint8_t, system::hash_cptr, uint8_t>{ "media", "hash", "type" },
method<"get_utxos_confirmed", uint8_t, system::hash_cptr, uint8_t>{ "media", "hash", "type" },
// mempool (json only)
method<"mempool", optional<true>, optional<false>>{ "verbose", "sequence" },
// info (json only)
method<"chain_information">{},
method<"mempool_information">{},
method<"fork_information", nullable<system::hash_cptr>>{ "hash" }
};evoskuil07:13
So now both
(edited)bitcoind-json-rpc and bitcoind-rest compatibility interfaces are defined and functional (incomplete) on the [bitcoind] endpoint, alongside the native [explore] http/ws rest interface (complete on http, ws functional) and embedded (or directory override) block explorer, and the [web] html embedded (or directory) dashboard/admin site (functional). Next is to get the [electrum] endpoint/interface up (json-rpc over tcp). That's all the endpoints apart from stratum (and [stratum_v1] is the same endpoint type as Electrum, just different methods).Tuesday, December 16, 2025
evoskuil09:21
I completed the implementation of a json-rpc http body template, derived from the json body. This parses/serializes json-rpc requests/responses over both http (bitcoind) and raw tcp (electrum/stratum-v1). That provides some simplification for the bitcoind endpoint, and the required transport for the electrum endpoint. Tests complete, hopefully hooked up tomorrow.
🎉 1
Once that's done, all that remains for all endpoints is method implementations (query/serialize) and then TLS on all endpoints.
Wednesday, December 17, 2025
evoskuil03:08
Found a dead body in the code today...
https://github.com/libbitcoin/libbitcoin-network/blob/master/include/bitcoin/network/messages/rpc/body.hpp#L100
Was like... what the...
Then I remembered that I had performed a replace of all
rpc_ with terminated_, within the rpc::body class.Thursday, December 18, 2025
evoskuil09:32
The above-mentioned integration of the json-rpc http body into the variant message is complete. So the variant message now detects json in the headers and selects the derived json-rpc reader (unless a flag is set to use its base body and return the plain parsed json model). Both plain json (explore) and json-rpc (bitcoind) are now flowing through the same pipeline. Next is to hook up the new json-rpc body as reader/writer for the non-http json-rpc interfaces (electrum/stratum_v1). These are slightly different because they are line terminated, and that's baked into the json-rpc body template. If the read/write is initiated in an http socket, it uses a different constructor so the body can distinguish. I hope to have this hooked up to the electrum channel and dispatching methods tomorrow.
🎉 1
Monday, December 22, 2025
evoskuil21:01
X (formerly Twitter)
Default sync (skipping signature validation until block 900,000) of Libbitcoin 4.0.0 to block 928,000 took my benchmark machine 1 hour 43 minutes.
🎉 3
This is 1Gbps Internet, 32GB, 12 slow threads, pcie3, avx2 and no shani.
(edited)Tuesday, December 23, 2025
stratan17:27
Hello there! I'm having trouble getting up and running with
libbitcoin-node with latest master. I'm not very familar so it's more than likely I've misunderstood something. Running bn --newstore on a fresh install fails after all database files are successfully created. The error message is confusing me. Ideally I'd like to create a regtest config.
I'm using the following minimal config:
$ cat /tmp/bn.cfg [log] path = /tmp/bn-logs [database] path = /tmp/bn-db [inbound] bind = 127.0.0.1:18445 connections = 1 [outbound] connections = 0 $ bn --config /tmp/bn.cfg --newstore
Output:
All files are created, opened, loaded, unloaded, and closed successfully: create::create_file(header_head) ... create::close_file(filter_tx_body) Database creation failed with error 'file failed to load, disk may be full'.
Any kind of pointer would be much appreciated (storage seems fine) 🙂
(edited)13 replies · pmienk, eynhaender, evoskuil, stratan · last Jan 08, 2026
eynhaender23:26
I just checked your config and the store has successfully been created with an initial size ob 14GB. This is the minimum you need to start. Then, starting the node without
(edited)-n but with outbound.connections = 0 in the config will cause the node to just idle and do nothing as it will not connect to any peers.👍🏻 1
evoskuil02:30
A local inbound connection could be made (if a peer has the address:port), on an empty chain. However by default inbound endpoints are suspended until the chain is current (configurable).
The error message indicates the disk is full, which seems straightforward. There are other possible causes, but that is the most likely. The startup/create requires about 25% of the mainnet full chain size by default (configurable).
Regtest (and testnet) is possible, but we might have to help out with the config settings, as I don't believe they are documented yet. And that hasn't been tested since v3, though I would expect it to work.
(edited)stratan15:17
Hello folks! I appreciate the prompt feedback and I'm very sorry for ghosting you as I disconnected for a while due to holidays 😅 I'll go over your pointers and get back to you. Thank you once again!
👍🏻 1
stratan19:09
Hello again! Your feedback regarding the disk space was correct. I wasn't expecting it to really be a disk issue as I had over 100gigs of disk space (don't remember the exact number). Now libbitcoin-node spawns and works just fine. I stumbled upon another "issue" when trying to port libbitcoin-node as a target for fuzzamoto. That is, fuzzamoto uses Nyx for deterministic snapshot fuzzing but due to its limited resources it doesn't have enough space to pre-allocate a regtest db profile. I played around and hacked a regtest config that seems to work ok and I would appreciate your feedback on whether I'm overdoing it or not. I haven't found any documentation around libbitcoin-node + regtest so I'm open to suggestions on that side too.
[log]
path = {logdir}
[bitcoin]
# Regtest genesis block (hex-encoded)
genesis_block = {genesis_hex}
# Regtest proof of work limit (0x207fffff = 545259519)
proof_of_work_limit = 545259519
# Accept any valid chain (no minimum work requirement)
minimum_work = 0000000000000000000000000000000000000000000000000000000000000000
[forks]
# Regtest allows easy mining without real proof of work
difficult = false
[network]
# Regtest network magic (0xDAB5BFFA = 3669344250)
identifier = 3669344250
[inbound]
# Listen for P2P connections
bind = 127.0.0.1:{p2p_port}
connections = 10
enable_loopback = true
[outbound]
# Disable outbound connections to prevent contacting mainnet seeds
connections = 0
host_pool_capacity = 0
[database]
path = {datadir}
# Tiny regtest DB profile (avoid multi-GB prealloc).
header_buckets = 1
header_size = 1000
txs_buckets = 1
txs_size = 1000
tx_buckets = 1
tx_size = 1000
point_buckets = 1
point_size = 1000
strong_tx_buckets = 1
strong_tx_size = 1000
input_size = 1000
ins_size = 1000
output_size = 1000
outs_size = 1000
candidate_size = 1000
confirmed_size = 1000
duplicate_buckets = 1
duplicate_size = 100
prevout_buckets = 0
prevout_size = 1
validated_bk_buckets = 1
validated_bk_size = 1000
validated_tx_buckets = 1
validated_tx_size = 1000
address_buckets = 1
address_size = 1000
filter_bk_buckets = 0
filter_tx_buckets = 0
[node]
# Don't delay inbound peers waiting for "current" chain state (regtest genesis is old).
delay_inbound = falseevoskuil03:44
Hi, you've definitely got the right idea. I haven't tested regtest or testnet in a long time, so it's good to see this. I’m traveling so can provide much more feedback, but it should work. Just note that one bucket means the hash table becomes a linked list. That's fine but of course will be a bit slow if there are a lot of entries. I do it by accident sometimes and am always surprised that’s it’s pretty fast considering. Also, I don't think you want loopback. That allows the node to connect to itself.
evoskuil03:44
Also, bn is no longer. You will need to move to bs for future work. Same config and command line otherwise.
stratan07:22
As far as I understand bs isn't at a “usable state” according to its README, which is why I pivoted to bn. How do you suggest I proceed?
(edited)stratan11:43
Looks like latest master (libbitcoin-server) works now (
--help and --version still reference bn) 🙂
./libbitcoin-server/console/bs --help
Usage: bn [-abdfhiklnrsv] [--config value] [--test value] [--write value]
Info: Runs a full bitcoin node.
Options (named):
-a [--slabs] Scan and display store slab measures.
-b [--backup] Backup to a snapshot (can also do live).
-c [--config] Specify path to a configuration settings file.
-d [--hardware] Display hardware compatibility.
-f [--flags] Scan and display all flag transitions.
-h [--help] Display command line options.
-i [--information] Scan and display store information.
-k [--buckets] Scan and display all bucket densities.
-l [--collisions] Scan and display hashmap collision stats (may exceed
RAM and result in SIGKILL).
-n [--newstore] Create new store in configured directory.
-r [--restore] Restore from most recent snapshot.
-s [--settings] Display all configuration settings.
-t [--test] Run built-in read test and display.
-v [--version] Display version information.
-w [--write] Run built-in write test and display. (edited)eynhaender12:43
Everything but the ---help message has moved up to server, the updated readme is currently under review. All build related stuff should should work for bs (-server) like it did for bn (-node) last week.
👍 1
Monday, December 29, 2025
evoskuil20:45
I just fixed a trivial regression in network that prevented returns from http endpoints.
evoskuil23:24
Yesterday I incorporated max request and min buffer config into all endpoints, which is applied consistently at the socket. This required a lot of injection, as the socket is the lowest level networking object, and is encapsulated by the proxy after construction by the acceptor or requestor. But it's all flowing nicely now.
[endpoint] maximum_request = 4,000,000 minimum_buffer = 4,000,000
I've also just updated the the
network::channel_rpc and ::protocol_rpc so that they are both templatized on a specific rpc (json-rpc) interface. This allows node::protocol_electrum and node::protocol_stratum_v1 to just derive from the network::protocol_rpc<interface::electrum> and implement the electrum rpc method handlers, and of course the same for network::protocol_rpc<interface::stratum_v1>.evoskuil23:33
Last week I completed work in network to add rpc parsing to the socket using the json-rpc http body (which is also used for bitcoind-rpc over http) over raw tcp. This json-rpc http body derives from the json http body, which encapsulates the boost::json parser and serializer. So now these are hooked up for json-rpc send/receive for electrum and stratum_v1 over tcp. stratum_v2 is stubbed in, but I'll defer socket implementation for that, since it's not a v4 feature. stratum_v1 is implemented only because everything is the same as electrum except the method handlers.
I don't yet have a good test harness for json-rpc over tcp at this point, but using telnet I can see that it functions as expected.
Tuesday, December 30, 2025
evoskuil01:39
And I just completed node subscription and handling of the first couple of methods in electrum and stratum_v1. So at this point all of the work to integrate all of the endpoints and protocols is complete, apart from individual method implementations. This includes:
[web] # admin website (embedded or local) # websockets enabled [explore] # html block explorer (embedded or local) # rest interface (json, hex, binary) # websockets enabled [bitcoind] # bitcoind json-rpc (post) # bitcoind rest (get) [electrum] # json-rpc over tcp [stratum_v1] # json-rpc over tcp [incoming] # p2p (tcp) [outgoing] # p2p (tcp) [manual] # p2p (tcp)(edited)
The only others envisioned are [zeromq] and [stratum_v2], which can easily drop into the same framework. Otherwise there's TLS, which is a few different protocols at the socket level, over all endpoints, and SOCKS5 proxy support (simple, also proxy level).
evoskuil01:47
The plan is to eventually move all of the non-p2p protocols up to libbitcoin-server, along with the bn executor classes, full command line parser, console, and client-server endpoint configuration settings. So bn would revert to the node library with trivial console and bs would again be the "product".
evoskuil07:18
Completed the first successful tests of Electrum connection to bn electrum endpoint (version message).
🎉 2🚀 2👌🏻 1
evoskuil21:09
I saw today that there is a new node project out there, vibe coded ~250,000 lines of code in 29 days.
evoskuil22:08
btcd is dying.
https://x.com/lopp/status/2005631549795099128?s=20
X (formerly Twitter)
Full validation sync of btcd v0.25.0 to block 928,000 took my benchmark machine 3 days, 11 hours, 28 minutes.
😂 1
evoskuil23:02
I just completed stubbing in all
stratum_v1 rpc handlers. These sit on the same base as electrum, so they everything is fully functional apart from populating the responses. Currently they all send error::not_implemented, eg:
bool protocol_stratum_v1::handle_mining_subscribe(const code& ec,
rpc_interface::mining_subscribe, const std::string& user_agent,
double extranonce1_size) NOEXCEPT
{
if (stopped(ec)) return false;
send_code(error::not_implemented);
return true;
}Wednesday, December 31, 2025
evoskuil06:04
Completed the Electrum version handshake, resolved an issue with json-rpc object serialization, implemented base class round-tripping of json-rpc version and message id (keeping it out of the method handlers), added a system level config class for working with serializable 4 part versions.
Used the Electrum wallet to fully implement and verify the handshake, with versioning all ready to go for each method.
enum class protocol_version
{
/// Invalid version.
v0_0,
/// 2011, initial protocol negotiation.
v0_6,
/// 2012, enhanced protocol negotiation.
v0_8,
/// 2012, added pruning limits and transport indicators.
v0_9,
/// 2013, baseline for core methods in the official specification.
v0_10,
/// 2014, 1.x series, deprecations of utxo and block number methods.
v1_0,
/// 2015, updated version response and introduced scripthash methods.
v1_1,
/// 2017, added optional parameters for transactions and headers.
v1_2,
/// 2018, defaulted raw headers and introduced new block methods.
v1_3,
/// 2019, removed deserialized headers and added merkle proof features.
v1_4,
/// 2019, modifications for auxiliary proof-of-work handling.
v1_4_1,
/// 2020, added scripthash unsubscribe functionality.
v1_4_2,
/// 2022, updated response formats and added fee estimation modes.
v1_6
};evoskuil06:10
I think tomorrow I'll take a look at brining in
WolfSSL sources and start looking at TLS for each of our protocols (http/s, ws/s, p2p/s, rpc/s [zmq/s, sv2/s]). I believe that each can be integrated in basically the same manner entirely at the socket level.I gotta say, it's nice to finally see this...
2025-12-31T06:13:05Z.2 Accepted electrum connection [127.0.0.1:64158] on binding [0.0.0.0:50001]. 2025-12-31T06:13:05Z.2 Inbound electrum channel start [127.0.0.1:64158] success 2025-12-31T06:13:06Z.0 Rpc request: [1024] server.version(...). 2025-12-31T06:13:06Z.0 Rpc response: [62], success.
evoskuil18:58
Still 55 minutes on maxi, so not losing a step with all of the client-server integration.
evoskuil21:12
X (formerly Twitter)
@Otoya_Coinjoin Same machine since 2018, yes. As you can see, they made some improvements to btcd. Libbitcoin also made massive improvements lately.
evoskuil21:20
Gocoin is essentially Core architecture but with an absolute constraint that the utxo data structure remain memory resident...
https://x.com/lopp/status/2003511137531470265
X (formerly Twitter)
Full validation sync of Gocoin 1.11.0 on my benchmark machine crashes at block 860,800 after 11 hours 6 minutes due to running out of memory.
X (formerly Twitter)
Full validation sync of Mako commit hash 41ef1040... took 1 day 18 hours 4 minutes to sync to block 928,000 on my benchmark machine.
X (formerly Twitter)
Full validation sync of Bitcoin Core 30.0 to block 928,000 took my benchmark machine 12 hours 7 minutes.
evoskuil21:28
Full validation with Core on my maxi takes at least this long, where bn takes 4.2 hours.
evoskuil21:52
The full review is now posted. JL did a solid job representing the various considerations.
https://blog.lopp.net/2025-bitcoin-node-performance-tests/
Cypherpunk Cogitations
Testing full validation sync performance of 5 Bitcoin node implementations.
evoskuil22:00
Note that we are the only node that was tested over an Internet connection.
According to his tracking, we download about 2.5% more than others. This is the result of aggressively dropping peers with blocks partially transmitted. Otherwise we don't perform any redundant downloading of blocks.
Also I was able to verify that despite the 32GB RAM, the maps were thrashing due to low CPU (12 slow threads). We'll need to do some more testing to find out where the CPU limit should be set for a 32GB machine.
🫡 3
eynhaender22:04
We're gaining traction and increasing our public recognition. It's almost as if the market was waiting for libbitcoin. 2026 is going to be a great year. It's wonderful that you're finally reaping the rewards of your hard work.
👍🏻 1👍 2
evoskuil22:04
I found this comment interesting:
https://x.com/CameronOtsuka/status/2006060329403637996
(edited)X (formerly Twitter)
@lopp I'm surprised by the dirty pages issue. I wonder if zram would be a way to "trick" the kernel into keeping more in memory too? Those swappiness settings are familiar: https://t.co/kS69jf6rNd
evoskuil22:06
I'm more considering the idea of a custom memory map, so taking the paging control away from the OS in these cases. IOW just allocate the RAM, which the OS seems to be fine with.
(edited)But it's possible that operating on a ram disk might achieve some similar result.
I mean, what happens when you mmap a disk that's already in mapped into ram?
(edited)