Bitcoin full node and query server
November 2025
Archive of the libbitcoin Slack #general channel.
Sunday, November 2, 2025
evoskuil07:38
The above described embedded sites are now complete. Everything as planned, streamed directly from POD types to the port, simple to embed, one source file per document. 5 documents per site. Currently two sites enabled (admin site and explorer). Also beefed up html logging a lot. It emits at the
[log] protocol = true level, or toggle on p <enter> at the console.The () is for body bytes, with (c) for
chunked encoding (where we don't have the size at log time). Pretty much everything out is chunked encoding presently.
Embed looks like this for each site:
Currently only bindings and connection count is required in config, but if you aren't synced there's one more setting required to have the site up. Also for logging you can configure the protocol level by default. There's a lot more, but this is the least:
[node] delay_inbound = false [log] protocol = true [web] connections = 1000 bind = 0.0.0.0:80 [explore] connections = 1000 bind = 0.0.0.0:81
The explore site is presently running the simplified
[block|header|tx]/hash query interface, with ?format=[json|text|data] formatting, as well as the embedded html.💯 2
evoskuil19:28
I've also given some thought to wire serialization of blocks/txs from the store. While performance is excellent, it's astounding how much translation is taking place. The store spans 8 tables and two hashmap headers (header & txs) to retrieve a block by hash, and (6 tables and one header - tx) to retrieve a tx by hash independent of a block. It then deserializes from store encoding to the native
chain:: object model. All objects, down to op.data stored on std::share_ptr<> instances. Then to serialize out the entire object model is walked on a system::ostream to create a single buffer. For text (hex) encoding the buffer is then translated to hex using system::base16_encode. For json the object modal is walked directly by boost::json.
For wire/text there is really no need whatsoever to translated through the normal form (object model). The wire serialization can be trivially produced directly onto the stream from store serialization. This would bypass countless allocations and copies. Scripts and witnesses for example, making up the bulk of all data, do not even require parse, they can just be written to the stream. (Though it would certainly be cleaner if no for the tragic segwit decision to serialize witnesses independently from inputs.) I'd estimate that this would cut large block deserialization time in half, and on my old machine that would put it under 20ms.evoskuil19:43
Implementation for this is just a set of store queries that operate over the stream, and swapping out the get block/tx calls that then serialize to wire with one that just returns the stream buffer. Blocks are stored with their serialized size (witnessed if enabled on the node), in the txs association table. So
(edited)system::stream can be used over a preallocated contiguous buffer. But eventually this could even just stream write out to the port in chunks. It's possible to write the message heading without reading the object, since we always have the size indexed. So stream direct from store into the port with zero translation of scripts, witnesses, and point hashes, and the rest is just just organizing the pieces. I'd estimate 2-3 days of work for the whole thing to serialize blocks and txs directly from store to wire. The store queries no more than 1 day.This of course can also be applied to non-p2p output streaming when it's not
json. Furthermore, for text/hex serialization, we can provide a encoding streamer, just like our system::sha256_writer for example, that performs hex encoding on the fly, writing 2 bytes for every 1, and with a single double buffer size requirement. So with the fixed buffer can also use the fast system::ostream. That cuts allocation from 3x to 2x, and of course allows it to be chunked.evoskuil19:59
Thinking about
json, that's a different problem, since boost::json operates over an object model. However the boost::json::value (intermediate json DOM) could be populated directly from store queries as well. In other words, have the query produce the model instance - instead of having the boost::json value convertor produce it from the annotated object model. This again would cut out the normal form queries. Those are really only necessary for validation. That's probably 2 days of work in the store - first day to figure out the proper boost::json pattern. That would cut way down on memory allocations and probably cut the json output time in half (presently 75ms for large blocks on my machine, vs. 40ms for hex, and 35ms for wire encoding). So maybe json in 40ms, hex in 20ms, and 18ms for wire. Cut that in half for a 2x faster CPU, as I'm running 2.1GHz threads and these are entirely CPU-bound single threaded operations.evoskuil21:58
I've changed config:
[node] priority = true
to
[node] thread_priority = true memory_priority = true
where "thread" really refers to "processing" (CPU). Note that the network threadpool always runs at "normal" (medium) thread priority and is not exposed to config. Both of the
node settings default to true, which raises the thread priority above network. Setting memory priority to false will lower the process memory priority below OS defaults, and that's a new option. Also, memory priority only functions on windows.If you have a
node.priority setting in config either remove or rename it for use with future builds, or you will get a error on node startup.evoskuil23:05
I've just updated the configuration so that the existence of the
path setting for a site will allow that to override the embedded site. This was the original intent, just wasn't thinking last night at 4am. That allows us to test changes without rebuilding, allows customers to build their own sites, and allows default config to be safe/easy. It'll be checked in in a bit, just waiting for the node repo to clear CI.👍 1
Monday, November 3, 2025
M
Marnix06:06
I'm unable to run bn when my disk/VM is like 50 GiB:
Database creation failed with error 'file failed to load, disk may be full'.
I don't want to do a full sync, just create the store and initial bootstrap. This worked in the past with only like 10GiB space available.
This change is on purpose?
evoskuil06:08
I don't know how it ever would have worked. The initialization is about 25% of the final store size, which is about 650GB, so over 150GB. You can modify the initial table sizes from their defaults. The defaults are listed in the
--settings option. You could divide them all by 10 by just dropping the last digit.👍 1
evoskuil06:14
The only way that would have ever worked is if I had given you reduced table size config in the past.
1 reply · Marnix · last Nov 03, 2025
M
Marnix06:21
IDK, I ran into this issue https://github.com/libbitcoin/libbitcoin-node/issues/832 when I was running with 20 GiB
evoskuil06:21
The first thing it does upon table creation and upon startup, is to size all of the tables to their configured minimum. Oh... that was broken on linux. That's why we had the disk full bug. It would report that it had sized the tables, but it wouldn't actually size them. So we fixed that bug and it no longer crashes when running out of space, because it actually guarantees it has the space.
(edited)👍 1
It's never been possible on Windows, but on macOS and Linux it was possible.
✅ 1
evoskuil12:45
2025-11-03T12:41:43Z.2 Accepted socket connection [192.168.0.219:51007] on binding [192.168.0.219:82].
2025-11-03T12:41:43Z.2 Inbound socket channel start [192.168.0.219:51007] success
2025-11-03T12:41:43Z.3 Response [Switching Protocols] http/1.1 (c) keep [192.168.0.219:51007] {}
2025-11-03T12:41:43Z.3 Websocket upgraded [192.168.0.219:51007]
eynhaender19:06
Before we changed the allocation method for the disk it was possible to reserve more space than what's available. If "it worked before" means before this change, it's a expected behavior. It now needs at least as much space as needed for the first chunk of whatever this happens to be, even for the start.
(edited)evoskuil20:46
The initial table sizes can actually be set to zero (or one?) bytes. This will cause the startup to be a little sluggish, but the growth rate increases (and these rates can be controlled for each table via config as well). Low growth rates defer the disk full condition to as late as possible. The default rate is 5% of the current allocation. I run 15% on the maxi just to reduce the number of remaps.
Thursday, November 6, 2025
Huddle with pmienk, Marnix, evoskuil · 188 min16:55
Friday, November 7, 2025
evoskuil08:13
Completed testing of the
hex_reader, so we now have streaming parse and serialization (hex_writer) for binary (p2p), base16 (text/hex), json (rpc, explore, etc.), files (html) - both embedded (POD buffer/span) and file system (file handle).evoskuil08:42
Back to websockets work tomorrow. I hope to have that wrapped up with full generalization of the universal socket class tomorrow. Then on to generalization of our current http variant message type into a universal message type. This means defining a header type for each transport (
(edited)http, ws, p2p, electrum/sv1, sv2, zeromq). That will drop into a header std::variant, which will result in the current universal http message (variant body with http header only) becoming a universal tcp message.
This will allow send/receive over the same proxy::write(message) and proxy::read(message) methods with pass-through to the contained socket's two analogous methods. All aspects of framing implemented in the headers (todo), all aspects of body parse and serialization contained in the bodies (which is complete). The socket already handles tcp/p2p, http/ws using a std::optional switch between the tcp and ws stream. This will become generalized to a variant socket, with four stream types, which you can think of as tcp, tls, tcp+ws, and tls+ws. tcp and tls can upgrade to ws (from http messaging), and tls or tcp will be the initialization of any endpoint (i.e. an endpoint is either always tls or not tls). ws is really about framing, but dropping the upgrade process down to the socket simplifies implementation, due to the complexity of ws framing. It really operates more like a transport layer.evoskuil08:53
Because streams are cumulative, we build
(edited)tls on tcp and ws on either. The upgrade from one to the next destroys the original socket, so the new one simply takes its place in the variant. So to support all forms of tcp transport, we will still only require the resources of a single socket instance (and the variant type flag). Same for the universal tcp message. One c++ type with just two variants (header and body), no other state. All aspects of the transport protocol are buried in the socket, all aspects of message framing are controlled by message headers and all aspects of message encoding are controlled by message bodies. Messages, being entirely decoupled from transport, framing, and encoding, define the application protocol. This cleanly separates the transport and message plumbing from application semantics. And as such we can run any application protocol in any encoding(s) over any transport(s).evoskuil08:59
Just to be clear,
tls above refers to http/s, also applicable as sv2/s (which is not http), as well as p2p/s (bip324) and zmq/s (curve-cp). All of these will be handled by the same tls socket abstraction and are distinct only in the handshake callback, which is handled by the channel base type that derives from the proxy, which embeds the socket.evoskuil09:04
I'm working toward completion of the above model with implementations for tls/ws, http/p2p, binary/hex/json/file/span encodings, for admin website, block explorer (html and API), both embedded with file system override, and p2p. Once these are clean on the abstractions the other will slot right in. Most of this is functionally complete but some still needs to be refactored into the above. Getting p2p into this set of abstractions with zero performance hit is a real coup.
(edited)Saturday, November 8, 2025
evoskuil01:58
I completed isolation of the http->websocket upgrade to the socket class, fixed up an issue with websocket stop handling, and set up a test of the websocket implementation by issuing requests via script from a served html page, displaying the request results on the page. Not as much as I wanted to get done, but steady progress.
🎉 3
Sunday, November 9, 2025
evoskuil03:07
Since the request bodies are now entirely independent of encoding, across all protocols and endpoints, the question becomes, what's the normal form for any type of request. The most general form we have is json-rpc. The c++ representation is independent of json, existing to define the mapping from the byte stream into json DOM and onto the objects. This can accommodate everything including REST object requests (explore), page/file requests (web admin site), json-rpc (electrum, stratum_v1, bitcoind), custom-serialized rpc (stratum_v2), and any of the above over other protocols, such as ws/zmq - to include subscription.
The channels are a man-in-the-middle between the proxy/socket and the protocols. Protocols deal with semantics and store/node interaction. Proxy/socket deals with transport-level considerations. So it's the channel's job to translate between the universal message type (to/from proxy) and subscribe/send of the protocols. Subscribers allow the protocols to bind arbitrary and fully-parameterized class methods to the channel's incoming message stream. The channel owns a distributor which holds protocol-subscribed closures and invokes them based on channel invoking distributor.notify(message).
So the current question is how to best generalize the distributor and keep the channels simple. We want a protocol for any endpoint handling a common request to be reusable across those endpoints. So for example, we can attach a "get chain object" protocol to any of the channels that provide that type of request, whether REST or RPC, whether HTTP or WS or JSON-RPC. No reason we should write a block, header, tx, address, etc. getter for each of these. We've already dealt with normalization across encodings, so how to generalize these calls, and keep each strongly-typed for the protocols.
By deserializing json-rpc from stream to our rpc object model (c++), and doing the same for all other request types (regardless of encoding or structure), we can then use a single pair of methods (one for positional and the other for named parameters) in the base distributor to map rpc parameters into strongly-typed method calls. So then the question becomes, how to establish the method binding.
Each rpc call has a "method" name, each p2p message has a "command" header field, and each REST getter has an object name that it intends to get (e.g. "block"). These text names map to an RPC "function", so they just get dropped into the normal form, as the request.method field of the universal rpc request object. In json-rpc request, this method is already embedded in the body of the request. In an http REST query, it's in the message header (as the target path). In an p2p message it's in the fixed-width header. So the method maps to the same field, and the parameters, such as the hash association for a REST block request, ends up in the request.params list.
Ok, so now the distributor can define a set of subscribers, one per method, with the strongly-typed argument list for each, and a mapping from method text name to subscriber produces dispatch to the appropriate method subscriber (we do this already for p2p). In p2p, that's it. Just forward the object to the subscriber, as all handlers have the same signature (the object). But with RPC we are now generalized into arbitrary method signatures, which we have required to be strongly-typed.
So how to efficiently validate, map, and invoke the selected method using the loose collections of parameters, either ordered or named? Enter the absolute magic of c++20. Given nothing but the method reference, and a variadic reference to it's arbitrary argument list, we can:
(1) map the arguments to the method (strongly typed) (2) validate args against the method - matching type - matching name (or position) - no excess args (3) invoke the method with the args
Each of these is just one line of code.
👍 1🙌 1
Tuesday, November 11, 2025
evoskuil23:20
Neck deep in C++20 metaprogramming... the base json-rpc distributor template is implemented and tested. Now generalizing it so we can just declare the method sets for each of the three public endpoints (
bitcoind, electrum, and stratum_v1).👍 1
Thursday, November 13, 2025
evoskuil09:05
A couple days ago I wrote the rpc message dispatcher, and yesterday I generalized it to a template that can accept an interface definition. I cleaned that up and finished its testing today. This is an example of the injectable interface definition:
/// rpc interface requires methods, type, size, and mode.
struct bitcoind
{
static constexpr std::tuple methods
{
method<"getbestblockhash">{},
method<"getblock", string_t, double>{ "blockhash", "verbosity" },
method<"getblockchaininfo">{},
method<"getblockcount">{},
method<"getblockfilter", string_t, string_t>{ "blockhash", "filtertype" },
method<"getblockfrompeer", string_t, double>{ "blockhash", "peer_id" },
method<"getblockhash", double>{ "height" },
method<"getblockheader", string_t, bool>{ "blockhash", "verbose" },
method<"getblockstats", string_t, array_t>{ "hash_or_height", "stats" },
method<"getchaintips">{},
method<"getchaintxstats", option<double, 42>, option<string_t, "42">>{ "nblocks", "blockhash" },
method<"getdifficulty">{},
method<"getmempoolancestors", string_t, bool>{ "txid", "verbose" },
method<"getmempooldescendants", string_t, bool>{ "txid", "verbose" },
method<"getmempoolentry", string_t>{ "txid" },
method<"getmempoolinfo">{},
method<"getrawmempool", bool, bool>{ "verbose", "mempool_sequence" },
method<"gettxout", string_t, double, bool>{ "txid", "n", "include_mempool" },
method<"gettxoutproof", array_t, string_t>{ "txids", "blockhash" },
method<"gettxoutsetinfo", string_t, bool, double>{ "hash_type", "use_index", "scan_blocks" },
method<"preciousblock", string_t>{ "blockhash" },
method<"pruneblockchain", double>{ "height" },
method<"savemempool">{},
method<"scantxoutset", string_t, array_t>{ "action", "scanobjects" },
method<"verifychain", double, double>{ "checklevel", "nblocks" },
method<"verifytxoutproof", string_t>{ "proof" },
};
using type = decltype(methods);
static constexpr auto size = std::tuple_size_v<type>;
static constexpr group mode = group::either;
};I realized as I was completing the testing that I had neglected optional parameters. These can occur in an array of parameters, where positions are nulled or trailing positions are unspecified, and can happed with named parameters, where the parameter is just omitted. So you can see a couple of the types above are wrapped with an option and default value of the option type:
option<double, 42>. I'll work this in tomorrow. The outcome will be that handlers can reliably bind to a fixed parameter set and the default values will simply be injected when they are allowed to be defaulted and are omitted from the call. If disallowed (not configured as optional) it's just an error for the parameter to be missing.evoskuil09:21
So we'll soon have an interface definition for
bitcoind, electrum, and stratum_v1 (whatever subsets of these can be supported with just a node). The handlers are just subscriptions, like any protocol. And implementation of will be straightforward to the extent that the required information is in the store. In v4 we won't have the tx pool, so those calls will return not implemented. This dispatcher operates over the json-rpc request_t (and response_t) message, which can be produced from any transport. The channel binds the socket to the endpoint, feeds the above distributor, and attaches protocols. The protocols subscribe to messages, which in this case are dispatched by name and parameters to the strongly typed handler created by the protocol, a closure over which is pushed to the distributor as a subscription.🙌 1
Huddle with pmienk, Marnix, eynhaender, Rowan Voskuil, evoskuil · 211 min16:55
Rowan Voskuil23:12
Screenshot of the UTXO visualization chart that I have been working on this week. Made without adding any chart dependency just react. It can be panned and zoomed in and out. I also am working on a address balance graph over time.
🔥 1
the bubbles can be clicked on and will bring the user to the tx that created the output.
The charts are collapsed by default as to not get in the way. And load asynchronously as a separate react component. So they wont slow down page load or responsiveness.
Friday, November 14, 2025
evoskuil03:24
I burned like a full day on this one issue of clean option syntax, totally dumb but I couldn't let it go 😐. Not totally satisfied, but it's pretty close, eg:
static constexpr std::tuple methods
{
method<"getbestblockhash">{},
method<"getblockhash", double>{ "height" },
method<"getblock", string_t, option<1.0>>{ "blockhash", "verbosity" },
method<"getblockheader", string_t, option<true>>{ "blockhash", "verbose" },
method<"getblockstats", string_t, array_t>{ "hash_or_height", "stats" },
method<"getchaintxstats", option<42.0>, option<"hello"_t>>{ "nblocks", "blockhash" },
};It's the
(edited)option<""_t> - couldn't get it down to just option<""> but the use of the existing text array conversion _t will at least will let me sleep well tonight. :? So for rpc default value types, the default is provided and the type is inferred. option<1.0> - requires the decimal, since these are doubles (json), and option<true> true or false. array_t and object_t don't have defaults, as they can just be empty collections.This could be generalized to any types added to the
value_t variant parameter value type, when it comes time to generalize beyond the json-rpc type system.Interesting that
(edited)method<"foo"> is perfectly achievable as a type, but this is not possible for a specialized class template (other types in the same option<...> tag. So the _t converts the string to a user-defined-literal, which is a std::array<char, N> which can be handled (because it's not a char*), and the option instance just exposes it as a std::string_view and types it as a string_t (one of our json-rpc types).So now on to the integration of these optional and defaulted parameters into the universal dispatch.
evoskuil04:08
WTF clang...
error: sorry, non-type template argument of type 'double' is not yet supported method<"getblock", string_t, option<1.0>>
@pmienk Can you please look at getting us up to
clang18 (~Feb 2024).2 replies · pmienk, evoskuil · last Nov 15, 2025
pmienk18:37
Pull was made last night, all repositories that were previously compiling continue to do so, no new warnings that I recognized.
👍🏻 1
Saturday, November 15, 2025
evoskuil19:41
Nice,
clang18 finally has consteval support, so we are down to one last c++20 workaround.Sunday, November 16, 2025
Monday, November 17, 2025
evoskuil04:20
Over the last couple days I completed the
(edited)rpc dispatch code, with complete testing, and just defined all of the json-rpc interfaces using it (bitcoind, electrum, and stratum_v1).
Also spent all day Friday on construction at a friend's house, so really just 2 days since the meeting. The bulk of the work over that time was in fully integrating support for optional (defaulted) and nullable (explicitly null passed) method arguments.
I also refactored the file and namespace layout so that rpc and json are now entirely independent of json-rpc. IOW we have rpc, and we have json parse/serialize. And since our rpc model is consistent with json-rpc, it inherently supports it just by serializing via json.Tuesday, November 18, 2025
evoskuil09:01
Crossed another threshold tonight, after a long day of test and debug. The rpc dispatch is now capable of dispatching p2p message traffic with nothing more than this interface definition. The rpc dispatch template will collapse all dispatch into these declarative interface definitions. No need for protobuf, etc. Just a little bit of magical metaprogramming.
🔥 1🎉 2
Wednesday, November 19, 2025
evoskuil06:04
http dispatch is now integrated into rpc, so that's all of the current interfaces:
bitcoind, electrum, stratum_v1, http/s, and p2p. One small template to generate the dispatch over the simple method signature declarations, and then just implement the message handlers vis subscription to the generated dispatcher, within any number of attached protocols. Any other messaging protocols will be trivial to add and all will perform just as well as p2p.evoskuil09:01
The
http interface above is really just for a generic directory/embedded server (e.g. admin) site. The block explore site will use a different interface (restful query). While it will also run over http/s and ws/s transports, as with all of the interfaces, it will be independent of transport, encoding, and serialization. The interface definitions above are about the message structure - supporting fully-generalized dispatch. I've previously completed most of the work to generalize transports (http/s, ws/s, p2p/s, tcp/s zmq/s, encodings (data, text, json, html), and serialization (json-rpc, peer, explore, web, sv2). The design is to use a single pair of proxy methods (send/receive), a single pair of messages (request/response), and a single interface (generic rpc dispatch of request), with multiple message bodies to provide parse/serialize from/to all encodings, multiple message headers to prove framing for all transports, including body selection on read, and two pairs of fundamental transports (tcp, tcp/s, ws, ws/s). Really that's just tcp and ws with tls added, though there are three forms of secure sockets (tls, CurveCP, and bitcoin p2p has its own special brand). Other transports like http, zmq, and p2p are just tcp from this perspective, as only ws and tls require socket integration (framing).evoskuil09:13
Most of this is done or working already, minus much of the method implementations. The big exception is secure sockets (tls), but I've worked out the details. Presently I'm finishing up the integration of p2p with the generic dispatch. I just did the first actual transport over it after integrating it into the http endpoints. So http and ws working on it now, p2p hopefully tomorrow. That cuts out a large amount of existing and future code. I'll also add in all of the methods and dispatch for electrum, bitcoind, and stratum_v1 soon.
evoskuil09:41
A big part of the work today, apart from the dispatcher integration, was working out how to patch arbitrary c++ objects into the rpc model, which is fundamentally based on the simple json type system (though in c++). So for example, how to make it carry a block, or an http request, as a function parameter, without reducing all objects to text or json structure. So this ended up being one additional type added to our value_t variant for json parameters.
struct value_t
{
using inner_t = std::variant
<
null_t,
boolean_t,
number_t,
string_t,
array_t,
object_t,
any_t
>;
...
};The new
any_t is our own class that wraps std::any. This provides entirely safe and strongly typed object containment with full type erasure. In order to not have this inflate the value_t variant it always contains a std::shared_ptr<Type> in the std::any and exposes it as the the shared pointer. The important thing is that it's just a single type added to the variant, but it's mapped in dispatch based on the inner Type. So when the http channel wants to dispatch a request, it just does this...
if (const auto code = dispatcher_.notify(request_t
{
.method = "get",
.params = { array_t{ any_t{ tag_request<verb::get>(request) } } }
}))
{
stop(code);
return;
}The
any_t wraps the std::shared_ptr<request>, not to be confused with request_t which is the rpc base object (e.g. a json-rpc request). The any_t drops into an array_t as the .params property of the rpc request. .params can be array_t (unnamed elements) or object_t (named elements), and the .method is a string_t that selects the rpc method. in a json-rpc call the whole thing is just deserialized off the wire using boost::json. Eventually that will be happening with all of the request types.
Note tag_request<verb::get>(...) above wraps the http request, which is itself a shared_ptr<http::request>. This is an "tag" that allows dispatch based on the inner type of the pointer. It adds no size to the original pointer, and aliases the request as get/put/post, etc. That routing is defined here:
struct http_methods
{
static constexpr std::tuple methods
{
method<"get", http::method::get::cptr>{},
method<"head", http::method::head::cptr>{},
method<"post", http::method::post::cptr>{},
method<"put", http::method::put::cptr>{},
method<"delete", http::method::delete_::cptr>{},
method<"trace", http::method::trace::cptr>{},
method<"options", http::method::options::cptr>{},
method<"connect", http::method::connect::cptr>{},
method<"unknown", http::method::unknown::cptr>{}
};
};Notice the method name "get" aligns with the rpc object (
request_t.method). This is how the request_t.params end up in a method call. Those are defined in a protocol, and subscribed to messages by the inner type:
void protocol_http::start() NOEXCEPT
{
BC_ASSERT(stranded());
if (started())
return;
SUBSCRIBE_CHANNEL(method::get::cptr, handle_receive_get, _1, _2);
SUBSCRIBE_CHANNEL(method::head::cptr, handle_receive_head, _1, _2);
SUBSCRIBE_CHANNEL(method::post::cptr, handle_receive_post, _1, _2);
SUBSCRIBE_CHANNEL(method::put::cptr, handle_receive_put, _1, _2);
SUBSCRIBE_CHANNEL(method::delete_::cptr, handle_receive_delete, _1, _2);
SUBSCRIBE_CHANNEL(method::trace::cptr, handle_receive_trace, _1, _2);
SUBSCRIBE_CHANNEL(method::options::cptr, handle_receive_options, _1, _2);
SUBSCRIBE_CHANNEL(method::connect::cptr, handle_receive_connect, _1, _2);
SUBSCRIBE_CHANNEL(method::unknown::cptr, handle_receive_unknown, _1, _2);
protocol::start();
}
// Handle disallowed-by-default methods (override to implement).
// ----------------------------------------------------------------------------
bool protocol_http::handle_receive_get(const code& ec,
const method::get::cptr& get) NOEXCEPT
{
send_method_not_allowed(*get, ec);
return true;
}
bool protocol_http::handle_receive_post(const code& ec,
const method::post::cptr& post) NOEXCEPT
{
send_method_not_allowed(*post, ec);
return true;
}
bool protocol_http::handle_receive_put(const code& ec,
const method::put::cptr& put) NOEXCEPT
{
send_method_not_allowed(*put, ec);
return true;
}
bool protocol_http::handle_receive_head(const code& ec,
const method::head::cptr& head) NOEXCEPT
{
send_method_not_allowed(*head, ec);
return true;
}
bool protocol_http::handle_receive_delete(const code& ec,
const method::delete_::cptr& delete_) NOEXCEPT
{
send_method_not_allowed(*delete_, ec);
return true;
}
bool protocol_http::handle_receive_trace(const code& ec,
const method::trace::cptr& trace) NOEXCEPT
{
send_method_not_allowed(*trace, ec);
return true;
}
bool protocol_http::handle_receive_options(const code& ec,
const method::options::cptr& options) NOEXCEPT
{
send_method_not_allowed(*options, ec);
return true;
}
bool protocol_http::handle_receive_connect(const code& ec,
const method::connect::cptr& connect) NOEXCEPT
{
send_method_not_allowed(*connect, ec);
return true;
}
bool protocol_http::handle_receive_unknown(const code& ec,
const method::unknown::cptr& unknown) NOEXCEPT
{
send_method_not_allowed(*unknown, ec);
return true;
}It's also possible to bind the handlers just based on the identity of the method, as opposed to the first param shared_ptr inner type, like this (test case):
instance.subscribe(
[&](const code& ec, mock::all_required, bool a, double b, std::string c)
{
called = true;
result_a = a;
result_b = b;
result_c = c;
promise8.set_value(ec);
return true;
});The
mock::all_required is the actual method type and uniquely identifies the handler as a unique signature. For object_t params, the method descriptors also support names for correlation (a, b, c, below)...
struct mock_methods
{
static constexpr std::tuple methods
{
method<"empty_method">{},
method<"all_required", bool, double, std::string>{ "a", "b", "c" },
method<"with_options", std::string, optional<4.2>, optional<true>>{ "a", "b", "c" },
method<"with_nullify", std::string, nullable<double>, nullable<bool>>{ "a", "b", "c" },
method<"with_combine", std::string, nullable<bool>, optional<4.2>>{ "a", "b", "c" },
method<"not_required", nullable<bool>, optional<4.2>>{ "a", "b" }
};
// Derive this from above in c++26 using reflection.
using empty_method = at<0, decltype(methods)>;
using all_required = at<1, decltype(methods)>;
using with_options = at<2, decltype(methods)>;
using with_nullify = at<3, decltype(methods)>;
using with_combine = at<4, decltype(methods)>;
using not_required = at<5, decltype(methods)>;
};The all becomes part of the toolkit, making it easy to extend the node/server with any sort of communication.
evoskuil09:47
And you can see above I've also dealt with
optional<> parameters, with defaults configured, and nullable<> parameters, where the json (or other) request can explicitly pass a null_t. In the former case, if the parameter is not provided, the default is injected into the method call. If a null is provided, the parameter will have no value (nullables get passed as a std::optional<T>). Optionals only trailing in the method signatures. All part of the magical dispatch, most of which is just compile time code. Very little actually happens at run time, and actually less than what was the case with the hand-rolled p2p dispatch. For example, the full dispatch map for each interface is defined and built statically, one for all connections to the node.evoskuil20:01
Two more interfaces. I'll probably extend the rpc interface for integral values, raw binary buffer (vs string) and optional-nullables to better support these.
👀 1
Thursday, November 20, 2025
Huddle with pmienk, evoskuil, Rowan Voskuil, eynhaender · 153 min16:55
evoskuil22:38
Along with http, peer messaging is now running through the new generated interface dispatch. I had a failure on the first attempt, where the
version message was getting handled, but the verack message failed to route. Turns out I had naively use the method names here, as opposed to the message names. This defines that named message gets routed to what method given what parameters. So this is the corrected version:
static constexpr std::tuple methods
{
method<"addr", messages::peer::address::cptr>{},
method<"alert", messages::peer::alert::cptr>{},
method<"block", messages::peer::block::cptr>{},
method<"filteradd", messages::peer::bloom_filter_add::cptr>{},
method<"filterclear", messages::peer::bloom_filter_clear::cptr>{},
method<"filterload", messages::peer::bloom_filter_load::cptr>{},
method<"cfilter", messages::peer::client_filter::cptr>{},
method<"cfcheckpt", messages::peer::client_filter_checkpoint::cptr>{},
method<"cfheaders", messages::peer::client_filter_headers::cptr>{},
method<"cmpctblock", messages::peer::compact_block::cptr>{},
method<"blocktxn", messages::peer::compact_transactions::cptr>{},
method<"feefilter", messages::peer::fee_filter::cptr>{},
method<"getaddr", messages::peer::get_address::cptr>{},
method<"getblocks", messages::peer::get_blocks::cptr>{},
method<"getcfcheckpt", messages::peer::get_client_filter_checkpoint::cptr>{},
method<"getcfheaders", messages::peer::get_client_filter_headers::cptr>{},
method<"getcfilters", messages::peer::get_client_filters::cptr>{},
method<"getblocktxn", messages::peer::get_compact_transactions::cptr>{},
method<"getdata", messages::peer::get_data::cptr>{},
method<"getheaders", messages::peer::get_headers::cptr>{},
method<"headers", messages::peer::headers::cptr>{},
method<"inv", messages::peer::inventory::cptr>{},
method<"mempool", messages::peer::memory_pool::cptr>{},
method<"merkleblock", messages::peer::merkle_block::cptr>{},
method<"notfound", messages::peer::not_found::cptr>{},
method<"ping", messages::peer::ping::cptr>{},
method<"pong", messages::peer::pong::cptr>{},
method<"reject", messages::peer::reject::cptr>{},
method<"sendaddrv2", messages::peer::send_address_v2::cptr>{},
method<"sendcmpct", messages::peer::send_compact::cptr>{},
method<"sendheaders", messages::peer::send_headers::cptr>{},
method<"tx", messages::peer::transaction::cptr>{},
method<"version", messages::peer::version::cptr>{},
method<"verack", messages::peer::version_acknowledge::cptr>{},
method<"wtxidrelay", messages::peer::witness_tx_id_relay::cptr>{}
};Where the
(edited)"verack" message routes to the version_acknowledge message subscribers. Should have a read on performance impact soon.
Friday, November 21, 2025
evoskuil01:02
I mentioned today that the network build is getting slow. Just hit this:
error C1128: number of sections exceeded object file format limit: compile with /bigobj
evoskuil04:34
Back up and running on p2p, with the generated dispatch. Performance is the same, as expected. Interestingly I ran it first without the custom memory allocator, which I forgot to reconnect. It was running at about 150% of the normal time. I hadn't looked at the impact of the allocator for a couple years. When I wrote it the impact was not as significant, because there were other issues. But as we've leaned out it's become more impactful, significant in fact. It was written originally to allow blocks to be move to a common strand, which pushed destruction to another thread. That led to huge deallocation costs. We abandoned that approach, but kept the allocator (which had resolved that issue), because it provided a modest perf benefit. Now that benefit a reduction in sync time by about 1/3. I can see this in the flame graph as well, it was at about 32% total time, just for block deallocation. Now it's back to about 7%.
evoskuil18:13
Did a full clean milestone maxi run to block
(edited)900,000 on the new dispatch architecture last night, 36 seconds better than the record run, at 59.3 mins.🎉 2
Saturday, November 22, 2025
evoskuil00:36
Trying to think of what rabbit comes out of the hat next year to keep us under 60 minutes on my 2.3 Gbps connection. The most likely target is reduction of the deserialization and deallocation costs, by denormalizing the wire deserialization for milestone runs. We don't need the full block object model for milestone, all we need is tx and header hashing. That already comes directly off of the wire (before object deserialization). That should knock at least 10% off of total milestone time, and blocks 900,001-950,000 should be roughly 10% of the total chain at that point.
(edited)3 replies · evoskuil · last Nov 23, 2025
evoskuil14:22
I took a look at this last night. This can also be implemented for full validation by just moving block.check() to the block validator (vs downloader). At that point the downloader is the same as for milestone, which is sufficient given that validation will perform the checks. That would materially reduce validation cost. As significantly more memory would be available for validation, and CPU and alloc/free costs for download would drop to near zero.
evoskuil14:36
This requires a new block and tx parser to pull the necessary database fields from the wire stream, and to write them to the store. We use a “skip parser” to pull out both witness and non-witness hashes from the wire stream. This uses a fast stream that allows forward and reverse movement with the stream.
This same technique can be used to avoid any allocations at all when moving a block/txs from the wire buffer to the store. We can literally just make a store::writer that operates over a store::stream and writes directly to tables, and a block::writer that operates over the incoming wire fast stream, and then just have the the writer write the reader. The writer flush just closes out the two phase commit.
evoskuil14:39
So we just drive scripts and witnesses directly into the store with nothing but length parse. The rest is just bookkeeping trivia.
Sunday, November 23, 2025
evoskuil14:22
replied to a thread: view
I took a look at this last night. This can also be implemented for full validation by just moving block.check() to the block validator (vs downloader). At that point the downloader is the same as for milestone, which is sufficient given that validation will perform the checks. That would materially reduce validation cost. As significantly more memory would be available for validation, and CPU and alloc/free costs for download would drop to near zero.
evoskuil14:36
replied to a thread: view
This requires a new block and tx parser to pull the necessary database fields from the wire stream, and to write them to the store. We use a “skip parser” to pull out both witness and non-witness hashes from the wire stream. This uses a fast stream that allows forward and reverse movement with the stream.
This same technique can be used to avoid any allocations at all when moving a block/txs from the wire buffer to the store. We can literally just make a store::writer that operates over a store::stream and writes directly to tables, and a block::writer that operates over the incoming wire fast stream, and then just have the the writer write the reader. The writer flush just closes out the two phase commit.
Monday, November 24, 2025
evoskuil00:53
I've just completed a significant refactor of network and therefore node configuration settings. This is just part of rationalizing p2p network services with others. The large set of
[network] settings has been reduced and there are now three new config sections: [outbound], [inbound], and [manual]. Other config sections are unaffected.
Note that there is no "seeding" section, despite there being a seeding and the above four p2p session types. This is because seeding is actually part of outbound from a configuration perspective.
Each new section drives from the same base configuration settings structure (tcp_server) as the other network services. There is also an intermediate http_server derivation, and html_server derived from that. In each case sharing common base properties.evoskuil01:00
[network] # common network settings # ... [inbound] # tcp_server properties secure = false connections = 100 bind = [::]:8333 bind = 0.0.0.0:8333 inactivity_minutes = 10 expiration_minutes = 60 # inbound properties enable_loopback = false self = 42.42.42.42:8333 self = [abcd:abcd:abcd:abcd::abcd:abcd:abcd]:8333 [outbound] # tcp_server properties # ... # outbound properties enable_ipv6 = true connect_batch_size = 5 host_pool_capacity = 10,000 seeding_timeout_seconds = 60 seed = seed1.libbitcoin.net:8333 seed = seed2.libbitcoin.net:8333 seed = seed3.libbitcoin.net:8333 [manual] # tcp_server properties # ... # manual properties peer = 123.123.123.123:8333 peer = 124.123.123.123:8333 [web] # tcp_server properties # ... # http_server properties server = libbitcoin/web host = localhost:80 host = my-server # html_server properties path = m:/server default = ws.html [explore] # tcp_server properties # ... # http_server properties # ... # html_server properties # ... [bitcoind] # tcp_server properties # ... # http_server properties # ... [electrum] # tcp_server properties # ... [stratum_v1] # tcp_server properties # ... [stratum_v2] # tcp_server properties # ...
1 reply · evoskuil · last Nov 24, 2025
evoskuil01:06
bind properties will apply to outbound and manual connections and eventually allow internal multihomed load balancing. connections will apply to manual as a way to optionally select a subset of listed peers. secure is transport layer security for the endpoint. That's not currently implemented and is presently a placeholder. Given that we may want to enable secure and non-secure endpoints for each of the above services concurrently, that my just result in a distinct section name for each (e.g. [web/s] and [web]).Also I haven't shown the current
[socket] section, as that will be folded into http properties. And I haven't shown the [zeromq] section because that's not stubbed in yet.Friday, November 28, 2025
evoskuil21:46
I started the week on the task to full generalize the request and response types. Presently the bodies are fully generalized, but the headers are limited to http. So all http message types are unified, which is a lot. But the other framing types (such as p2p, electrum/sv1, sv2) are independently processed over a raw tcp socket. But my grand scheme to unify these fell apart when I got into the details of beast header processing. It became a choice between doing a lot of it from scratch, or not being able to unify with http. So I scrapped a couple days worth of work on it and moved to the next thing.
I completed the URL target path to request parser. This can drop in to an http body type and produce an rpc request object, which can then be passed directly to the dispatcher typed over that interface. This required some further generalization of the dispatcher, so that nullable parameters are also optional parameters, and so that complex types can also be nullable. That came out well, and allows us to define the interface as:
static constexpr std::tuple methods
{
// The block hash or height is logically required.
method<"block", uint8_t, nullable<system::hash_cptr>, nullable<uint32_t>>{ "version", "hash", "height" },
method<"header", uint8_t, nullable<system::hash_cptr>, nullable<uint32_t>>{ "version", "hash", "height" },
method<"filter", uint8_t, nullable<system::hash_cptr>, nullable<uint32_t>>{ "version", "hash", "height" },
method<"block_txs", uint8_t, nullable<system::hash_cptr>, nullable<uint32_t>>{ "version", "hash", "height" },
// The block hash or height is logically required.
// The position parameter is out of URL order because it's required.
method<"block_tx", uint8_t, uint32_t, nullable<system::hash_cptr>, nullable<uint32_t>>{ "version", "position", "hash", "height" },
// All parameters are required.
method<"transaction", uint8_t, system::hash_cptr>{ "version", "hash" },
method<"input", uint8_t, system::hash_cptr, uint32_t>{ "version", "hash", "index" },
method<"inputs", uint8_t, system::hash_cptr>{ "version", "hash" },
method<"input_script", uint8_t, system::hash_cptr, uint32_t>{ "version", "hash", "index" },
method<"input_scripts", uint8_t, system::hash_cptr>{ "version", "hash" },
method<"input_witness", uint8_t, system::hash_cptr, uint32_t>{ "version", "hash", "index" },
method<"input_witnesses", uint8_t, system::hash_cptr>{ "version", "hash" },
method<"output", uint8_t, system::hash_cptr, uint32_t>{ "version", "hash", "index" },
method<"outputs", uint8_t, system::hash_cptr>{ "version", "hash" },
method<"output_script", uint8_t, system::hash_cptr, uint32_t>{ "version", "hash", "index" },
method<"output_scripts", uint8_t, system::hash_cptr>{ "version", "hash" },
method<"output_spender", uint8_t, system::hash_cptr, uint32_t>{ "version", "hash", "index" },
method<"output_spenders", uint8_t, system::hash_cptr>{ "version", "hash" },
method<"address", uint8_t, system::hash_cptr>{ "version", "hash" }
};That defies the interface, and dropped into the dispatch template, that implements the routing from the request message (parsed from the request URL path), from this pattern:
/// Pagination and filtering are via query string.
enum explore_targets
{
/// /v[]/block/hash/[bkhash] {1}
/// /v[]/block/height/[height] {1}
block,
/// /v[]/block/hash/[bkhash]/header {1}
/// /v[]/block/height/[height]/header {1}
header,
/// /v[]/block/hash/[bkhash]/filter {1}
/// /v[]/block/height/[height]/filter {1}
filter,
/// -----------------------------------------------------------------------
/// /v[]/block/hash/[bkhash]/transactions {all txs in the block}
/// /v[]/block/height/[height]/transactions {all txs in the block}
block_txs,
/// /v[]/block/hash/[bkhash]/transaction/[position] {1}
/// /v[]/block/height/[height]/transaction/[position] {1}
block_tx,
/// /v[]/transaction/[txhash] {1}
transaction,
/// -----------------------------------------------------------------------
/// /v[]/input/[txhash]/[index] {1}
input,
/// /v[]/inputs/[txhash] {all inputs in the tx}
inputs,
/// /v[]/input/[txhash]/[index]/script {1}
input_script,
/// /v[]/input/[txhash]/scripts {all input scripts in the tx}
input_scripts,
/// /v[]/input/[txhash]/[index]/witness {1}
input_witness,
/// /v[]/input/[txhash]/witnesses {all witnesses in the tx}
input_witnesses,
/// -----------------------------------------------------------------------
/// /v[]/output/[txhash]/[index] {1}
output,
/// /v[]/outputs/[txhash] {all outputs in the tx}
outputs,
/// /v[]/output/[txhash]/[index]/script {1}
output_script,
/// /v[]/output/[txhash]/scripts {all output scripts in the tx}
output_scripts,
/// /v[]/output/[txhash]/[index]/spender {1 - confirmed}
output_spender,
/// /v[]/output/[txhash]/spenders {all}
output_spenders,
/// -----------------------------------------------------------------------
/// /v[]/address/[output-script-hash] {all}
address
};Here's an example of the request parser doing it's thing in one line (path_to_request):
BOOST_AUTO_TEST_CASE(path_to_request__block_tx_height_valid__expected)
{
request_t request{};
BOOST_REQUIRE(!path_to_request(request, "/v42/block/height/123456/transaction/7"));
BOOST_REQUIRE_EQUAL(request.method, "block_tx");
BOOST_REQUIRE(request.params.has_value());
const auto& params = request.params.value();
BOOST_REQUIRE(std::holds_alternative<object_t>(params));
const auto& object = std::get<object_t>(request.params.value());
BOOST_REQUIRE_EQUAL(object.size(), 3u);
const auto version = std::get<uint8_t>(object.at("version").value());
BOOST_REQUIRE_EQUAL(version, 42u);
const auto height = std::get<uint32_t>(object.at("height").value());
BOOST_REQUIRE_EQUAL(height, 123456u);
const auto position = std::get<uint32_t>(object.at("position").value());
BOOST_REQUIRE_EQUAL(position, 7u);
}So by just calling
path_to_request(out, url.path) we get the routable request (out), and by calling dispatcher_.notify(out) we invoke all subscribers for the given request method. Defining the dispatcher amounts to:
using explore = publish<explore_methods>; using interface = rpc::interface::peer::dispatch; using dispatcher = rpc::dispatcher<interface>;
Where
explore_methods is defined by the interface above.1 reply · evoskuil · last Nov 28, 2025
evoskuil22:17
Should read:
using explore = rpc::publish<explore_methods>; using dispatcher = rpc::dispatcher<explore>;
evoskuil22:02
The rest is just dropping in a handler for each method defined by the interface, invoking the database query, and posting the http response with the object in its body. I still need to add one parameter to each of the interface methods to provide the caller's accept media type (enum), so that the handler can set the return media type/serialization correctly. So prob another day or two and the full
(edited)explore endpoint will be up and running.🔥 1
1 reply · evoskuil · last Dec 01, 2025
evoskuil06:17
The node is now dispatching explore requests via the generalized dispatcher, with serialization tied to http header accept type and query string format, and with the witness option as applicable. Just need to hook up the queries for the remaining methods now. These are complete:
/v1/block/hash/[hash] /v1/block/height/[height] /v1/block/hash/[hash]/header /v1/block/height/[height]/header /v1/transaction/[hash]
with (optional):
?format=json|text|data&witness=true|false
🎉 2
evoskuil22:09
Today I took on some backlog. The dispatcher has replaced the old macro-based distributor (which was limited to the specifics of p2p). It's also been used to implement http messaging. But we had another similar mechanism in the broadcaster. This is what allows a protocol (on a given channel) to push out a message (e.g. p2p message object) to internal subscribers. So it subscribes to a message type on the distributor, and pushes out a message via the broadcaster. Subscription to broadcaster messages is available to any part of the node (channels, sessions, logging, chasers, etc).
Since this is process-wide (lives on the network/node/server object), so not limited to a channel, it requires explicit desubscription. That requires a desubscriber object as opposed to the subscriber/unsubscriber objects used internally by the dispatcher. So this required an interface expansion on the dispatcher, which I ended up implementing as a
broadcaster<Interface> derived from the existing dispatcher<Interface. So now all dispatch/broadcast is implemented in the same templates, and the big macro-based hacks are gone.