Bitcoin full node and query server
October 2025
Archive of the libbitcoin Slack #general channel.
Wednesday, October 1, 2025
Thursday, October 2, 2025
Huddle with evoskuil, Marnix, eynhaender, Rowan Voskuil, cguida · 144 min16:58
7 replies · eynhaender, cguida, evoskuil · last Oct 03, 2025
evoskuil00:34
Currently putting together HTTP (Web, REST, and json-rpc), next is WebSocket, then ElectrumX, and later TLS and Socks5.
🔥 1
Friday, October 3, 2025
evoskuil03:17
Completed the generalization of http channels to use boost beast async for read/write and parse/serialize. This reduced the full server implementation to just a few lines of interesting code.
🙌 2
evoskuil16:02
This is the entire implementation of web services over the previously-existing p2p stack. All but the first three functions are just boilerplate wrappers for hiding, error normalization, thread safety, and exception safety.
// Channel... this declares all state maintained by the derived http channel (channel_client).
subscriber<http_string_request> subscriber_;
http_flat_buffer request_buffer_;
bool reading_{};
// Channel... this listens for a request object (string).
void channel_client::read_request() NOEXCEPT
{
if (stopped() || paused() || reading_)
return;
// HTTP is half duplex.
reading_ = true;
// Use shared pointer to capture request in completion handler.
const auto request = to_shared<http_string_request>();
// Post handle_read_request to strand upon stop, error, or buffer full.
read(request_buffer_, *request,
std::bind(&channel_client::handle_read_request,
shared_from_base<channel_client>(), _1, _2, request));
}
// Channel... this notifies subscribers of the request, stopping channel on read failure.
void channel_client::handle_read_request(const code& ec, size_t,
const http_string_request_ptr& request) NOEXCEPT
{
BC_ASSERT_MSG(stranded(), "strand");
if (stopped())
return;
if (ec)
{
stop(ec);
return;
}
// HTTP is half duplex.
reading_ = false;
subscriber_.notify(error::success, *request);
}
// Channel... this sends a response object (string, file, or binary), stopping channel on send failure.
template <class Message>
void send(Message&& response, result_handler&& handler) NOEXCEPT
{
BC_ASSERT_MSG(stranded(), "strand");
const auto ptr = system::make_shared(std::forward<Message>(response));
// Use shared pointer to capture response in completion intermediate handler.
auto complete = [self = shared_from_base<channel_client>(), ptr,
handle = std::move(handler)](const code& ec, size_t) NOEXCEPT
{
if (ec) self->stop(ec);
handle(ec);
};
write(*ptr, std::move(complete));
} (edited)🎉 1🔥 1
///////////////////////////////////////////////////////////////////////////////
// This is all just boilerplate...
///////////////////////////////////////////////////////////////////////////////
// Proxy... these just hide the socket from channels.
// Reads.
// ----------------------------------------------------------------------------
void proxy::read(http_flat_buffer& buffer, http_string_request& request,
count_handler&& handler) NOEXCEPT
{
socket_->http_read(buffer, request, std::move(handler));
}
// Writes.
// ----------------------------------------------------------------------------
// Writes are composed but http is half duplex so there is no interleave risk.
void proxy::write(const http_string_response& response,
count_handler&& handler) NOEXCEPT
{
socket_->http_write(response, std::move(handler));
}
void proxy::write(const http_data_response& response,
count_handler&& handler) NOEXCEPT
{
socket_->http_write(response, std::move(handler));
}
void proxy::write(http_file_response& response,
count_handler&& handler) NOEXCEPT
{
socket_->http_write(response, std::move(handler));
}
// Socket... these just ensure that socket calls are stranded (thread safe).
// There is no actual post to the strand if the caller is already stranded.
// HTTP I/O.
// ----------------------------------------------------------------------------
void socket::http_read(http_flat_buffer& buffer, http_string_request& request,
count_handler&& handler) NOEXCEPT
{
boost::asio::dispatch(strand_,
std::bind(&socket::do_http_read_string_buffered, shared_from_this(),
std::ref(buffer), std::ref(request), std::move(handler)));
}
void socket::http_write(const http_string_response& response,
count_handler&& handler) NOEXCEPT
{
boost::asio::dispatch(strand_,
std::bind(&socket::do_http_write_string,
shared_from_this(), std::cref(response), std::move(handler)));
}
void socket::http_write(const http_data_response& response,
count_handler&& handler) NOEXCEPT
{
boost::asio::dispatch(strand_,
std::bind(&socket::do_http_write_data,
shared_from_this(), std::cref(response), std::move(handler)));
}
void socket::http_write(http_file_response& response,
count_handler&& handler) NOEXCEPT
{
boost::asio::dispatch(strand_,
std::bind(&socket::do_http_write_file,
shared_from_this(), std::ref(response), std::move(handler)));
}
// Socket... these ensure that socket calls are exception safe (and manage read buffer consumption).
// http executors (private).
// ----------------------------------------------------------------------------
void socket::do_http_read_string_buffered(
const std::reference_wrapper<http_flat_buffer>& buffer,
const std::reference_wrapper<http_string_request>& request,
const count_handler& handler) NOEXCEPT
{
BC_ASSERT_MSG(stranded(), "strand");
auto complete = [=](const code& ec, size_t size) NOEXCEPT
{
buffer.get().consume(size);
handler(ec, size);
};
try
{
// This operation posts handler to the strand.
boost::beast::http::async_read(socket_, buffer.get(), request.get(),
std::bind(&socket::handle_http,
shared_from_this(), _1, _2, std::move(complete)));
}
catch (const std::exception& LOG_ONLY(e))
{
LOGF("Exception @ do_http_read_string_buffered: " << e.what());
handler(error::operation_failed, zero);
}
}
void socket::do_http_write_string(
const std::reference_wrapper<const http_string_response>& response,
const count_handler& handler) NOEXCEPT
{
BC_ASSERT_MSG(stranded(), "strand");
try
{
// This operation posts handler to the strand.
boost::beast::http::async_write(socket_, response.get(),
std::bind(&socket::handle_http,
shared_from_this(), _1, _2, handler));
}
catch (const std::exception& LOG_ONLY(e))
{
LOGF("Exception @ do_http_write_string: " << e.what());
handler(error::operation_failed, zero);
}
}
void socket::do_http_write_data(
const std::reference_wrapper<const http_data_response>& response,
const count_handler& handler) NOEXCEPT
{
BC_ASSERT_MSG(stranded(), "strand");
try
{
// This operation posts handler to the strand.
boost::beast::http::async_write(socket_, response.get(),
std::bind(&socket::handle_http,
shared_from_this(), _1, _2, handler));
}
catch (const std::exception& LOG_ONLY(e))
{
LOGF("Exception @ do_http_write_data: " << e.what());
handler(error::operation_failed, zero);
}
}
void socket::do_http_write_file(
const std::reference_wrapper<http_file_response>& response,
const count_handler& handler) NOEXCEPT
{
BC_ASSERT_MSG(stranded(), "strand");
const auto writer = std::make_shared<http_file_serializer>(response.get());
auto complete = [writer, handler](const code& ec, size_t size) NOEXCEPT
{
handler(ec, size);
};
try
{
// This operation posts handler to the strand.
boost::beast::http::async_write(socket_, *writer,
std::bind(&socket::handle_http,
shared_from_this(), _1, _2, std::move(complete)));
}
catch (const std::exception& LOG_ONLY(e))
{
LOGF("Exception @ do_http_write_file: " << e.what());
handler(error::operation_failed, zero);
}
}
// Socket... this just normalizes to our return codes (from boost codes).
// handlers (private).
// ----------------------------------------------------------------------------
void socket::handle_http(const error::boost_code& ec, size_t size,
const count_handler& handler) NOEXCEPT
{
BC_ASSERT_MSG(stranded(), "strand");
if (error::asio_is_canceled(ec))
{
handler(error::channel_stopped, size);
return;
}
// Translate other beast error codes and invoke caller handler.
handler(error::beast_to_error_code(ec), size);
} (edited)The request and response objects are fully-parsed and directly usable by the protocols. Protocols just subscribe to the request and send a response.
There's also the subscriber in the channel...
template <class Message, typename Handler>
void subscribe(Handler&& handler) NOEXCEPT
{
BC_ASSERT_MSG(stranded(), "strand");
subscriber_.subscribe(std::forward<Handler>(handler));
}Saturday, October 4, 2025
evoskuil00:50
I've completed a full generalization of request dispatch by http::method (GET, POST, PUT, ...) using the existing message distributor and wrapping each request by method in an empty struct that carries the request pointer. This allows for compile-time binding of request subscription and cleanly isolates handlers by request method.
M
Marnix06:09
Is there an automated way to generate/update the example config file from the parser?
(edited)evoskuil15:07
bn -s These are the configuration settings that can be set. [bitcoin] # The activation time for bip16 in unix time, defaults to 1333238400. bip16_activation_time = <value> # The number of blocks considered for bip34 style soft fork activation, defaults to 1000. bip34_activation_sample = <value> # The number of new version blocks required for bip34 style soft fork activation, defaults to 750. bip34_activation_threshold = <value> # The number of new version blocks required for bip34 style soft fork enforcement, defaults to 950. bip34_enforcement_threshold = <value> # The block height to freeze the bip34 softfork for bip90, defaults to 227931. bip34_freeze = <value> # The block height to freeze the bip65 softfork for bip90, defaults to 388381. bip65_freeze = <value> # The block height to freeze the bip66 softfork for bip90, defaults to 363725. bip66_freeze = <value> # The hash:height checkpoint for bip9 bit0 activation, defaults to 000000000000000004a1b34462cb8aeebd5799177f7a29cf28f2d1961716b5b5:419328. bip9_bit0_active_checkpoint = <value> # The target block period, defaults to 600. block_spacing_seconds = <value> # The blockchain checkpoints, defaults to the consensus set. checkpoint = <value> # The genesis block, defaults to mainnet. genesis_block = <value> # The initial block subsidy, defaults to 50. initial_block_subsidy_bitcoin = <value> # A block presumed to be valid but not required to be present, defaults to 000000000000000000010538edbfd2d5b809a33dd83f284aeea41c6d0d96968a:900000. milestone = <value> # The minimum work for any branch to be considered valid, defaults to 000000000000000000000000000000000000000052b2559353df4117b7348b64. minimum_work = <value> # The proof of work limit, defaults to 486604799. proof_of_work_limit = <value> # The difficulty retargeting factor, defaults to 4. retargeting_factor = <value> # The difficulty retargeting period, defaults to 1209600. retargeting_interval_seconds = <value> # The subsidy halving period, defaults to 210000. subsidy_interval = <value> # The future timestamp allowance, defaults to 7200. timestamp_limit_seconds = <value> [database] # The number of buckets in the address table head, defaults to '1' (0|1 disables). address_buckets = <value> # The percentage expansion of the address table body, defaults to '5'. address_rate = <value> # The minimum allocation of the address table body, defaults to '1'. address_size = <value> # The percentage expansion of the candidate table body, defaults to '5'. candidate_rate = <value> # The minimum allocation of the candidate table body, defaults to '2575500'. candidate_size = <value> # The percentage expansion of the candidate table body, defaults to '5'. confirmed_rate = <value> # The minimum allocation of the candidate table body, defaults to '2575500'. confirmed_size = <value> # The minimum number of buckets in the duplicate table head, defaults to '1024'. duplicate_buckets = <value> # The percentage expansion of the duplicate table, defaults to '5'. duplicate_rate = <value> # The minimum allocation of the duplicate table body, defaults to '44'. duplicate_size = <value> # The number of buckets in the filter_bk table head, defaults to '0' (0 disables). filter_bk_buckets = <value> # The percentage expansion of the filter_bk table body, defaults to '5'. filter_bk_rate = <value> # The minimum allocation of the filter_bk table body, defaults to '1'. filter_bk_size = <value> # The number of buckets in the filter_tx table head, defaults to '0' (0 disables). filter_tx_buckets = <value> # The percentage expansion of the filter_tx table body, defaults to '5'. filter_tx_rate = <value> # The minimum allocation of the filter_tx table body, defaults to '1'. filter_tx_size = <value> # The number of buckets in the header table head, defaults to '386364'. header_buckets = <value> # The percentage expansion of the header table body, defaults to '5'. header_rate = <value> # The minimum allocation of the header table body, defaults to '21000000'. header_size = <value> # The percentage expansion of the input table body, defaults to '5'. input_rate = <value> # The minimum allocation of the input table body, defaults to '92500000000'. input_size = <value> # The percentage expansion of the ins table body, defaults to '5'. ins_rate = <value> # The minimum allocation of the point table body, defaults to '8550000000'. ins_size = <value> # The percentage expansion of the output table body, defaults to '5'. output_rate = <value> # The minimum allocation of the output table body, defaults to '25300000000'. output_size = <value> # The percentage expansion of the puts table body, defaults to '5'. outs_rate = <value> # The minimum allocation of the puts table body, defaults to '3700000000'. outs_size = <value> # The blockchain database directory, defaults to 'blockchain'. path = <value> # The number of buckets in the spend table head, defaults to '1365977136'. point_buckets = <value> # The percentage expansion of the point table body, defaults to '5'. point_rate = <value> # The minimum allocation of the point table body, defaults to '25700000000'. point_size = <value> # The minimum number of buckets in the prevout table head, defaults to '0'. prevout_buckets = <value> # The percentage expansion of the prevout table, defaults to '5'. prevout_rate = <value> # The minimum allocation of the prevout table body, defaults to '1'. prevout_size = <value> # The number of buckets in the strong_tx table head, defaults to '469222525'. strong_tx_buckets = <value> # The percentage expansion of the strong_tx table body, defaults to '5'. strong_tx_rate = <value> # The minimum allocation of the strong_tx table body, defaults to '2900000000'. strong_tx_size = <value> # The number of buckets in the tx table head, defaults to '469222525'. tx_buckets = <value> # The percentage expansion of the tx table body, defaults to '5'. tx_rate = <value> # The minimum allocation of the tx table body, defaults to '17000000000'. tx_size = <value> # The number of buckets in the txs table head, defaults to '900001'. txs_buckets = <value> # The percentage expansion of the txs table body, defaults to '5'. txs_rate = <value> # The minimum allocation of the txs table body, defaults to '1050000000'. txs_size = <value> # The number of buckets in the validated_bk table head, defaults to '900001'. validated_bk_buckets = <value> # The percentage expansion of the validated_bk table body, defaults to '5'. validated_bk_rate = <value> # The minimum allocation of the validated_bk table body, defaults to '1700000'. validated_bk_size = <value> # The number of buckets in the validated_tx table head, defaults to '1'. validated_tx_buckets = <value> # The percentage expansion of the validated_tx table body, defaults to '5'. validated_tx_rate = <value> # The minimum allocation of the validated_tx table body, defaults to '1'. validated_tx_size = <value>
[forks] # Add check-sequence-verify op code, defaults to true (soft fork). bip112 = <value> # Use median time past for locktime, defaults to true (soft fork). bip113 = <value> # Segregated witness consensus layer, defaults to true (soft fork). bip141 = <value> # Witness version 0 (segwit), defaults to true (soft fork). bip143 = <value> # Prevent dummy value malleability, defaults to true (soft fork). bip147 = <value> # Add pay-to-script-hash processing, defaults to true (soft fork). bip16 = <value> # Disallow collision of unspent transaction hashes, defaults to true (soft fork). bip30 = <value> # Require coinbase input includes block height, defaults to true (soft fork). bip34 = <value> # Witness version 1 (taproot), defaults to true (soft fork). bip341 = <value> # Validation of taproot script, defaults to true (soft fork). bip342 = <value> # Finite monetary supply, defaults to true (soft fork). bip42 = <value> # Add check-locktime-verify op code, defaults to true (soft fork). bip65 = <value> # Require strict signature encoding, defaults to true (soft fork). bip66 = <value> # Add relative locktime enforcement, defaults to true (soft fork). bip68 = <value> # Assume bip34, bip65, and bip66 activation if enabled, defaults to true (hard fork). bip90 = <value> # Require difficult blocks, defaults to true (use false for testnet). difficult = <value> # Retarget difficulty, defaults to true. retarget = <value> # Fix target overflow for very low difficulty, defaults to false (hard fork). retarget_overflow_patch = <value> # Use scrypt hashing for proof of work, defaults to false (hard fork). scrypt_proof_of_work = <value> # Fix time warp bug, defaults to false (hard fork). time_warp_patch = <value> [log] # Enable application logging, defaults to true. application = <value> # Enable local fault logging, defaults to true. fault = <value> # The maximum byte size of each pair of rotated log files, defaults to 1000000. maximum_size = <value> # Enable news logging, defaults to true. news = <value> # The log files directory, defaults to empty. path = <value> # Enable protocol logging, defaults to false. protocol = <value> # Enable proxy logging, defaults to false. proxy = <value> # Enable quitting logging, defaults to false. quitting = <value> # Enable remote fault logging, defaults to true. remote = <value> # Enable session logging, defaults to true. session = <value> # Path to windows debug build symbols file (.pdb). symbols = <value> # Enable verbose logging, defaults to false. verbose = <value> [network] # The lower bound for address selection divisor, defaults to 5. address_lower = <value> # The upper bound for address selection divisor, defaults to 10. address_upper = <value> # IP address to bind, multiple entries allowed, defaults to 0.0.0.0:8333. bind = <value> # IP address to disallow as a peer, multiple entries allowed. blacklist = <value> # The age limit for any connection, defaults to 1440. channel_expiration_minutes = <value> # The time between ping messages, defaults to 5. channel_heartbeat_minutes = <value> # The inactivity time limit for any connection, defaults to 10. channel_inactivity_minutes = <value> # The number of concurrent attempts to establish one connection, defaults to 5. connect_batch_size = <value> # The time limit for connection establishment, defaults to 5. connect_timeout_seconds = <value> # Enable address messages, defaults to true. enable_address = <value> # Enable address v2 messages, defaults to false. enable_address_v2 = <value> # Enable alert messages, defaults to false. enable_alert = <value> # Enable enable compact block messages, defaults to false. enable_compact = <value> # Enable internet protocol version 6 (IPv6), defaults to false. enable_ipv6 = <value> # Allow connections from the node to itself, defaults to false. enable_loopback = <value> # Enable reject messages, defaults to false. enable_reject = <value> # Enable transaction relay, defaults to true. enable_relay = <value> # Enable witness transaction identifier relay, defaults to false. enable_witness_tx = <value> # The time limit to complete the connection handshake, defaults to 30. handshake_timeout_seconds = <value> # The maximum number of peer hosts in the pool, defaults to 10000. host_pool_capacity = <value> # The magic number for message headers, defaults to 3652501241. identifier = <value> # The target number of incoming network connections, defaults to 100. inbound_connections = <value> # The advertised services that cause a peer to be dropped, defaults to 176. invalid_services = <value> # The maximum allowable channel clock skew, defaults to 120. maximum_skew_minutes = <value> # The minimum retained read buffer size, defaults to 4000000. minimum_buffer = <value> # The target number of outgoing network connections, defaults to 100. outbound_connections = <value> # The peer address cache file directory, defaults to empty. path = <value> # A persistent peer node, multiple entries allowed. peer = <value> # The maximum network protocol version, defaults to 70012. protocol_maximum = <value> # The minimum network protocol version, defaults to 31800. protocol_minimum = <value> # The peer download rate limit in bytes per second, defaults to 1024 (not implemented). rate_limit = <value> # The time delay for failed connection retry, defaults to 1. retry_timeout_seconds = <value> # A seed node for initializing the host pool, multiple entries allowed. seed = <value> # The time limit for obtaining seed addresses, defaults to 30. seeding_timeout_seconds = <value> # IP address to advertise, multiple entries allowed. self = <value> # The maximum services exposed by network connections, defaults to 9 (full node, witness). services_maximum = <value> # The minimum services exposed by network connections, defaults to 9 (full node, witness). services_minimum = <value> # The minimum number of threads in the network threadpool, defaults to 16. threads = <value> # The node user agent string, defaults to '/libbitcoin:4.0.0/'. user_agent = <value> # Validate the checksum of network messages, defaults to false. validate_checksum = <value> # IP address to allow as a peer, multiple entries allowed. whitelist = <value> [node] # Block deserialization buffer multiple of wire size, defaults to 20 (0 disables). allocation_multiple = <value> # Allowable underperformance standard deviation, defaults to 1.5 (0 disables). allowed_deviation = <value> # Limit of per channel cached peer block and tx announcements, to avoid replaying (defaults to 42). announcement_cache = <value> # Time from present that blocks are considered current, defaults to 60 (0 disables). currency_window_minutes = <value> # Delay accepting inbound connections until node is current, defaults to true. delay_inbound = <value> # Maximum number of blocks to download concurrently, defaults to '50000' (0 disables). maximum_concurrency = <value> # Maximum block height to populate, defaults to 0 (unlimited). maximum_height = <value> # Set the validation threadpool to high priority, defaults to true. priority = <value> # Sampling period for drop of stalled channels, defaults to 10 (0 disables). sample_period_seconds = <value> # The number of threads in the validation threadpool, defaults to 32. threads = <value> [server] # IP address to bind, multiple entries allowed, defaults to 0.0.0.0:8080. bind = <value> # The target number of incoming network connections, defaults to 0. inbound_connections = <value>
4 replies · Marnix, evoskuil · last Oct 04, 2025
evoskuil15:27
Though this has grown so much that it's prob not a good example. I would like to have this spit out the actual values as well, including the defaulted ones, but that's low pri work.
👍 1
evoskuil15:08
bn -h
Usage: bn [-abdfhiklnrstvw] [--config value]
Info: Runs a full bitcoin node.
Options (named):
-a [--slabs] Scan and display store slab measures.
-b [--backup] Backup to a snapshot (can also do live).
-c [--config] Specify path to a configuration settings file.
-d [--hardware] Display hardware compatibility.
-f [--flags] Scan and display all flag transitions.
-h [--help] Display command line options.
-i [--information] Scan and display store information.
-k [--buckets] Scan and display all bucket densities.
-l [--collisions] Scan and display hashmap collision stats (may exceed
RAM and result in SIGKILL).
-n [--newstore] Create new store in configured directory.
-r [--restore] Restore from most recent snapshot.
-s [--settings] Display all configuration settings.
-t [--test] Run built-in read test and display.
-v [--version] Display version information.
-w [--write] Run built-in write test and display.Sunday, October 5, 2025
evoskuil18:46
[admin] # user interface (http/s, stateless html) bind = 192.168.0.219:8082 bind = 192.168.254.85:8083 connections = 1000 host = localhost path = m:/bitcoin/server/admin [explore] # block explorer (http/s, stateless html) bind = 192.168.0.219:8180 connections = 1000 host = localhost path = m:/bitcoin/server/explore [rest] # native RESTful query interface (http/s, stateless json) bind = 192.168.0.219:8280 connections = 1000 host = localhost [websocket] # native websocket query interface (http/s->tcp/s, json, upgrade handshake) bind = 192.168.0.219:8380 connections = 1000 host = localhost [bitcoind] # bitcoind compatibility interface (http/s, stateless json-rpc-v2) bind = 192.168.0.219:8480 connections = 1000 host = localhost [electrum] # electrum compatibility interface (tcp/s, json-rpc-v2) bind = 192.168.0.219:8580 connections = 1000 [stratum.v1] # stratum v1 compatibility interface (tcp/s, json-rpc-v1, auth handshake) bind = 192.168.0.219:8680 connections = 1000 [stratum.v2] # stratum v2 compatibility interface (tcp[/s], binary, auth/privacy handshake) bind = 192.168.0.219:8790 connections = 1000
🚀 1
1 reply · evoskuil · last Oct 05, 2025
Monday, October 6, 2025
1 reply · Marnix · last Oct 06, 2025
Tuesday, October 7, 2025
4 replies · Marnix, pmienk · last Oct 07, 2025
pmienk05:41
Because it's passthrough configuration.
--help displays script arguments, --without-tests is a configure.ac argument and CMakeLists.txt depending on build system.pmienk05:44
anything the script doesn't recognize gets passed to every invoked configuration (with some questions surrounding boost and other dependencies which don't conform to standard configuration). So it'll go out to, say, icu which may or may not recognize it. but standard behavior is to ignore unrecognized parameters, which is why we do it.
pmienk04:48
though passing it will pass it to all projects as it's passthrough to configuration. probably still see a test run with no tests execute
👍 1
Wednesday, October 8, 2025
evoskuil05:30
The generalized web server endpoints are now fully functional. The node serves a configured set of endpoints, from a configured directory, with a configured default page. Here it is serving up the Bitbikers site from the local file system, running entirely on the existing asio/proactor network infrastructure.
🔥 1🚀 2🎉 1
Thursday, October 9, 2025
Huddle with pmienk, Marnix, evoskuil, Rowan Voskuil, parsevalbtc, eynhaender · 144 min16:59
1 reply · Rowan Voskuil · last Oct 09, 2025
pmienk17:07
In an attempt to accommodate people some frequent attendees we will be moving the Developer Weekly Meetup 1 hour earlier in the following weeks. Any feedback is welcome.
👍 2👎 1👍🏻 1🤙🏻 1
Friday, October 10, 2025
evoskuil01:46
I moved the timeouts (inactivity and expiration) from
channel_peer to channel and integrated them with channel_client, so now the client-server endpoints will timeout connections. Initially I'm just using the [network] (p2p) config settings for this.P
parsevalbtc14:44
I've just discovered I have sha_ni instructions on my PC . To enable them in the compilation process with
install.sh script I can just put the --enable-shani option right? https://github.com/libbitcoin/libbitcoin-system/blob/ae55f24704c6c9660fccf6e98ec1e61865583574/configure.ac#L15441 replies · parsevalbtc, eynhaender, pmienk, evoskuil · last Oct 11, 2025
eynhaender15:55
CFLAGS="-msse4.1 -msha -mavx512f -mavx512dq -mavx512bw -O3" CXXFLAGS="-msse4.1 -msha -mavx512f -mavx512dq -mavx512bw -O3" ./install-cmake.sh --prefix=/home/eynhaender/development/libbitcoin/build/mmap_patch/ --build-dir=/home/eynhaender/development/libbitcoin/build --build-secp256k1 --build-boost --disable-sharedP
parsevalbtc15:59
Thanks @eynhaender! Checking the log it seems that it does those flags everywhere exept to libbitcoin-system"
(edited)eynhaender16:13
yes. and of course you need to adapt the options for your specific hardware. -mavx2 will add axv2
👍 1
pmienk16:26
With the cmake installer and cmake in general, arguments at the configuration level are written
-Denable-shaniP
parsevalbtc16:29
@pmienk it's strange the usual script does not work... if you want I can provide the log in DM
pmienk16:29
this matches the convention of that platform.
configure.ac is an autotools artifact. passing the cflags directly without signaling to our build system will not fully inform the compilation as far as I'm aware as it fails to define WITH_SHANI which was relied upon to enable the codepathsP
parsevalbtc13:09
Eventually built with cmake recipe. Is fast like a rocket with the optimizations compiled and ssd disk 🚀🚀🚀
evoskuil14:49
It will start getting slower as the chain grows, because of the limited RAM. You only got to about 25% chain.
(edited)P
parsevalbtc14:52
in a past dev meeting you mentioned that at least you have to store the table's head in the RAM. Actually I don't remember how big those are...
P
parsevalbtc14:54
the question is, if the RAM size is below heads' size, the node put's them between RAM and disk or only to the disk?
evoskuil14:55
It's all in RAM, but these are frequently and randomly accessed during writes. The tables are not, mostly append.
P
parsevalbtc15:10
you mean table's heads are always loaded in RAM (and eventually swap for the exceeding part)?
(edited)P
parsevalbtc20:56
With external disk,as expected, is running significantly slower, but it is not stalling. So keep you posted!
Saturday, October 11, 2025
E
expatriotic is me15:33
Hello, having a time of it trying to get this up and running. Just got this error "configure: error: Boost 1.86.0 or later is required but was not found."
After running
sudo ./install.sh --prefix=/usr --build-boost --build-secp256k1(edited)
evoskuil15:39
Hi, probably your installed boost is missing parts. Avoid sudo, build into a home subdirectory
--prefix, use --disabled-shared to avoid inadvertent linkage to your installed boost.E
expatriotic is me17:28
I was building everything manually since the script failed. That also did not work. So trying ./install.sh again without sudo
./install.sh --prefix=$HOME/libbitcoin --build-boost --build-secp256k1 --disable-shared
Why does boost default to 1.89.0 though?
(edited)236 replies · pmienk, evoskuil, expatriotic is me, otto +1 · last Oct 13, 2025
pmienk17:41
The issue is not sudo, but instead the likelihood of an existing installation of boost which does not fully satisfy our requirements but is detected and preferred via the boost/autoconf archive/cmake provided discovery mechanism. A prefix directory explicitly provides a preferred first path of discovery.
evoskuil18:01
It would be nice if Boost could be relied upon to just work across version upgrades, but this is often not the case. There are frequently issues we have to work around and sometimes ones that we cannot. In those cases we hold off supported version updates until they patch.
E
expatriotic is me18:12
Asking for a friend:
"you could ask how to compile the main project without having to install the others.
I compiled the others, but don't want to install them.
If the build process can reference the other directories, nice"
evoskuil18:17
You don't have to install anything, everything builds and into the prefix directory, no sudo required. If you just want to run the node, it's just a single file binary with no dependencies (for the static build).
evoskuil18:18
The node project depends on the other three projects (system, network, database), Those must be built. Also secp256k1 and boost.
pmienk18:25
@expatriotic is me, @evoskuil the install script in each project installs that project and its dependencies. If you just want system, install system. If the issue is installation to the system as a whole, Eric is right, prefix installation scopes where everything lives and allows for placement in paths or not as the user dictates.
👍 1
pmienk18:27
If the goal is development and building instead of installation, this requires a greater degree of sophistication, and I can share my approach on Linux, as can a few others.
E
expatriotic is me18:39
https://bin.ronindojo.io/?639871059b0ec067#J4J11U5gpAbrpzmEtCngCYUoTmQmpFUdjZ9Pm1NRhxXG
mkdir: cannot create directory '/home/me': Permission denied
I've also tried a sandboxed approach [https://bin.ronindojo.io/?6525653b57ed9616#3LH1o2YKdWofngao2pN5NGWe6zwSQCoQt7FmmReYMKo9]... No matter what I do, I can't get it to run the command
bx version
E
expatriotic is me18:41
I'm curious. I didn't see in the documentation what to do after the script finishes to verify if it worked.
Gemini says
nano ~/.bashrc export PATH="$HOME/libbitcoin/bin:$PATH" bx version
evoskuil18:43
I'm not sure about your directory create question, but the pastebin is showing that you are building libbitcoin-system from the version3 branch. Is this your intent?
evoskuil18:45
Not if that's what you are trying to build. But I thought you were after the current node build.
pmienk18:46
Trying to create a directory under /home outside a user directory typically requires privileges
(edited)evoskuil18:46
Ok, then you want to get the
install.sh from the libbitcoin-node repo and the master branch.E
expatriotic is me18:47
I just ran:
git clone https://github.com/libbitcoin/libbitcoin-system.git cd libbitcoin-system ./install.sh --prefix=$HOME/libbitcoin --build-boost --build-secp256k1 --disable-shared
evoskuil18:49
libbitcoin-network is the p2p network stack, but has no connection to a database (i.e. can be used by wallets, etc).evoskuil18:50
libbitcoin-node pulls together the other three and builds a chain off the network and stores it in the database.E
expatriotic is me19:18
Version Information...
libbitcoin-node: 4.0.0
libbitcoin-database: 4.0.0
libbitcoin-network: 4.0.0
libbitcoin-system: 4.0.0
E
expatriotic is me19:20
Gemini says now I:
source ~/.bashrc mkdir -p ~/.config/bitcoin nano ~/.config/bitcoin/bn.cfg [database] directory = /home/satoshi/libbitcoin-data [node] network = mainnet mkdir ~/libbitcoin-data screen -S node bn
evoskuil19:21
bn --settings shows all of the current config file settings. you don't need to set any of them.evoskuil19:25
These are runtime options you can type at the console with the node running:
2025-10-11T19:24:23Z.0 Toggle: [a]pplication 2025-10-11T19:24:23Z.0 Toggle: [n]ews 2025-10-11T19:24:23Z.0 Toggle: [s]ession 2025-10-11T19:24:23Z.0 Toggle: [p]rotocol 2025-10-11T19:24:23Z.0 Toggle: [x]proxy 2025-10-11T19:24:23Z.0 Toggle: [r]emote 2025-10-11T19:24:23Z.0 Toggle: [f]ault 2025-10-11T19:24:23Z.0 Toggle: [q]uitting 2025-10-11T19:24:23Z.0 Toggle: [o]bjects 2025-10-11T19:24:23Z.0 Toggle: [v]erbose 2025-10-11T19:24:23Z.0 Option: [b]ackup the store 2025-10-11T19:24:23Z.0 Option: [c]lose the node 2025-10-11T19:24:23Z.0 Option: [e]rrors in store 2025-10-11T19:24:23Z.0 Option: [g]o network communication 2025-10-11T19:24:23Z.0 Option: [h]old network communication 2025-10-11T19:24:23Z.0 Option: [i]nfo about store 2025-10-11T19:24:23Z.0 Option: [m]enu of options and toggles 2025-10-11T19:24:23Z.0 Option: [t]est built-in case 2025-10-11T19:24:23Z.0 Option: [w]ork distribution 2025-10-11T19:24:23Z.0 Option: [z]eroize disk full error
E
expatriotic is me19:32
ts:20250903/
2025-10-11T19:32:02Z.1 Outbound [45.129.99.150:8333] version (70016) /Satoshi:24.99.0/
2025-10-11T19:32:02Z.5 Insufficient services (1032) by [45.129.99.150:8333] showing (67112013) /Satoshi:24.99.0/
(edited)2025-10-11T19:32:12Z.0 CONSOLE: 'bn -d'
2025-10-11T19:32:17Z.1 Outbound [123.100.246.80:8333] version (70016) /Satoshi:29.1.0/
2025-10-11T19:32:17Z.1 Outbound [159.89.121.240:8333] version (70016) /Satoshi:29.1.0/
2025-10-11T19:32:25Z.1 Outbound [203.11.72.30:8333] version (70016) /Satoshi:29.1.0/E
expatriotic is me19:34
hmm... It's running though. Do I open a new terminal window? THis is a screen. I was about to detach and sleep
evoskuil19:35
You can run multiple instances, but they can't share a store. That can be varied in the config file. But this is just a console dump of build info.
E
expatriotic is me19:36
satoshi@ubuntu-bitcoin:~$ bn -d Hardware configuration... arm..... platform:0. intel... platform:1. avx512.. platform:0 compiled:0. avx2.... platform:1 compiled:0. sse41... platform:1 compiled:0. shani... platform:0 compiled:0. satoshi@ubuntu-bitcoin:~$
evoskuil19:37
So your hardware supports avx2. Not great because no shani, but better than nothing. You should rebuild with that enabled.
E
expatriotic is me19:38
It's an Optiplex running Proxmox.
So wipe it and redo.
How would I enable
avx2?E
expatriotic is me23:13
Once this is up, I can use as node for Sparrow + my HWW (Passport). I.e., I am not limited to just creating a seed + making txs in the native Libbitcoin CLI interphase. Explorer I think it's called.
bx👍🏻 1
E
expatriotic is me23:49
expatriotic is me
``satoshi@ubuntu-bitcoin:~$ bn -d
Hardware configuration...
arm..... platform:0.
intel... platform:1.
avx512.. platform:0 compiled:0.
avx2.... platform:1 compiled:0.
sse41... platform:1 compiled:0.
shani... platform:0 compiled:0.
satoshi@ubuntu-bitcoin:~$``O
otto00:19
Hey, I was the "friend" he asked about. @evoskuil send me the link weeks ago, but I didn't have the time.
I compiled it yesterday, now will do it again with the CPU flags
O
otto01:08
^^^ Seems to work to detect the flags without having to first compile bn:
grep -Eo "avx5|avx2|sse|sha" /proc/cpuinfo | sort -uevoskuil01:17
It ensures consistency, because there are some subtleties (like multiple variants of avx512, not all of which are sufficient).
✅ 1
E
expatriotic is me01:37
I updated from 8GB which was the arbitrary amount of RAM I had given my LXC "ubuntu-bitcoin"
But now it's gotten worse, it's only 2GB being used. The image is of the proxmox host "PVE2"
E
expatriotic is me01:39
This is what the container is doing
Status
running
CPU usage
0.49% of 4 CPU(s)
Memory usage
3.99% (1.24 GiB of 31.23 GiB)
SWAP usage
0.00% (0 B of 512.00 MiB)
(edited)E
expatriotic is me01:42
Could I sync to my NVME?
root@pve2:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 2T 0 loop
sda 8:0 0 3.6T 0 disk /mnt/data
nvme0n1 259:0 0 931.5G 0 disk
├─nvme0n1p1 259:1 0 1007K 0 part
├─nvme0n1p2 259:2 0 1G 0 part /boot/efi
└─nvme0n1p3 259:3 0 930G 0 part
├─pve-swap 252:0 0 8G 0 lvm [SWAP]
└─pve-root 252:1 0 922G 0 lvm /
root@pve2:~#
E
expatriotic is me01:42
I have been putting my templates/container etc in NVME and the data itself to the HDD. Is that creating a bottleneck?
evoskuil01:46
I'm not familiar with the environment, but clearly I/O delay is a problem. I don't really know what that's about, but if you aren't running off of an internal SSD your performance will be horrible.
O
otto01:52
FWIW, it was quite simple to compile inside a debian docker container. I just removed --quiet from wget cause it took was taking a while to download boost (I thought I was having VPN connection issues inside docker).
Also had to build secp256k1. I plan to leave this syncing overnight
O
otto02:01
Weird, it failed to find other nodes and stopped:
2025-10-13T02:00:16Z.0 Please wait while network is starting... 2025-10-13T02:00:16Z.1 Candidate top [000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f:0]. 2025-10-13T02:00:16Z.1 Fork point (0) unassociated (0). 2025-10-13T02:00:16Z.1 Seeding because of insufficient (0 of 500) address quantity. 2025-10-13T02:00:16Z.1 Connecting to seed [seed.bitcoin.sipa.be:8333] 2025-10-13T02:00:16Z.1 Connecting to seed [dnsseed.bluematt.me:8333] 2025-10-13T02:00:16Z.1 Connecting to seed [dnsseed.bitcoin.dashjr-list-of-p2p-nodes.us:8333] 2025-10-13T02:00:16Z.1 Connecting to seed [seed.bitcoin.jonasschnelli.ch:8333] 2025-10-13T02:00:16Z.1 Connecting to seed [seed.btc.petertodd.net:8333] 2025-10-13T02:00:16Z.1 Connecting to seed [seed.bitcoin.sprovoost.nl:8333] 2025-10-13T02:00:16Z.1 Connecting to seed [dnsseed.emzy.de:8333] 2025-10-13T02:00:16Z.1 Connecting to seed [seed.bitcoin.wiz.biz:8333] 2025-10-13T02:00:16Z.1 Connecting to seed [seed.mainnet.achownodes.xyz:8333] 2025-10-13T02:00:32Z.1 Failed to connect seed address [dnsseed.bluematt.me:8333] operation timed out 2025-10-13T02:00:32Z.1 Failed to connect seed address [seed.bitcoin.wiz.biz:8333] operation timed out 2025-10-13T02:00:32Z.1 Failed to connect seed address [seed.mainnet.achownodes.xyz:8333] operation timed out 2025-10-13T02:00:32Z.1 Failed to connect seed address [dnsseed.emzy.de:8333] operation timed out 2025-10-13T02:00:32Z.1 Failed to connect seed address [seed.btc.petertodd.net:8333] operation timed out 2025-10-13T02:00:32Z.1 Failed to connect seed address [seed.bitcoin.sipa.be:8333] operation timed out 2025-10-13T02:00:32Z.1 Failed to connect seed address [dnsseed.bitcoin.dashjr-list-of-p2p-nodes.us:8333] operation timed out 2025-10-13T02:00:32Z.1 Failed to connect seed address [seed.bitcoin.jonasschnelli.ch:8333] operation timed out 2025-10-13T02:00:32Z.1 Failed to connect seed address [seed.bitcoin.sprovoost.nl:8333] operation timed out 2025-10-13T02:00:32Z.0 Node failed to start with error 'seeding unsuccessful'. 2025-10-13T02:00:32Z.1 Seed session complete. 2025-10-13T02:00:32Z.0 Please wait while network is stopping...
evoskuil02:03
It's not going to keep looping over the seed nodes, so it stops if it fails to get addresses.
O
otto02:06
Is that port (8333) correct for lookup, though? Cause I get many peers without it, but not with it:
libbitcoin$ nslookup seed.bitcoin.sprovoost.nl 2>&1 | grep Address | cut -d' ' -f2 Address: 192.168.1.1#53 45.33.1.178 143.198.210.127 23.28.188.241 213.141.251.4 173.46.87.142 89.106.27.201 45.79.22.148 89.23.119.185 37.182.152.42 73.216.78.94 79.151.229.127 89.106.27.218 155.4.26.146 83.32.167.77 174.140.231.184 98.114.36.130 172.233.151.68 35.242.187.91 170.187.181.164 103.246.186.204 75.82.57.33 47.14.96.94 47.160.10.230 2a05:3580:dc0b:1600:28dc:ab23:609f:8c3 2a00:23a8:83b:2201:dc52:e0c7:9e99:a076 2600:1702:7aa0:10b0:266e:96ff:fe46:9060 2a02:220b:2000:f800:fb46:25d3:84c7:20c0 2a01:6380:fffe:73:10fb:d012:8581:b4d7 2601:84:c900:1b90:be24:11ff:fe8d:f86f 2a11:6100:0:5216:: 2003:e7:2f1d:2601:be24:11ff:fe4b:9928 2001:41d0:303:89af::1 2a02:aa10:480:6f00:5ccb:4df0:367f:c503 2804:14d:5c78:82db::1000 2600:1900:40a0:98ea:0:2:: 2605:59c2:14ae:2520:20c:29ff:fe23:a4fd libbitcoin$ nslookup seed.bitcoin.sprovoost.nl:8333 2>&1 | grep Address | cut -d' ' -f2 Address: 192.168.1.1#53
evoskuil02:07
2025-10-13T02:07:33Z.0 Please wait while network is starting... 2025-10-13T02:07:34Z.1 Candidate top [00000000000000000001b7350b4b22eb496c952012d94773bdbb304c1bdda002:918789]. 2025-10-13T02:07:34Z.1 Fork point (550718) unassociated (0). 2025-10-13T02:07:34Z.1 Seeding because of insufficient (0 of 500) address quantity. 2025-10-13T02:07:34Z.1 Connecting to seed [seed.bitcoin.sipa.be:8333] 2025-10-13T02:07:34Z.1 Connecting to seed [dnsseed.bluematt.me:8333] 2025-10-13T02:07:34Z.1 Connecting to seed [dnsseed.bitcoin.dashjr-list-of-p2p-nodes.us:8333] 2025-10-13T02:07:34Z.1 Connecting to seed [seed.bitcoin.jonasschnelli.ch:8333] 2025-10-13T02:07:34Z.1 Connecting to seed [seed.btc.petertodd.net:8333] 2025-10-13T02:07:34Z.1 Connecting to seed [seed.bitcoin.sprovoost.nl:8333] 2025-10-13T02:07:34Z.1 Connecting to seed [dnsseed.emzy.de:8333] 2025-10-13T02:07:34Z.1 Connecting to seed [seed.bitcoin.wiz.biz:8333] 2025-10-13T02:07:34Z.1 Connecting to seed [seed.mainnet.achownodes.xyz:8333] 2025-10-13T02:07:34Z.1 Inbound [[2604:2dc0:145:c900::]:8333] version (70016) /Satoshi:29.0.0/ 2025-10-13T02:07:34Z.1 Accepted (1000>748>747) addresses from seed [[2604:2dc0:145:c900::]:8333]. 2025-10-13T02:07:34Z.1 Seed stop [[2604:2dc0:145:c900::]:8333] seeding complete 2025-10-13T02:07:34Z.0 Node is started. 2025-10-13T02:07:34Z.1 Accepting 100 peers on 2 bindings.
O
otto02:13
yeah, found it weird that was the output when port 8333 was used.
Getting many peers now
👍🏻 1
O
otto02:29
Is there a way to track progress? All I see is a lot of messages about outbound peers being connected to
O
otto02:29
Also, is it expected to have the computer close to unusable while during this initial sync?
evoskuil02:31
Yes, the sync is intended to be fast. If you want it to be slow you can reduce the thread count and/or priority. It will use whatever resources the OS gives it. It's the job of the OS to keep the computer responsive, not applications running on the computer.
evoskuil02:32
I can sync the entire chain in < 60 minutes on my 9 year old windows PC and keep working like nothing's happening.
evoskuil02:34
Both over limit resources and kill processes when there are plenty of resources, and also fail to fair allocate them when they are sparse.
evoskuil02:35
You can request info at the console, and also vary the level of console/log output.
m <enter> for the options.evoskuil02:36
But there is also a high speed event log, which puts out readable events. That's easiest for just tracking block progress. Keep in mind that blocks are downloaded and validated out of order (concurrently).
(edited)O
otto02:37
right, it works. I didn't see anything echoed after I typed
i, didn't realise i had to type enter tooevoskuil02:40
The <i>nformation option is a little hard to pick apart, it's really for us to get insight into store logical behavior.
evoskuil02:42
header_organized.... 918318 27704 header_archived..... 918319 27982 header_organized.... 918319 27982 header_archived..... 918320 29198 header_organized.... 918320 29198 header_archived..... 918321 29250 header_organized.... 918321 29250
evoskuil02:42
block_organized..... 899992 4062 block_confirmed..... 899993 4062 block_organized..... 899993 4062 block_confirmed..... 899994 4062 block_organized..... 899994 4062 block_confirmed..... 899995 4062 block_organized..... 899995 4062 block_confirmed..... 899996 4062 block_organized..... 899996 4062 block_confirmed..... 899997 4062 block_organized..... 899997 4062 block_confirmed..... 899998 4062 block_organized..... 899998 4062 block_confirmed..... 899999 4062 block_organized..... 899999 4062 block_confirmed..... 900000 4062 block_organized..... 900000 4062
evoskuil02:44
If we spit out these entries to the console we'd spend more time writing the console than syncing the blocks.
O
otto02:45
block_organized..... 150070 1534 block_validated..... 150071 1536 block_confirmed..... 150071 1536 block_organized..... 150071 1536 block_validated..... 150072 1536 block_confirmed..... 150072 1536 block_organized..... 150072 1536
evoskuil02:47
block_archived...... 150070 85 block_validated..... 150070 85 block_confirmed..... 150070 85 block_organized..... 150070 85
evoskuil02:57
VPN and wifi both, prob rough.
Also on Linux you probably won't get close to full RAM access, probably limited to like 6-7GB. To get full access to the RAM you need to set something like this:
sudo sysctl vm.dirty_ratio=80 sudo sysctl vm.dirty_background_ratio=50 sudo sysctl vm.dirty_expire_centisecs=6000 sudo sysctl vm.dirty_writeback_centisecs=1000
evoskuil02:58
That will make a big difference. Apart from your network speed, it should perform well given your specs.
O
otto02:58
It seems the SSD is PCIe4, not sure about the motherboard. Will try to measure speed, the computer is a little slow
evoskuil03:04
We can't go any faster than the theoretical limit (wire serialization bytes / DL bandwidth). :)
evoskuil03:14
But anything less than 32GB is not a target platform for us, so when you're limited to < 6GB (or even < 24GB) it just thrashes.
(edited)E
expatriotic is me04:31
Same, mine went to less RAM after fiddling with it. Can't be the storage disk as that factor didn't change. Gonna remove these settings from sysctl.conf:
sudo sysctl vm.dirty_ratio=80
sudo sysctl vm.dirty_background_ratio=50
sudo sysctl vm.dirty_expire_centisecs=6000
sudo sysctl vm.dirty_writeback_centisecs=1000
evoskuil05:08
Contact @eynhaender to see what he thinks. This has worked consistently, but it's been his project.
E
expatriotic is me08:38
I.e., have I just made my HWW useless? And I'd need to make the seed hot to spend? Likewise with my hot wallet Ashigaru, Dojo is built with Core in mind. Just want to understand for myself and others, what they can do with this tech, as cool as it is, if people don't build on it... 👀
TBH, this will be my first node. I ran a rockpine POS with dojo, but fulcrum was buggy and would corrupt a lot. And I moved behind the great firewall of china where tor is unusable for nodes... So if this node gets stood up, it'll kind of be my first. Been borrowing other peoples nodes to power my wallets
evoskuil08:38
It can’t be paired with anything except code presently, as the client-server interface is a work in progress. Bx will also work with it when it’s updated. The client server interface will have Electrum, json-rpc and stratum compatibility modes, as well as a full REST API and block explorer.
E
expatriotic is me08:46
Okay, thank you for clarifying. That makes perfect sense.
So for a non-developer like me, once the node is fully synced, what is the immediate, practical use case? It feels like I've successfully built a powerful backend/dev-kit, but I'm trying to understand what I can do with it today from a user's perspective, before the new client-server interface is ready.
E
expatriotic is me09:32
checking for libsecp256k1 >= 0.1.0.20... no
configure: error: libsecp256k1 >= 0.1.0.20 is required but was not found.
real 2m7.815s
user 3m30.237s
sys 0m29.384s
satoshi@ubuntu-bitcoin:~/libbitcoin-node$ ./install.sh --prefix=$HOME/libbitcoin --build-boost --disable-shared --enable-avx2 --build-libsecp256k1
E
expatriotic is me09:33
But I nuked the container and now it hangs asking for it. So I added the flag... Still halting saying it is needed
E
expatriotic is me09:34
Rebuilt the container to use the 1TB SSD and all the RAM... but now stuck on the build again
O
otto14:17
maybe it you installed it from apt in the previous attempt. On Debian I had to build it as the default repo doesn't have that version
eynhaender14:58
These are lots of different setups now tbh. I am working on 3 different Ubuntu machines simultaneously, 2 with 32 GB and one with 96. 2 different Kernels, all different CPUs and SSDs. Then you have different binaries too (release, debug, other flags). What I want to say that it's hard to tell from here which aspect leads to limiting RAM on your @otto machine. I also don't know anything about what will happen to mem usage/disk access in containers or VMs. I'd absolutely love to know more about it though 🙂
You can absolutely drop me a line to look into your issues and I'll have a look at it when possible.
O
otto15:22
Just a feedback on the progress, it looks extremely slow here:
block_organized..... 851185 43022 block_organized..... 851189 43487 block_organized..... 851190 43570 block_organized..... 851552 44175 block_organized..... 852078 44226(edited)
evoskuil15:55
That seems backwards. It will start full validation above the configured milestone block, which is 900,000 by default. This will be significantly slower. I think what you are seeing is much higher CPU utilization, not faster completion.
O
otto15:58
Hmm right, it's "stuck" on 900.989 now.
CPU usage's been high all this time, unlike RAM (4-7GB most)
evoskuil15:59
This times above are very slow
One second...
block_organized..... 898291 3862 block_confirmed..... 898292 3862 block_organized..... 898292 3862 block_confirmed..... 898293 3862 block_organized..... 898293 3862 block_confirmed..... 898294 3862 block_organized..... 898294 3862 block_confirmed..... 898295 3862 block_organized..... 898295 3862 block_confirmed..... 898296 3862 block_organized..... 898296 3862 block_confirmed..... 898297 3862 block_organized..... 898297 3862 block_confirmed..... 898298 3862 block_organized..... 898298 3862 block_confirmed..... 898299 3862 block_organized..... 898299 3862 block_confirmed..... 898300 3862 block_organized..... 898300 3862 block_confirmed..... 898301 3862 block_organized..... 898301 3862 block_confirmed..... 898302 3862 block_organized..... 898302 3862 block_confirmed..... 898303 3862 block_organized..... 898303 3862 block_confirmed..... 898304 3862 block_organized..... 898304 3862 block_confirmed..... 898305 3862 block_organized..... 898305 3862 block_confirmed..... 898306 3862 block_organized..... 898306 3862 block_confirmed..... 898307 3862 block_organized..... 898307 3862 block_confirmed..... 898308 3862 block_organized..... 898308 3862 block_confirmed..... 898309 3862 block_organized..... 898309 3862 block_confirmed..... 898310 3862 block_organized..... 898310 3862 block_confirmed..... 898311 3862full file
evoskuil16:02
The reason CPU has been high is that the low RAM is causing thrashing, which is occupying all of the CPU time.
O
otto16:02
I assume CPU usage will slow down a lot after this initial sync?
Not sure if it has an "IBD mode" like Core
evoskuil16:04
It takes 16GB to hold the table heads, which should remain memory resident. So most of the time is spent just moving these on and off the disk.
evoskuil16:05
That's fine when it's only processing a few hundred blocks, but not when it's got 900k to work on.
evoskuil16:06
It's some Linux (and macOS) configuration issue. It was unresolved for a long time, but recently we are able to match Windows performance on Linux.
👍 1
evoskuil16:15
If you wanna see what it should look like on your specs, you are welcome to try out a windows build.
O
otto19:14
Sure.
I'll post it later on X and ping you. Also how I built it. There are more people willing to try, but they don't know how to do it
evoskuil19:16
People who are diligent and technical enough to seek us out and go through the process provide good input to the process. But the masses are another story at this stage.
(edited)✅ 1
eynhaender19:45
I will adapt the README.md over the next days. Then we will see if it can cover enough obstacles for the very early adopters to not get frustrated while playing around. Keep you posted.
🔥 1
O
otto20:14
Trying to build it on arch to compare, but the script is failing.
(edited)./install.sh --prefix=$HOME/libbitcoin --build-boost --build-secp256k1 --disable-shared --enable-avx2 --enable-shani --enable-sse41
On this call:
create_from_github bitcoin-core secp256k1 v0.7.0 yesO
otto20:14
Inside the function there is:
push_directory "$BUILD_SRC_DIR"
this variable isn't assigned anywhereevoskuil20:18
You already ran on arch, where you build it doesn’t matter. The install/build isn’t tested on Arch, just Ubuntu/Debian, so there are likely issues.
O
otto20:20
I found a couple unrelated issues with the script:
if [[ DISPLAY_VERBOSE ]]; then
VERBOSITY="VERBOSE=1"
fi
SEQUENTIAL=1
# Avoid setting -j1 (causes problems on single threaded systems [TRAVIS]).
if [[ $JOBS > $SEQUENTIAL ]]; then
make -j"$JOBS" "$@" $VERBOSITY
else
make "$@" $VERBOSITY
fiDISPLAY_VERBOSE is missing the $
> doesn't work for numbers, AFAIK. Should be -gtevoskuil20:25
Running it on Arch vs. Debian may make a difference. The issue is the OS management of mmap paging.
👍 1
O
otto21:05
@pmienk I'm curious, what is this
(edited)BUILD_SRC_DIR? It's not mentioned anywhere elsepmienk21:18
An artifact of writing codegen for 4 install scripts and attempting to maximize common code use. I'm testing rename of BUILD_DIR to BUILD_SRC_DIR to confirm working fix, as BUILD_DIR for install.sh/install-cmake.sh/install-cmakepresets.sh are meant to be for end users and so don't distinguish src/obj directories while developer_setup.sh does.
pmienk21:19
oddly enough, i don't see a failure in CI, so I'm curious if it's defined in a way I don't remember offhand.
👍 1
E
expatriotic is me17:33
Why is there no security? AKA checking sigs of packages?
If you were to build secp256k1 alone for example they have a security page on their github for pulling in .asc and checking for good sig
Is that what you're referring to? Boost? Or my second question
pmienk17:38
Boost is 1.86.0 providing minimum required version. Signature checking, providing binaries, etc is a release related issue.
✍️ 1
E
expatriotic is me19:36
replied to a thread: view
satoshi@ubuntu-bitcoin:~$ bn -d Hardware configuration... arm..... platform:0. intel... platform:1. avx512.. platform:0 compiled:0. avx2.... platform:1 compiled:0. sse41... platform:1 compiled:0. shani... platform:0 compiled:0. satoshi@ubuntu-bitcoin:~$
Sunday, October 12, 2025
P
parsevalbtc15:53
@evoskuil in events.log
block_confirmed means it has a valid PoW and block_organized means, in addition, that it belong to the strong branch, is it?evoskuil16:01
That's close.
block_confirmed means that the block is valid and will always be confirmable. It's a terminal validation state. block_organized means that it's been moved into the strong chain. It can be reorganized out and back again, but will never be validated/confirmed again.👍 1
P
parsevalbtc16:21
so between
block_validated and block_confirmed the difference lies in the valid PoW threshold to put it at least in a weak branch?evoskuil16:25
It won't be worked on unless it's in a stronger branch. Once there is a confirmable stronger branch, it's blocks get reorganized into the confirmed chain.
(edited)the event
block_confirmed corresponds to the "confirmable" state. The block will always be confirmable, and may be organized (confirmed). The event names are a simplification.The "candidate" chain is always as strong or stronger than the "confirmed" chain, and all download/validation/confirmation work takes place on headers/blocks in the candidate chain. A sparse in-memory header tree is maintained relative to the candidate chain and reorgs into the candidate chain when a strong branch is found. When the header chain is confirmed to a point stronger than the confirmed chain, it is reorganized into the confirmed chain. So there is header reorg (tree to candidate) and block reorg (candidate to confirmed).
P
parsevalbtc16:38
but validation means all consensus rules are checked including sufficient PoW threshold?
evoskuil16:40
Without validating work how could we know what the strength of the chains are as described above?
(edited)"all consensus rules" are ALWAYS validated. A milestone validation is based on the assertion by the node operator that a given block is valid. Given that we can safely avoid certain checks that are assured by that assertion. That doesn't include header validation, as otherwise we would be subject to trivial DoS attacks. And header validation is itself trivial cost.
(edited)P
parsevalbtc16:53
so in which case a
(edited)block_validated doesn't become block_confirmed ? Maybe when it is a branch which is really in the past (for example a valid block e.g. #800000 which is valid but which will be hardly be confirmed? (meaning it will be difficult that additional +100k blocks will become from this fork the strong branch)evoskuil16:55
No, none of that.
If it includes a double spend of an ancestor (i.e. in its branch) or if it spends an output that is not in its ancestry. That's the only way it becomes "unconfirmable" (another terminal state) once it's been validated. Though this can also be caused by one of its ancestors becoming unconfirmable (which can be caused by the above, or by invalidity). Unconfirmability propagates to all descendant blocks.
(edited)👍 1
evoskuil17:01
A candidate branch that is stronger than confirmed will be worked on, and will be reorged out to the header tree if it shrinks due to an invalid block and therefore becomes weaker than the confirmed chain. So it's possible that a confirmable block never becomes confirmed, as that requires that its candidate chain is stronger than the confirmed chain.
P
parsevalbtc17:06
Thanks now it's clear! I think that under core blocks containing double spent are threaded as invalid altogether
validation can be fully parallelized by block, confirmation is strictly sequential.
In Core, coupling these together, makes all validation strictly sequential.
(edited)Which is one of the reasons that the CPU runs at 10% vs. 100% when validating.
🙌 1
evoskuil17:28
Update on macOS: I've verified that our custom
fallocate on macOS is working as expected. The disk full fault we had on Linux was a result of relying on sparse allocation via ftruncate . That's resolved by using fallocate but macOS doesn't implement it. So disk full should be working properly on macOS now. This is the proof of proper behavior, with file system sparse and full sizes compared against the Windows sizes, with default initial sizes and files closed vs. open. Exact same to the byte, and physically allocated.
FILES CLOSED ++++++++++++ evoskuil@Big-Mac bitcoin % ls -l total 88 -rw-r--r-- 1 evoskuil staff 0 Oct 12 11:40 address.data -rw-r--r-- 1 evoskuil staff 98 Oct 12 11:40 archive_header.data -rw-r--r-- 1 evoskuil staff 79 Oct 12 11:40 archive_input.data -rw-r--r-- 1 evoskuil staff 13 Oct 12 11:40 archive_ins.data -rw-r--r-- 1 evoskuil staff 81 Oct 12 11:40 archive_output.data -rw-r--r-- 1 evoskuil staff 5 Oct 12 11:40 archive_outs.data -rw-r--r-- 1 evoskuil staff 39 Oct 12 11:40 archive_point.data -rw-r--r-- 1 evoskuil staff 65 Oct 12 11:40 archive_tx.data -rw-r--r-- 1 evoskuil staff 10 Oct 12 11:40 archive_txs.data -rw-r--r-- 1 evoskuil staff 3 Oct 12 11:40 candidate.data -rw-r--r-- 1 evoskuil staff 3 Oct 12 11:40 confirmed.data -rw-r--r-- 1 evoskuil staff 0 Oct 12 11:40 duplicate.data -rw-r--r-- 1 evoskuil staff 0 Oct 12 11:40 filter_bk.data -rw-r--r-- 1 evoskuil staff 0 Oct 12 11:40 filter_tx.data drwxr-xr-x 20 evoskuil staff 640 Oct 12 11:39 heads -rw-r--r-- 1 evoskuil staff 0 Oct 12 11:40 prevout.data -rw-r--r-- 1 evoskuil staff 11 Oct 12 11:40 strong_tx.data -rw-r--r-- 1 evoskuil staff 0 Oct 12 11:40 validated_bk.data -rw-r--r-- 1 evoskuil staff 0 Oct 12 11:40 validated_tx.data Directory of M:\bitcoin 10/12/2025 11:58 AM <DIR> . 10/12/2025 11:58 AM <DIR> .. 10/12/2025 11:58 AM 0 address.data 10/12/2025 11:58 AM 98 archive_header.data 10/12/2025 11:58 AM 79 archive_input.data 10/12/2025 11:58 AM 13 archive_ins.data 10/12/2025 11:58 AM 81 archive_output.data 10/12/2025 11:58 AM 5 archive_outs.data 10/12/2025 11:58 AM 39 archive_point.data 10/12/2025 11:58 AM 65 archive_tx.data 10/12/2025 11:58 AM 10 archive_txs.data 10/12/2025 11:58 AM 3 candidate.data 10/12/2025 11:58 AM 3 confirmed.data 10/12/2025 11:58 AM 0 duplicate.data 10/12/2025 11:58 AM 0 filter_bk.data 10/12/2025 11:58 AM 0 filter_tx.data 10/12/2025 11:58 AM <DIR> heads 10/12/2025 11:58 AM 0 prevout.data 10/12/2025 11:58 AM 11 strong_tx.data 10/12/2025 11:58 AM 0 validated_bk.data 10/12/2025 11:58 AM 0 validated_tx.data 18 File(s) 407 bytes 3 Dir(s) 2,119,607,652,352 bytes free ==================== evoskuil@Big-Mac heads % ls -l total 28693968 -rw-r--r-- 1 evoskuil staff 8 Oct 12 11:40 address.head -rw-r--r-- 1 evoskuil staff 1545460 Oct 12 11:40 archive_header.head -rw-r--r-- 1 evoskuil staff 5 Oct 12 11:40 archive_input.head -rw-r--r-- 1 evoskuil staff 4 Oct 12 11:40 archive_ins.head -rw-r--r-- 1 evoskuil staff 5 Oct 12 11:40 archive_output.head -rw-r--r-- 1 evoskuil staff 4 Oct 12 11:40 archive_outs.head -rw-r--r-- 1 evoskuil staff 10927817096 Oct 12 11:40 archive_point.head -rw-r--r-- 1 evoskuil staff 1876890104 Oct 12 11:40 archive_tx.head -rw-r--r-- 1 evoskuil staff 4500010 Oct 12 11:40 archive_txs.head -rw-r--r-- 1 evoskuil staff 3 Oct 12 11:40 candidate.head -rw-r--r-- 1 evoskuil staff 3 Oct 12 11:40 confirmed.head -rw-r--r-- 1 evoskuil staff 4100 Oct 12 11:40 duplicate.head -rw-r--r-- 1 evoskuil staff 3 Oct 12 11:40 filter_bk.head -rw-r--r-- 1 evoskuil staff 5 Oct 12 11:40 filter_tx.head -rw-r--r-- 1 evoskuil staff 5 Oct 12 11:40 prevout.head -rw-r--r-- 1 evoskuil staff 1876890104 Oct 12 11:40 strong_tx.head -rw-r--r-- 1 evoskuil staff 3600008 Oct 12 11:40 validated_bk.head -rw-r--r-- 1 evoskuil staff 10 Oct 12 11:40 validated_tx.head Directory of M:\bitcoin\heads 10/12/2025 11:58 AM <DIR> . 10/12/2025 11:58 AM <DIR> .. 10/12/2025 11:58 AM 8 address.head 10/12/2025 11:58 AM 1,545,460 archive_header.head 10/12/2025 11:58 AM 5 archive_input.head 10/12/2025 11:58 AM 4 archive_ins.head 10/12/2025 11:58 AM 5 archive_output.head 10/12/2025 11:58 AM 4 archive_outs.head 10/12/2025 11:58 AM 10,927,817,096 archive_point.head 10/12/2025 11:58 AM 1,876,890,104 archive_tx.head 10/12/2025 11:58 AM 4,500,010 archive_txs.head 10/12/2025 11:58 AM 3 candidate.head 10/12/2025 11:58 AM 3 confirmed.head 10/12/2025 11:58 AM 4,100 duplicate.head 10/12/2025 11:58 AM 3 filter_bk.head 10/12/2025 11:58 AM 5 filter_tx.head 10/12/2025 11:58 AM 5 prevout.head 10/12/2025 11:58 AM 1,876,890,104 strong_tx.head 10/12/2025 11:58 AM 3,600,008 validated_bk.head 10/12/2025 11:58 AM 10 validated_tx.head 18 File(s) 14,691,246,937 bytes 2 Dir(s) 2,119,607,652,352 bytes free FILES OPEN (network halted at startup) ++++++++++++++++++++++++++++++++++++++ evoskuil@Big-Mac bitcoin % ls -l total 345171760 -rw-r--r-- 1 evoskuil staff 1 Oct 12 12:38 address.data -rw-r--r-- 1 evoskuil staff 21000000 Oct 12 12:38 archive_header.data -rw-r--r-- 1 evoskuil staff 92500000000 Oct 12 12:38 archive_input.data -rw-r--r-- 1 evoskuil staff 8550000000 Oct 12 12:38 archive_ins.data -rw-r--r-- 1 evoskuil staff 25300000000 Oct 12 12:38 archive_output.data -rw-r--r-- 1 evoskuil staff 3700000000 Oct 12 12:38 archive_outs.data -rw-r--r-- 1 evoskuil staff 25700000000 Oct 12 12:38 archive_point.data -rw-r--r-- 1 evoskuil staff 17000000000 Oct 12 12:38 archive_tx.data -rw-r--r-- 1 evoskuil staff 1050000000 Oct 12 12:38 archive_txs.data -rw-r--r-- 1 evoskuil staff 2575500 Oct 12 12:38 candidate.data -rw-r--r-- 1 evoskuil staff 2575500 Oct 12 12:38 confirmed.data -rw-r--r-- 1 evoskuil staff 44 Oct 12 12:38 duplicate.data -rw-r--r-- 1 evoskuil staff 1 Oct 12 12:38 filter_bk.data -rw-r--r-- 1 evoskuil staff 1 Oct 12 12:38 filter_tx.data -rw-r--r-- 1 evoskuil staff 0 Oct 12 12:38 flush.lock drwxr-xr-x 20 evoskuil staff 640 Oct 12 11:39 heads -rw-r--r-- 1 evoskuil staff 1 Oct 12 12:38 prevout.data -rw-r--r-- 1 evoskuil staff 0 Oct 12 12:38 process.lock -rw-r--r-- 1 evoskuil staff 2900000000 Oct 12 12:38 strong_tx.data -rw-r--r-- 1 evoskuil staff 1700000 Oct 12 12:38 validated_bk.data -rw-r--r-- 1 evoskuil staff 1 Oct 12 12:38 validated_tx.data Directory of M:\bitcoin 10/12/2025 12:15 PM <DIR> . 10/12/2025 12:15 PM <DIR> .. 10/12/2025 12:15 PM 1 address.data 10/12/2025 12:15 PM 21,000,000 archive_header.data 10/12/2025 12:15 PM 92,500,000,000 archive_input.data 10/12/2025 12:15 PM 8,550,000,000 archive_ins.data 10/12/2025 12:15 PM 25,300,000,000 archive_output.data 10/12/2025 12:15 PM 3,700,000,000 archive_outs.data 10/12/2025 12:15 PM 25,700,000,000 archive_point.data 10/12/2025 12:15 PM 17,000,000,000 archive_tx.data 10/12/2025 12:15 PM 1,050,000,000 archive_txs.data 10/12/2025 12:15 PM 2,575,500 candidate.data 10/12/2025 12:15 PM 2,575,500 confirmed.data 10/12/2025 12:15 PM 44 duplicate.data 10/12/2025 12:15 PM 1 filter_bk.data 10/12/2025 12:15 PM 1 filter_tx.data 10/12/2025 12:15 PM 0 flush.lock 10/12/2025 11:58 AM <DIR> heads 10/12/2025 12:15 PM 1 prevout.data 10/12/2025 12:15 PM 0 process.lock 10/12/2025 12:15 PM 2,900,000,000 strong_tx.data 10/12/2025 12:15 PM 1,700,000 validated_bk.data 10/12/2025 12:15 PM 1 validated_tx.data 20 File(s) 176,727,851,049 bytes 3 Dir(s) 1,942,879,830,016 bytes free ==================== evoskuil@Big-Mac heads % ls -l total 28693968 -rw-r--r-- 1 evoskuil staff 8 Oct 12 12:38 address.head -rw-r--r-- 1 evoskuil staff 1545460 Oct 12 12:38 archive_header.head -rw-r--r-- 1 evoskuil staff 5 Oct 12 12:38 archive_input.head -rw-r--r-- 1 evoskuil staff 4 Oct 12 12:38 archive_ins.head -rw-r--r-- 1 evoskuil staff 5 Oct 12 12:38 archive_output.head -rw-r--r-- 1 evoskuil staff 4 Oct 12 12:38 archive_outs.head -rw-r--r-- 1 evoskuil staff 10927817096 Oct 12 12:38 archive_point.head -rw-r--r-- 1 evoskuil staff 1876890104 Oct 12 12:38 archive_tx.head -rw-r--r-- 1 evoskuil staff 4500010 Oct 12 12:38 archive_txs.head -rw-r--r-- 1 evoskuil staff 3 Oct 12 12:38 candidate.head -rw-r--r-- 1 evoskuil staff 3 Oct 12 12:38 confirmed.head -rw-r--r-- 1 evoskuil staff 4100 Oct 12 12:38 duplicate.head -rw-r--r-- 1 evoskuil staff 3 Oct 12 12:38 filter_bk.head -rw-r--r-- 1 evoskuil staff 5 Oct 12 12:38 filter_tx.head -rw-r--r-- 1 evoskuil staff 5 Oct 12 12:38 prevout.head -rw-r--r-- 1 evoskuil staff 1876890104 Oct 12 12:38 strong_tx.head -rw-r--r-- 1 evoskuil staff 3600008 Oct 12 12:38 validated_bk.head -rw-r--r-- 1 evoskuil staff 10 Oct 12 12:38 validated_tx.head Directory of M:\bitcoin\heads 10/12/2025 11:58 AM <DIR> . 10/12/2025 11:58 AM <DIR> .. 10/12/2025 11:58 AM 8 address.head 10/12/2025 11:58 AM 1,545,460 archive_header.head 10/12/2025 11:58 AM 5 archive_input.head 10/12/2025 11:58 AM 4 archive_ins.head 10/12/2025 11:58 AM 5 archive_output.head 10/12/2025 11:58 AM 4 archive_outs.head 10/12/2025 11:58 AM 10,927,817,096 archive_point.head 10/12/2025 11:58 AM 1,876,890,104 archive_tx.head 10/12/2025 11:58 AM 4,500,010 archive_txs.head 10/12/2025 11:58 AM 3 candidate.head 10/12/2025 11:58 AM 3 confirmed.head 10/12/2025 11:58 AM 4,100 duplicate.head 10/12/2025 11:58 AM 3 filter_bk.head 10/12/2025 11:58 AM 5 filter_tx.head 10/12/2025 11:58 AM 5 prevout.head 10/12/2025 11:58 AM 1,876,890,104 strong_tx.head 10/12/2025 11:58 AM 3,600,008 validated_bk.head 10/12/2025 11:58 AM 10 validated_tx.head 18 File(s) 14,691,246,937 bytes 2 Dir(s) 1,942,879,830,016 bytes free DU/STAT ++++++++++++++++++++ ls -l doesn't ensure we are seeing actually allocated space, but these do evoskuil@Big-Mac bitcoin % du -h archive_input.data 86G archive_input.data evoskuil@Big-Mac bitcoin % du -h archive_output.data 24G archive_output.data evoskuil@Big-Mac bitcoin % du -h archive_point.data 24G archive_point.data evoskuil@Big-Mac bitcoin % stat -f "%z %b %B" archive_input.data 92500000000 180664072 1760283584 evoskuil@Big-Mac bitcoin % stat -f "%z %b %B" archive_output.data 25300000000 49414072 1760283584 evoskuil@Big-Mac bitcoin % stat -f "%z %b %B" archive_point.data 25700000000 50195328 1760283584 evoskuil@Big-Mac bitcoin %
WRT mmap paging, we are still seeing the same problem that we were formerly seeing on Linux, so macOS is not high performance presently (and Linux requires some simple config). The above change rules out sparse allocation as the issue. More likely just restrictive OS behavior that (unlike Linux) is not configurable. Windows dynamically manages this behavior, being optimal without any configuration required. Grok describes it as follows:
The problem isn't allocation—it's mmap itself on macOS for large files (>1GB per mapping, especially on APFS). Windows (SetEndOfFile / CreateFileMappingW) and Linux (mmap with native fallocate) handle large shared mappings reliably, but macOS has a long-standing kernel bug/limitation causing mmap failures or hangs for large files, even with preallocation. Here's the breakdown and fixes.(edited)
So on my "Big Mac", which has 768 GB of physical RAM, the process never gets above 26.5 GB usage (3.5%), because the OS is aggressively discarding pages. This causes thrashing despite extremely low resource utilization. This is the same behavior we saw on Linux before recent changes and a couple lines of config.
(edited)Worse yet, macOS may terminate the process due to low memory under the above conditions (of 3.5% memory utilization).
macOS's mmap kernel implementation fails with ENOMEM (out of memory) or EOVERFLOW for files > ~1-2 GB, even if RAM + swap is sufficient and the file is preallocated/non-sparse. This is a kernel VM limit, not a disk issue—it's been reported since macOS 10.5 (Leopard) and persists in Sonoma/Ventura (as of 2024 reports).
evoskuil17:44
Even worse yet, macOS doesn't simply kill the process, it goes into full on kernel panic as a result of this 3.5% memory usage. I've seen that several times, presently and in the past, and have never seen it simply kill the process.
In this case a kernel panic can indeed force a full system restart instead of just terminating the process. This isn't uncommon with the Darwin kernel's handling of massive shared mappings (> tens of GB), where the upfront VM address space claim (even if physical RAM/swap is available) can trigger a panic if it fragments or overflows internal limits, rather than gracefully failing with ENOMEM. Your preallocation via fallocate rules out disk-related causes, so it's almost certainly the mmap VM bug I mentioned.
This Big Mac is a BEAST and yet it gets sluggish with almost no resource utilization.
Based on your description (low CPU, 700GB free RAM, but system-wide sluggishness during large mmap operations on APFS), this isn't a RAM/CPU bottleneck but likely a kernel-level virtual memory (VM) issue tied to macOS's handling of massive shared mappings. Even with free physical RAM, mmap reserves virtual address space upfront, and on APFS, lazy page faults for large files (e.g., your 92G+ mappings) can cause thrashing or fragmentation, leading to system-wide I/O stalls or thermal throttling without spiking CPU. This matches reports of similar "ghost" slowdowns where Activity Monitor looks fine, but the kernel is busy with metadata/VM ops.
evoskuil17:49
Yeah, the funny thing is that the sole rationalization for the choices is to "ensure system stability". What a macPOS.
(edited)Also, it's not
ulimit, the OS shows no resource limits (unlimited) across the board.evoskuil17:58
evoskuil@Big-Mac bitcoin % ulimit -a -t: cpu time (seconds) unlimited -f: file size (blocks) unlimited -d: data seg size (kbytes) unlimited -s: stack size (kbytes) 8192 -c: core file size (blocks) 0 -v: address space (kbytes) unlimited -l: locked-in-memory size (kbytes) unlimited -u: processes 11136 -n: file descriptors 2560
evoskuil19:10
Reviewing some macOS logs, I'm finding that this ridiculous operating system is actually monitoring disk writes (actually mmap page dirtying) to prevent SSD wear, and even though the node is append-only for all but file heads, which makes it very SSD friendly, macOS is seeing a very high write RATE and triggering its SSD protection.
8.59 GB in 33 minutes at ~4.34 MB/s, exceeding the ~99 KB/s average limit
WTF kind of ridiculous limiting is this. We write at ~ network download rate, which here is 2Gb/s, or 250MB/s, and it wants us under 100KB/s?
evoskuil19:22
This is the beefiest mac pro I could get. This clown car is designed for server workloads.
pmienk19:28
It sounds pretty laughable. I suppose this stuff isn't giving graphics artists pause? Do they do everything over network because I thought video editing was disk and ram intensive
evoskuil20:00
I can't find any proof that it implements throttling on the SSD writes, but it certainly spits out warnings about limits exceeded. System monitor shows less than 5% memory, cpu, network, disk... almost the same as at idle. If I halt the network, the OS is responsive, when I reenable the network, there's a small bump of disk writes, the whole OS becomes very unresponsive, and still almost no resource usage. The files mapped are 200GB, and the overall RAM is 768GB. It's a 16 dual core Mac Pro with a 4TB SSD and 2.5Gbps NIC on a 2Gbps WAN. It should just be screaming into RAM and not even touching the disk until fully synced.
(edited)Done with this for now... back to you @eynhaender
🤝🏻 1
The operating system has one job, allocate resources.
evoskuil20:43
In other work, I've moved on to work on the websocket interface. I'm deferring moving client-server work into node until it's all ready for message processing. SO after websockets its json-rpc v1/v2 processing (over websockets, tcp, and http). I've got all of the design work on these done, just plugging it in now.
🙌 2
Tuesday, October 14, 2025
evoskuil19:35
The directory-based
web service is fully operational. This is the admin console for the server. The explore service is a RESTful service for chain queries, and the HTML accept type will be the block explorer that @Rowan Voskuil is currently working on. That will use explore for HTML initial page loads (based on accept header) and the websocket JSON service for background content loading. He's currently mocking the websocket interface on third party public data services. I'll soon enable the HTML common page response on explore , for all requests. By restricting the HTML block explorer to fixed content responses the security guarantees become much easier. The JSON explorer requests, and websocket (JSON) requests, are limited to store query.
So with these three endpoints we get an admin console, which can be isolated, a RESTful block explorer on HTML and JSON, and dynamic JSON content, for both the explorer and admin console via WebSockets. We'll be able to drop in TLS over all endpoints as well.evoskuil19:46
The
bitcoind, electrum and stratum_v1 services are json-rpc, versions 1 and 2 and over tcp, http, and websockets. These are materially different from a REST interface, and share a common parse mechanism. I'm getting close to completion of the json-rpc parser that can drop into the above transport services allowing us to seamlessly support these as well. websockets and http processing are facilitated by boost::beast and with this parser, implementation of the API's is straightforward. stratum_v1 is a proprietary interface, so it just ends up with a custom parser, similar to the json-rpc implementation. Finally tls can be transparently layered over all of these using the same implementation.
So for now I've activated the endpoints, and a couple are useful. I'll be adding to these rapidly. Presently this is all implemented in libbitcoin-network apart from settings parsing, which is in libbitcoin-node, but as soon as I start implementing store queries it all all move up to node. Eventually we'll probably move the endpoint specific behavior into a new libbitcoin-server, leaving the primitives (e.g. the parsers and base types) in network.👍 3🚀 1
Rowan Voskuil20:32
Compared to the mock version of the explorer that I showed last week I have rewritten it so that pages can load even before they have received the data. For example if a user attempts to navigate to a block the page will immediately load and display "loading block". And since last week I have like Eric said implemented blockstream as a external data source so that I have been able to continue to test and develop the explorer before I can get it hooked up to libbitcoin node.
🔥 2👏 1
Rowan Voskuil20:43
I also yesterday implemented pagination for displaying transactions blocks etc. So without a reload the user can now simply click load more transactions for example and it goes and makes a async request and then adds them if the request was successful.
👍 1🔥 2
Wednesday, October 15, 2025
evoskuil04:48
[explore] bind = 127.0.0.1:80 connections = 1000 path = m:/server #default = index.html(edited)
It's in. There's some regression in the config parsing, so ports are not getting picked up with DNS names, but it you stick with numeric host addresses and explicit ports it should be fine.
Should work exactly as I described. A path with no extension gets loaded from the default page, as does the default path. A path with an extension gets loaded as specified. So you can load page content from various URLs, but non-content URLs all go to the home page.
I should have store queries connected tomorrow. Maybe even a json object REST response.
🎉 3
Thursday, October 16, 2025
evoskuil06:34
In order for web services to activate in the latest build, the node must either be current (top confirmed block 60 minutes from present time by default, change with
currency_window_minutes), or at/above configured maximum_height (not configured by default), or delay_inbound set to false.
[node] delay_inbound = false maximum_height = 900,000 currency_window_minutes = 60
This affects both inbound p2p channels and server services. We'll want to allow the admin service to function regardless, so I'll change that one soon.
I've ported all of the services to node from network, with the base functionality remaining in network. This means that the client-server protocols can now interact with the store.
(edited)evoskuil06:42
That was a long data of refactoring and organizing. The services are simple and now well organized, and cleanly integrated into the startup sequence. I've also incorporated white/black listing (using the same lists as for p2p [network] config), automatic service disable/restart when the p2p network is paused (manually, or during snapshot or disk full), and the above [node] config. This is mostly just a consequence of the existing system design, with a little config. Also, if the node falls behind (outside the currency window), inbound channels and services automatically pause IAW the above config.
(edited)Hoping to have the json-rpc parser integrated tomorrow. The parser is complete and mostly tested - and maybe websocket service handling as well, and at least one store query response.
(edited)Huddle with pmienk, Marnix, eynhaender, neillm, Rowan Voskuil, evoskuil, parsevalbtc · 186 min15:55
Friday, October 17, 2025
evoskuil07:36
Just completed movement of service settings from network to node, and under their own config branch called server. Also moved the remaining non-network aspects up to node, with a bunch of refactoring, and stubbed in websockets channel classes, with handshake and running protocols. In good shape to go back to the json-rpc parser and hopefully have that integrated for request response via a custom message distributor in the bitcoind http service.
🎉 2
Got my network back up to snuff, and completed a < 1hr sync on the maxi while running compiles. So good to see that all of the additional class definitions and changes haven't had an adverse impact on p2p perf.
🔥 2
evoskuil19:04
It looks like we'll want to multiplex websockets onto the same
ip:port bindings as the RESTful block explore service, to avoid routing of non-standard ports. This means that there ultimately won't be config for the current [socket] interface, it will just be incorporated into the [explore] interface. This does imply however that the admin [web] interface will still end up on independent bindings, which I think is important for default security config. Integrating the two sites ([web] and [explore]) will be feasible, but either on distinct IP addresses, or on distinct ports. Distinct IP addresses is ideal if you want public routing (using default ports), but locally (or over a VPN) the use of non-standards ports wouldn't matter.I'm going to keep
(edited)[socket] bindings to independent ip:port and with their own config for now, just because it's a little simpler, but won't be hard to multiplex, that's mostly internal config as long as it's planned for.Sunday, October 19, 2025
Monday, October 20, 2025
Wednesday, October 22, 2025
D
David Chocholatý13:10
Hi all, I wanted to thank you for letting me join the community. I won’t be able to contribute anymore, so I’ll be logging out. Wishing you all the best with libbitcoin! 🔥
👍 1👋 1👊 1❤️ 1
Thursday, October 23, 2025
Huddle with pmienk, Marnix, evoskuil, Rowan Voskuil · 110 min15:54
Friday, October 24, 2025
evoskuil05:37
Completed the
json::rpc::request parser and json::rpc::response serializer, as well as a json::rpc::request serializer to facilitate test. Good test coverage, but I'll add a bunch more tomorrow. This also includes the type system and a reader/writer json::body<parser> for boost::asio::async_read and ::async_write, which utilizes the request parser and response serializer.
The serializers are straightforward writes to a stream buffer. Currently this uses std::ostringstream, but the performance of this could be bumped up using system::ostream, as we do with p2p network and store.
The parser is streaming, so can accommodate boost::asio piecemeal asynchronous buffer updates for a given request. Additionally the parser is non-allocating apart from the final copy/move of buffer values to the request object fields. Instead of pushing characters into a buffer, it moves views over the network buffer.
I've dusted off the boost::json declarative serialize/deserialize that was formerly applied to all of the chain objects (and actually reimplemented both of the above serializers with it but decided in favor of the custom implementation). These conversions are almost entirely declarative and very performant.
This will be used to convert the request.params object or array types from string blobs passed via in the json-rpc request/response DOM between the channel/proxy and the protocol implementations. As I outlined in the meeting, this deferred serialization simplifies the rpc layer, keeping native objects from being unnecessarily parsed in/out of a document object model.
Just as with http::request and http::response objects, we'll pass these json::request and json::response objects via a message distributor template instance for the various json-rpc based channels. Protocols just subscribe and handle the request, sending the response - just as with the p2p protocols.🙌 2
N
noah.309:14
It's an illuminating experience to read your architecture notes, decisions, and milestones as you make progress. I appreciate the learnings. Every post leads to a personal TIL moment. Keep up the good work, Team.
💯 1👍 2👍🏻 1
1 reply · josie · last Oct 24, 2025
josie15:49
wanted to plus one this. i’ve been busy with a few other projects these days so haven’t had time to actively engage yet, but i’ve been really enjoying following along and am learning a lot from lurking in this chat. keep up the good work and thanks for taking the time to explain your reasoning here, @evoskuil
(edited)❤️ 1👊 1
josie15:49
replied to a thread: view
wanted to plus one this. i’ve been busy with a few other projects these days so haven’t had time to actively engage yet, but i’ve been really enjoying following along and am learning a lot from lurking in this chat. keep up the good work and thanks for taking the time to explain your reasoning here, @evoskuil
(edited)❤️ 1👊 1
Tuesday, October 28, 2025
evoskuil16:32
Did a couple of full runs on the maxi yesterday, after some generalizations in the network stack. No changes expected, none observed. 58 mins to 900k. But I also managed to get a good baseline for my new test WAN, which is now dual-homed with two 2.5Gbps NICs, each on their own dedicated 2.5 Gbps router and modem, one to a fiber line and the other to cable. Each measures 2.3 Gbps independently on speedtest.net, though upload on the fiber is 10x that of the cable (400 Mbps vs. 40). The upload speed doesn't have have a material impact on perf at either level.
evoskuil16:37
The noticeable difference now that each is independently the same, is that the load is well balanced. However the performance is not cumulative at all. It was almost identical to a single NIC, with bandwidth utilization at about 1.2 on each NIC. Hardware utilization is otherwise the same as on a single, plenty of usable RAM (which it will use if it needs it), barely touching the SSD (periodic flush to keep RAM low), 30-40% CPU, plenty of threads available and allocated. Nothing interesting in the heatmap. This indicates that it's still network limited, which is consistent with the current multi-homed setup.
Current: ┌────────────────────────────────────┐ │ Single io_context (all channels) │ ← One slow peer blocks *all* └────────────────────────────────────┘ │ ├──> NIC A: 2.3 Gbps → drops to 1.8 Gbps └──> NIC B: 0.1 Gbps Total: ~2.9 Gbps(edited)
Fixed: ┌─────────────────┐ ┌─────────────────┐ │ nic_a.ioc │ │ nic_b.ioc │ ← Fully independent │ (channels on A) │ │ (channels on B) │ └─────────────────┘ └─────────────────┘ │ │ └────> 2.3 Gbps └────> 2.3 Gbps Total: 4.6 Gbps
A couple of images that Grok drew me that explain the issue. Each channel operates on its own strand. Inbound channels (not material here) are bound to a NIC. Outbound channels use boost::asio's default NIC selection. Each channel is on it's own proactor strand, but all share the same network thread pool. So where the software can otherwise easily concurrency across the two independent channels, it's effectively serialized. I also verified that the hardware I/O bus can handle it. Not a problem at all, can handle about 20x more than it's getting.
evoskuil16:49
The solution is to configure outbound NICs for explicit binding (and/or default the explicit binding), and put each NIC on its own thread pool. This can be generalized to inbound and outbound. Each inbound address binding has an independent
acceptor object that iterates over individual new connection attempts (basically the same for manual outbound connections - using the connector). Inbound connections are tied to an address, which means a NIC, so the number of acceptors is limited. Outbound connections also use a connector to establish the connection, and then the connector goes out of scope. The connector and acceptor objects control socket construction and thread pool ( asio iocontext) association, with each socket establishing and owning the asio strand instance for all operations on the connection.So there is a master network strand, which drives the connection process, and protects shared resources, like the address pool and certain config, and there is a strand for each channel. These all operate on the single network thread pool. There is only one other thread pool, which is to isolate CPU intensive prevout query and script validation. The database is synchronous and lock-free, so all database calls are just simple function calls from strands on either of these pools. Channels directly read/write the store, with the major block commitments coming concurrently from the channels that downloads the blocks.
The important observation is that despite having each channel on the same thread pool, logically they operate entirely independently, with the strand ensuring thread safety only internally to the channel. There are also
chaser objects that manage cross-channel state computations, such as performance deviations and validation. These also operate on their own strands and on the network thread pool, but are generally very low cost (apart from the aforementioned script validation, which uses an independent pool). So given that these are already logically independent (concurrent) logical strands of execution within a single thread pool, it is trivial to factor them into independent thread pools - based on NIC selection.The slightly more challenging issue is just to determine which NIC is more heavily loaded. So it requires cross-NIC monitoring, similarly to what we presently do for cross-channel download performance monitoring. Given a balanced distribution of work across thread pool-associated NICs, we should be able to run them in parallel and get most of the expected cumulative benefit. This is basically running a load balanced internally. This can be applied externally as well, but the OS doesn't do this.
evoskuil17:10
A simple naïve implementation would just accept a configured list of outbound connection bindings to local NICs, as with inbound, and then just round-robin connections to each NIC, with a thread pool associated to each - for all connection types. That's probably one day's effort. Small change to config and the connector, dispatch fully contained within the outbound session.
(edited)Note that since p2p and client-server connections all sit on this same network stack, the benefit would accrue to all connections, not just p2p. So a fully internally load balancing multihomed server.
evoskuil17:20
So interesting learnings from spare time during the day setting up the new hardware. But I won't be working on that in the near term. It's back to the client-server endpoints. I've got updates on the above thread on the general purpose
json-rpc parser/serializer and json parse/serialize in general, as well as on generalization of http request/response dispatch/send into boost::beast design patterns.👍 1
Jack Minnick19:53
Hey, what READMEs do I need to pour through before I start asking my stupid questions here?
Context: I am trying to Build Libbitcoin-server on a Linux machine.
For a full node implementation that can get highest chain stack, query, sign, and broadcast, do I need to build any other programs/ services? If so, what READMEs will help me there. Thank you.
4 replies · Marnix, eynhaender, Jack Minnick · last Oct 29, 2025
M
Marnix01:48
READMEs haven't been updated yet. You can find some v4 info here https://marnixcroes.github.io/libbitcoin-doc/
👀 1
eynhaender20:45
I have a current README for building the node on Linux in my repo: https://github.com/eynhaender/libbitcoin-node/blob/master/README.md
evoskuil19:54
you'll want to start at libbitcoin-node, since server has not been modified for v4, and node has everything presently. eventually some of node may be factored into server again. @Marnix?
evoskuil19:59
I wouldn't spend time on 3.x. 4.0 is not released, but node works very well presently.
👍 1
Regarding json parse/serialize, as I mentioned a few days ago, the custom streaming
json-rpc parser was completed, and very efficient. No allocations until moving completed pieces, no intermediate copying. Streaming SAX parser, one char at a time. This was about 6 days work, and another day for the custom serializer. The serializer is not streaming however. I also implemented both using boost::json parse/serialization, which is a DOM model, not SAX. This is a highly efficient library, and easy to use, however the DOM model results in a full copy of the object in/out of an intermediate state. Additionally I didn't believe that there was an associated streaming parser. Turns out that there is, both parse and serialize can be streamed on an individual char boundary (so a benefit on the serialization side). After discovering this I took another look at the tradeoffs between the existing custom implementation and a boost::json based impl.evoskuil20:50
The boost approach is based on mapping c++ classes in/out of the json DOM (
boost::json::value) abstraction. This generally comes down to trivial almost declarative mappings of objects, and looks very much like the custom serializer. But with the custom types annotated you get bidirectional streaming (and non-streaming), for any such types. And common types are automatic (like a vector of array of strings for example).
The objective has always been to annotate any transported types, which we implemented a couple years ago for all of the system::chain objects (like block, tx, etc.) just to verify it. These currently run in our system unit tests using non-streaming parse/serialize. To implement this for json-rpc just required creating a c++ object model for the required classes, which are network::json::request and ::response. These represent the structure of RPC calls and responses, with parameters (::request::params) as an array [...] or object {...} representation, and similar for ::response::result. The contained parameters can be any c++ objects/types, represented in JSON. The intent was to contain parameter values as strings and defer parse/serialize. So the custom parse/serialize would be used for the request/response and boost::json for parameter values.evoskuil20:59
However, given that
boost::json does in fact provide full streaming capability to/from its DOM, it becomes very attractive to allow the network layer to simply parse/serialize to/from the json DOM for both the json-rpc request/response classes, and the contained request::params/response::result objects. These objects we defined as (tiny bit simplified):
namespace libbitcoin {
namespace network {
namespace json {
/// Enumeration of JSON-RPC protocol versions.
enum class version
{
/// Undefined version (not set).
undefined,
/// JSON-RPC 1.0 (bitcoin core).
v1,
/// JSON-RPC 2.0 (electrum daemon, electrum protocol, stratum v1).
v2,
/// Determine the version from the jsonrpc element.
any,
/// The jsonrpc value is invalid.
invalid
};
/// Forward declaration.
struct value_t;
struct null_t {};
using code_t = int64_t;
using boolean_t = bool;
using number_t = double;
using string_t = std::string;
using array_t = std::vector<value_t>;
using object_t = std::unordered_map<string_t, value_t>;
using identity_t = std::variant
<
null_t,
code_t,
string_t
>;
using id_option = std::optional<identity_t>;
struct value_t
{
using type = std::variant
<
null_t,
boolean_t,
number_t,
string_t,
array_t,
object_t
>;
type inner{};
};
using value_option = std::optional<value_t>;
using params_t = std::variant
<
array_t,
object_t
>;
using params_option = std::optional<params_t>;
struct BCT_API result_t
{
code_t code{};
string_t message{};
value_option data{};
};
using error_option = std::optional<result_t>;
struct BCT_API response_t
{
version jsonrpc{ version::undefined };
id_option id{};
error_option error{};
value_option result{};
};
struct BCT_API request_t
{
version jsonrpc{ version::undefined };
id_option id{};
string_t method{};
params_option params{};
};
} // namespace json
} // namespace network
} // namespace libbitcoin (edited)evoskuil21:05
So by annotating these objects for
boost::json we get the streaming and non-streaming parse/serialize for free. This is what that looks like, using a couple of our libbitcoin-system macros to reduce the syntactic overhead:
namespace libbitcoin {
namespace network {
namespace json {
DECLARE_JSON_TAG_INVOKE(version);
DECLARE_JSON_TAG_INVOKE(value_t);
DECLARE_JSON_TAG_INVOKE(identity_t);
DECLARE_JSON_TAG_INVOKE(request_t);
DECLARE_JSON_TAG_INVOKE(response_t);
} // namespace json
} // namespace network
} // namespace libbitcoinnamespace libbitcoin {
namespace network {
namespace json {
// version
// ----------------------------------------------------------------------------
DEFINE_JSON_FROM_TAG(version)
{
switch (instance)
{
case version::v1: value = "1.0"; break;
case version::v2: value = "2.0"; break;
default: value = {}; break;
}
}
DEFINE_JSON_TO_TAG(version)
{
if (value.is_string())
{
const auto& text = value.as_string();
if (text == "1.0") return version::v1;
if (text == "2.0") return version::v2;
}
return version::invalid;
}
// value_t
// ----------------------------------------------------------------------------
DEFINE_JSON_FROM_TAG(value_t)
{
std::visit(overload
{
[&] (null_t) NOEXCEPT
{
value = nullptr;
},
[&](boolean_t visit) NOEXCEPT
{
value = visit;
},
[&](number_t visit) NOEXCEPT
{
value = visit;
},
[&](const string_t& visit) NOEXCEPT
{
value = visit;
},
[&](const array_t& visit) NOEXCEPT
{
value = value_from(visit);
},
[&](const object_t& visit) NOEXCEPT
{
value = value_from(visit);
}
}, instance.inner);
}
DEFINE_JSON_TO_TAG(value_t)
{
if (value.is_null())
{
return { std::in_place_type<null_t> };
}
else if (value.is_bool())
{
return { std::in_place_type<boolean_t>, value.as_bool() };
}
else if (value.is_number())
{
return { std::in_place_type<number_t>, value.to_number<number_t>() };
}
else if (value.is_string())
{
return { std::in_place_type<string_t>, value.as_string() };
}
else if (value.is_array())
{
return { std::in_place_type<array_t>, value_to<array_t>(value) };
}
else if (value.is_object())
{
return { std::in_place_type<object_t>, value_to<object_t>(value) };
}
throw ostream_exception{ "value_t" };
}
// identity_t
// ----------------------------------------------------------------------------
DEFINE_JSON_FROM_TAG(identity_t)
{
std::visit(overload
{
[&] (null_t) NOEXCEPT
{
value = boost::json::value{};
},
[&](code_t visit) NOEXCEPT
{
value = visit;
},
[&](const string_t& visit) NOEXCEPT
{
value = visit;
}
}, instance);
}
DEFINE_JSON_TO_TAG(identity_t)
{
if (value.is_null())
{
return { null_t{} };
}
else if (value.is_number())
{
return { code_t{ value.to_number<code_t>() } };
}
else if (value.is_string())
{
return { string_t{ value.as_string() } };
}
throw ostream_exception{ "identity_t" };
}
// request_t
// ----------------------------------------------------------------------------
DEFINE_JSON_FROM_TAG(request_t)
{
json_object object{};
if (instance.jsonrpc != version::undefined)
{
object["jsonrpc"] = value_from(instance.jsonrpc);
}
if (instance.id.has_value())
{
object["id"] = value_from(instance.id.value());
}
if (!instance.method.empty())
{
object["method"] = value_from(instance.method);
}
if (instance.params.has_value())
{
if (const auto& params = instance.params.value();
std::holds_alternative<array_t>(params))
{
object["params"] = value_from(std::get<array_t>(params));
}
else
{
object["params"] = value_from(std::get<object_t>(params));
}
}
value = object;
}
// Bogus warnings on `it` not checked.
BC_PUSH_WARNING(NO_UNGUARDED_POINTERS)
DEFINE_JSON_TO_TAG(request_t)
{
request_t request{};
const auto& object = value.as_object();
// jsonrpc
if (const auto it = object.find("jsonrpc"); it != object.end())
{
request.jsonrpc = value_to<version>(it->value());
}
// method
if (const auto it = object.find("method"); it != object.end())
{
request.method = it->value().as_string();
}
// id
if (const auto it = object.find("id"); it != object.end())
{
request.id = value_to<identity_t>(it->value());
}
// params
if (const auto it = object.find("params"); it != object.end())
{
if (const auto& params = it->value(); params.is_array())
{
request.params = params_t
{
std::in_place_type<array_t>, value_to<array_t>(params)
};
}
else if (params.is_object())
{
request.params = params_t
{
std::in_place_type<object_t>, value_to<object_t>(params)
};
}
}
return request;
}
// response_t
// ----------------------------------------------------------------------------
DEFINE_JSON_FROM_TAG(response_t)
{
json_object object{};
if (instance.jsonrpc != version::undefined)
{
object["jsonrpc"] = value_from(instance.jsonrpc);
}
if (instance.id.has_value())
{
object["id"] = value_from(instance.id.value());
}
if (instance.error.has_value())
{
const auto& result = instance.error.value();
object["error"] =
{
{ "code", result.code },
{ "message", result.message }
};
if (result.data.has_value())
{
object["data"] = value_from(result.data.value());
}
}
if (instance.result.has_value())
{
object["result"] = value_from(instance.result.value());
}
value = object;
}
DEFINE_JSON_TO_TAG(response_t)
{
response_t response{};
const auto& object = value.as_object();
// jsonrpc
if (const auto it = object.find("jsonrpc"); it != object.end())
{
response.jsonrpc = value_to<version>(it->value());
}
// id
if (const auto it = object.find("id"); it != object.end())
{
if (const auto& id = it->value(); id.is_null())
{
response.id = { null_t{} };
}
else if (id.is_number())
{
response.id = { id.to_number<code_t>() };
}
else if (id.is_string())
{
response.id = { string_t{ id.as_string() } };
}
}
// error
if (const auto it = object.find("error"); it != object.end())
{
result_t result{};
const auto& error = it->value().as_object();
if (const auto code_it = error.find("code");
code_it != error.end())
{
result.code = code_it->value().to_number<code_t>();
}
if (const auto message_it = error.find("message");
message_it != error.end())
{
result.message = message_it->value().as_string();
}
if (const auto data_it = error.find("data");
data_it != error.end())
{
result.data = value_to<value_option>(data_it->value());
}
response.error = result;
}
// result
if (const auto it = object.find("result"); it != object.end())
{
response.result = value_to<value_option>(it->value());
}
return response;
}
BC_POP_WARNING()
} // namespace json
} // namespace network
} // namespace libbitcoin (edited)Similarly, annotations that allow support for
system::chain::transaction (for example) look like this:
DEFINE_JSON_TO_TAG(transaction)
{
return
{
value.at("version").to_number<uint32_t>(),
value_to<inputs>(value.at("inputs")),
value_to<outputs>(value.at("outputs")),
value.at("locktime").to_number<uint32_t>()
};
}
DEFINE_JSON_FROM_TAG(transaction)
{
value =
{
{ "version", instance.version() },
{ "inputs", value_from(*instance.inputs_ptr()) },
{ "outputs", value_from(*instance.outputs_ptr()) },
{ "locktime", instance.locktime() }
};
}
DEFINE_JSON_TO_TAG(transaction::cptr)
{
return to_shared(tag_invoke(to_tag<transaction>{}, value));
}
DEFINE_JSON_FROM_TAG(transaction::cptr)
{
tag_invoke(from_tag{}, value, *instance);
}That's it, for any class. The
json-rpc annotations are more complex because of the use of variants. The chain objects contain sub-objects on vectors of shared_ptr, so they also have a little additional complexity (the last to declarations).So I decided that the additional performance cost of the DOM vs. SAX parser was worth the benefit in simplicity and flexibility, and streaming response serialization is a material benefit. I dumped the 7 days worth of work, and spent another day working in the
boost::json based json-rpc and all is good.Having moved on from that, it was time to tackle integration with the network layer. We have socket/proxy abstractions now for complete serialization and deserialization of
boost::beast::http request/response types. These are not polymorphic, but instead duck-typed using common outer templates. The http header parse/serialize and containing object is common across the various types, which include (TCP is for p2p):
/// TCP.
/// -----------------------------------------------------------------------
/// Read fixed-size TCP message from the remote endpoint into buffer.
virtual void read(const asio::mutable_buffer& buffer,
count_handler&& handler) NOEXCEPT;
/// Send a complete TCP message to the remote endpoint.
virtual void write(const asio::const_buffer& payload,
count_handler&& handler) NOEXCEPT;
/// HTTP Readers.
/// -----------------------------------------------------------------------
/// Read full http variant request from the socket, using provided buffer.
virtual void read(http::flat_buffer& buffer, http::request& request,
count_handler&& handler) NOEXCEPT;
/// Read full http string request from the socket, using provided buffer.
virtual void read(http::flat_buffer& buffer, http::string_request& request,
count_handler&& handler) NOEXCEPT;
/// Read full http json request from the socket, using provided buffer.
virtual void read(http::flat_buffer& buffer, http::json_request& request,
count_handler&& handler) NOEXCEPT;
/// HTTP Writers.
/// -----------------------------------------------------------------------
/// Write full http variant response to the socket (json buffer in body).
virtual void write(http::response& response,
count_handler&& handler) NOEXCEPT;
/// Write full http string response to the socket.
virtual void write(http::string_response& response,
count_handler&& handler) NOEXCEPT;
/// Write full http json response to the socket (serialize buffer in body).
virtual void write(http::json_response& response,
count_handler&& handler) NOEXCEPT;
/// Write full http data response to the socket.
virtual void write(http::data_response& response,
count_handler&& handler) NOEXCEPT;
/// Write full http file response to the socket.
virtual void write(http::file_response& response,
count_handler&& handler) NOEXCEPT;evoskuil21:24
The
http::json_request and http::json_response are new. These are defined using a custom network::http::body class that populates the boost::beast::http::request<> and ::response<> templates. The body is built over the boost::json parse/serialize annotations on the custom objects. One might wonder why boost::beast doesn't include this OOTB, but alas. The json body template impl looks like this (note that this is generic to any json, not limited to json-rpc:
namespace libbitcoin {
namespace network {
namespace json {
// reader
// ----------------------------------------------------------------------------
TEMPLATE
void CLASS::reader::init(const http::length_type& length,
error_code& ec) NOEXCEPT
{
const auto value = length.get_value_or(zero);
if (system::is_limited<size_t>(value))
{
ec = boost::system::errc::protocol_error;
return;
}
expected_ = system::possible_narrow_cast<size_t>(value);
parser_.reset();
total_ = zero;
ec.clear();
}
TEMPLATE
size_t CLASS::reader::put(const buffer_type& buffer, error_code& ec) NOEXCEPT
{
try
{
const auto data = system::pointer_cast<const char>(buffer.data());
const size_t parsed = parser_.write_some(data, buffer.size(), ec);
total_ = system::ceilinged_add(total_, parsed);
if (!ec && total_ > expected_.value_or(max_size_t))
ec = boost::system::errc::protocol_error;
return parsed;
}
catch (...)
{
ec = boost::system::errc::protocol_error;
return {};
}
}
TEMPLATE
void CLASS::reader::finish(error_code& ec) NOEXCEPT
{
parser_.finish(ec);
if (!ec)
payload_.model = parser_.release();
}
// writer
// ----------------------------------------------------------------------------
TEMPLATE
void CLASS::writer::init(error_code& ec) NOEXCEPT
{
if (!payload_.buffer)
{
// Caller controls max_size and other buffer behavior by assigning it.
payload_.buffer = std::make_shared<http::flat_buffer>(4096);
}
else
{
// Caller has assigned the buffer (or just reused the response).
payload_.buffer->consume(payload_.buffer->size());
}
ec.clear();
serializer_.reset(&payload_.model);
}
TEMPLATE
CLASS::writer::out_buffer CLASS::writer::get(error_code& ec) NOEXCEPT
{
ec.clear();
if (serializer_.done())
return {};
try
{
// Always prepares the configured max_size.
const auto size = payload_.buffer->max_size();
const auto buff = payload_.buffer->prepare(size);
const auto data = system::pointer_cast<char>(buff.data());
const auto view = serializer_.read(data, buff.size());
const auto done = serializer_.done();
payload_.buffer->commit(view.size());
return out_buffer{ std::make_pair(boost::asio::buffer(view), !done) };
}
catch (...)
{
ec = boost::system::errc::protocol_error;
return boost::none;
}
}
} // namespace json
} // namespace network
} // namespace libbitcoinAnd by just declaring this body as such, we get the capability to receive/respond using native c++ objects only at the protocol layer (above the proxy).
namespace libbitcoin {
namespace network {
namespace json {
using parser = boost::json::stream_parser;
using serializer = boost::json::serializer;
} // namespace json
namespace http {
/// http::json::request
using json_parser = json::parser;
using json_serializer = json::serializer;
using json_request = boost::beast::http::request<json_body>;
using json_response = boost::beast::http::response<json_body>;
using json_request_cptr = std::shared_ptr<const json_request>;
using json_request_ptr = std::shared_ptr<json_response>;
} // namespace http
} // namespace network
} // namespace libbitcoinThe request/response objects come/go at the channel with fully-parsed http headers and a DOM-based json "body". Converting the DOM to/from c++ to RPC at the channel's message distributor allows subscribing protocols to get (and send) full c++ strongly-typed RPC function call parameters.
So RPC channels sit on top of the basic JSON request/response objects, validate RPC and map to-from "function" calls by broadcasting to protocol subscribers. Otherwise it's fully json generic. This means that the same request/response objects can just as easily support non-RPC json serialization endpoints/formats.
evoskuil21:47
And in that case there is a potential complexity, as one endpoint may be listening for various types of requests. Someone might post a JSON object or a document request for example. Also a protocol sends an response object but based on the requester's accept types, or query string, the response will need distinct encoding. A transaction may arrive as json or as text (hex) or as binary (wire). One might be returned in any of these formats as well. Due to the above-mentioned lack of polymorphism, dispatching in these cases can create a combinatorial type explosion. We would need to listen for headers, parse them and dispatch to another listener to read/parse the body into a certain format and construct a different request type using the header object and unique body type (e.g. json, string, data, file, empty).
So to manage this I created a variant request/response body (encapsulating the json body above, along with predefined
boost::beast::http bodies):
namespace libbitcoin {
namespace network {
namespace http {
using json_body = json::body<json::parser, json::serializer>;
using empty_body = boost::beast::http::empty_body;
using data_body = boost::beast::http::vector_body<uint8_t>;
using string_body = boost::beast::http::string_body;
using file_body b= oost::beast::http::file_body;
using json_value = json_body::value_type;
using data_value = data_body::value_type;
using file_value = file_body::value_type;
using string_value = string_body::value_type;
using empty_value = empty_body::value_type;
using json_reader = json_body::reader;
using data_reader = data_body::reader;
using file_reader = file_body::reader;
using string_reader = string_body::reader;
using empty_reader = empty_body::reader;
using json_writer = json_body::writer;
using data_writer = data_body::writer;
using file_writer = file_body::writer;
using string_writer = string_body::writer;
using empty_writer = empty_body::writer;
using value_variant = std::variant
<
empty_value,
json_value,
data_value,
file_value,
string_value
>;
using reader_variant = std::variant
<
empty_reader,
json_reader,
data_reader,
file_reader,
string_reader
>;
using writer_variant = std::variant
<
empty_writer,
json_writer,
data_writer,
file_writer,
string_writer
>;
struct variant_payload
{
mutable std::optional<value_variant> inner{};
};
/// boost::beast::http body template for all message types.
/// This encapsulates a variant of supported body types, selects a type upon
/// reader or writer construction, and then passes all calls through to it.
struct body
{
using value_type = variant_payload;
class reader
{
public:
using buffer_type = asio::const_buffer;
template <bool IsRequest, class Fields>
inline explicit reader(header<IsRequest, Fields>& header,
value_type& payload) NOEXCEPT;
inline void init(const length_type& length, error_code& ec) NOEXCEPT;
inline size_t put(const buffer_type& buffer, error_code& ec) NOEXCEPT;
inline void finish(error_code& ec) NOEXCEPT;
private:
reader_variant reader_;
};
class writer
{
public:
using const_buffers_type = asio::const_buffer;
using out_buffer = get_buffer<const_buffers_type>;
template <bool IsRequest, class Fields>
inline explicit writer(header<IsRequest, Fields>& header,
const value_type& payload) NOEXCEPT;
inline void init(error_code& ec) NOEXCEPT;
inline out_buffer get(error_code& ec) NOEXCEPT;
private:
writer_variant writer_;
};
};
} // namespace http
} // namespace network
} // namespace libbitcoinnamespace libbitcoin {
namespace network {
namespace http {
inline bool is_attachment(auto& head) NOEXCEPT
{
const auto& disposition = head[field::content_disposition];
const auto content = system::ascii_to_lower(disposition);
return content.find("filename=") != std::string::npos;
}
// reader
// ----------------------------------------------------------------------------
template <class Body>
inline reader_variant reader_from_body(auto& head,
variant_payload& pay) NOEXCEPT
{
BC_ASSERT(pay.inner.has_value());
using reader = typename Body::reader;
using value = typename Body::value_type;
return reader_variant{ std::in_place_type<reader>, head,
std::get<value>(pay.inner.value()) };
}
/// Select reader based on content-type header.
inline reader_variant to_reader(auto& head, variant_payload& pay) NOEXCEPT
{
switch (content_mime_type(head[field::content_type]))
{
case mime_type::application_json:
{
pay.inner = json_value{};
return reader_from_body<json_body>(head, pay);
}
case mime_type::text_plain:
{
pay.inner = string_value{};
return reader_from_body<string_body>(head, pay);
}
case mime_type::application_octet:
{
if (is_attachment(head))
{
pay.inner = file_value{};
return reader_from_body<file_body>(head, pay);
}
else
{
pay.inner = data_value{};
return reader_from_body<data_body>(head, pay);
}
}
default:
{
pay.inner = empty_value{};
return reader_from_body<empty_body>(head, pay);
}
}
}
template <bool IsRequest, class Fields>
inline body::reader::reader(header<IsRequest, Fields>& header,
value_type& payload) NOEXCEPT
: reader_{ to_reader(header, payload) }
{
}
inline void body::reader::init(const length_type& length,
error_code& ec) NOEXCEPT
{
std::visit(overload
{
[&](auto& read) NOEXCEPT
{
try
{
read.init(length, ec);
}
catch (...)
{
ec = boost::system::errc::io_error;
}
}
}, reader_);
}
inline size_t body::reader::put(const buffer_type& buffer,
error_code& ec) NOEXCEPT
{
return std::visit(overload
{
[&](auto& read) NOEXCEPT
{
try
{
return read.put(buffer, ec);
}
catch (...)
{
ec = boost::system::errc::io_error;
return size_t{};
}
}
}, reader_);
}
inline void body::reader::finish(error_code& ec) NOEXCEPT
{
return std::visit(overload
{
[&](auto& read) NOEXCEPT
{
try
{
return read.finish(ec);
}
catch (...)
{
ec = boost::system::errc::io_error;
}
}
}, reader_);
}
// writer
// ----------------------------------------------------------------------------
template <class Body>
inline writer_variant writer_from_body(auto& head,
const variant_payload& pay) NOEXCEPT
{
BC_ASSERT(pay.inner.has_value());
using writer = typename Body::writer;
using value = typename Body::value_type;
return writer_variant{ std::in_place_type<writer>, head,
std::get<value>(pay.inner.value()) };
}
/// Create writer matching the caller-defined body.inner (variant) type.
inline writer_variant to_writer(auto& head,
const variant_payload& pay) NOEXCEPT
{
// Caller should have set inner, otherwise set it to empty (it's mutable).
if (!pay.inner.has_value())
pay.inner = empty_value{};
return std::visit(overload
{
[&](const json_value&) NOEXCEPT
{
return writer_from_body<json_body>(head, pay);
},
[&](const string_value&) NOEXCEPT
{
return writer_from_body<string_body>(head, pay);
},
[&](const data_value&) NOEXCEPT
{
return writer_from_body<data_body>(head, pay);
},
[&](const file_value&) NOEXCEPT
{
return writer_from_body<file_body>(head, pay);
},
[&](const empty_value&) NOEXCEPT
{
return writer_from_body<empty_body>(head, pay);
}
}, pay.inner.value());
}
template <bool IsRequest, class Fields>
inline body::writer::writer(header<IsRequest, Fields>& header,
const value_type& payload) NOEXCEPT
: writer_{ to_writer(header, payload) }
{
}
inline void body::writer::init(error_code& ec) NOEXCEPT
{
return std::visit(overload
{
[&] (auto& write) NOEXCEPT
{
try
{
write.init(ec);
}
catch (...)
{
ec = boost::system::errc::io_error;
}
}
}, writer_);
}
inline body::writer::out_buffer body::writer::get(error_code& ec) NOEXCEPT
{
return std::visit(overload
{
[&](auto& write) NOEXCEPT
{
try
{
return write.get(ec);
}
catch (...)
{
ec = boost::system::errc::io_error;
return out_buffer{};
}
}
}, writer_);
}
} // namespace http
} // namespace network
} // namespace libbitcoinThis is just a passthrough wrapper for each body type that selects the body type (parser) on request construct, based on the parsed request header. On the response the body is the response content (typed by the responder, including header content-type) and on writer construct the body serializer is selected for emitting the object. So we can pass around this single type which at almost no runtime cost passes through to a single contained type with parser/serializer automatically selected based on declared content-type.
So for endpoints that support multiple content formats, their channel can still use just a single request/response pair of types (
http::request and http::response), and a single proxy method for each:
/// HTTP Readers.
/// -----------------------------------------------------------------------
/// Read full http variant request from the socket, using provided buffer.
virtual void read(http::flat_buffer& buffer, http::request& request,
count_handler&& handler) NOEXCEPT;
...
/// HTTP Writers.
/// -----------------------------------------------------------------------
/// Write full http variant response to the socket (json buffer in body).
virtual void write(http::response& response,
count_handler&& handler) NOEXCEPT;
... (edited)For endpoints that are dedicated to a single format, such as the
bitcoind compat json-rpc endpoint, the non-variant types can be used directly:
/// HTTP Readers.
/// -----------------------------------------------------------------------
...
/// Read full http json request from the socket, using provided buffer.
virtual void read(http::flat_buffer& buffer, http::json_request& request,
count_handler&& handler) NOEXCEPT;
/// HTTP Writers.
/// -----------------------------------------------------------------------
...
/// Write full http json response to the socket (serialize buffer in body).
virtual void write(http::json_response& response,
count_handler&& handler) NOEXCEPT;evoskuil22:03
The proxy is stateless and unconfigured, so the caller (channel) retains control over read buffering options via the passed buffer, and
json serialization (which also requires a buffer) buffer control is maintained by its exposure on the http::json::body object, accessible to the caller as response::body().buffer. In both cases this allows the channel to reuse a single buffer for all reads and for json writes for a given channel instance. So dynamic allocation can be eliminated, allowing the buffers to remain stack allocated. If the json serialization buffer is not provided, it will get allocated at a hardwired size internally. The payload object here is provided by the boost::beast construction of the request/response as the .body().
struct payload
{
/// JSON DOM.
json_value model{};
/// Writer serialization buffer (max size, allocated on write).
mutable http::flat_buffer_ptr buffer{};
};evoskuil23:07
Above constitutes nearly the entire implementation of
(edited)json and json-rpc base implementation from wire to channel and back to wire. boost::json on boost::beast on boost::asio. Next is to set up the message distributors and send templates on channels, to provide subscription and full abstraction from encodings for protocols. Then it's just protocols subscribing with strongly-typed handlers, making the queries and invoking send(...) with the resulting objects.Wednesday, October 29, 2025
evoskuil08:45
Test complete for variant
http::request / http::response and http::json_request / http::json_response. On to the channel and protocol integration tomorrow!🎉 2
Jack Minnick20:41
Docs read, running Libbitcoin-node for science. 32gig ram 12 core Linux system, loaded all Libraries successfully, still doing IBD, getting hundreds of outbound connections. Still reading V3 source code (quite a bit there).
evoskuil20:42
Great! Why v3? Really outdated. To get anywhere close to peak download perf on Linux you need a couple config changes... @eynhaender
What's your WAN bandwidth? And SSD perf.
(edited)Jack Minnick20:46
I kept seeing that Branch wasn't usable in current state so I went back until it was.
evoskuil20:46
Right, that's a warning against production use. But you def don't wanna be testing out v3.
(edited)👍🏼 1
I also was using an HDD, but I will retry later with SSD.
Internal SSD is the only reasonable perf.
Unless you have enough RAM to store most of the chain, then it's ok. 🙂
Jack Minnick20:50
I'll give it another go with current branch. I wasn't min/maxing opti, I am just looking for general functionality and what I can build on top when running.
I do have 8 slots on this machine, but not enough sticks at the moment to max out.
evoskuil20:52
You'll be able to do a lot more on v4, and you'll be able to see that very soon with the public service endpoints starting to light up.
🙏🏼 1
web console, integrated block explorer - RESTful with json serialization, bitcoind json-rpc node API, ElectrumX server API, eventually Stratum v1/v2.
👀 1
Jack Minnick20:56
I am definitely interested in the Stratum V2 future applications. Wallet stuff is cool too but I want to get people looking at taker part of the block market again.
evoskuil20:57
Plenty of wallet APIs in libbitcoin-system, but the node is not a wallet, so no wallet APIs there.
evoskuil23:37
I completed the integration of the new variant request/response types into
network::http_channel and network::http_protocol. This enables node::protocol_explore and node::protocol_html receipt of any http::request contained variant type at the same endpoint, and the ability to construct and send back any http::response contained variant type. This code barely changed, all it required was replacement of http::string_request and http::string_response with the corresponding variant types. Sending doesn't require the variant type, it can just be sent with any type using polymorphism, works either way. But I expect that the variant response will end up becoming a useful mechanism for similar late branching of serialization if that becomes beneficial.
We're now up and running the internal html site on the new types, and right now I'm gonna start sending some text and json from a test client, and pick it up in the node and then start sending some back. I expect that to got well, and if so I'll be building some actual application/json (json encoded btc wire protocol) and text/plain (hex encoded) and application/octet-stream (binary encoded btc wire protocol) requests and responses.🎉 1
Thursday, October 30, 2025
evoskuil05:12
The first ever http database requests from libbitcoin.
(1) json (pretty printed by Chrome)
(2) text (hexadecimal wire encoding)
(3) binary (wire encoding) captured as a file by Chrome
(4) none-of-the-above returns default page (for all URLs)
Accept media type set as URL query parameter.
Accept header supported (lower priority than parameter).
Single hash token interpreted as tx (no parse/dispatch yet).
(edited)🚀 5👾 1
1 reply · evoskuil · last Oct 30, 2025
evoskuil06:07
/// Pagination and filtering are via query string.
enum targets
{
/// /v[]/block/hash/[bkhash] {1}
/// /v[]/block/height/[height] {1}s
block,
/// /v[]/block/hash/[bkhash]/filter {1}
/// /v[]/block/height/[height]/filter {1}
filter,
/// /v[]/block/hash/[bkhash]/header {1}
/// /v[]/block/height/[height]/header {1}
header,
/// /v[]/transaction/hash/[txhash] {1}
/// /v[]/block/hash/[bkhash]/transaction/[position] {1}
/// /v[]/block/height/[height]/transaction/[position] {1}
transaction,
/// /v[]/block/hash/[bkhash]/transactions {all txs in the block}
/// /v[]/block/height/[height]/transactions {all txs in the block}
transactions,
// --------------------------------------------------------------------
/// /v[]/input/[txhash]/[index] {1}
input,
/// /v[]/inputs/[txhash] {all inputs in the tx}
inputs,
/// /v[]/input/[txhash]/[index]/script {1}
input_script,
/// /v[]/input/[txhash]/scripts {all input scripts in the tx}
input_scripts,
/// /v[]/input/[txhash]/[index]/witness {1}
input_witness,
/// /v[]/input/[txhash]/witnesses {all witnesses in the tx}
input_witnesses,
// --------------------------------------------------------------------
/// /v[]/output/[txhash]/[index] {1}
output,
/// /v[]/outputs/[txhash] {all outputs in the tx}
outputs,
/// /v[]/output/[txhash]/[index]/script {1}
output_script,
/// /v[]/output/[txhash]/scripts {all output scripts in the tx}
output_scripts,
/// /v[]/output/[txhash]/[index]/spender {1 - confirmed}
output_spender,
/// /v[]/output/[txhash]/spenders {all}
output_spenders,
// --------------------------------------------------------------------
/// /v[]/address/[output-script-hash] {all}
address
};evoskuil08:14
A little fancier with the hack URI path decoding, so now we can specify block, header or tx, format="json|text|data", and witness="true|false". Rendering is stupid fast.
(edited)
🚀 1🎉 1
Huddle with pmienk, evoskuil, Marnix, eynhaender, Rowan Voskuil, jungly · 216 min15:53
Friday, October 31, 2025
evoskuil03:43
Turns out the inflation from wire protocol to json is 2.5x, just based on pulling down some blocks in each format and comparing the file sizes. And even though we don't know the json size before we write it, and we need the buffer to do that, we can use the wire serialization size, which we get for free from the store-deserialized c++ instance of any object. For header this is just a static fixed value. But it varies a lot for others. So a 2.5x buffer estimate will eliminate most iteration over the buffer. Or we can accept a smaller number of iterations to preserve memory. Also resizing down is a cost in terms of reallocations, so there's always a tradeoff.
This is a cold start block 900,000 (783vKB) json + witness:
And this is the refresh (hitting the node):
Notice that the green line is consistent at 75ms. There is no server caching, apart from the effect of the store being paged into RAM. So the refresh time at 35% of initial time is just a consequence of Chrome caching.
Same block text (hex) wire encoding:
This shows the data (binary) format download in the second row (still showing the hex page in the browser because the binary is offloaded to a file). Note that the green line is not discernable because of scale, but it shows in the detail view as 32.33ms:
(edited)
Binary (wire) size is 1,920,777 bytes (hex is 2x), json is 4,207,065 bytes (~2.2x)
Server time for
data/binary = 32ms, text/hex = 37ms, and json = 75ms.
Payload size for data/binary = 1x, text/hex = 2x, and json = ~2.2x.For the html site, node times are barely perceptible in ms....
(edited)
evoskuil18:12
Pages use the streaming
file_body http response, so pass directly off of the file system to the port. With the embedded site it will be the same or faster, since the pages will be POD-typed, and using the buffer_body response will stream directly from the application's data segment without translation or copy.The response time for p2p block output is certainly less than the data format above, despite having the same serialization, as there is no http overhead. So I think we can lay to rest any concerns about not being able to rapidly serve blocks to peers. If we had the blocks already stuffed on the SSD in wire serialization, they could maybe be served a tiny bit faster. But then of course every other query would be horrendous.
evoskuil18:27
Also note that this is single thread retrieval on a 10yo machine. I'd expect these times to get cut in half by just modern hardware. Then of course we can just apply c++20 vectorization to blocks, processing txs on independent threads. I just haven't bothered to write the queries for this yet. We do use this technique in one of the confirmation queries, as confirmation runs on a single thread, but can be vectorized within a block. It's as simple as this:
std::transform(parallel, txs.begin(), txs.end(), block.begin(), to_block);(edited)
🙌 1