Archive of the libbitcoin Slack #general channel.
bigmac RAM at 32GB and called it bigmac32. That is now completing in the same timeframe as Linux and Windows, and both of those are also optimized down to 8.6 hrs on the the minis (neidermeyer and stork). Milestones are all consistent across the board at high and low RAM (write pressure). Basically I've tightened the custom memory map implementation so it's the same dynamic behavior as Windows, while dealing with the macOS and Linux idiosyncrasies.bigmac milestone sync. archive scale is dwarfed by validation (no-op) and confirmation (tx state changes). Cool to see it this way.DoS protection throttle. This operates in the network proxy (the socket encapsulator and base class for all channels). The real work was done in every piece of network code over the past 5 years. Protection operates on backpressure, a design principle I picked up from the late great Pieter Hintjens. So really there's nothing to do except to set a rate limit, so a timer, a handler, and a config setting. The config setting was already in there, and now the rest is done. I will probably also drop in a per service override config setting.
There is no limit whatsoever on how much can be sent to the node/server. The limit is on what we send back. Queries are small, responses are large and costly - and generally the larger the more costly (though not always, so there's some room for tuning this). The interesting part about the handling is that there is no moving window, accumulation or averaging. There's just a rate at which we reply (or notify), in bytes per time.
The send is emitted (at network speed of course), and then the handler invocation is delayed (asynchronously) until the time allocated for that number of bytes is reached. So if the configured rate is faster than network speed, there's no limit. Otherwise the channel is reduced to the configured speed. The total cost is just the timer invocation and its handler indirection, which are trivia in terms of network perf.stork) and macOS (bigmac) milestone and full validation projects based on a per-tx rate between 750-850k blocks. Due to it's dynamic native memory management stork consistently beats bigmac in the low RAM regime despite significantly lower CPU power. stork is network limited (milestone) at 24GB+ and bigmac at 32GB+. Validation doesn't become truly CPU limited until there is full RAM, thought the RAM effects become very small at high RAM. There is no stork above 32GB for comparison.
Windows is able to validate down to 18GB platforms and macOS down to 24GB platforms. That's the legacy of Unix overcommit killing the process. Windows continues to validate at any RAM level, but the time cost goes parabolic below 18GB. These observations of viability hold regardless of CPU power. 32GB is a good RAM target for macOS and 32GB for Windows, as RAM effects are modest and smooth above those points, but 24GB is viable for macOS and 18GB for Windows.neidermeyer (Linux) quite an unfair advantage by making it headless. Good work @eynhaender! Though stork (Windows) does have a faster processor.secp256k1 to the brand new v0.8.0.0 release, which reportedly speeds sig verify by ~11%. NuGet packaged published for vc145, projects all bumped, verified in CI in my repo. Waiting on performance runs before merge.no longer use the previously-required Linux OS paging config (could actually be negative). macOS is now competitive, and Windows remains so. secpp256k1 is updated 0.8.0.0, with published NuGet package. Network DoS protection is in, including global and per-service rate-limiting.MANAGE_STAGING preprocessor define to get the mmap optimizations but that will be removed in the next database PR.
neidermeyer (Linux) finished the 16GB full validation run to 950k @ with load factor 5.0 in 8.52 hrs. That's down from 11.5 hrs at LF 2.5.stork (Windows) finished the 16GB full validation run to 950k @ with load factor 3.75 in 11.3 hrs. That's down from [took forever and was stopped] at LF 2.5 - which is awesome. But I'll run this one again at LF 5.0 for a straight comparison with mac/linux.bigmac (macOS) [with another recent mmap patch] is running the 16GB full validation with LF 5.0 and is running well behind the others- - projecting ~ 15 hrs, but is running on 16GB! On earlier runs macOS could not complete even the 16GB milestone run (projected over 28 hrs) and before the custom mmap it couldn't even complete a 768 GB milestone run.bigmac was grinding through its full validation steadily until it was jetsam-killed at around 850k. I've projected a completion along its curve because it wasn't parabolic - it was steady.
Notice that the leftmost solid circle shows the point after which the LF2.5 runs went parabolic with latest optimizations. The open circles show the same code running at an increased LF. For Windows the LF3.75 was better than LF5.0, since the header residency problem is less pronounced for it and the higher LF increases body paging.
So you can see that we have a pretty clear pattern. The different mmap implementations and underlying support handle memory pressure differently. In full validation by 32GB this is mostly phased out, and similarly for milestone by 24GB. Yet we are avoiding thrashing down to 16GB at least, with the exception of macOS, which got close. I suspect the duplicate probing starts to hurt in this regime, which I'll verify at some point.
The LF impact implies that we should tune the heads given a default store creation, and vary that by platform.
There is damage on high RAM platforms in Linux, as a consequence of aggressively shedding body pages. I've disabled that above 32GB but I don't have a platform to verify it.
The next round of tests will hit milestone with the latest optimizations and LF variations. It's possible and I've seen a couple of times, that optimization for one scenario hurts another, but at this point I believe it's optimal across validation and milestone. I expect that, as with the validation runs, the LF increase will significantly bend down the low end of the milestone curves, as the heads are only required for reads, and the increased LF hurts reads, but we don't read much in the big heads for milestone. So it's more like free RAM at no cost. Ideally those would entirely flatten to network limited all the way to 16GB.
noexcept enabled...
A race that corrupts state kills the process with SIGSEGV/SIGBUS on a wild access. noexcept is invisible to that: it governs C++ exception propagation, not signal delivery. The .ips report and any core capture the faulting instruction and all thread stacks regardless. Here noexcept changes the shape of the dump, in our favor: the throw hits the noexcept boundary and std::terminate fires at the throw site, without unwinding — SIGABRT with the origin frames still on the stack (abort ← terminate ← __cxa_throw ← the actual thrower). In a debug build with NOEXCEPT off, the exception would unwind — and if anything up-stack caught it, the origin would vanish entirely. So for localization, release-with-noexcept is arguably the better trap.
macOS completed its 16GB milestone run in 1.61 hrs.
The 8GB @ LF10 runs hit a wall, so I backed that off to 12GB @ LF 7.25 to find the inflection point.
But it's safe to say at this point that all OSs are pretty close to network limited starting from 16GB, validation is reasonable on all three down to 16GB, and excellent from 24GB.
12GB milestone at LF7.25 in 1.02 hrs completely flat, actually beat it's 16GB time.12GB milestone at LF7.25 in 1.05 hrs completely flat, almost beat it's 16GB time.neidermeyer at 16GB @ LF5.0, right up to 950k in 9.51 hrs.svg favicon for libbitcoin.info, so just added it to the build as a shared resource for both embedded sites, same byte size as the old one.
macOS completed 850k milestone on 12GB in 70 mins. Inadvertently hit a disk full, and then recovered and continued when I cleared off the earlier runs. But stopped it to do some other testing. Mac has come a long way, and significantly, this run finally corrected our accounting issues on Darwin, and it's looking good on all platforms right now.Windows just completed 950k block pruned milestone sync on an 8GB mini at 2.5Gbps in 41.4 mins!Linux just completed a 40.6 min milestone to block 950k on 8GB RAM. Slightly lesser CPU but more free RAM (due to being headless).pruned store size of 441GB at 950k blocks. This increases to 450GB at default LF2.5.
address index from ~156GB including table head to just the ~6GB table head. That's still big for a table head, and can be reduced (e.g. ~3GB at 2x LF), but would still keep it optional, though maybe on by default. This will also reduce the additional sync time for the address index to almost nothing. The cost is increased search cost, which I'm still investigating. But the search is so fast that the storage space is generally a much bigger issue, and may itself offset the search cost in reduced paging.bip324 (opportunistic encryption) a clean integration. The p2p protocol is now p2p/s with a proper socket variant. I also cleaned up the confusing services configuration settings and integration, which is where p2p privacy is enabled. Still have to do some acceptance testing, but otherwise this is complete.
[peer] enable_privacy = true # new, opportunistic encryption (bip324) invalid_services = 176 # shitcoins (existing config, retained) #services_minimum = 42 # obsolete/invalid #services_maximum = 42 # obsolete/invalid [node] # these are new (replace services_minimum/maximum) limited_blocks = false provide_blocks = true require_blocks = true provide_witness = true require_witness = true provide_filters = false(edited)
2026-08-16T06:46:29Z.5 Insufficient services (3080) by [144.91.83.152:8333] showing (67112013) /Satoshi:31.0.0/ <<unhandled exception - start trace>> exception 0xc0000005 on thread 21128 libbitcoin::network::socket::do_handshake -> C:\Source\evoskuil\libbitcoin-network\src\net\socket_connect.cpp(218) libbitcoin::network::socket::do_handshake -> C:\Source\evoskuil\libbitcoin-network\src\net\socket_connect.cpp(218) boost::asio::detail::range_connect_op<boost::asio::ip::tcp,boost::asio::any_io_executor,boost::asio::ip::basic_resolver_results<boost::asio::ip::tcp>,boost::asio::detail::default_connect_condition,std::_Binder<std::_Unforced,void (__cdecl libbitcoin::network::socket::*)(boost::system::error_code const &,boost::asio::ip::basic_endpoint<boost::asio::ip::tcp> const &,std::function<void __cdecl(std::error_code const &)> const &) noexcept,std::shared_ptr<libbitcoin::network::socket>,std::_Ph<1> const &,std::_Ph<2> const &,std::function<void __cdecl(std::error_code const &)> const &> >::process<boost::asio::ip::basic_resolver_iterator<boost::asio::ip::tcp> > -> C:\Source\evoskuil\.nuget\packages\libbitcoin-boost.1.91.0\lib\native\include\boost\asio\impl\connect.hpp(355) boost::asio::detail::executor_function::complete<boost::asio::detail::binder1<boost::asio::detail::range_connect_op<boost::asio::ip::tcp,boost::asio::any_io_executor,boost::asio::ip::basic_resolver_results<boost::asio::ip::tcp>,boost::asio::detail::default_connect_condition,std::_Binder<std::_Unforced,void (__cdecl libbitcoin::network::socket::*)(boost::system::error_code const &,boost::asio::ip::basic_endpoint<boost::asio::ip::tcp> const &,std::function<void __cdecl(std::error_code const &)> const &) noexcept,std::shared_ptr<libbitcoin::network::socket>,std::_Ph<1> const &,std::_Ph<2> const &,std::function<void __cdecl(std::error_code const &)> const &> >,boost::system::error_code>,std::allocator<void> > -> C:\Source\evoskuil\.nuget\packages\libbitcoin-boost.1.91.0\lib\native\include\boost\asio\detail\executor_function.hpp(114) boost::asio::detail::executor_op<boost::asio::detail::executor_function,std::allocator<void>,boost::asio::detail::scheduler_operation>::do_complete -> C:\Source\evoskuil\.nuget\packages\libbitcoin-boost.1.91.0\lib\native\include\boost\asio\detail\executor_op.hpp(71) boost::asio::detail::strand_executor_service::run_ready_handlers -> C:\Source\evoskuil\.nuget\packages\libbitcoin-boost.1.91.0\lib\native\include\boost\asio\detail\impl\strand_executor_service.ipp(153) boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>,4> const ,void>::operator() -> C:\Source\evoskuil\.nuget\packages\libbitcoin-boost.1.91.0\lib\native\include\boost\asio\detail\impl\strand_executor_service.hpp(122) boost::asio::io_context::basic_executor_type<std::allocator<void>,4>::execute<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>,4> const ,void> > -> C:\Source\evoskuil\.nuget\packages\libbitcoin-boost.1.91.0\lib\native\include\boost\asio\impl\io_context.hpp(218) boost::asio::detail::strand_executor_service::do_execute<boost::asio::io_context::basic_executor_type<std::allocator<void>,4> const ,boost::asio::detail::executor_function,std::allocator<void> > -> C:\Source\evoskuil\.nuget\packages\libbitcoin-boost.1.91.0\lib\native\include\boost\asio\detail\impl\strand_executor_service.hpp(251) boost::asio::execution::detail::any_executor_base::execute_ex<boost::asio::strand<boost::asio::io_context::basic_executor_type<std::allocator<void>,4> > > -> C:\Source\evoskuil\.nuget\packages\libbitcoin-boost.1.91.0\lib\native\include\boost\asio\execution\any_executor.hpp(902) boost::asio::detail::win_iocp_socket_connect_op<boost::asio::detail::range_connect_op<boost::asio::ip::tcp,boost::asio::any_io_executor,boost::asio::ip::basic_resolver_results<boost::asio::ip::tcp>,boost::asio::detail::default_connect_condition,std::_Binder<std::_Unforced,void (__cdecl libbitcoin::network::socket::*)(boost::system::error_code const &,boost::asio::ip::basic_endpoint<boost::asio::ip::tcp> const &,std::function<void __cdecl(std::error_code const &)> const &) noexcept,std::shared_ptr<libbitcoin::network::socket>,std::_Ph<1> const &,std::_Ph<2> const &,std::function<void __cdecl(std::error_code const &)> const &> >,boost::asio::any_io_executor>::do_complete -> C:\Source\evoskuil\.nuget\packages\libbitcoin-boost.1.91.0\lib\native\include\boost\asio\detail\win_iocp_socket_connect_op.hpp(123) boost::asio::detail::win_iocp_io_context::do_one -> C:\Source\evoskuil\.nuget\packages\libbitcoin-boost.1.91.0\lib\native\include\boost\asio\detail\impl\win_iocp_io_context.ipp(499) boost::asio::detail::win_iocp_io_context::run -> C:\Source\evoskuil\.nuget\packages\libbitcoin-boost.1.91.0\lib\native\include\boost\asio\detail\impl\win_iocp_io_context.ipp(229) std::thread::_Invoke<std::tuple<`libbitcoin::network::threadpool::threadpool'::`4'::<lambda_1> >,0> -> C:\Program Files\Microsoft Visual Studio\18\Community\VC\Tools\MSVC\14.51.36231\include\thread(60) thread_start<unsigned int (__cdecl*)(void *),1> -> minkernel\crts\ucrt\src\appcrt\startup\thread.cpp(97) BaseThreadInitThunk @ 0x00007ff9b160ccb7 <<unhandled exception - end trace>>
p2p/s activated. It may even be faster than p2p but I don't expect too much difference.p2p/s (bip324) at 90% channel participation is milestone performance neutral on our minimal platforms, with possibly a slight increase in cost on the Linux machine. There is no indication of skew over the runs toward non-private connections, so individual channel performance remained consistent. However these are milestone runs, with plenty of excess CPU. The continuous ChaCha20-Poly1305 increases CPU utilization by 40% on all machines. This hits the CPU-weakest machine (Linux) hardest in terms of wall time (40.3 -> 41.9 mins => +4.1%). That will translate into a material cost during validation.
"Each v1 P2P message uses a double-SHA256 checksum truncated to 4 bytes. Roughly the same amount of computation power is required for encrypting and authenticating a v2 P2P message as proposed."(edited)
shelly completed a pruned milestone sync to 950k in 22.3 mins -- including the full chain address index (ElectrumX). Total store size with full constant time bidirectional indexation is 444GB.150GB index) in favor of dynamic in-place hashing over the full script , the queries are still fast. The large address query takes 8.5ms (data), 10ms (text), and 13.7ms (json). This returns an array of 7,453 serialized output points:
[
{
"point": {
"hash": "db9847575ef720b8b5582bac74bcedef1b6a08903ddc27638024b8d0d3ba0400",
"index": 0
},
"value": 1270486495
},
...
]107,147,568 bytes - or 2,435,172 output points - (with coin values) in 1.5 secs (1.9 secs with turbo disabled).227,897,252 bytes - or 5,179,483 output points - (with coin values) in 2.6 secs (3.4s cold).150GB to 15GB. It's inconsequential enough that we can just leave it on by default. Sync time on shelly with a full address index pruned milestone at default LF2.5 was 22.75 mins.
libbitcoinkernel : bitcoinbackbone.org
Long-term goal would be to integrate libbitcoin's libbicoin-system as a second validation backend:
github.com/libbitcoin/libbitcoin-system/issues/1923
That way avoid social and technical dependency on a single validation backend.
And allow backbone node operators to pick up the validation engine they wish at build time.
Voila, voila 🙂psbt and descriptors for support of the bitcoind jrpc interface. Weird that they have psbt as a client-server interface, since it's computation only, but added for compat.
Also merged an optimized database query to populate utxo totals also for bitcoind jrpc. Since we don't actually maintain a utxo accumulator, this is a full store scan. However the use cases for these are limited to wallet recovery and auditing consensus. The scan is highly optimized, using a parallel query across the store for accumulation. There is also a pair of bitcoind methods that export and import an "assume utxo" blob, for which we reject support (return not_implemented).[x] Linux OOM kill at store creation on 8GB (initialize on disk) [x] restore after height index regress (change to headers) [x] broken store with no snap should create empty store [x] restore after prevouts prune (unsafe) [x] archive immutable store settings in genesis [x] API from ultrafast to detect GPU usability [x] milestone prompt when not explicitly configured
expected element count, from which the store derives optimal boom filter k at create (formerly suboptimal), persisted via the settings envelope across load/restore. The bs defaults are recalibrated to a mainnet 950k basis: expected element counts, buckets sized for a 2.5 load factor, and minimum body allocations at 1% of the pruned body with 1% growth.
I am continuing to optimize the custom posix mmap implementations, which has resulted in more performance improvement at the low RAM end*. The default allocations and growth rates are dramatically smaller than previous defaults. They no longer have a significant impact on sync performance, and this is a much more flexible configuration. With this done I'll be moving on to default store config automation as a function of available RAM.
(*) e.g. Linux 8 GB RAM mini pruned with full address index now completes a 950k milestone run on 2.5Gbps in 38 minutes.
Also removed the slab scan (-a) command line option, as it has been obsoleted for some time due to a redesign of the ins/points tables. ins and outs are an association record table that each point to inputs and outputs slab tables. So in/out element counts are simply array size.