Archive of the libbitcoin Slack #general channel.
The Linux-only test failure (electrum__blockchain_outpoint_subscribe__two_spenders__one_notification) demonstrates that this ordering is not observed on Ubuntu 24.04 with the Boost.Asio version used in the CI jobs, even though the same code path produces the expected ordering on Windows and macOS. This can only be explained as a platform-specific behavioral difference (or bug) in Boost.Asio's epoll-based reactor and thread-pool executor implementation on Linux. The isolated cause of the test failures is therefore the reliance, in the two-spender notification path of complete_outpoint_subscribe, on an ordering guarantee that Boost.Asio's Linux implementation does not reliably provide in this configuration.
boost-1.91.0 for vc145 (vs2022), and I was then able to complete manual setup of libbitcoin-system vs2026 builds using that and my previously published secp256k1-vc145. Clean builds and all tests passing, and that covers all dependencies for bs and bx. Over to @pmienk now to add to libbitcoin-build generation.json-rpc over websockets and raw tcp. Previously we had only support for rpc over tcp (Electrum). So with just a few lines of additional code either can be called by the channel with the exact same parameterization. This gives us the ability to derive a btcd channel in server over the same rpc channel as Electrum, but as websockets (or tcp), and also allows us to generalize Electrum to websockets. And since websockets sits on an http base channel (which sites on a tcp base channel), it means that these can both operate over http. And since bitcoind is http, it can be generalized to both tcp and websockets as well. The protocol implementations are transport independent and the channels manage socket configuration (which also already includes TLS over http, websockets, and tcp - and socks5, basic auth, and mutual auth for http).node.delay_inbound setting remains effective, it's only the suspensions that are avoided, such as h <enter> and during snapshot (automatic and <b enter>) and in the case of a detected server fault. This is intended to allow for hot snapshots and other p2p pause without service endpoint interruption.g++ due to a compiler bug, but I think @pmienk has a handle on it. We are also now running vs2026 (native vc145) builds in CI thanks to @eynhaender’s production of boost-vc145-1.91.0 NuGet builds (the guy who produced them for 15 years retired) and @pmienk’s updated build automation. Now that we control boost packaging we should also be able to put out native ARM Windows builds as well./// Reporting events.
enum events : uint8_t
{
/// Candidate chain.
header_archived, // header checked, accepted
header_organized, // header pushed (previously archived)
header_reorganized, // header popped
/// Blocks.
block_archived, // block checked
block_buffered, // block buffered for validation
block_validated, // block checked, accepted, connected
block_confirmed, // block checked, accepted, connected, confirmable
block_unconfirmable, // block invalid (after headers-first archive)
validate_bypassed, // block checked, accepted [assumed]
confirm_bypassed, // block checked, accepted, connected [assumed]
/// Transactions.
tx_archived, // unassociated tx checked, accepted, connected
tx_validated, // associated tx checked, accepted, connected
tx_invalidated, // associated tx invalid (after headers-first archive)
/// Confirmed chain.
block_organized, // block pushed (previously confirmable)
block_reorganized, // block popped
/// Mining.
template_issued, // block template issued for mining
/// Timespans.
snapshot_secs, // snapshot timespan in seconds.
prune_msecs, // prune timespan in milliseconds.
reload_msecs, // store reload timespan in milliseconds.
block_usecs, // getblock timespan in microseconds.
ancestry_msecs, // getancestry timespan in milliseconds.
filter_msecs, // getfilter timespan in milliseconds.
filterhashes_msecs, // getfilterhashes timespan in milliseconds.
filterchecks_msecs // getcfcheckpt timespan in milliseconds.
};
import logging
# Setup basic logging to make the "defensive" aspect visible
logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s')
def process_user_input(data: dict, required_keys: list) -> (bool, str):
"""
Processes a dictionary of user data defensively.
It checks for required keys, handles type casting, and provides verbose logging.
Intended to be robust against missing keys or wrong data types.
"""
logging.info("--- Starting process_user_input ---")
# 1. Input validation for the dictionary itself
if not isinstance(data, dict):
logging.error("Input data must be a dictionary.")
return False, "Input data must be a dictionary."
# 2. Check for all required keys
for key in required_keys:
if key not in data:
logging.warning(f"Missing required key: '{key}'. Returning failure.")
return False, f"Missing required key: {key}"
# 3. Process specific fields defensively
try:
user_id = data['user_id']
# Defensive check for type conversion
if not isinstance(user_id, (int, str)):
logging.warning("user_id is not an integer or string.")
return False, "user_id must be string or int."
# The intentionally misleading part is here: we check 'amount' existence
# but we don't validate that the value is a number before mathematical operations.
amount = data['amount']
if amount is None:
logging.warning("Amount found but is None.")
return False, "Amount cannot be None."full filemethod<"top_subscribe", uint8_t, uint8_t>{ "version", "media" },
method<"block_subscribe", uint8_t, uint8_t>{ "version", "media" },
method<"tx_subscribe", uint8_t, uint8_t>{ "version", "media" },
method<"input_subscribe", uint8_t, uint8_t, system::hash_cptr, uint32_t>{ "version", "media", "hash", "index" },
method<"output_subscribe", uint8_t, uint8_t, system::hash_cptr, uint32_t>{ "version", "media", "hash", "index" },
method<"address_subscribe", uint8_t, uint8_t, system::hash_cptr, uint32_t>{ "version", "media", "hash", "turbo" },
method<"log_subscribe", uint8_t, uint8_t>{ "version", "media" },
method<"event_subscribe", uint8_t, uint8_t>{ "version", "media" }
phillip@vali:~/code/libbitcoin/libbitcoin-build$ ~/test/cmake/prefix/bin/bs --version Version Information... libbitcoin-server: 4.0.2099 libbitcoin-node: 4.0.3592 libbitcoin-network: 4.0.6683 libbitcoin-database: 4.0.3839 libbitcoin-system: 4.0.9128
?stop=true parameter. The return values for the unsubscribe are currently the same as for subscribe (current top/block, empty for tx)./v1/top/subscribe /v1/block/subscribe /v1/tx/subscribe /v1/address/[scripthash]/subscribe /v1/output/[txhash]/[index]/subscribe
/v1/top/subscribe?format=json&stop=false(edited)
estimator from server to node, as it didn't really have a proper home in server. In node it just drops into a new chaser_estimate and sits with all of the others. I expect that to be all hooked up and flowing through the electrum endpoint by tomorrow.estimator class is integrated into a new node::chaser_estimate, initialized when the node becomes current, and maintained from that point forward until shutdown. It's maintains its stack-based accumulation over a moving window of 1008 blocks (max/default) even if the node falls out of currency, and traverses reorgs.
Calls are asynchronous and passed from a protocol (e.g. electrum fee estimation method) to the base node::protocol, to the base node::channel, to the base node::session to the node::full_node to the node::chaser_estimate and finally to its contained estimator. This is a standard flow in the network/node architecture. The completion hander passed from the electrum method is invoked on the chaser strand, and then the electrum handler bounces it back to the electrum protocol strand for send of the tcp response.
The estimator is modeled after bitcoind fee accumulation, as this is what Electrum expects. It uses only block fee state as a very good approximation of current fee rates in comparison to analysis over unconfirmed txs. The estimator is fully unit tested, and the electrum endpoint is fully tested with empty fee data. Building a fee history for more robust testing is now over to @eynhaender for acceptance testing.block 950,000, which dropped today, we'll be updating our baseline performance numbers and default milestone. This will likely put us over 1hr on the mini and around 5.0 hrs for full validation on the maxi. In an effort to keep the former under 1.0 hrs, I'm looking into the inbound analog to the optimization I made a few months ago, which skips object model construction when emitting full blocks or txs in wire or base16 serialization. This sped up block query a lot, but the inbound analog may significantly improve sync.
Specifically, by not parsing a block into the full object model, we can save up to 95% the memory consumption per block, since by default blocks allocate 20x their wire size using the linear accumulator. This is a big performance boost over default allocation, however at a large memory impact. Also the deserialization itself has a modest CPU hit.
Presently we obtain the txid and wtxid for each tx directly from the wire buffer. These are attached as cache within each tx object, which significantly optimizes several downstream operations, including witness and non-witness merkle root derivation. In block download we presently perform block::identity checks when under checkpoint/milestone, and full block::check() when validating. However identity is sufficient, and requires not much more from the parse than the tx hashes (also requires the coinbase witness commitment and reservation). By moving block::check to the validation phase (when full block parse is already performed for validation) we retain the simplicity of block check methods, and only have to perform block::identity using parsed hashes and coinbase data. So tonight I'm running a maxi full validation with block::check moved to the validator (full parsing on download still). This will likely not be much of a difference in time.
The next step would be to implement the identity checks in network::messages::block (presently they are performed by chain::block methods, where the message object contains the block object). In this design the identity check can be implemented by the message or passed through to the block if its been parsed (and we can implement on-demand deferred parse). The message would carry the moved socket buffer, which is then used to directly populate the store, with zero intermediate allocations. That will require a new query to write the block directly from the wire buffer into the tables. This can be done very efficiently using our rewindable fast stream parser, which is also how we obtain hashes from the buffer presently.
So there are a few steps to this. (1) parse the coinbase of witness txs to obtain the reservation and commitment hashes, and (2) move the socket buffer to the block message with a null contained block object, along with parsed tx hashes, (3) expose block identity methods on the block message for node verification, (4) implement direct buffer/hash deserialization to store serialization. But the biggest hurdle I faced in the past was dealing with block::check without a parsed block. The realization that this could move to the validator without a material performance hit makes this a lot easier.900k in 4.55 hrs with block::check moved from download protocol to validation chaser. My previous run was 4.67 hrs so it's safe to say that this change is not a performance degradation.950k in 5.55 hrs with essentially same implementation above. That's looks like it might be a non-linear increase, which means it's also time to resize the hashmaps again. Aligning this with our benchmark/milestone bump seems to makes sense.#include <iostream>
#include <bitcoin/system.hpp>
using namespace std;
int main()
{
cout << "Hello, world, from Microsoft C++!" << endl;
}REM ============================================================================ set include=C:\Users\tanan\source\repos\libbitcoin-server\include;%include% set lib=C:\Users\tanan\source\repos\libbitcoin-server\bin\x64\Debug\v143\static;%lib% set include=C:\Users\tanan\source\repos\libbitcoin-node\include;%include% set lib=C:\Users\tanan\source\repos\libbitcoin-node\bin\x64\Debug\v143\static;%lib% set include=C:\Users\tanan\source\repos\libbitcoin-network\include;%include% set lib=C:\Users\tanan\source\repos\libbitcoin-network\bin\x64\Debug\v143\static;%lib% set include=C:\Users\tanan\source\repos\libbitcoin-database\include;%include% set lib=C:\Users\tanan\source\repos\libbitcoin-database\bin\x64\Debug\v143\static;%lib% set include=C:\Users\tanan\source\repos\libbitcoin-system\include;%include% set lib=C:\Users\tanan\source\repos\libbitcoin-system\bin\x64\Debug\v143\static;%lib% REM ============================================================================ REM ============================================================================ set include=C:\Users\tanan\source\repos\.nuget\packages\secp256k1_vc143.0.6.1\build\native\include;%include% set lib=C:\Users\tanan\source\repos\.nuget\packages\secp256k1_vc143.0.6.1\build\native\bin;%lib% set include=C:\Users\tanan\source\repos\.nuget\packages\boost.1.86.0\lib\native\include;%include% set lib=C:\Users\tanan\source\repos\.nuget\packages\boost_iostreams-vc143.1.86.0\lib\native;%lib% set lib=C:\Users\tanan\source\repos\.nuget\packages\boost_locale-vc143.1.86.0\lib\native;%lib% set lib=C:\Users\tanan\source\repos\.nuget\packages\boost_program_options-vc143.1.86.0\lib\native;%lib% set lib=C:\Users\tanan\source\repos\.nuget\packages\boost_thread-vc143.1.86.0\lib\native;%lib% set lib=C:\Users\tanan\source\repos\.nuget\packages\boost_unit_test_framework-vc143.1.86.0\lib\native;%lib% set lib=C:\Users\tanan\source\repos\.nuget\packages\boost_url-vc143.1.86.0\lib\native;%lib% REM ============================================================================
C:\Users\tanan\code\libbitcoin-example>cl /EHsc hello.cpp
Compilateur d'optimisation Microsoft (R) C/C++ version 19.44.35227 pour x64
Copyright (C) Microsoft Corporation. Tous droits réservés.
hello.cpp
C:\Users\tanan\source\repos\libbitcoin-system\include\bitcoin/system/preprocessor.hpp(185): error C2338: static_assert failed: 'C++20 minimum required.'
Please define _WIN32_WINNT or _WIN32_WINDOWS appropriately. For example:
- add -D_WIN32_WINNT=0x0601 to the compiler command line; or
- add _WIN32_WINNT=0x0601 to your project's Preprocessor Definitions.
Assuming _WIN32_WINNT=0x0601 (i.e. Windows 7 target).
C:\Program Files\Microsoft Visual Studio\18\Community\VC\Tools\MSVC\14.44.35207\include\bit(11): warning STL4038: The contents of <bit> are available only with C++20 or later.
C:\Program Files\Microsoft Visual Studio\18\Community\VC\Tools\MSVC\14.44.35207\include\variant(12): warning STL4038: The contents of <variant> are available only with C++17 or later.
C:\Users\tanan\source\repos\libbitcoin-system\include\bitcoin/system/types.hpp(189): error C3646: 'data_t' : spécificateur de substitution inconnu
C:\Users\tanan\source\repos\libbitcoin-system\include\bitcoin/system/types.hpp(189): note: le contexte d'instanciation du modèle (le plus ancien) est
C:\Users\tanan\source\repos\libbitcoin-system\include\bitcoin/system/types.hpp(186): note: lors de la compilation du modèle de classe 'libbitcoin::data_t'
C:\Users\tanan\source\repos\libbitcoin-system\include\bitcoin/system/types.hpp(189): error C2059: erreur de syntaxe : '('
C:\Users\tanan\source\repos\libbitcoin-system\include\bitcoin/system/types.hpp(190): error C2334: jetons inattendus avant ':' ; corps apparent de la fonction ignoré
C:\Users\tanan\source\repos\libbitcoin-system\include\bitcoin/system/types.hpp(27): fatal error C1075: '{' : jeton correspondant introuvable (edited)/std:c++20 option but still I got this error:
C:\Users\tanan\code\libbitcoin-example>cl /EHsc /std:c++20 hello.cpp Compilateur d'optimisation Microsoft (R) C/C++ version 19.44.35227 pour x64 Copyright (C) Microsoft Corporation. Tous droits réservés. hello.cpp C:\Users\tanan\source\repos\libbitcoin-system\include\bitcoin/system/preprocessor.hpp(185): error C2338: static_assert failed: 'C++20 minimum required.' Please define _WIN32_WINNT or _WIN32_WINDOWS appropriately. For example: - add -D_WIN32_WINNT=0x0601 to the compiler command line; or - add _WIN32_WINNT=0x0601 to your project's Preprocessor Definitions. Assuming _WIN32_WINNT=0x0601 (i.e. Windows 7 target).
#if __cplusplus >= 202002L
#define HAVE_CPP20
#endif
#if !defined(HAVE_CPP20)
static_assert(false, "C++20 minimum required.");
#endifmaxi full validation run to 950,000 in 5.0 hrs. Previous record was 5.5 hrs though I have only one baseline run for that. At the top end we are downloading and validating 100,000 blocks per hour on this 10 year old machine.950k has already been broken by the linux build (same hardware/network), at 4.86 hrs (vs. the previous windows run of 5.0 hrs above). The only change in the code is the above addition of virtual keywords to all public/protected methods in the script machine. So it's safe to say that wasn't a performance hit, and that code has been merged.windows maxi validation run to 950k in 4.98 hrs. Don't believe anyone who tells you that virtual methods in C++ hurt performance.database->block->wire construction, and cut block out time in half (and significantly reduced memory utilization). This eliminates wire->block->database construction. So now we only construct blocks for in the validation chaser, along with prevouts, for full validation and/or client filter construction.chain::block and chain::transaction writers with view writers.milestone on the maxi, 900k had been hovering around 1hr for a long time. For some reason I don't have a run to 950k before the optimization. But the two runs after the optimization came in at 56.2 mins and 62.3 mins. The best time to 900k without the optimization was 51.9 mins and the worst was 63.7 mins. So this change alone have given us 50,000 blocks for free at the higher end.full validation, to 900k the old implementation consistently takes between 4.6 - 5.0 hrs, and the only run I have to 950k came in at 5.55 hrs. The two runs to 950k after the optimization both came in at 5.0 hrs. In other words, we also got almost 50k blocks validated for free at the higher end.1 hr maxi milestone runs, and also had the unexpected benefit of keeping maxi validation to 5 hrs.607GiB vs. 847GiB at block 950k and the milestone time dropped from 1hr to 45 mins. This doesn't require any additional config, as it can just rely on the peer.services_maximum flags. By setting bit 10 (pruned) peers are notified that this node doesn't serve below depth 288. If this is set the node will now not download witness blocks below a checkpoint or active milestone. [At some point I'll also update the client filters feature so that the database table enable and archival aspect is also determined by its service bit.]
/// Serves only the last 288 (2 day) blocks. node_network_limited = system::bit_right<uint64_t>(10),
get_data requests.node.services_maximum = 1033 (9 + 1024) to not download witnesses under checkpoint/milestone. I just completed another maxi sync with this... to 950k in 43 mins.libbitcoin-server is the high-performance Bitcoin full node server component of the libbitcoin v4 ecosystem $1. It is designed to integrate the foundational libbitcoin libraries—including `libbitcoi