Bitcoin full node and query server
June 2025
Archive of the libbitcoin Slack #general channel.
Sunday, June 1, 2025
evoskuil06:57
Tapscript now also validating. Still have to update the unit tests, but Taproot is up and validating blocks on mainnet now.
(edited)
evoskuil14:46
That's just the difference between writing software and producing a software product.
💯 1
Monday, June 2, 2025
evoskuil14:23
First complete (taproot active) validation sync, to
(edited)block 896,070 complete in 17,512 secs. Before enabling taproot it was 17,331 secs. So it will be well under 5 hrs on the maxi for full validation to block 900,000, and as I suspected, the additional taproot signature validation was not a material additional cost.🔥 1
Note that the maxi is a slow bus (pcie3) 8+ year old Win10 pc, with 2.1GHz threads, avx2, and without shani. Its advantage is that it has 32 dual cores and 256GiB RAM.
evoskuil14:34
Snapshot time
88 secs, stop time 15 secs. Store create 9 secs, and initial header sync (up to 200 competing peers) 83 secs. No seeding on this run, though that's usually only a few seconds.evoskuil14:41
Note that taproot spends are pretty common now, but they constitute only about 7.5% of the chain, and the validation costs are slightly lower.
(edited)1 + (221,000,000 taproot / 2,982,883,321 total) * 4.8 hrs = 4.87 hrs1 reply · evoskuil · last Jun 02, 2025
evoskuil19:35
4.90 hrs total to block 899,536 on the maxi (current top). I estimate that on a modern machine of the same size RAM and threads, this would be about half the time, so about 2.5 hrs. I think it's feasible presently to complete full validation to 900k in under an hour on a 2.5Mbps WAN.Tuesday, June 3, 2025
evoskuil01:16
Getting a (almost) 900k run on the mini with taproot enabled. Passing 750k validated and running about 6 minutes ahead of the previous record. Likely due to some small optimizations in signature hash caching along the way.
(edited)evoskuil06:51
Despite adding taproot the 850k portion achieved the same time as the previous record due to other improvements. Apart from the point table, table heads haven't yet been optimized for the 900k level so we can probably get total time down some.
1 reply · Marnix · last Jun 03, 2025
M
Marnix19:44
I was looking and referring to libbitcoin-system, https://github.com/libbitcoin/libbitcoin-system/tree/master/src/wallet.
address generation, is with explorer indeed my bad
Wednesday, June 4, 2025
evoskuil02:21
There are no segwit or taproot updates to the system wallet capabilities, which extends to explorer.
12 replies · neillm, evoskuil, Marnix · last Jun 04, 2025
neillm02:30
neillm11:49
bx of course isn't updated, but these used to work: https://github.com/libbitcoin/libbitcoin-explorer/blob/master/src/commands/ec-to-witness.cppneillm11:49
neillm11:50
evoskuil14:58
Yeah, I totally forgot. I was recalling all the times where people asked and I had to tell them no. I even did a ton of work on it aboot 5 years ago when I was working on the client side. That got interrupted as I moved from bx to system and then to node via network and database. Never got back to it.
👍 1
1 reply · neillm · last Jun 04, 2025
evoskuil23:08
I've been working on network protocols, which is super straightforward compared to optimizing the node. I've added support for the handing the
(edited)sendaddrv2 message in the version handshake, by adding a derived protocol_version_70016 and which will be functional once I add derived protocol_address_in_70016 and protocol_address_out_70016. I've also implemented protocol_header_out_70012.
Next is protocol_block_out, which is very similar to header_out (and I completed the query for it along with the header_out, since they are basically the same). Then just need to add header announcements (sendheaders message gets incorporated in protocol_header_out_70012) and block announcements in protocol_block_out (which is only for blocks-first support).
At that point it's a fully functional blocks-only node, with blocks first and headers first protocol support. We'll want to add the compact filters support (bip157 at protocol level 70015) because we have the store/query available. But there's no reason to implement the compact blocks, mempool, or tx relay protocols until we have a tx pool. Everything else is complete up to bip155 (sendarrdv2, @ network level 70016) and bip340/341/342 (taproot, consensus). I expect all of the above to be complete in a couple of weeks.🚀 3
Thursday, June 5, 2025
evoskuil00:02
Mini milestone sync to 899,500 at 59 mins with 28 sec snapshot and 20 sec stop time. That's using a 50k download window. With an unrestricted window it will beat 60 mins to 900,000 (sometime tomorrow).
🎉 3
Huddle with pmienk · 15 min16:55
Friday, June 6, 2025
u32Luke14:55
Does anyone have an estimation of how long Core/Bitcoind can do full sync/validation before it eventually halts?
C
cguida16:18
Depends wildly on your hardware. On a[n optimal] US$350 machine you can expect ~18 hours if you turn off assumevalid
Saturday, June 7, 2025
u32Luke03:06
@cguida @evoskuil has said that at some point Core will be unable to sync due to linear growth in chain size. I'm wondering if anyone has done some math on this
15 replies · neillm, cguida, u32Luke, evoskuil +1 · last Aug 13, 2025
neillm03:09
Software changes over time -- personally, I wouldn't speculate on this. That may be true for current rates, but I expect that will change in time before any kind of crisis point.
👍 1
C
cguida03:09
I think it has less to do with the chain size and more to do with the utxoset size. Given that, I'd say it depends mostly on how bad the spam attacks get
C
cguida03:10
Core kinda cheats by not validating most of the chain during IBD though. So given that weaker security model it could last quite a lot longer
u32Luke03:12
Bitcoin takes off, gets captured by institutions, and fights the plebs trying to fully validate
👀 1
neillm03:14
Fortunately,
libbitcoin is here for a reason, and if you've ever tried building something on Bitcoin (core), core is a large web and doesn't play that well (imho).🙏 1👍 1
u32Luke03:17
This is an incredible weak point. I wish more of the op_return/knots energy would be directed to development here.
Is there a roadmap for V4 release?
neillm03:25
No, not officially a roadmap (that I know of), but I believe the v4 node is on track for a release by the end of the year. It's been completely overhauled so many times since v3, so it's hard to track exactly, but it appears we're narrowing the functionality down
🙏 1
neillm03:27
This is an incredible weak point. I wish more of the op_return/knots energy would be directed to development here.
Bring it? 🤷♂️
u32Luke03:29
Yes! I'm not as good with C++, but maybe I can help with documentation or developer relations
👍 1
evoskuil03:48
If they don’t rearchitect it then eventually it won’t be able to validate. They resisted doing this forever, instead working toward mechanisms that weaken the security model by not validating. But now that we have shown that it’s a software issue and not a Bitcoin issue, they have been forced to face these design flaws. @neillm is right, someone will eventually do this work at some point. They are working on a technique that allows someone to download the uxto state produced by someone else, and assume valid faster by relying on this. But of course this presumes someone can actually produce it. And it assumes there’s a reasonably trustworthy source for it. Then they will eventually do the significant amount of work required to not only validate concurrently, but do so at full capacity. This means making the network stack asynchronous, and creating a massively concurrent store.
😓 2💯 2
Kurtis10:31
I wish more of the op_return/knots energy would be directed to development here.
Hi @neillm , I'm a designer (with a background in product management/ business analysis), currently helping knots with branding and to improve their UX.
I want to help here too. If anyone building wants input on anything, ping me - would be glad to help
(edited)👍🏻 1👍 1🙏 1
Monday, June 9, 2025
u32Luke03:33
@neillm are these community servers still up?
https://github.com/libbitcoin/libbitcoin-server/wiki/Community-Servers
13 replies · neillm, u32Luke · last Sep 04, 2025
neillm03:35
They are, but they are unresponsive, so they aren't. For some reason, the firewall (or something) is not permitting access to them. I am moving them over to new servers (this month) that should be accessible at some point after that (likely with a new firewall entirely). Apologies for now.
👍 2
neillm03:47
No, but I also didn't monitor logs constantly. I would have a fixed peer that was one of my core nodes, so in that sense, no.
(edited)👍 1
neillm12:45
I believe these servers are reachable now -- please test them and let me know. Note that mainnet2/testnet2 is no longer mapped in DNS, but the IP address is the same (72.65.246.84). Mainnet2 and Testnet1 are still catching up:
$ version3/install/bin/bx fetch-height tcp://mainnet1.libbitcoin.net:9081 'WrOzx:DPxBh)R$vL9JEUuclRVx5swjN+v^Xak3VM' 903525 $ version3/install/bin/bx fetch-height tcp://72.65.246.84:9091 897422 $ version3/install/bin/bx fetch-height tcp://72.65.246.84:19091 3611865 $ version3/install/bin/bx fetch-height tcp://mainnet1.libbitcoin.net:19091 2951381
Wednesday, June 11, 2025
evoskuil03:43
You just set the database address table bucket count above 1. But this should be properly sized for perf reasons, and the initial table size should be as well. Otherwise it could be very slow. I’ll post some current values for you in a bit.
2 replies · Marnix, evoskuil · last Jun 11, 2025
evoskuil04:47
# disabled by default (address) #----------------------------- address_buckets = 1,315,297,540 address_size = 30,000,000,000 # disabled by default (compact filters) #----------------------------- filter_bk_buckets = 850,001 filter_bk_size = 6,810,000 filter_tx_buckets = 850,001 filter_tx_size = 2,700,000,000(edited)
👍 1
M
Marnix17:34
For who is interested, I started writing documentation for Libbitcoin (v4).
https://github.com/MarnixCroes/libbitcoin-doc
feedback and contributions welcome
🔥 5❤️ 1
evoskuil18:24
Back from 6 days on the road. Time to get the network protocols wrapped up for node!
🎉 3
Thursday, June 12, 2025
Huddle with pmienk, evoskuil, Marnix, Enligths, parsevalbtc · 112 min16:53
3 replies · neillm, Enligths, parsevalbtc · last Jun 12, 2025
P
parsevalbtc18:21
CFLAGS="-msse4.1 -msha -mavx512f -mavx512dq -mavx512bw -O3" CXXFLAGS="-msse4.1 -msha -mavx512f -mavx512dq -mavx512bw -O3" ./install-cmake.sh --prefix=/home/... --disable-shared --build-boost --build-secp256k1 | tee ./compile_logs/install_20250602_1.log
Friday, June 13, 2025
evoskuil02:07
Since our seed nodes are not active yet I've been using the current bitcoind embedded addresses in my config, which works fine:
[network] seed = seed.bitcoin.sipa.be:8333 seed = dnsseed.bluematt.me:8333 seed = dnsseed.bitcoin.dashjr-list-of-p2p-nodes.us:8333 seed = seed.bitcoin.jonasschnelli.ch:8333 seed = seed.btc.petertodd.net:8333 seed = seed.bitcoin.sprovoost.nl:8333 seed = dnsseed.emzy.de:8333 seed = seed.bitcoin.wiz.biz:8333 seed = seed.mainnet.achownodes.xyz:8333
1 reply · evoskuil · last Jun 13, 2025
evoskuil21:35
These have been incorporated into the node config overrides of default network settings. Node starts up, gets seeded, drops seeds, gets outbounds, and obtains the first header in < 4 seconds from startup.
evoskuil14:54
Ping me with your desired email address if you need an invite. It's on the google thing.
evoskuil20:56
There are a large number of peers, entirely in the version 27.1 - 27.2 range, that have a significant clock skew in their version message. There are a very small number over the entire network that exceed even +/- 1 minute, but these versions commonly reach as high as 5,100 minutes behind (never ahead). We've started dropping peers for excessive skew, which I've made configurable. Anyone aware of such an issue with Core nodes?
(edited)Just a small sampling...
These are more broadly spread across versions and small enough to be expected (I've also seen some small deviations like this in the positive (ahead) range):
evoskuil21:35
replied to a thread: view
These have been incorporated into the node config overrides of default network settings. Node starts up, gets seeded, drops seeds, gets outbounds, and obtains the first header in < 4 seconds from startup.
evoskuil22:43
Just set a
milestone 900k record on the maxi of 59.8 minutes. This is almost as fast as the mini (59 mins) despite not having shani, and having half the RAM/SSD/Bus speed. That also included seeding against the 8 nodes, which was completed in 3 seconds.🔥 4🎉 1
Sunday, June 15, 2025
evoskuil21:50
Conditionality in the network protocol is about as thick as in consensus.
enum : feature_t
{
no_flags = 0,
all_flags = system::bit_all<uint32_t>,
/// -----------------------------------------------------------------------
/// Set at 106 (n/a) [version, verack].
version = system::bit_right<uint32_t>(0),
/// Set at 106 (n/a) [ping].
ping = system::bit_right<uint32_t>(1),
/// Set at 106 (n/a), disabled by config.enable_address [addr].
address = system::bit_right<uint32_t>(2),
/// Set at 209 (n/a), disabled by config.enable_address [getaddr].
get_address = system::bit_right<uint32_t>(3),
/// Set at 311 (n/a), deprecated, enabled by config.enable_alert [alert].
alert = system::bit_right<uint32_t>(4),
/// Set at 31402 (n/a), disabled by enable_address [addr.timestamp].
address_timestamp = system::bit_right<uint32_t>(5),
/// Set at 31800 (n/a) [headers, getheaders].
headers = system::bit_right<uint32_t>(6),
/// Set at 60001 (bip31), nonce and pong messages added [ping.nonce, pong].
ping_pong = system::bit_right<uint32_t>(7),
/// Set at 70001 (bip37) [mempool].
mempool = system::bit_right<uint32_t>(8),
/// Set at 70001 (n/a) [notfound].
not_found = system::bit_right<uint32_t>(9),
/// Set at 70002 (n/a) [multiple inv response].
mempool_multiple = system::bit_right<uint32_t>(10),
/// Set at 70002 (bip61), deprecated, enabled by config.enable_reject [reject].
reject = system::bit_right<uint32_t>(11),
/// Set at 70013 (bip133), enabled by non-zero config.minimum_fee [feefilter].
fee_filter = system::bit_right<uint32_t>(12),
/// -----------------------------------------------------------------------
/// Set by handshake message negotiation, at 70012 (bip130) [sendheaders].
send_headers = system::bit_right<uint32_t>(15),
/// Disabled by config.enable_compact.
/// [sendcmpct, blocktxn, cmpctblock, getblocktxn]
/// Set by post-handshake message negotiations, at 70014 (bip152).
/// Negotiation can establishes multiple versions (only one is defined).
/// Version information is maintained by the compact blocks protocol(s).
send_compact = system::bit_right<uint32_t>(16),
/// Disabled by config.enable_address_v2 (independent of enable_address).
/// Set by handshake message negotiation, at 70016 (bip155) [sendaddrv2, addrv2].
send_address_v2 = system::bit_right<uint32_t>(17),
/// Disabled by config.enable_witness_tx.
/// Set by handshake message negotiation, at 70016 (bip339) [wtxidrelay].
send_witness_tx = system::bit_right<uint32_t>(18),
/// [sendtxrcncl, reqrecon, sketch, reqsketchext, reconcildiff].
/// Disabled by config.enable_erlay, requires wtxidrelay [not supported].
/// Set by handshake message negotiation, at 70016 (bip330).
send_reconcile = system::bit_right<uint32_t>(19),
/// -----------------------------------------------------------------------
/// [getblocks, inv, getdata, block, tx]
/// Set by service::node_network [0], at 106 (excluding 32000 - 32400).
blocks = system::bit_right<uint32_t>(13),
/// Disabled by config.enable_relay.
/// Set by service::node_network [0], at 106.
/// Negated by the version.relay flag, at 70001 (bip37) [version.relay].
transactions = system::bit_right<uint32_t>(14),
/// [merkleblock, filterload, filteradd, filterclear]
/// Set at 70001 (bip37) [not supported, queries allowed but ignored].
/// Set by service::node_bloom [2], at 70011 [otherwise false] (bip111).
bloom = system::bit_right<uint32_t>(20),
/// Set by service::node_witness [3], adds MSG_WITNESS_* inv (bip144).
witness = system::bit_right<uint32_t>(21),
/// Set by service::node_client_filters [6], at 70015 (bip157).
filters = system::bit_right<uint32_t>(22)
}; (edited)Thursday, June 19, 2025
Huddle with pmienk, Marnix, neillm, evoskuil, Rowan Voskuil · 162 min16:53
1 reply · Rowan Voskuil · last Jun 19, 2025
Sunday, June 22, 2025
evoskuil17:50
I've wrapped up the protocol work. This adds
protocol_block_out_106 , protocol_block_out_70012 (disables block inv announcement when sendheaders received), protocol_header_out_31800 and protocol_header_out_70012 (enables headers announcement when sendheaders received).
I also added automatic snapshotting when the node is "recent". This includes new is_recent() methods from protocol->session->full_node, a new chaser notification type chase::recent , which is posted by confirmation when a just-organized block is at the configured maximum height or the chain is current (based on is_current() , which is based on configured time from the top confirmed block timestamp (defaults to 60 minutes). Recency is logged at confirmation chaser startup and if it changed to true.
The chase::recent notification is therefore posted 0..1 times for the process. This is caught only by the snapshot chaser, which then takes a snapshot. The snapshot causes all channels to be dropped, the network paused, and restarted when complete. Attachment of protocols to channels tests is_recent() when attaching inbound channels, if delay_inbound is set in node.config , and only attaches block and header inbound protocols when chain is recent. So the recency snapshot serves to reset channels for inbound traffic, and unlike others is not optional/configurable.
I also added duration based events to log the time cost in milliseconds of the snapshot (and a store restart following a disk full resume). Snapshots are typically around a minute. Also added console logging for recency, initial positive state or transition. Finally, added base session_inbound connection acceptance disable and verbose logging, with override in node to disable until is_recent , also only if delay_inbound is configured.🙌 3
evoskuil18:37
and only attaches block and header inbound protocols
Should say "outbound protocols". IOW, if
(edited)delay_inbound is set we don't accept inbound connections or attach outbound protocols (to outbound session channels) until the confirmed chain is current or at maximum configured height ("recent").👀 1
Thursday, June 26, 2025
Huddle with pmienk, Marnix, evoskuil, eynhaender, parsevalbtc · 181 min16:52
Friday, June 27, 2025
evoskuil00:06
On the call today I was trying to remember how the
(edited)chase::recent event is consumed. It is handled by the snapshot chaser, which causes a snapshot to be taken (once the node is "recent" - which means current or max height). A snapshot causes all channels to be dropped, which results in no store activity, which allows the table heads and bodies to be flushed to disk, and the heads copied. This also means that the channels are reopened as "recent" - which results in outbound protocols being attached. This is enforced - not configurable. The node will always snapshot once it is recent.evoskuil00:14
We don't allow block to be taken in from incoming channels (protocol never attached). We don't attach outbound protocols (txs, blocks|headers) unless the node is recent. Channels have time limits, so don't stay attached perpetually. Recency can regress (e.g. if the node is disconnected) and if it's significant the channels will adapt to the regress as they are connected. The only thing that is sent one time and does not adapt is the
(edited)chase::recent notification, and that is only used to establish a snapshot, which also resets all channels. We wouldn't want a cycling recency state to cause cycling snapshots. We want just one.Monday, June 30, 2025
evoskuil17:08
Just set a
milestone 900k record on the maxi of 56.6 minutes. Consistently under 57 mins now. Somehow it keeps getting a little faster. As I add new features, which should have a cost, I also end up optimizing little things. I've started dropping peers for protocol non-conformance. Specifically a lot of Core 25.0 and 25.1 peers send tx inventory despite it not being requested. Also a large number of heavily clock-skewed Core 27.1 and 27.2 nodes, though it appears that these are tied to single owner with a lot of addresses, and not necessarily the node impl.🔥 2