Bitcoin full node and query server
February 2023
Archive of the libbitcoin Slack #general channel.
Wednesday, February 1, 2023
evoskuil05:34
Verify more after getting the node up and running.
Added the alert protocol, so we now see the alert messages if protocol is below a certain level (peers assume we are core and need an upgrade, lol). Also improved the presentation for reject message (deprecated), and improved logging for a log of others. Give a better look at what's happening on the wire.
Added config for
network.user_agent string (was hardwired). If you see /panopticon:2.0.2.0/ out there that's me 🙂 Also added config for network.enable_alert (deprecated message) and network.enable_reject (deprecated message) and renamed relay transactions to network.enable_relay.Verified behavior of
network.self and network.inbound_port , which set up inbound connections (self is how own node is advertised). Set up my port forwarding and firewall, listening now for inbound. If someone wants to set up a manual connection to me let me know.Outbound and seed sessions are thoroughly verified as of yesterday. Manual connections verified today. Just waiting on some inbound now to verify the inbound sessions live, then sessions and base protocols are a wrap.
Verified incoming txs yesterday. No problems. Decided to set min services requires on incoming to zero, because it makes sense and we were dropping bots like bitnodes.io.
So I consider network a wrap at this point, apart from tweaking log messaging and levels as we go. I’ll spend a couple days adding log levels, console input, and making a couple enhancements to address pooling (timeout, dropping failed addresses, storage of services, type and time, shared pointer-based address set parameterization to avoid copies).
🎉 1
Thursday, February 2, 2023
evoskuil11:09
More good stuff today. Decided to do the work to buffer and total outgoing data per channel. This is the basis of the DoS strategy that was put off forever. Turned out to be super simple. No change to the channel/proxy public interfaces, as it remains just asynchronous. Handlers still invoked in order for each queued write. It's just that they are queued in a deque as opposed to the opaque asio work buffer. This also closes an issue that probably existed with interleave of large (>64k, at least on Win32) writes on a given channel. It's all there now, including the config, tracking of total send and current backlog, log messages, etc. Just need to decide on the formulation for channel dropping based on buffer excess and peer rate limit.
👍 1
Also did some work to make the address protocol a little more advanced, optimized message checksumming (double sha) for empty messages (always the same int). Was able to derive this as constexpr using the system hash and endian utils, which is cool - not embedding magic numbers.
pmienk17:00
Topic: Dev Weekly Status
Time: February 2, 2023 01:00pm Pacific Time (US and Canada)
Join Zoom Meeting
https://us05web.zoom.us/j/87666370902?pwd=bEZDUlJ0MGJ5V3dVMUhVdHl5QzdFdz09
Meeting ID: 876 6637 0902
Passcode: 85U0bF
Sunday, February 5, 2023
evoskuil08:23
Worked off some technical debt in address pool management today. This included adding services and timestamps to address pool serialization/deserialization, which required a new config::address class (these classes wrap objects to provide command line and settings file parsing, but in this case for the hosts file. Now we can avoid connecting to hosts with insufficient services, which will reduce effective connection time (by not wasting connections), which is a material factor in initial block download.
Also implemented checkout/in of addresses form the pool. This reduces the instances of attempting connection to hosts we are already connected to, which will also reduce effective connection time.
Actually we get pretty good connection time regardless, almost instant, because of batch connection. But this will allow us to reduce the batch quantity to achieve the same rate. Probably from 10 down to 5 or less concurrent attempts per intended outbound connection.
So hosts file format now like this (ip/timestamp/services):
145.255.105.37:8333/1674843314/1032 [2601:18c:8b80:29cb:c149:75bc:5486:d716]:8333/1673430801/1037 80.220.68.138:8333/1675384869/1033 88.10.167.189:8333/1675548091/1033 185.107.56.236:8333/1673037105/1037 85.253.135.250:8333/1674765984/1032 143.244.60.173:8333/1675336113/1032 64.250.72.172:8333/1675567362/1033 [2001:9b1:28fd:1401:51c9:f320:52e4:7c19]:8333/1673262494/1032 73.222.63.19:8333/1674103782/1033 27.122.115.1:8333/1675295192/1037 5.9.109.174:8333/1675542544/18446744073709551615 24.231.203.26:8333/1675204643/1033 37.19.206.112:8333/1675286235/67109965 211.30.164.119:8333/1673641425/1032 172.100.101.232:8333/1674691795/1037 87.95.184.34:8333/1673688936/1033 [2404:440c:2329:2300:ca0f:33d5:3f8a:f62b]:8333/1674578566/1033 [2403:5803:1e29:aa::7ce]:8333/1673974513/1033 70.175.51.156:8333/1674456305/1033 94.31.82.133:8333/1675498459/1033 46.56.243.16:8333/1673850313/1032
👍 1
Tuesday, February 7, 2023
evoskuil06:13
================= startup Mon Feb 6 22:11:39 2023 ================== Using config file: "bn.cfg" Press CTRL-C to stop the node. Please wait while the node is starting... Bypassed seeding because outbound connections disbled. Maintaining manual connection to [127.0.0.1:8333] Accepting up to 1 connections on port 8333. Not configured for outbound connections. Node is started. Peer [127.0.0.1:8333] protocol version (70015) user agent: "/panopticon:2.0.2.0/" Peer [127.0.0.1:57492] protocol version (70015) user agent: "/panopticon:2.0.2.0/" Manual channel start [127.0.0.1:8333] success Inbound channel start [127.0.0.1:57492] success Please wait while the node is stopping... Manual channel stop [127.0.0.1:8333] service stopped Inbound channel stop [127.0.0.1:57492] peer disconnect Node stopped successfully.
Loopback just works, which was nice to see. Very handy for protocol debugging.
evoskuil09:38
Worked off some test backlog, added whitelist config, standardized file path config across [log], [network], [database] settings (each has one directory), reduced and improved logging, restricted address request and acceptance for inbound connections. Also did a lot of live runs. Tested out intentional loopback to own node. Spent some time verifying that I'm properly listening on ipv4 and ipv6, and that the reason I'm not making any ipv6 outbound connections is just that I don't have an ipv6 default gateway on my LAN.
Finally found a legit issue in the network stack. The three main session types (inbound, outbound, manual) have a timer (from session base) embedded in the main retry loop, to prevent tight looping in the case of conditions like network offline, no addresses, etc. But the timer is a member of the session, so it has one clock. This gets reset each time there is a new failure event. That's common in the case of these failures, because sessions manage groups of connection attempts. So I need to rethink the approach to these delays, or just remove them. They are mitigated somewhat by the fact that each action is posted to the p2p (network or node) strand.
(edited)evoskuil09:47
The only approach that I can think of that makes sense is to create a timer for each event, register timer cancel with the stop subscriber, and capture the timer in its stop and timeout handlers. This keeps stop responsive and allows for a delay on each manual connect failure and batch connect failure, and won't get reset by any other events. This also allows the session member timer to be removed. Th overhead shouldn't be an issue, as the whole point is intentional idling.
(edited)evoskuil09:56
The work I did on using shared pointer movement of address messages reduced memory consumption by about 20%, and reducing console logging verbosity by default makes everything more responsive. The node is basically an address relay. Otherwise it just ping/pongs and maintains connections. That's a pretty good foundation for adding the block/header/tx relay protocols. I've been maintaining between 500-1000 concurrent connections overnight, with no leakage or problems at all.
👍 1
Wednesday, February 8, 2023
evoskuil23:40
I noticed that there are a few class C networks that repeatedly connect and disconnect with various user agents from various hosts on the class C. Since one may want to white/blacklist subnets, I've added CIDR notation to the
config::authority class. https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing#CIDR_notationBy excluding /0 (not useful) the default of zero implies a single host, but otherwise the CIDR value is retained as a member and used in the ==/!= overload. For consistency it's also added to serialization. Comparison of two subjects results in false unless they are identical. Port is allowed to be combined in serialization and comparison, with an unspecified port (retained as zero) matching all ports.
evoskuil23:50
So...
[network] # block one host incoming/outgoing from/to all ports. blacklist = 209.222.252.42 # block one host incoming/outgoing from/to port 8333. blacklist = 209.222.252.42:8333 # block class C subnet incoming/outgoing from/to all ports. blacklist = 209.222.252.0/24 # block class C subnet incoming/outgoing from/to port 8333. blacklist = 209.222.252.0:8333/24 # allow one host incoming from all ports. whitelist = 209.222.252.42 # allow one host incoming from port 4242. whitelist = 209.222.252.42:4242 # allow class C subnet incoming from all ports. whitelist = 209.222.252.0/24 # allow class C subnet incoming from port 4242. whitelist = 209.222.252.0:4242/24
evoskuil23:55
While parsing is supported, the following form isn't very useful: 209.222.252.42/24, as it is the same as 209.222.252.42 in terms of matching. Also, incoming ports aren't so useful (outbound may be for cooperative scenarios).
(edited)Thursday, February 9, 2023
evoskuil08:31
================= startup Thu Feb 9 00:27:57 2023 ================== Using config file: "bn.cfg" Press CTRL-C to stop the node. Please wait while the node is starting... Bypassed seeding because outbound connections disabled. Accepting up to 10 connections on port 8333. Not configured for outbound connections. Node is started. Dropping blacklisted connection [162.218.65.24:18248] Dropping blacklisted connection [162.218.65.31:23281] Dropping blacklisted connection [188.166.69.73:43191] Peer [185.220.101.184:28069] protocol version (70016) user agent: /Satoshi:24.0.0/ Inbound channel start [185.220.101.184:28069] success Sending (1000) addresses to [185.220.101.184:28069] Inbound channel stop [185.220.101.184:28069] peer disconnect Dropping blacklisted connection [162.218.65.24:14308] Dropping blacklisted connection [162.218.65.31:17041] Dropping blacklisted connection [162.218.65.31:16394] Dropping blacklisted connection [162.218.65.31:63358] Dropping blacklisted connection [162.218.65.31:5248] Dropping blacklisted connection [162.218.65.31:37596] Dropping blacklisted connection [162.218.65.31:63656] Dropping blacklisted connection [162.218.65.24:5154] Dropping blacklisted connection [162.218.65.31:27888] Dropping blacklisted connection [162.218.65.31:54807] Dropping blacklisted connection [162.218.65.31:27783] Dropping blacklisted connection [162.218.65.31:39967] Dropping blacklisted connection [162.218.65.31:39529] Dropping blacklisted connection [162.218.65.24:38102] Dropping blacklisted connection [162.218.65.31:6165] Dropping blacklisted connection [162.218.65.24:54071] Dropping blacklisted connection [162.218.65.31:57533] Dropping blacklisted connection [162.218.65.24:10376] Dropping blacklisted connection [162.218.65.31:51198] Dropping blacklisted connection [162.218.65.24:3481] Dropping blacklisted connection [162.218.65.31:48515] Dropping blacklisted connection [162.218.65.24:62271] Peer [88.99.167.175:35307] protocol version (70016) user agent: /bitnodes.io:0.3/ Dropping blacklisted connection [162.218.65.31:54326] Dropping blacklisted connection [162.218.65.24:52904] Inbound channel start [88.99.167.175:35307] success Inbound channel stop [88.99.167.175:35307] peer disconnect Dropping blacklisted connection [162.218.65.31:10826] Dropping blacklisted connection [162.218.65.24:38934] Dropping blacklisted connection [162.218.65.31:42167] Dropping blacklisted connection [162.218.65.24:18184] Dropping blacklisted connection [162.218.65.31:43590] Dropping blacklisted connection [162.218.65.24:56634] Dropping blacklisted connection [162.218.65.31:39968] Dropping blacklisted connection [162.218.65.24:42574] Dropping blacklisted connection [162.218.65.31:26926] Dropping blacklisted connection [162.218.65.24:50835] Dropping blacklisted connection [162.218.65.31:44381] Dropping blacklisted connection [162.218.65.24:7831] Dropping blacklisted connection [162.218.65.31:64237] Dropping blacklisted connection [162.218.65.24:39074] Dropping blacklisted connection [162.218.65.31:41292] Dropping blacklisted connection [162.218.65.24:34834] Dropping blacklisted connection [162.218.65.31:40323] Dropping blacklisted connection [162.218.65.24:4489] Dropping blacklisted connection [162.218.65.31:7271] Dropping blacklisted connection [162.218.65.24:17343] Dropping blacklisted connection [162.218.65.31:32737] Dropping blacklisted connection [162.218.65.24:59027] Dropping blacklisted connection [162.218.65.31:40058] Dropping blacklisted connection [162.218.65.24:21570] Dropping blacklisted connection [162.218.65.31:62057] Dropping blacklisted connection [162.218.65.24:22570] Dropping blacklisted connection [162.218.65.31:40824] Dropping blacklisted connection [162.218.65.24:62587] Dropping blacklisted connection [162.218.65.31:34703] Dropping blacklisted connection [162.218.65.24:11505] Dropping blacklisted connection [162.218.65.31:63285] Dropping blacklisted connection [162.218.65.24:15993] Dropping blacklisted connection [162.218.65.31:64586]
That's obviously some verbose logging. Took a little longer than I expected, but had to learn a few more ASIO and IP things, which simplified some code and got this subnet masking working for black and whitelisting.
[network] # /bitnodes.earn.com:0.1/ sends unsolicited addresses. blacklist = 188.166.69.73 # /nodes.mom.market:0.2/ sends unversioned sendaddrv2. blacklist = 94.237.44.67 # /Satoshi:21.99.0/ sends unversioned sendaddrv2. blacklist = 173.48.121.181 # wallet networks that constantly drop and reconnect blacklist = 209.222.252.0/24 blacklist = 162.218.65.0/24 blacklist = 91.198.115.0/24
evoskuil08:59
I just noticed someone pushing private addresses into the address pool, causing me to attempt loopback connections (which get dropped by loopback detection). But blacklisting prevents these from going out (and of course you can still enable them for local work):
# en.wikipedia.org/wiki/Private_network blacklist = 10.0.0.0/8 blacklist = 127.0.0.0/8 blacklist = 172.16.0.0/12 blacklist = 192.168.0.0/16 blacklist = 169.254.0.0/16 blacklist = 100.64.0.0/10 blacklist = [fd00::]/8 blacklist = [fe80::]/10 blacklist = [::1](edited)
👍 1
evoskuil15:38
Accepted (0 of 1 of 1) addresses from [68.105.220.26:8333]. Dropping not whitelisted connection [209.222.252.85:64448] Dropping not whitelisted connection [209.222.252.217:15312] Accepted (0 of 2 of 2) addresses from [54.212.175.16:8333]. Dropping not whitelisted connection [209.222.252.70:33723] Dropping not whitelisted connection [209.222.252.85:37115] Dropping not whitelisted connection [209.222.252.217:32660] Dropping not whitelisted connection [88.99.167.175:15149] Accepted (0 of 1 of 1) addresses from [81.170.255.240:8333]. Dropping not whitelisted connection [209.222.252.70:26772] Accepted (1 of 4 of 4) addresses from [47.203.150.77:8333]. Accepted (0 of 1 of 1) addresses from [81.6.45.155:8333]. Dropping not whitelisted connection [209.222.252.85:30428] Dropping not whitelisted connection [209.222.252.217:31736] Accepted (0 of 1 of 1) addresses from [77.0.64.205:8333]. Accepted (0 of 1 of 1) addresses from [88.67.80.89:8333]. Accepted (0 of 1 of 1) addresses from [207.182.146.85:8333]. Accepted (0 of 1 of 1) addresses from [66.38.94.13:8333]. Accepted (0 of 1 of 1) addresses from [185.209.70.17:8333]. Dropping not whitelisted connection [209.222.252.70:26812] Accepted (0 of 2 of 2) addresses from [38.114.113.52:8333]. Dropping not whitelisted connection [209.222.252.85:12199] Dropping not whitelisted connection [209.222.252.217:64097] Accepted (0 of 1 of 1) addresses from [34.222.125.43:8333]. Accepted (0 of 1 of 1) addresses from [174.89.180.238:8333]. Accepted (0 of 1 of 1) addresses from [65.21.142.118:8333]. Dropping not whitelisted connection [209.222.252.70:18069] Dropping not whitelisted connection [209.222.252.85:57776] Dropping not whitelisted connection [209.222.252.217:52986] Accepted (0 of 1 of 1) addresses from [47.149.172.35:8333]. Accepted (0 of 1 of 1) addresses from [99.29.112.187:8333]. Accepted (0 of 1 of 1) addresses from [144.76.58.238:8333]. Accepted (0 of 1 of 1) addresses from [185.140.253.169:8333]. Dropping not whitelisted connection [209.222.252.70:61538] Dropping not whitelisted connection [209.222.252.85:22483] Dropping not whitelisted connection [209.222.252.217:36892] Accepted (0 of 1 of 1) addresses from [99.6.8.35:8333]. Accepted (1 of 3 of 3) addresses from [184.147.137.197:8333]. Accepted (0 of 1 of 1) addresses from [14.200.28.19:8333]. Dropping not whitelisted connection [209.222.252.70:34006] Dropping not whitelisted connection [209.222.252.85:61254] Accepted (0 of 1 of 1) addresses from [111.90.158.46:8333]. Dropping not whitelisted connection [209.222.252.217:62302]
[network] whitelist = 127.0.0.1 whitelist = 209.237.0.0/16 whitelist = 94.0.0.0/8 whitelist = 188.166.69.0/24 whitelist = 173.0.0.0/8
evoskuil15:43
Whitelisting is applied to inbound channels only (blacklisting to in/out). The both use the same ip/subnet address matching. I put some openings and closures against a few common annoying subnets above and all working great. Still haven't hand a single hand/crash or unexplained behavior since I started running the node. white/black is all done, which makes traffic a lot nicer to work with. Next is moving the tight-loop-retry-delay-timers into stop subscriber captures instead of session base. Then on to log levels and dynamic console input.
pmienk18:32
Topic: Dev Weekly Status
Time: February 9, 2023 01:00pm Pacific Time (US and Canada)
Join Zoom Meeting
https://us05web.zoom.us/j/87666370902?pwd=bEZDUlJ0MGJ5V3dVMUhVdHl5QzdFdz09
Meeting ID: 876 6637 0902
Passcode: 85U0bF
Friday, February 10, 2023
evoskuil12:30
Took care of the retry timer issue today. To do this I implemented a "resubscriber" template. We had one of these (as opposed to the simpler subscriber we have) a long time back, but it was sketchy, so I removed it. With proper use of the strand now, it's straightforward. So you register a handler for notifications, and it can be removed externally by unique id, or by false return from a notification. We have a number of places where this would simply/rationalize things a bit, so I'll apply that soon. But working flawlessly with retry timers, where we now can have a retry timer for each connector/acceptor that might fail. Previously a second failure would fire the handler for the previous timer, so with a network failure it didn't do its job (no real delay). It's working great now.
Still haven't had a single hang, crash, or even unexplained behavior of the node since I got it running - and with all of these changes and additions. Also shutdown remains near instant in all scenarios. Feeling pretty good about that.
(edited)evoskuil12:43
I've pushed the numbers up to crazy levels, and it's solid and responsive. For a basic bitcoind standard config of like 8 outbound and inbound and 10k address pool it flatlines at 4mb RAM, just pulsing on ping/pong and address relay. At 1,000 outbound and 100k address pool it's about 64mb RAM.
evoskuil12:53
Forgot to mention, I also tested the retry logic by disabling/enabling the ethernet NIC during large runs, and disabling during startup and during shutdown. No increased latency or problems, just picks up where it can and cancels immediately.
evoskuil23:42
Looking at the per connection sizing, it's about 60kb per connection. Most connections receive a maximally-sized address message, which resizes the incoming buffer to 30kb, so half of the per channel memory is this. This could be scaled down during quiet periods, but it will eventually grow to max block weight (4,000,000 bytes). Currently I'm opting for avoiding dynamic allocation, with the expectation that peer counts are more traditional (on the order of 10-100). But with 1,000 peers pushing 1 max block, that's 4GB RAM. So with very high peer counts we want to provide for dynamic allocation. I'll place a config setting that shrinks the buffer after it exceeds the configured bytes, defaulting to 4,000,000.
Saturday, February 11, 2023
evoskuil01:33
Yeah, making the read buffer fully dynamic cuts the 1,000peer/10,000address memory consumption from 46Mb to 32mb. And instead of climbing asymptotically, it starts as before but drops off rapidly to a flatline as communication trickles and buffers are resized down. The basic/standard config drops from 4mb to 2mb.
evoskuil01:41
This is the 1000 peer profile. The waves at the start are sets of connection timeouts (connection attempts aren't staggered - I used to do that but I think it just causes delay).
By increasing connection batching from 10 to 50, the establishment time is reduced proportionally. The subsequent two steps are completion of waves of address messages from the new connections.
By increasing it further to 100, nearly all connections are established within the initial timeout window, so just one wave.
In other words, it takes between 50 and 100 unique addresses from the address pool (after removing blocked and provably bogus ones) to get 1 good connection. The I/O shown is verbose logging, due to the high channel count.
(edited)The reason this matters (a lot) to initial block download performance is that we want to always find the fastest available peers. Diversity doesn't matter in terms of block fill. That's only important for discovery of the header chain (which is not optimized for fast peers). To find faster peers you drop the slow ones and try another. The time spent waiting on another is costly, and if too long ends up requiring that you just maintain more channels, but this causes more work factoring. As I found in prototyping work, getting channels quickly makes a big difference. But I was never able to operate at this density before. With batches of 100 we can establish a connection almost instantly.
🙌 1
evoskuil06:36
Turns out the last image was pathological. I was hitting this issue by attempting more that 5,000 concurrent outbound connections: https://learn.microsoft.com/en-us/troubleshoot/windows-client/networking/connect-tcp-greater-than-5000-error-wsaenobufs-10055
evoskuil06:43
The I/O in pink was verbose retry logging. I've been looking into this and other causes of low address quality. Given I don't have an IPv6 default gateway, all IPv6 addresses fail immediately. I've added config to disable IPv6 so these get filtered out, and maybe that can be automated at some point, that's about 20% of the address pool. I've also determined that about 26% of all addresses don't advertise node service (which we filter out), and another 3% include "invalid" (shitcoin) service identifiers. That's 49,000 addresses out of 100,000 taking up address pool space, so I'm going to prefilter those out as the addresses are received and before a random portion is selected for retention. This will reduce connect time a good amount.
(edited)Sunday, February 12, 2023
evoskuil03:17
I updated my local win32 socket limit, tested with different timeouts, and did some feature/logging work. Came back to the 100k connections (1000 x 100 at a time) image, and it's now legit as I expected - one big spike to get most connections established.
(edited)
evoskuil03:23
There's a gradual climb as the 1000 channels build up the message buffer with 30kb address messages (one per connection) and then as these are cleared and read buffer memory shrunk on each channel, the RAM profile flatlines - with occasional spikes as channels drop out and are replaced, and as inbound connections arrive.
(edited)
evoskuil03:39
The rapid drop is curious. It's a 3 second span in which 35mb is dropped. Given that CPU is almost zero, it's all I/O (address processing). The addresses are not serialized to disk until shutdown, so the holdup is just the processing of the addresses on the network strand (single in-memory pool, so unlike the blockchain store, no concurrency). This is a pretty extreme setup, with 1000 connections being held with up to 100,000 being concurrently attempted, and it takes 5 minutes to clear their work. For a typical node startup, the 10 connections are established and addresses cleared 4 secs from process start.
Monday, February 13, 2023
evoskuil06:33
Took the time today to centralize and isolate regex parsing (only in network). This will make it much easier to replace the boost::regex with std::regex, allowing us to drop another boost dependency lib. Also added a lot of testing and fully normalized the address, authority and endpoint parsers.
// C++11: use std::regex.
using namespace boost;
// en.wikipedia.org/wiki/List_of_URI_schemes
// Schemes of p2p network and our zeromq endpoints.
#define SCHEME "(tcp|udp|http|https|inproc):\\/\\/"
#define IPV4 "([0-9.]+)"
#define IPV6 "\\[([0-9a-f:]+)]"
#define HOST "([^:?/\\\\]+)"
#define PORT ":([1-9][0-9]{0,4})"
#define CIDR "\\/([1-9][0-9]{0,2})"
// Excludes ipv4 mapped, unbracketed, and ports allowed by make_address.
bool parse_host(asio::address& ip, const std::string& value) NOEXCEPT
{
static const regex regular
{
"^(" IPV4 "|" IPV6 ")$"
};
sregex_iterator token{ value.begin(), value.end(), regular }, end{};
return token != end
&& make_address(ip, (*token)[1]);
}
// Excludes ipv4 mapped to ipv6.
bool parse_authority(asio::address& ip, uint16_t& port, uint8_t& cidr,
const std::string& value) NOEXCEPT
{
static const regex regular
{
"^(" IPV4 "|" IPV6 ")(" PORT ")?(" CIDR ")?$"
};
sregex_iterator token{ value.begin(), value.end(), regular }, end{};
return token != end
&& make_address(ip, (*token)[1])
&& to_integer(port, (*token)[5])
&& to_integer(cidr, (*token)[7])
&& ((ip.is_v4() && cidr <= maximum_cidr_ip4) ||
(ip.is_v6() && cidr <= maximum_cidr_ip6));
}
// Excludes ipv4 mapped to ipv6.
bool parse_endpoint(std::string& scheme, std::string& host, uint16_t& port,
const std::string& value) NOEXCEPT
{
static const regex regular
{
"^(" SCHEME ")?(" IPV4 "|" IPV6 "|" HOST ")(" PORT ")?$"
};
sregex_iterator token{ value.begin(), value.end(), regular }, end{};
return token != end
&& to_string(scheme, (*token)[2])
&& to_string(host, (*token)[3])
&& to_integer(port, (*token)[8]);
}That's the only regex in the full set of libs, so if anyone wants to do the conversion to std (the model is a little different), then have at it.
evoskuil06:47
Also took the time to change address storage to shared pointer per address in the circular buffer. Each address is 30 bytes, so this made each effectively 38 bytes. For a 100k address pool that's 800kb more data. This eliminates the dynamic conversion/copy cost to shared pointer I/O from the pool. But it's not worth it, as the size increase isn't offset by any noticeable performance benefit, even with huge connectivity. So backed that out.
evoskuil10:04
Took the time to fully verify all of the various ipv4/ipv6 (enabled/disabled), insufficient/unsupported service adverts, black/white-listing, via incoming/outgoing, with loopback, hosts/ports/subnets. etc. on the live net... all perfect. Gonna add the code tomorrow to the address/seed protocols to fully prefilter address message payloads. Then console input and log levels.
Tuesday, February 14, 2023
evoskuil10:24
Spent some time today on technical debt. Tracked down a debug assertion showing up in runtime. Turned out to be inconsequential. Resolved that and added logging for stopped-on-destruct of dynamic objects (in addition to the refcount logging that's implemented via CRTP). Applied the new resubscriber template for session stop handling, which allows the seed session to be removed once complete (vs. staying around until shutdown). That worked out well, so I'll end up applying that in a couple more places.
I also applied it to public channel and stop subscribers. This allows the subscriber to return true/false in the event handler, to direct continuing subscription vs. unsubscribe. It also provides a handle upon registration which can be used to explicitly unsubscribe. This makes the subscribers much more dynamic (formerly process lifetime). Finally, I also applied this dynamic model to the optionally-retained handler for manual connections. By returning false at any connect success/fail even, the subscription (and manual connection cycle for it) is dropped. And while a connection exists, the subscriber can just stop it at any time. So good flexibility all around on subscriptions now.
(edited)evoskuil10:30
Also noticed that because we inject a logger into most test case now, logging can be trivially emitted for a test case by just providing a handler that writes to a buffer or console out. That's pretty handy.
🎉 1
Still no problems or unexpected behavior, with the exception of that assertion. Ran a huge run with object logging in debug. Took 10 minutes just to clear the log on stop, but all perfecto. Pretty amazing picture of the internal behavior, watching the trail of object creation and destruction for up to 100,000 connections, asynchronously, across 32 threads.
🙌 1
Also spent some time tracking down why a std lib API returns false for directory create, after creating the directory that didn't exist (false implies existed). Turned out to be a consequence of a trailing slash on the path. So trim resolves that.
Wednesday, February 15, 2023
This image on the right is a few days ago, and on the left is now. The x axis doesn't stretch, so those are same time per pixel. Both 1000 peers batched by 100. Notice the area maxes out much more quickly and there is much less network communication due to the completion window being about 50% shorter in time. This is the result of randomizing timeouts for connect and retry. Experimentation showed randomizing the top 50% of 10/5 secs was pretty good. I had several runs where there was never a drop-off. So basically twice as fast for these high volume scenarios. Simple stuff that I had experimented with in previous iteration.
(edited)evoskuil08:50
I also tracked down a periodic error on loading the hosts file. Had a deserialization failure for IPv6-Compatible addresses. I was intentionally not supporting these because they aren't necessary and are deprecated by RFC, but some were coming in over the network and I was serializing them, but didn't have the deserialization. So did the full denormalization to ensure we write those as IPv4 as well. The idea is to keep the hosts file simple as IPv4 and IPv6 in a single format each, no compat modes. I also modified the reader to skip invalid lines instead of aborting at that point. Added some logging as well so it's easy to see what's getting read/lost from the file.
Added some overrides and tests and simplified some -system string tools with C++20 std::erase/erase_if.
evoskuil09:09
Quite a difference also in terms of useful work earlier in the run (vs. this one from a few days ago). I've moved the blocks of address validation from the session (central strand) to each channel (independent strand per channel). Strands of execution are non-concurrent, but concurrent vs. different strands. So this allowed the work to spread out nicely. This is what will happen with tx/block loads as well.
(edited)
evoskuil10:39
After fully implementing address filtering (against invalid, etc), this is 1,000 connections batched by 25, 10, 5, and 1. The batch size required to reach the same number of connections quickly is dramatically reduced, from 50-100 down to 5-10.
(edited)
The last one is a very gradual climb that reaches 1000 right at the end. The memory totals displayed are off because they are after shutdown. But there is a clear memory-speed tradeoff. Also, at some point increased batch also slows returns. For this 5 seemed about right. But this also varies with timeout settings / network latency.
evoskuil12:10
Here's a final shot for the night. This is reaching 500 channels batched by 5, with address relay turned off. You can see on both RAM and I/O it sort of asymptotically approaches the target connection count. That's to be expected, faster to a number just means more batch or more total.
I used a connect handler to capture channel count in the console, and to stop at 500, as well as a stop handler to log the total numbers at stop...
Outbound [177.5.100.90:8333] protocol version (70016) user agent: /Satoshi:22.0.0/ Channel count (480). Outbound [34.172.31.246:8333] protocol version (70016) user agent: /Satoshi:22.0.0/ Outbound [85.250.100.142:8333] protocol version (70016) user agent: /Satoshi:24.0.1/ Outbound [73.108.249.219:8333] protocol version (70016) user agent: /Satoshi:0.21.1/ Outbound [45.13.7.221:8333] protocol version (70016) user agent: /Satoshi:24.0.0/ Outbound [178.63.52.205:8333] protocol version (70015) user agent: /Satoshi:0.19.1/ Outbound [168.119.86.21:8333] protocol version (70016) user agent: /Satoshi:0.21.1/ Outbound [173.249.56.164:8333] protocol version (70016) user agent: /Satoshi:24.0.1/ Outbound [70.233.173.241:8333] protocol version (70016) user agent: /Satoshi:24.0.0/ Outbound [200.150.112.190:8333] protocol version (70016) user agent: /Satoshi:22.0.0/ Outbound [34.106.249.67:8333] protocol version (70015) user agent: /Satoshi:0.20.1/ Outbound [109.248.251.140:8333] protocol version (70016) user agent: /Satoshi:22.0.0/ Outbound [178.62.15.92:8333] protocol version (70016) user agent: /Satoshi:0.21.1/ Outbound [136.244.65.212:8333] protocol version (70015) user agent: /Satoshi:0.19.99/ Outbound [54.159.193.149:8333] protocol version (70016) user agent: /Satoshi:23.0.0/Knots:20220529/ Outbound [46.28.205.68:8333] protocol version (70015) user agent: /Satoshi:0.17.1/ Outbound [159.100.245.237:8333] protocol version (70016) user agent: /Satoshi:24.0.0/ Outbound [99.237.146.168:8333] protocol version (70016) user agent: /Satoshi:22.0.0(FutureBit-Apollo-Node)/ Outbound [8.210.18.56:8333] protocol version (70015) user agent: /Satoshi:0.20.1/ Outbound [74.91.115.229:8333] protocol version (70016) user agent: /Satoshi:22.0.0/ Outbound [137.184.186.253:8333] protocol version (70016) user agent: /Satoshi:22.0.0/ Outbound [63.228.117.99:8333] protocol version (70016) user agent: /Satoshi:22.0.0/ Outbound [89.244.182.121:8333] protocol version (70016) user agent: /Satoshi:0.21.0/ Outbound [15.204.199.249:8333] protocol version (70016) user agent: /Satoshi:24.0.1/ Channel count (490). Outbound [34.229.155.152:8333] protocol version (70016) user agent: /Satoshi:24.0.1/ Outbound [47.75.194.174:8333] protocol version (70016) user agent: /Satoshi:24.0.1/ Channel count (490). Outbound [23.108.108.110:8333] protocol version (70016) user agent: /Satoshi:0.21.0/ Channel count (490). Outbound [176.9.17.121:8333] protocol version (70016) user agent: /Satoshi:24.0.1/ Outbound [31.104.58.236:8333] protocol version (70016) user agent: /Satoshi:24.0.1/ Outbound [75.176.70.180:8333] protocol version (70016) user agent: /Satoshi:22.0.0/ Outbound [98.179.228.56:8333] protocol version (70016) user agent: /Satoshi:23.0.0/ Outbound [162.248.95.197:8333] protocol version (70016) user agent: /Satoshi:22.0.0/ Outbound [173.249.9.243:8333] protocol version (70016) user agent: /Satoshi:24.0.1/ Outbound [86.90.214.244:8333] protocol version (70016) user agent: /Satoshi:24.0.1/ Outbound [161.117.239.176:8333] protocol version (70016) user agent: /Satoshi:22.0.0/ Outbound [162.55.238.126:8333] protocol version (70016) user agent: /Satoshi:23.0.0/ Outbound [13.231.241.62:8333] protocol version (70015) user agent: /Satoshi:0.20.1(Omni:0.11.0)/ Outbound [217.26.32.10:8333] protocol version (70015) user agent: /Satoshi:0.20.1/ Outbound [62.210.132.30:8333] protocol version (70016) user agent: /Satoshi:23.0.0/ Invalid heading magic (0x48545450) from [192.99.14.77:8333] Outbound [171.96.86.246:8333] protocol version (70016) user agent: /Satoshi:22.0.0/ Outbound [82.64.247.247:8333] protocol version (70016) user agent: /Satoshi:0.21.1/ Outbound [134.122.38.34:8333] protocol version (70015) user agent: /Satoshi:0.20.1/ Outbound [97.99.5.48:8333] protocol version (70016) user agent: /Satoshi:22.0.0/ Outbound [83.198.77.75:8333] protocol version (70016) user agent: /Satoshi:22.0.0/ Outbound [184.101.44.89:8333] protocol version (70016) user agent: /Satoshi:22.0.0/ Outbound [80.241.209.102:8333] protocol version (70016) user agent: /Satoshi:22.0.0/ Outbound [95.79.53.19:8333] protocol version (70016) user agent: /Satoshi:22.0.0/Knots:20211108/ Channel count (500). Stopping at channel target (500). Please wait while the node is stopping... Closing with (501) channels and (74197) addresses. Node stopped successfully.
There is typically that first dip in the RAM, which is due to the fact that all connections are queued at once, so a big batch of fails, and then those that fail (or time out semi-randomly) loop back with another address after a semi-random retry delay. The variation typically smooths out the rest of the display. Prob not worth issuing the first batch with delays.
(edited)evoskuil12:18
Overlaying the address traffic (typically 30kb per connection) disguises this clean asymptotic iteration. There's also a lot of connection dropping (over 800 completed handshakes to reach 500 holding connections) after the handshake, probably because of lack of feature support. That should improve as more is added. But filtering the address pool made a huge difference.
Thursday, February 16, 2023
evoskuil11:48
Worked on consolidation and better organization of subscribers, possible from the creation of the "resubscriber" template, which allows dropping of subscriptions, and targeting notification to one element by id. Was able to consolidate the session stop, retry and connection-pending subscribers into just one stop subscriber.
This took a good amount of refactor so that sessions and channels are constructed with unique identifiers that can be used for desubscription. But came out better/simpler as a result. connectors/accepter now create only running sockets, not channels. Channels are now created after socket is obtained and verified (blacklist, etc) by session. Session then encapsulates the socket in channel, as before but now in one place (session base) vs. connector and acceptor.
Now that this is done, I can finally remove the last vector of objects (channels) that's being retained by any class for enumeration. I'll create a message broadcaster object, based on the resubscriber. This will allow channels to subscribe for the message types to relay, instead of exposing this publicly for p2p to push through. This will then be symmetrical with incoming message type subscription and will simplify more complex protocol design. Been wanted this change for a long time.
Also tracked down an issue in tests, where the injected log was running past containing object destruct. The fix to the hundreds of tests was to create the log stopped for these tests, except a few to test a handler to block destruct until the messages are cleared. Made a similar update to the node executable wrapper (bn.exe), as shutdown was technically unsafe without a log handler blocking until cleared.
Also added a couple of node config settings to facilitate manual live test runs. A stop target in a channel handler, based on number of channels, and emission of log messaging based on address pool size and channel count. Good was to exercise use of the public subscription interface.
(edited)evoskuil12:05
So tomorrow I'll be creating the broadcaster, which may be the same as the current incoming message pump after a little refactor. Then hoping I'll be back to getting log levels/timestamps and console input.
pmienk17:00
Topic: Dev Weekly Status
Time: February 16, 2023 01:00pm Pacific Time (US and Canada)
Join Zoom Meeting
https://us05web.zoom.us/j/87666370902?pwd=bEZDUlJ0MGJ5V3dVMUhVdHl5QzdFdz09
Meeting ID: 876 6637 0902
Passcode: 85U0bF
Tuesday, February 21, 2023
evoskuil07:26
Despite the fact that it's pretty easy to capture ~100,000 unique IPv4 addresses from the p2p network, bitnodes.io claims there are 5,471 reachable IPv4 nodes on the net. I was able to reach 4,675 (I don't have IPv6 routing ability from here), at which point the node just churns on address messages trying to find valid ones it doesn't have... a slow trickle of growth from there.
evoskuil08:23
After @neillm found the host pool div by zero bug I started looking closely at an issue whereby I could never quite reach a targeted number of connections. @pmienk looked at it with me for a few hours last week. It was finally determined that it was the result of a race in the connector class. Despite being stopped (and fully stranded) it could still be running until handlers were executed. Since it was reused, this could cause state corruption despite no actual concurrency (by interleaving). I resolved this today, and now am no longer losing an occasional connector.
I did some work on the address pool, though I'm not really happy with that yet. That's really the major performance issue in terms of connection speed at this point. Also got rid of another boost dependency.
🎉 1
Thursday, February 23, 2023
evoskuil10:02
Finished up the work on the address pool. There's always more that could be done, but we're clocking in stable connection creation from the address pool at 5-8 connections per second depending on config, up to thousands of connections. The upper bound seems to be available hosts/ports. So prob. good enough for any scenario imaginable for a long time.
evoskuil10:19
Been working off some technical debt. We have three places where we fan out asynchronous operations and have to resynchronize them. This is a little different than startup/shutdown, because sessions and channels are never really recombined, they just are terminated and the process waits on threadpool join. When the threadpool is continuing it's a different model. I have dealt with this before, but never satisfactorily.
Just finished up three small templates that provide similar but distinct capability for this. I call these racers. I have looked at boost asio tools for this in the past but never found anything that was quite right. These are tools that operate on a single common strand, with asynchronous events (like timer, connect, accept, cancel) moving handlers to a strand for ordered execution. So there's no issue with actual concurrency, just a need to convert multiple events into one handler invoke based on various criteria. Each accepts a hander on start and fires it upon being invoked with completion arguments a certain number of times and/or with certain values.
So I have a "speed racer" that invokes the handler after template N finishes, passing the arguments of the first. IOW the first finisher wins (and should cancel the others) but the invoke doesn't occur until all are finished. Also a "quality racer" that invokes after parameter N finishes, with the first successful finisher winning (getting its arguments passed on). And finally a "volume racer", with a "complete" handler invoked when all N have finished, and a "sufficient" handler invoked as soon as a volume (such as count of connections) is reached.
These are used in the
connector (speed race between timer and socket connect), session_outbound (quality race between a set of connectors, with the first successful connection winning), and session_seed (volume race, completing when all seed nodes close and sufficient when a target number of addresses is obtained - or all nodes are closed). These then can be easily fully tested and reduced some longstanding complex code to trivial, just a couple lines in each usage. All working well, tests almost done.Friday, February 24, 2023
evoskuil06:05
I added RFC 3339 timestamps to the log. This is the shutdown sequence from 1,006 connections (1 incoming, 6 manual, 1,000 outbound).
2023-02-24T06:03:48Z Please wait while the node is stopping...
2023-02-24T06:03:48Z {in:1}{ch:1007}{rv:1007}{nc:1006}{ad:10000}{bs:1007}{ss:5}{cs:1}.
2023-02-24T06:03:48Z {in:1}{ch:1007}{rv:1007}{nc:1006}{ad:10000}{bs:1007}{ss:0}{cs:1}.
2023-02-24T06:03:48Z Inbound channel stop [177.227.58.83:26077] service stopped
2023-02-24T06:03:48Z Manual channel stop [5.14.19.0:8333] service stopped
2023-02-24T06:03:48Z Manual channel stop [65.109.113.126:8333] service stopped
2023-02-24T06:03:48Z Manual channel stop [86.104.228.11:8333] service stopped
2023-02-24T06:03:48Z Manual channel stop [34.222.125.43:8333] service stopped
2023-02-24T06:03:48Z Manual channel stop [89.35.142.168:8333] service stopped
2023-02-24T06:03:48Z Manual channel stop [51.79.80.166:8333] service stopped
2023-02-24T06:03:48Z Node stopped successfully.
2023-02-24T06:03:48Z ====================== shutdown ======================evoskuil10:00
Also added high resolution timer events to the log, with event id and arbitrary numeric field (no text). This uses the same pervasive logging object, but is on an independent subscriber. The event ID allows events to be filtered in the event handler. These are meant to be piped out to a service for static or dynamic reporting of fine-grained internal events (as via statsd). As with LOG writes, EVENT firings can also be compiled out.
Also noticed while running that manual connections are can sometimes compete with their own addresses arriving via the pool (which get trapped later as a duplicate connection attempt), so added configured manual peers to filtering. Dynamic manual connections (added through the API) are still subject to competition.
Saturday, February 25, 2023
evoskuil05:02
Cleaned up a couple more things pertaining to address pool management. Got connection rate up to almost 10 per second to 1000 connections.
evoskuil12:13
Added logging levels, with optional compile-time exclusion by level. Levels can filtered using a switch in the log event subscriber(s). The other approach would be to add a subscriber per level into the logger, but that's not as flexible and create more overhead.
Monday, February 27, 2023
evoskuil02:20
Added console input, similar to logger class. Now we can capture dynamic configuration changes (and non-signal stop) via the console.
evoskuil02:52
The catch with console input is always that the console input needs to terminate the input, since reading from
std::istream is always a blocking call and there's no way to unblock it. Terminating absent a successful read implies thread termination, which is process corrupting. So the capture class launches one thread for the blocking read, and another for posting notification and accepting subscriptions via its own internal strand (just like logger). These are set to high priority, so accelerate actions posted via the console, such as stop. Since the threads are almost entirely idle, this has no material impact on the process. The only downside is that in the case of a self-termination (i.e. the node decides it must stop), the process cannot terminate. It just await console input in order to unblock the input thread. So it posts a line to the console (via logger) to hit <enter> to close.
Otherwise the interface is very clean, just construct on std:cin& and subscribe to lines captured from input. It's nice that, at least on win32, a signal such as <ctrl-c> causes both an interrupt and an input event, so there is no need to wait on input. And of course if input terminates (e.g. user types "q" <enter>), then the input can terminate as well. So only on self-termination must it wait. The subscription model allows for input to be written back to the log (explicit and controlled echo). I turn of console echo in construct, as otherwise input text gets interleaved with console output and this is presumably not thread safe.I played a bit with modifying certain configuration settings from the console, which works fine as long as the application adapts to the change. Settings are passed pervasively using a single
const& though the application. This can be modified by the node class owner, though in some cases the use of the setting would require a notification. So eventually the settings class will carry a subscriber, and any class that accepts settings and wants to update on a change can subscribe to change notification. This would then just require that the setting change be notified by type - which we can now do at the console. Eventually this can be done via an zeromq admin interface. So no restarts for many config changes.evoskuil03:14
In other exciting news, a minor breakthrough...
2023-02-27T02:22:22Z.8 Node stopped successfully. 2023-02-27T02:22:22Z.8 ====================== shutdown ====================== 2023-02-27T02:22:22Z.8 Press <enter> to exit... <<unhandled exception start>> std::promise<std::error_code>::~promise<std::error_code> -> C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.34.31933\include\future(1186) std::promise<std::error_code>::~promise<std::error_code> -> C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.34.31933\include\future(1186) libbitcoin::system::main -> D:\Source\evoskuil\libbitcoin-node\console\main.cpp(89) libbitcoin::system::call_utf8_main -> D:\Source\evoskuil\libbitcoin-system\src\unicode\utf8_everywhere\environment.cpp(249) wmain -> D:\Source\evoskuil\libbitcoin-node\console\main.cpp(44) __scrt_common_main_seh -> D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl(288) <<unhandled exception end>>
This is a hand-spun win32 stack trace emitted to the console. I wrote this a week ago but could not get it to emit anything meaningful when I manufactured a failure. I just produced the first unintended unhandled exception since integrating it, and it worked. Given that I do most of my work in vc++, and that there is never a stack trace, this is a pretty awesome change. It will also be useful in customer environments.
(edited)evoskuil03:33
This trace came from a release build, with maximal compiler optimizations, but there is an associated
.pdb file that provides instrumentation (by default in the same directory).What's also very interesting is that the stack trace was obtained despite
NOEXCEPT on libbitcoin::system::call_utf8_main, which is something I've been concerned about given the pervasive use of NOEXCEPT.👍 1
Tuesday, February 28, 2023
evoskuil06:56
I recently removed the channel registration vector, which was required for both stop notification and message broadcast, and replaced it with a de-subscribable (new) stop subscriber. This was a half-step toward creating a broadcaster, to which channels subscribe for both stop notification and messages broadcast from outside the node or across channels. This cleaned up the control inversion of the node maintaining a channel vector, but it left for future work the task of creating the broadcaster.
Today I laid a bunch of the groundwork for the broadcaster. This an analog to the message subscriber (pump), which protocols use to obtain messages incoming from their channel's peer. So I normalized a few things in the pump implementation with the anticipated broadcaster. This also included doing some longstanding work in hash optimization.
Bitcoin p2p performs a double hash on each message and graphs 4 bytes as a checksum, placed into the header. This can be bypassed when reading messages, but is necessary for outgoing - in case another node is validating. For empty messages this is a fixed value, so free. For some messages, the hash is required after reading, and generally available when writing (to the net). This applies most obviously to blocks/headers and txs.
Obtaining the hash from a read is most optimally done via the read buffer. Previously we would reserialize the tx just to get its hash, which is a huge expense. Also, when we get a block/header/tx from the store - we have the hash. So I worked it out a while back that message objects would have the ability to ferry their identity hash and/or checksum, so that we could eliminate all of the redundant and hugely expensive hashing. I just stubbed in the simple remaining changes to eliminate all redundant hashing, and to ferry the hash and/or checksum as applicable for certain message, on a message-by-message type basis.
The consequence is that we will never compute hashes on hash/block/tx objects retrieved from the database for sending on the wire, never serialize an object off the wire to obtain its hash, either for identity or checksum validation, never compute a hash redundantly (for both checksum validation and identity), and never compute a hash/checksum for a message that is read and then forwarded.
I contemplated storing checksums in the database for certain message types, but the storage cost is large and there is no advantage for header/block/tx objects, as the checksum is a cast of the database natural key. There really aren't any other p2p message types for which it could be stored.
evoskuil07:11
So once I've completed this optimization, I'll finish up the message broadcaster. This object presents an interesting question. A broadcast message should be serialized once for all peers, as there may be many and reserialization for each is costly. Previously we serialized for each peer. So imagine, with 10 peers, each rebroadcast tx was serialized to a buffer for each peer and each was serialized again to obtain its hash - for each peer - to obtain the network checksum. Where N is peer count, that's 2*N (vs. 1) serializations and N double hashes (vs. 0) for each broadcast tx.
But our message serialization is purposely protocol version dependent, as there are cases where protocol may vary serialization. This means that each channel can potentially require independent serialization. Fortunately the need for this is limited, and we can rely on canonical serialization in most situations. So the broadcaster serializes and then pushes out the buffer (including header/checksum) to each peer for send. But a subscribing protocol may need to inspect the message, and not just forward it blindly. So the original un-serialized object is provided, as well as a buffer - both passed by shared pointer.
This is basically the same as the message pump except that the subscribers are not receiving notifications on their own strand, they are receiving notifications on the singleton network strand - and each must therefore bounce over to its own strand for thread safety. But this doesn't affect the design of the broadcaster (vs. the pump). The main distinction is that, as a subscriber on an independent strand, the protocols/channel must handle completion for subscription in order to not miss messages. But since, unlike channel protocol, broadcast is best effort - this shouldn't be necessary.
(edited)evoskuil07:17
But in the case of a connection obtained for the purpose of sending (eg tx broadcast), it's still an issue - so should ensure broadcast subscriptions are complete before providing channel notification.