Bitcoin full node and query server

January 2023

Archive of the libbitcoin Slack #general channel.

Sunday, January 1, 2023

Happy New Year!
🎉 1🥂 1💪🏾 1
evoskuil22:28
I've been working my way through database query testing. 60 out of 98 queries compete. Wanted to have it complete by EOY, but should be done by next meeting.
👍🏾 1
evoskuil23:39
That is completion to release quality of system, network, consensus, protocol, and database. That will leave node and blockchain to get to a functional node.
🎉 2

Wednesday, January 4, 2023

evoskuil08:35
12 methods left to test in database. Should finish up above 95% coverage.
👍 1

Thursday, January 5, 2023

pmienk17:00
Topic: Dev Weekly Status Time: January 5, 2023 01:00pm Pacific Time (US and Canada) Join Zoom Meeting https://us05web.zoom.us/j/87666370902?pwd=bEZDUlJ0MGJ5V3dVMUhVdHl5QzdFdz09 Meeting ID: 876 6637 0902 Passcode: 85U0bF
6 replies · neillm, pmienk · last Jan 05, 2023
neillm20:12
I'm planning to drop in today, but have to go after about 30 minutes unfortunately
neillm21:37
Sorry had to run, thanks everyone!!
pmienk21:38
Great to have you, catch you later
🙏 1
neillm23:42
Looks like mainnet.libbitcoin.net is mapped to 209.181.64.147. The old one (well, mine anyway) is at 72.65.246.83
neillm23:54
Well, now updated for the new secure testnet key ...

Saturday, January 7, 2023

evoskuil07:55
Database all tested. Just having some fun adding/testing a few more advanced wallet queries.
👍🏾 1👍 1

Sunday, January 8, 2023

evoskuil03:06
Calling it complete for now, on to node/blockchain tomorrow!
💥 1🎉 1🙏 1

Thursday, January 12, 2023

pmienk17:00
Topic: Dev Weekly Status Time: January 12, 2023 01:00pm Pacific Time (US and Canada) Join Zoom Meeting https://us05web.zoom.us/j/87666370902?pwd=bEZDUlJ0MGJ5V3dVMUhVdHl5QzdFdz09 Meeting ID: 876 6637 0902 Passcode: 85U0bF
evoskuil22:18
👍🏻

Thursday, January 19, 2023

pmienk17:00
Topic: Dev Weekly Status Time: January 19, 2023 01:00pm Pacific Time (US and Canada) Join Zoom Meeting https://us05web.zoom.us/j/87666370902?pwd=bEZDUlJ0MGJ5V3dVMUhVdHl5QzdFdz09 Meeting ID: 876 6637 0902 Passcode: 85U0bF
jungly22:12
https://github.com/libbitcoin/libbitcoin-system/actions/runs/3934527793/jobs/6729355343 This job always takes too long. Execute build is almost all the time.
evoskuil22:27
I should add the both templatization and forced inline significantly increase build time and build size.
These have the greatest impact in script and hash/scrypt functions.
(edited)
evoskuil22:39
I just encountered SYSCONFDIR not being defined in non-MSVC builds. This code was formerly operative but had become inadvertently excluded for some time. It seems that this may be something that we need to define in the build process. Otherwise I don't know why it would have always worked before but not now.
pmienk22:41
I'll look for a change in -build history, I don't recall removal or even it being previously provided
evoskuil22:42
It would have been years ago.
(edited)
pmienk22:53
looks like its defined in -explorer only
should be working, if you need it moved to system it's just adding the flag
evoskuil23:11
Hm, maybe that code was moved over from bx when it broke. That's about 2 years ago.
(edited)
I'll default it to "" if not defined, but we should define it for system (and therefore depending projects).
evoskuil23:30
Windows x32 release build time cut in half (47m 22s). Windows x64 release is materially slower because it is both fully optimized (unlike debug) and force-inlined.
✅ 1

Friday, January 20, 2023

evoskuil00:21
Given that x64 took 87 mins, I suspect the inlining (as well as the warnings about it) is a significant cost - which the debug versions are avoiding.
(edited)

Monday, January 23, 2023

evoskuil20:27
Completed a basic log implementation with the following:
  • log reference injected alongside config, through construct of p2p or full_node:p2p or server::full_node::p2p and then all other objects that require logging.
  • class tracker (base class to log object refcounts) accepts log object and logs checked (debug) builds.
  • logger posts each message to its own strand on its own low priority threadpool.
  • logger exposes standard subscriber for accepting log messages.
  • log handler in bn.exe subscribes to log and emits messages to console.
  • message notifications preserve order and with handlers invoked non-concurrently.
  • logger can be easily expanded to differentiate category and parameterization, either with multiple subscribers or via filtering within a subscriber.
  • Macro invocation both preserves serialization formatting and allows full exclusion at compile time.
(edited)
🎉 1
    LOG("Reject message '" << reject->message << "' ("
        << static_cast<uint16_t>(reject->code) << ") from [" << authority()
        << "] with reason: " << reject->reason);
    LOG("Received " << reject->message << " reject ("
        << static_cast<uint16_t>(reject->code) << ") from ["
        << authority() << "] '" << reject->reason << "'"
        << (reject->message == block::command ||
            reject->message == transaction::command ? " [" +
            encode_hash(reject->hash) + "]." : ""));
These compile to nop when the macro is disabled.
Preserving serial formatting while also posting discrete messages requires construction of a closure for each write, which is an additional cost that is excluded when messages are compiled out. Each message is serialized to a stringstring, which is then flushed to post. But the posted messages are then captured by subscription handlers, and these can queue up writes to the log output (such as the console or file) without flushing them on each write. This is a material performance improvement.
evoskuil20:38
The log is now fully thread safe and supports any intermediate and final formatting, including zeromq endpoints, and can be trivially expanded or any level of complexity given that it is a single object reference passed into each loggable context. These is no global or static context.
The one unhandled aspect at this point is file emission and rotation. Currently the bn.exe subscriber just uses a closure to handle messages and emit to std::cout.
evoskuil23:00
So working on a simple log rotator, with configurable rotation and size limit. Should be simple given the existing database file utils.
(edited)

Tuesday, January 24, 2023

jungly00:27
Sweet
evoskuil06:39
That’s done, except for the test and integration. Should be wrapped up tomorrow.
evoskuil22:29
Fully tested and integrated rotable log, just waiting on the 3 PRs. Can later trivially apply this to each log of various levels, but right now there's just a single log level. On to node block sync!
🎉 1
I'll end up differentiating logging and levels once I go back to revise configuration settings and parsing. That's just waiting on full implementation, so that the full set of config is known.

Thursday, January 26, 2023

pmienk17:00
Topic: Dev Weekly Status Time: January 26, 2023 01:00pm Pacific Time (US and Canada) Join Zoom Meeting https://us05web.zoom.us/j/87666370902?pwd=bEZDUlJ0MGJ5V3dVMUhVdHl5QzdFdz09 Meeting ID: 876 6637 0902 Passcode: 85U0bF
👍🏻 1
2 replies · jungly, pmienk · last Jan 26, 2023
jungly21:17
Missing tonight. Fire at day job
pmienk21:17
good luck
evoskuil22:34
"All temporary objects are destroyed as the last step in evaluating the full-expression that (lexically) contains the point where they were created, and if multiple temporary objects were created, they are destroyed in the order opposite to the order of creation. This is true even if that evaluation ends in throwing an exception." https://en.cppreference.com/w/cpp/language/lifetime
So, for this class...
class BCT_API logger
{
public:
    class writer final
    {
    public:
        DELETE_COPY_MOVE(writer);

        inline writer(const logger& log) NOEXCEPT
          : log_(log)
        {
        }

        inline ~writer() NOEXCEPT
        {
            BC_PUSH_WARNING(NO_THROW_IN_NOEXCEPT)
            log_.notify(error::success, stream_.str());
            BC_POP_WARNING()
        }

        template <typename Type>
        inline std::ostream& operator<<(const Type& message) NOEXCEPT
        {
            BC_PUSH_WARNING(NO_THROW_IN_NOEXCEPT)
            stream_ << message;
            BC_POP_WARNING()

            return stream_;
        }

    private:
        const logger& log_;
        std::ostringstream stream_{};
    };

    typedef std::function<void(const code&, const std::string&)> handler;

    logger() NOEXCEPT;

    writer write() const NOEXCEPT;
    void subscribe(handler&& handler) NOEXCEPT;

    /// Stop the subscriber/pool with a final message posted to subscribers.
    void stop(const std::string& message) NOEXCEPT;
    void stop(const code& ec, const std::string& message) NOEXCEPT;

protected:
    void notify(const code& ec, std::string&& message) const NOEXCEPT;

private:
    threadpool pool_;
    asio::strand strand_;
    subscriber<const code&, const std::string&> subscriber_;

    void do_subscribe(const handler& handler) NOEXCEPT;
    void do_notify(const code& ec, const std::string& message) const NOEXCEPT;
    void do_stop(const code& ec, const std::string& message) NOEXCEPT;
};
This should be safe, right (where logger_ is a persistent instance)?
logger_.write() << "Channel inactivity timeout [" << authority() << "]" << std::endl;
(edited)
pmienk22:37
sounds right, given the reverse order destruct
👍🏻 1

Friday, January 27, 2023

evoskuil05:05
Found my intermittent shutdown delay. During handshake the channels exist in base session closures. They aren't registered for message broadcast and stop invocation until handshake is complete. So the handshake timeout is the max stop delay (as I was seeing).
Before handshake I "pend" the nonce of the channel, in order to detect loopback connections during the handshake, unpending after and immediately before channel broadcast/stop registration. So will need to pend full channels and stop them on shutdown.
evoskuil05:40
All done, super fast stop on every run now. :)
🎉 1
Node now implements the --initchain option to construct the database, and opens/closes the database on runs.
(edited)
💯 1
evoskuil06:39
Pictures of building up to 1,000 peer connections and 10,000 unique address pool entries.
image.pngimage.png
evoskuil06:44
There's only address and ping protocol after the handshake, and you can see the traffic tail off after the initial connections and bulk address transfers. This was batched 3 connections at a time to achieve one - so 3,000 connections were fired off immediately. This was executed on a network threadpool with just one thread.
😮 1
evoskuil07:02
The 1,000 connections, with a few in transition.
image.png
evoskuil07:52
Despite the large system lib size (174MB), the fully-speed-optimized bn.exe is just 3.8MB.
4 replies · jungly, evoskuil · last Jan 27, 2023
jungly07:53
Hmmm. This is interesting. I am curious about sizes on linux with g++ now.
jungly07:53
Will try to build stuff over the next few days.
jungly07:54
Wonder if we should add the build size as an output in the CI.
evoskuil08:00
Hopefully the linker is as optimizing as msvc.
😉 1
evoskuil07:56
Spun up 1000 peers on 32 threads with no prob. Shutdown time without console logging after or during establishment of 1000 peers is consistently less than 1 sec. I'm letting an instance run overnight as a leak check, but it's looking very stable. I've had no failures or other bad behavior (apart from the handshake timeout on shutdown which is fixed) since starting working with the node process over the past couple days.
The network framework has achieved the objective of greatly simplifying protocol design. All protocols (version, address, ping, seed) have been flawless, despite not being tested (everything else in network is fully covered). Next step is to work up basic header/block protocols and populate the store. After that I'll set up test for all of the protocols.
(edited)
🎉 3
jungly21:28
This was an interesting read https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packagers-security-nightmare/ Singles out rust, go and python
Michał Górny
One of the most important tasks of the distribution packager is to ensure that the software shipped to our users is free of security vulnerabilities. While finding and fixing the vulnerable code is…
evoskuil21:29
Saw your post.
😉 1
jungly21:30
There's another thing I read a while back about how people pull in dependencies in rust left right and centre cause the stdlib isn't large enough yet. Sounded like another nightmare
evoskuil21:30
yep
(edited)
evoskuil22:16
They'll reinvent a boost lib, then work for a few decades to get it into their standard lib.
✔️ 1

Sunday, January 29, 2023

evoskuil03:42
Decided to really clean things up in logging, Made a device to take the place of the log rotator. Our streams are just boost "devices" that boost constructs a stream around. So now we have a class that implements std::ostream and writes to a log rotation sequence. As with the class impl, construct takes the file naming and size limit params. This allows console out and file out to be normalized to a std::ostream sink.

Monday, January 30, 2023

evoskuil07:02
This worked out nicely, tested and merged. I've added a bn command line option that disables console out. This dramatically reduces shutdown latency with high volume logging. I'm now testing at 5,000 connections, batched 5 at a time (25,000 connection attempts). These levels of connectivity used to crash my router. The new one has had no problems. CPU is almost nothing (~.15%), memory is a modest linear scale, and I/O is also small. I've got 32 of my 64 cores configured on network, but it's never using more than 10. Still not a single crash or deadlock and shutdown is almost instantaneous regardless of the connection count.
(edited)
👍 1
4 replies · jungly, evoskuil · last Jan 31, 2023
jungly07:27
Sweet! Nice that you are stress testing it so.
jungly07:28
Are you making all those connections by bumping you the connection limits?
evoskuil17:54
The outbound connection count and total connection limit are controlled via configuration.
jungly07:12
Yup, that's what I thought. Was curious if you were doing something special

Tuesday, January 31, 2023

evoskuil04:05
Spent some time with live seeding today. Found the same issue with seeding channels and I had with outbound channels during handshake. If a close occurred during seeding the close even wasn't getting to the seed channels, so they would delay shutdown until timeout. The seed channels stay open until they either time out (30 secs) or obtain an address message. Fixed that, so still immediate shutdown.
Just needed to register the channels outside of the normal channel registration (as with handshake) since normal channel registration allows message broadcast, and that starts as soon as there are enough addresses (doesn't wait for seed channels to all complete).
evoskuil04:12
Also verified proper operation of the node with config file, no problems. So all external files are now being used in my live testing (full database file set, two file rotating log, config settings, address pool).
evoskuil04:27
I stressed the address pool, easily pulls in 100,000 addresses, stores and loads them in no noticeable time from disk.
Changed seed address minimum to the amount configured batch*outbound, so that there are alt least as many seeded as we will attempt to open.
(edited)
Also worked out a plan for an object to capture std::cin and invoke a subscriber. This will operate on an independent threadpool and strand, basically like the logger. Pretty trivial way to feed events into the running node. Later (like logging) this can be exposed via zeromq, as an administrative endpoint. Should be one day of work to implement and test. Simpler than logging because no file rotation.
(edited)
1 reply · jungly · last Jan 31, 2023
jungly07:12
This is so cool
🤓 1
evoskuil04:57
The idea is to to work towards dynamic reconfiguration, so the node doesn't require a restart for reconfig. Since we pass a settings object through all major objects that accept config, this just entails making the config classes into subscribers and having each class that supports dynamic config just subscribe for update events.
2 replies · jungly, evoskuil · last Jan 31, 2023
jungly07:14
Hmm. Like change connection limits and all sorts of things on a live node. 🔥
evoskuil07:18
Yep, enabling endpoints, adding certificates, submitting txs directly, etc.