October 2021

These are the location of boost sources at compile time. The binary is statically linked, making these irrelevant.

that binary is still using /Users/Mica/explorer-qrcode-prefix/etc/libbitcoin/bx.cfg for the default config location.

Ok

Not sure how that is, because the path is computed from the current default directory at runtime, but in any case you can change it or compile it yourself.

Updates: I've got a monster PR in libbitcoin-system. It's ready to merge, though SourceForge is down again (bad certs), so builds are failing. Travis is giving us free builds, but have to re-request them every couple weeks or so (pain in the ass). The -system changes break the other repos, so I'm working my way through that now. It'll take a while, because I'm doing other work as I go. Currently reorganizing -network and adding test coverage. Network component tests appear to be failing because of public server or DNS outage, but need to work that out. Amir finally has libbitcoin.org and will transfer it to me eventually. We met in Amsterdam a couple of weeks ago. The Libbitcoin Institute did not get IRS 501c3 status, so that is being wound down, but another entity will be set up in its place with the same name and purpose (minus the tax benefit). Tom and I are still working on fundraising and have some new leads based on Amir stirring the pot yesterday. I'm not planning on running the conference next year, and will put my efforts into the v4 release. Possibly the following year. Book translations are starting to come in, and Max from Wasabi just posted the audiobook.

Github actions claim to be free for public repos: https://github.blog/2019-08-08-github-actions-now-supports-ci-cd/ Worth considering a move? Happy to put in the work. I was struggling with a PR today, but that was largely because of sourceforge's certs for boost downloads.

I was looking at it, but got sidetracked. If the actions include the actual build hosts, then I'm all for it. I see the builds worked, did you fix the SF issue? FYI, I'm trying to keep Travis time down at the moment. MacOS chews up like 6x the credits of the other platforms. So if you kick off a build, stop all but one gcc build until it's done, and if that's good restart the others once it looks ready to go.

Good advice about checking with a single build first.

Yeah, the free thing was ridiculous. But they are giving us more whenever we run out. They are just making it a hassle.

re: sourceforge certs, it seemed to have resolved itself after a while. I think their load balancer resolved the host to a different one and my build was lucky to go through.

all good

list of lists!

Yep, that would be AWESOME

We need libbitcoin! There is no two ways around it.

I think you'll appreciate the changes in system. Moved a lot of stuff out, added a lot of test. redesigned low level code (streams, serializers, endians, proptree, etc). Improved a some real perf issues in streams.

Many interface changes? That is, do we need to get the rest of the libs in line with system?

Yes, that's what I'm working on.

😱

Didn't I already tell you this?

I stopped keeping an eye over summer, I used to enjoy going through the diffs as emails came through

Cleaning house 😏

spring clean!

Lots of cruft collected over the years.

ok.. i am seeing doing gh actions for that branch, and then making sure it works with c++20 and boost 1.76... Those unlock coroutines, which will make my, and hopefully other people's lives easier.

Revamping libbitcoin-network. The object model remains close to what's there now, but with some material changes to threading. I put together this simplified model to help me visualize relationships. The main simplification is that derived session and protocol classes are not shown.

The extensible (yellow) classes are modified by deriving from p2p, overriding the virtual attach methods to attach custom derived sessions, and overriding session attach methods to attach custom derived protocols. This is how libbitcoin-node is constructed (adding transaction and header protocols). Of course, the same can be done in libbitcoin-node, and libbitcoin-server inherits from it (though does not change its protocols).

libbitcoin-network is also the threading engine for libbitcoin-node and libbitcoin-server. Node introduces additional thread pools for parallel validation, though these are isolated. Server introduces one thread per endpoint worker (well isolated within libbitcoin-protocol). This rework dramatically simplifies the threading model, so that we are operating on one dedicated thread per socket/channel. While the implementation remains entirely asynchronous, this will eliminate a lot of complexity and resolve certain issues in our use of boost::asio. It will also reduce our DOS attack surface to almost nothing. I expect it will also finally resolve the shutdown hang (deadlock). If it resurfaces, it will certainly be much easier to troubleshoot. We may possibly experience a performance hit, as we will not be queueing up arbitrary amounts of work when our processing speed is out of sync with network speed. However, this is the DOS vector. If there's a material hit I would create a managed work buffer, dropping outbound messages or blocking inbound messages as necessary (which is how we manage ZeroMQ buffering).

Note that in the model, 'protocol' (actually one derived class per protocol semantic) receives and originates all p2p messages (apart from explicit broadcasts). Each protocol subscribes to the message type(s) of interest. This is part of the elegance of the design. Each derived protocol class entirely encapsulates the semantic (such as compact blocs, ping/pong, headers, txs, blocks, etc.). These are versioned and are attached independently on each channel (by the session), conforming to the negotiated protocol version of the channel. They are very simple and clearly document each independent protocol.

Even without protocol customization, a p2p instance provides new channel notification, subscription/notification of messages on a channel and/or all channels, send of messages on a channel and/or broadcast on all channels, as well as channel and p2p stop subscription/notification.

See above, I'm moving to one asio thread per channel. Otherwise we end up blocking our way to the same thing, in order to guard the socket and to sequence messages.

After getting about half way through this work I found in boost documentation a resolution to the main issue we had with socket write composition. So with a shared threadpool of any size we can still ensure order on each channel. I’ve also been able to eliminate the execution of message handlers under lock, which eliminates the deadlock hazard. And I’m finally implementing the DOS protection strategy that has been envisioned for some time. This borrows from lessons learned working from ZeroMQ.

ZeroMQ uses transparent and configurable work queues for both send and receive. This is a necessary strategy, and it’s sort of surprising that asio doesn’t natively expose controls on its queue. But as all problems in computer science can be resolved with another layer of abstraction, I’ve created a buffered_socket::socket, which maintains size/count read and write message queues. This will suspend read when the read buffer is full (blocking the peer) and overflow the send queue (drop writes) when the write buffer is full. The latter may cause an interruption of the expected protocol semantics, resulting in a dropped connection, though this is acceptable and unavoidable given the p2p protocol design. The other alternative is to drop the peer.

The message queues are thus buried in the socket (as with ZeroMQ), maintaining the simplicity of protocol implementation. Ideally we would want to suspend performing work that could not be sent to a peer, but this at least puts an upper bound on the resources that a peer can consume. It will also be possible to provide a cost metric per incoming message type, to limit peer requests by actual resource consumption, suspending the reader until queued work drops below the configured limit.

I could not figure out why I was not syncing correctly…..now I am, as it turns out the time on the system has to be correct….I feel stupid.

Slowly catching up to v3 changes. I recall the qrcode/png updates happening not that long ago, but am not sure what's changed there yet. Is code like this dead? https://github.com/libbitcoin/libbitcoin-system/blob/version3/include/bitcoin/system/wallet/qrcode.hpp#L29 (i.e. WITH_QRENCODE and WITH_PNG as it seems there's no supported way to push those through via install.sh). Maybe I'm missing something?

It’s dead in master, not sure if I backported it all. Qrencode is now embedded and I wrote a tiff encode to get rid of png.

Looks like it's dead in v3, but I'm not sure how to use the new API. Any quick pointers?

Check the test cases.

I think I also hooked it up in bx.

Ah, I see -- it looks like master is ok, but version3 is not backported. For this particular case I was looking at version3 primarily, but I'll take a look at master then