May 2023

It looks like we can get nearly the full benefit of the no-iostream reader implementations that used above but in a fully generalized manner. The reader/writer/flippers accept an IStream/OStream/IOStream template argument. This isn't required to derive from the std iostream interface, just to implement a subset of it. By implementing the simple subset used by reader and writer, this can be passed into any of the byte or byte reader/writer/flipper implementations and based on a comparison of my ad-hoc reader/writer that cut confirmation time by around 55%, it should be very similar. This supports store serialization, network serialization, header/tx identity hashing, and consensus serialization for signature hashing. Basically everywhere we stream in/out of a fixed buffer. We could also implement a variation for dynamic buffering, but there isn't presently call for it. It looks to be about 1 days work to implement IStream/OStream and one that inherits both (IOStream), including tests. Applying these in the various scenarios is then trivial work, just swapping stream types.

It's not entirely clear to me why the std::iostreams produce so much overhead. This is a set of boost templates that we use to build the stream objects. But there's clearly a significant amount of overhead. It appears to be largely in the construction of the stream, as reported by the perf tool.

Complete and tested in system, istream, ostream, iostream. Will integrate with database tomorrow.

I just upgraded to Pro. As I understand it, the old messages should now be visible. Please let me know how it looks. 30 days free then I'll put it on my account.

We've got some lesser donations to cover this sort of thing.

Topic: Dev Weekly Status Time: May 4, 2023 01:00pm Pacific Time (US and Canada) Join Zoom Meeting https://us05web.zoom.us/j/87666370902?pwd=bEZDUlJ0MGJ5V3dVMUhVdHl5QzdFdz09 Meeting ID: 876 6637 0902 Passcode: 85U0bF

I merged updates to system that patch a conditional build problem that arose after @pmienk incorporated the defines defines. Also a new simple iostream object that maps a buffer without implementing the full std::iostream interface or being built from boost::iostream devices. Initial tests showed this to reduce the cost of database operations to 40% of the cost under iostreams. This was identified using the msvc performance profiler.

Just completed a quick run of set_strong using the new generalized simple iostreams. These are passed into the system read/writer as opposed to the previous iteration which ad-hocked a reader and writer, so there is one level of indirection/containment here that wasn't present in the previous test. The is resulted in a 123% of the previous run. The previous run was 35% of the std::iostreams unoptimzed run, and this new run is 43% of that run. New run in gray.

While we can still use the fully-optimized streamers for most database calls, we must either use this new mostly-optimized form for input/output script and witness reads or must extend the fully self-contained reader/writers to implement the bytereader/bytewriter interface. So while that remains possible this is a far more elegant and simple optimization and gets us very close.

This is partial completion of the fully-optimized full store read in relation to the same optimizations. Note that this one is significantly better because the "pointer" optimization did not optimize for scripts and witnesses.

Here's the complete results from the above set of runs.

But as for the effectiveness of the stream optimization on reads, a projection (i.e. not memory bound) of the gray run completes at 3.75 hours whereas the blue run projects to 5.83 hours. The optimization itself is 64% of the original time and 69% of the partial (orange) optimization. In other words, current implementation can sequentially read and deserialize the full chain in 3.75 hours if not memory bound (or if caching is somehow more optimally performed).

Here's full confirmation runs (vs. just set_strong) relative cost (vs. cumulative). The higher level abstraction creates an expected cost of a few percent, with no offsetting benefit - since only script/witness would be improved and there's none of that in confirmation. But I can't yet explain why (gray) performance is up to 3x worse (than blue) when blocks are small. It should never be worse than the full stream generalization. But the total impact is modest. 2.62 hours to confirm vs. 2.25.

Ok, ran some tests and determined that the difference with the early blocks is entirely caching. The OS caches memory map between runs, and I don't yet have a reliable way to clear the cache. With previously-cached tables the performance is consistent across the block range. The preceding scenarios had both been run with caching in the early stages. In the later stages (fuller blocks) the relative benefit is practically eliminate.

This (series 4/yellow) is a full sync run with the optimized stream. Sync runs currently rly on a single peer speed, and I seem to have lost my two performance-reliable peers (became unreliable). I couldn't find one that was comparable, so ran against one that was about 50% slower in terms of full header sync. This still outperformed the previous runs.

Completed the integration of the simple iostream with all network serialization/deserialization and also header/tx hash and signature hash computations. Hard to get a good measure of benefit without a consistent/baseline peer, but the performance analyzer is now showing a pattern I haven't seen before. With validation disabled there are no actual hotspots, tiny values all over. And with validation enabled all hotspots are in libsecp256k1 (signature validation) and a much more modest level in signature hashing and script execution generally. This is what we should expect to see under ideal circumstances.

But first I'm going to take a little detour into a script serialization optimization. Planning to replace the use of shared_ptr op-data with unique_ptr. These were originally created under the assumption that shared_ptr would be the basis of script evaluation, but the generalization to external_ptr made that unnecessary. unique_ptr eliminates one word per op-data instance and eliminates mutex locks internal to shared_ptr. And since a large percentage of these are nullptr and scripts are the bulk of the chain, this should be a material optimization, and simple.

Overall the store work has come out exceptionally well. system, store, network, and node have proven exceedingly reliable - in fact I'd even dare to say perfectly reliable. In the store work the main issues I had to work through were performance optimization around table structure and some regressions in chain validation due to the full rewrite. The only significant issue, which should not be understated, is thrashing of the memory maps when memory becomes full during significant reading.

Topic: Dev Weekly Status Time: May 11, 2023 01:00pm Pacific Time (US and Canada) Join Zoom Meeting https://us05web.zoom.us/j/87666370902?pwd=bEZDUlJ0MGJ5V3dVMUhVdHl5QzdFdz09 Meeting ID: 876 6637 0902 Passcode: 85U0bF

Topic: Dev Weekly Status Time: May 25, 2023 01:00pm Pacific Time (US and Canada) Join Zoom Meeting https://us05web.zoom.us/j/87666370902?pwd=bEZDUlJ0MGJ5V3dVMUhVdHl5QzdFdz09 Meeting ID: 876 6637 0902 Passcode: 85U0bF