Bitcoin full node and query server
August 2024
Archive of the libbitcoin Slack #general channel.
Thursday, August 1, 2024
Huddle with pmienk, Tanay Raikhere, evoskuil, Shashank Shekhar Singh · 50 min19:50
Tuesday, August 6, 2024
evoskuil19:04
After a couple weeks away and then a return to the task of caching blocks, I can report that the approach is working as expected.
evoskuil19:14
It took some time to build up the infrastructure to reliably deserialize all aspects of a block into a custom allocator, however the amount of new code is very small, and the related changes to chain object deserialization are simple. Having done that it is now possible to destruct blocks on any thread without any memory deallocation cost whatsoever. The memory allocation cost is also close to zero.
In comparison to a full chain sync using the default memory allocator, the new custom linear allocator is slightly faster even when blocks are deallocated on the allocating thread. However when blocks are projected to a common strand for caching, the increased cost is only about 10%, none of which is in deallocation - just posting the closures to a work queue.
Previously, using the default memory allocator, the increased cost was totally prohibitive. The total run time went from about 60 minutes to at least 24 hours if not longer (never finished). So that aspect of the issue has been resolved.
The cost of using the custom allocator is a fixed memory allocation for each network thread, equal to the largest expected block object size possible. Some measurements demonstrated that all but 1 of the 840k blocks have an object size under 12MB (wire serialization is limited to 4MB).
🙇🏾 1
evoskuil19:26
That 1 outlier block is over 28MB deserialized. Turns out that's because it has 500,003 witness elements. It also happens to be the block that caused both LND and btcd to stall. Since each witness element is a smart pointer to byte vector, this more than doubles the typical object ratio. Not a problem for us, just need to get the max sizing correct.
(edited)
evoskuil19:46
There is a simple option here to conditionally (based on config) use the custom block allocator. This provides a modest performance advantage but may increase memory contention on smaller machines, haven't tested that yet.
But the objective was to take blocks that are downloaded, deserialized, checked, and stored concurrently across channels, and forward them to a common stranded handler for caching. This step cannot reasonably rely on the per-channel single block linear buffer, and expanding that buffer into a circular buffer presents the same ultimate limits and more complexity.
However, once a block is deserialized using the custom allocator, all of its many thousands of individual allocations comprise a single contiguous memory block. This means that the block object also can be copied and serialized (e.g. to/from disk) at almost no cost.
evoskuil19:56
So, in order to achieve the objective with minimal cost and complexity, I'm planning to allocate a chunk of memory equal to the deserialized block and copy the block into it. Freeing this allocation consists only of calling
free(ptr) on this one shared_ptr<block>, which will be constructed with a deleter that simply calls free on its own address and otherwise does nothing (e.g. no destructors get invoked). This is possible because all objects contained within a block have only default destructors with the exception of std::shared_ptr and std::vector and these only invoke other destructors and/or free the same underlying memory.
Since the objects are all const there is no downstream risk apart from metadata setters, though these should only be used by the code that allocates the block in the first place.
Note that block destruct is 7% of the run cost, even without memory deallocations. So this will more than offset the alloc/copy cost of the implementation.So after every block is copied to its new buffer, which requires only a single
malloc(size) and copy_n(size) , the fixed per-channel block linear allocator just resets back to its buffer start, as it does now. This allows me to remove the tracking shared_ptr that was retained in each block, for the purpose of holding a mutex to block use of the buffer until the block is deleted. Now the block simply lives as long as it wants and deletes its memory when the shared pointer refcount goes to zero. So much simpler.Basically we get super-efficient and simple block object allocation and free deallocation. No custom memory allocator can beat a single-threaded linear allocator, as it's just a couple of math ops. We can then move the blocks across threads at zero additional deallocation cost, where previously that cost was entirely prohibitive, and don't even have to worry about deallocation order.
One side benefit, we now also have a precise and free measurement of the allocated size of block objects (the only ones that really matter). We can also apply this same allocation strategy to block and prevout deserialization from the store for the purpose of validation (in the case where the block object or prevout isn't cached).
(edited)Wednesday, August 7, 2024
evoskuil22:46
X (formerly Twitter)
Bitcoin Core has announced migration from Autotools to CMake, which it rightly states is "massive" and which will be "painful". It is a big project, but as a full node and wallet dev toolkit spanning 10 repositories, Libbitcoin is much bigger...
https://t.co/paqP20NPZz
✅ 1
pmienk23:38
Until the next great migration, where you have to tease out the salient details from cmake cruft
👍🏻 1
Thursday, August 8, 2024
M
mariano20:15
I’m finally in argentina and finishing setup the server … its a i9 with 64gb ram and 50tb in raid for testing … i’m going to setup some apps for source code control and give you access to
Friday, August 9, 2024
Thursday, August 15, 2024
Huddle with pmienk, evoskuil, Shashank Shekhar Singh · 75 min20:06
Shashank Shekhar Singh21:39
Is the Boost package version less than 1.75 creating this issue. I searched it on the internet and it said that "Boost.JSON was introduced in Boost
1.75.0 "
1 reply · Shashank Shekhar Singh · last Aug 17, 2024
Friday, August 16, 2024
evoskuil21:59
Recent optimizations to memory allocation have brought the 840k sync time (with confirmation, under milestone) under 60 minutes for the first time.
evoskuil23:57
The no-destructor run was actually a little slower than the run with all objects in the block being destructed. But since the only actions are object memory deallocations, I suppose that the destructors are actually fully optimized out, making it essentially the same code. This is good because it's much better to have them in there.
Saturday, August 17, 2024
evoskuil00:03
After all that work the real perf of the default allocator and the custom allocator, which performs no actual allocations and invokes no destruct, is about the same - within the margin of error. At least with unlimited RAM, I haven't tried it in a memory constrained run. So that's still good, but the point was to resolve the thread-traversing-deallocation issue, which remains unresolved.
Tuesday, August 20, 2024
evoskuil16:25
The problem I reported on July 2 is now finally resolved. The objective is to be able to cache downloaded blocks for validation so that they don't have to be read back out of the store to create the partial ordering when validating. This goes back to the v4 redesign work in 2021. The problem was memory deallocation, which was explosive and entirely prohibitive. Allocating and deallocating thousands of small chunks on the same threads is not a problem, but deallocating them on a different thread creates heavy contention, increasing sync time by an order of magnitude. This collection of blocks into a strand for caching is necessary in order to implement it in a lock-free manner. This therefore implies block destruction in distinct threads from block allocation...
(edited)evoskuil16:30
The solution consists of controlling memory deallocation for blocks, which implies controlling their memory allocation. Other aspects of memory within the node can rely on default allocation and perform well, and of course the store mmap is entirely controlled. Blocks are 99.9% of dynamic allocation, and 100% of this particular problem. So this solution required only a specialized block memory allocator, which allowed it to be fully optimized to the task (not generalized)...
The most efficient way to deallocate memory that has been allocated on distinct threads is to not deallocate it at all. The most efficient way to allocate on independent threads is to allocate once for a full block, on each thread separately, and reuse the allocation for each block in order on that thread. So I started with this...
(edited)
jungly16:33
So there's one strand for each block? Essentially leaving all memory alloc/free in the same thread for the same block?
Strands are ordered but do not own a thread.
ASIO can impose non-concurrency (unordered), non-concurrency with order, and concurrency (disorder).
But you cannot use it control which thread you operate on, it just pulls from a pool.
So in this case, if there is a 16 thread pool feeding a strand, then the strand can be operating on any one of the 16 threads, but preserves order and non-concurrency.
So each block is deserialized on the channel strand, which means any one of 16 threads. And memory being a shared global resource, it's the threads that matter.
jungly16:39
So basically now, even if a different thread comes to serve a block's next state, we are able to avoid the deallocation cause we don't dealloc the memory from the different thread? We just reuse the memory when processing the next block?
evoskuil16:40
Internally, thread safe allocators (like new/delete) have to sequence access to the single memory resource. They optimize this by taking smaller chunks from a per-thread larger allocation. So that's very efficient. But when it comes time to free the memory, and you aren't on the thread that allocated it, that can be expensive (like a lock). With little contention it's fine, but this is deallocating millions of little chunks per second.
(edited)👍🏾 1
So the initial approach was to allocate a fixed buffer, exceeding the max deserialized block allocation requirement, on each thread. This is easy, just create the allocator on a thread static. Since the calls are infrequent (once per block) this is low contention (C++ must still sequence the thread static).
A custom allocator was applied to the entire block, which was the most challenging aspect, but simple now that it's done. Then just ran with a default custom allocator that logs allocations, making sure every byte is accounted for.
Then swapped that out for a linear allocator, which is most efficient. The single allocation per thread is reset to the first byte at the beginning of each block. It's the same allocation for all blocks on that thread. Then the allocator just provides memory as requested by advancing a pointer. It's single threaded, so this is absolutely trivial and cannot be outperformed.
Then the block is stored and released, on the same thread (as before). The release invokes the thousands of destructing deallocations, but the deallocation part is a nop, so that's also as cheap as it gets. I also skipped the destructions, but that saves nothing, since they are all optimized out. So those remain for safety.
Despite the customization overhead, this resulted in slightly faster overall sync performance, but didn't finish the solution. Blocks need to be forwarded to another thread and archived, so they need to own their own memory. That also implies they need to release the one bug chunk upon destruct.
(edited)So the next step was to allocate a copy of the block into a new allocation, once the block was constructed and the required byte size was known. The allocation and copy were fast, and the release of the single chunk was invoked via the block destructor. This appeared to work, and performed well when forwarding blocks to another thread. However there was a stupid design flaw which I failed to forsee.
Any guess?
evoskuil16:54
Not really, exact memory for each block, plus the buffer. True, but that's not really a problem. I'm talking prohibitive.
It can copy the entire mmap to another location. Same basic idea, same basic problem.
(edited)When you move the memory it feels like data, but it's also self-referential pointers into absolute locations. All of the shared_ptr values become invalidated (for example).
This does sound like a problem that could benefit from a memory buffer that all these databases build for themselves. It's very very similar to your hashtable designs
But maybe that's an overkill
But good to know I'm not alone in missing that design flaw. 😅
😄 1
So... the reason I went with copying out of the buffer, which was really just three lines of code, was that detaching the initial buffer implies massive overallocations. The largest block is like 200x the wire size, and it can be much worse. So that' prohibitive unless the allocations can be trimmed back before dispatching the block to cache.
There is a C++ call to (
srd::realloc(ptr) ) to truncate (or expand) an existing allocation. But the truncation can and does move the memory sometimes, generally when it's smaller.evoskuil17:06
Copying allowed me to control the initial size of the allocation to be copied into, but there remained the copy pointer invalidation issue.
So the next approach was to detach the very large allocation in full, without the copying. I did this and completed a run on my big memory machine, with about a +10% sync time hit for the large allocations.
Then I tested with std::realloc, trimming the big allocations back, and it performed a little better, but as I shrunk the allocation as a test, I noticed it started moving memory. So like you said, unreliable.
Then I realized I could use mmap to truncate allocations, however this may not be worth doing. In the mean time, I wanted to deal with the fact that there are some very large deserialized blocks, disproportionate from wire size. The largest is almost 52 MB, and the next largest art under 20MB. But only .15% of blocks get above 20x wire size.
😯 1
So the next step was to use the allocated block to link to subsequent blocks (linked list), managing this entirely internal to the allocator. This allowed me to reduce the malloc chunk size to anything, and use this as the allocation granularity. So trimming becomes much less important, possibly counterproductive. So I change the block size config parameter to a multiple of wire size parameter, defaulted to 20, and implemented the malloc'd linking and chained free upon release.
This is release of the block, which frees all of its memory. This invalidates any objects within the block, such as a
shared_ptr<tx> that someone might copy from the block. This is an unexpected behavior that has to be managed. The block clears all of its memory upon destruct, even that owned by internally-maintained shared pointers. But that's how we avoid the large number of releases upon destruct.For public C++ interfaces a deep copy of the block (or contained objects) could be emitted, which is plenty fast enough.
So even though we are chaining allocations, the quantity is controllable vs. the excess allocation. And allocation is probably trimmable as well (I coded it up, but using realloc, but with the smaller allocations this remaps very frequently). The very low quantity of allocations, even with small granularity, resolves the forwarded block deallocation problem. Since all allocations apart from new chunks are entirely linear, and all deallocations are nop except for the chunks, it's very performant. Broke 60 minutes yesterday - which includes setting confirmation metadata (ready-to-run node).
(edited)👏🏾 1
I'll be coding up some more unit tests, taking a look at how holding on to the allocated blocks in cache affect subsequent allocations, and looking into truncation. But I consider this resolved at this point.
Yes, the difference between a wire serialized block and a deserialized block with cached hashes for all txs (2 each because of segwit) is significant. Shared pointers, vectors, full sized integers... typical is under 10x, and outliers are over 20x.
Wednesday, August 21, 2024
Shashank Shekhar Singh07:31
Can anyone help me out, tackle the issue in the above screen shot.
Whenever I do a new clone, and run autogen.sh and then install.sh I get this error.
5 replies · pmienk, Shashank Shekhar Singh · last Aug 21, 2024
pmienk16:08
Second, the screenshot provided does not provide enough detail related to your stated error.
pmienk16:09
Third, you might want to start by describing what you want to accomplish, as your methodology does not suggest your goal
pmienk16:18
The screenshot provided indicates a dependency failure. Please provide the command line argument(s) which led you here. Additionally, information as to how you are satisfying the boost dependency would be helpful. Otherwise I assume this is a boost versioning satisfaction issue. My guess which is heavily based on assumption is that you have a boost install for your system which provides an incompatible version, you haven't prefixed your invocation of install.sh and/or haven't allowed install.sh to
--build-boost3 replies · pmienk, Shashank Shekhar Singh · last Aug 22, 2024
pmienk16:11
The above does not state this error. Please describe or provide a log of the steps taken and the intentions underlying the actions
pmienk20:36
I'm glad the issue was resolved, but the described need for autogen is still concerning given that the cloned repo stated wouldn't be used without environmental variables related to github actions.
Shashank Shekhar Singh19:46
I missed the --build-boost flag, therefore the program wasnt' able to compile and generate the .configure file I guess.
Thursday, August 22, 2024
1 reply · evoskuil · last Aug 22, 2024
Huddle with pmienk, Shashank Shekhar Singh, Tanay Raikhere, evoskuil · 62 min19:59
2 replies · Shashank Shekhar Singh, pmienk · last Sep 05, 2024
Shashank Shekhar Singh20:59
In file included from ./include/bitcoin/node.hpp:19,
from console/executor.hpp:27,
from console/executor.cpp:19:
./include/bitcoin/node/block_arena.hpp:46:12: error: looser exception specification on overriding virtual function ‘virtual size_t libbitcoin::node::block_arena::detach() noexcept (false)’
46 | size_t detach() THROWS override;
| ^~~~~~
In file included from /usr/local/include/bitcoin/system/allocator.hpp:28,
from /usr/local/include/bitcoin/system.hpp:17,
from /usr/local/include/bitcoin/database.hpp:17,
from console/executor.hpp:26,
from console/executor.cpp:19:
/usr/local/include/bitcoin/system/arena.hpp:62:20: note: overridden function is ‘virtual size_t libbitcoin::arena::detach() noexcept’
62 | virtual size_t detach() NOEXCEPT = 0;
| ^~~~~~
In file included from ./include/bitcoin/node/block_memory.hpp:24,
from ./include/bitcoin/node/full_node.hpp:24,
from src/chasers/chaser_populate.cpp:26:
./include/bitcoin/node/block_arena.hpp:46:12: error: looser exception specification on overriding virtual function ‘virtual size_t libbitcoin::node::block_arena::detach() noexcept (false)’
46 | size_t detach() THROWS override;
| ^~~~~~
In file included from /usr/local/include/bitcoin/system/allocator.hpp:28,
from /usr/local/include/bitcoin/system.hpp:17,
from ./include/bitcoin/node/chasers/chaser_populate.hpp:22,
from src/chasers/chaser_populate.cpp:19:
/usr/local/include/bitcoin/system/arena.hpp:62:20: note: overridden function is ‘virtual size_t libbitcoin::arena::detach() noexcept’
62 | virtual size_t detach() NOEXCEPT = 0;
| ^~~~~~
In file included from ./include/bitcoin/node.hpp:19,
from console/main.cpp:21:
./include/bitcoin/node/block_arena.hpp:46:12: error: looser exception specification on overriding virtual function ‘virtual size_t libbitcoin::node::block_arena::detach() noexcept (false)’
46 | size_t detach() THROWS override;
| ^~~~~~
In file included from /usr/local/include/bitcoin/system/allocator.hpp:28,
from /usr/local/include/bitcoin/system.hpp:17,
from /usr/local/include/bitcoin/database.hpp:17,
from ./include/bitcoin/node.hpp:17,
from console/main.cpp:21:
/usr/local/include/bitcoin/system/arena.hpp:62:20: note: overridden function is ‘virtual size_t libbitcoin::arena::detach() noexcept’
62 | virtual size_t detach() NOEXCEPT = 0;
| ^~~~~~
In file included from ./include/bitcoin/node/block_memory.hpp:24,
from ./include/bitcoin/node/full_node.hpp:24,
from src/chasers/chaser_block.cpp:25:
./include/bitcoin/node/block_arena.hpp:46:12: error: looser exception specification on overriding virtual function ‘virtual size_t libbitcoin::node::block_arena::detach() noexcept (false)’
46 | size_t detach() THROWS override;
| ^~~~~~
In file included from /usr/local/include/bitcoin/system/allocator.hpp:28,
from /usr/local/include/bitcoin/system.hpp:17,
from ./include/bitcoin/node/chasers/chaser_block.hpp:22,
from src/chasers/chaser_block.cpp:19:
/usr/local/include/bitcoin/system/arena.hpp:62:20: note: overridden function is ‘virtual size_t libbitcoin::arena::detach() noexcept’
62 | virtual size_t detach() NOEXCEPT = 0;
| ^~~~~~
In file included from ./include/bitcoin/node/block_memory.hpp:24,
from ./include/bitcoin/node/full_node.hpp:24,
from src/chasers/chaser_confirm.cpp:26:
./include/bitcoin/node/block_arena.hpp:46:12: error: looser exception specification on overriding virtual function ‘virtual size_t libbitcoin::node::block_arena::detach() noexcept (false)’
46 | size_t detach() THROWS override;
| ^~~~~~
In file included from src/block_arena.cpp:19:
./include/bitcoin/node/block_arena.hpp:46:12: error: looser exception specification on overriding virtual function ‘virtual size_t libbitcoin::node::block_arena::detach() noexcept (false)’
46 | size_t detach() THROWS override;
| ^~~~~~
In file included from /usr/local/include/bitcoin/system/allocator.hpp:28,
from /usr/local/include/bitcoin/system.hpp:17,
from /usr/local/include/bitcoin/database.hpp:17,
from ./include/bitcoin/node/chasers/chaser_confirm.hpp:22,
from src/chasers/chaser_confirm.cpp:19:
/usr/local/include/bitcoin/system/arena.hpp:62:20: note: overridden function is ‘virtual size_t libbitcoin::arena::detach() noexcept’
62 | virtual size_t detach() NOEXCEPT = 0;
| ^~~~~~
In file included from /usr/local/include/bitcoin/system/allocator.hpp:28,
from /usr/local/include/bitcoin/system.hpp:17,
from ./include/bitcoin/node/block_arena.hpp:23,
from src/block_arena.cpp:19:
/usr/local/include/bitcoin/system/arena.hpp:62:20: note: overridden function is ‘virtual size_t libbitcoin::arena::detach() noexcept’
62 | virtual size_t detach() NOEXCEPT = 0;
| ^~~~~~
In file included from ./include/bitcoin/node/block_memory.hpp:24,
from src/block_memory.cpp:19:
./include/bitcoin/node/block_arena.hpp:46:12: error: looser exception specification on overriding virtual function ‘virtual size_t libbitcoin::node::block_arena::detach() noexcept (false)’
46 | size_t detach() THROWS override;
| ^~~~~~
In file included from /usr/local/include/bitcoin/system/allocator.hpp:28,
from /usr/local/include/bitcoin/system.hpp:17,
from /usr/local/include/bitcoin/network.hpp:17,
from ./include/bitcoin/node/block_memory.hpp:23,
from src/block_memory.cpp:19:
/usr/local/include/bitcoin/system/arena.hpp:62:20: note: overridden function is ‘virtual size_t libbitcoin::arena::detach() noexcept’
62 | virtual size_t detach() NOEXCEPT = 0;
| ^~~~~~
In file included from ./include/bitcoin/node/block_memory.hpp:24,
from ./include/bitcoin/node/full_node.hpp:24,
from src/chasers/chaser.cpp:25:
./include/bitcoin/node/block_arena.hpp:46:12: error: looser exception specification on overriding virtual function ‘virtual size_t libbitcoin::node::block_arena::detach() noexcept (false)’
46 | size_t detach() THROWS override;
| ^~~~~~
In file included from /usr/local/include/bitcoin/system/allocator.hpp:28,
from /usr/local/include/bitcoin/system.hpp:17,
from /usr/local/include/bitcoin/database.hpp:17,
from ./include/bitcoin/node/chasers/chaser.hpp:22,
from src/chasers/chaser.cpp:19:
/usr/local/include/bitcoin/system/arena.hpp:62:20: note: overridden function is ‘virtual size_t libbitcoin::arena::detach() noexcept’
62 | virtual size_t detach() NOEXCEPT = 0;
| ^~~~~~
In file included from ./include/bitcoin/node/block_memory.hpp:24,
from ./include/bitcoin/node/full_node.hpp:24,
from src/full_node.cpp:19:
./include/bitcoin/node/block_arena.hpp:46:12: error: looser exception specification on overriding virtual function ‘virtual size_t libbitcoin::node::block_arena::detach() noexcept (false)’
46 | size_t detach() THROWS override;
| ^~~~~~
In file included from /usr/local/include/bitcoin/system/allocator.hpp:28,
from /usr/local/include/bitcoin/system.hpp:17,
from /usr/local/include/bitcoin/database.hpp:17,
from ./include/bitcoin/node/full_node.hpp:22,
from src/full_node.cpp:19:
/usr/local/include/bitcoin/system/arena.hpp:62:20: note: overridden function is ‘virtual size_t libbitcoin::arena::detach() noexcept’
62 | virtual size_t detach() NOEXCEPT = 0;
| ^~~~~~
In file included from ./include/bitcoin/node/block_memory.hpp:24,
from ./include/bitcoin/node/full_node.hpp:24,
from src/chasers/chaser_header.cpp:24:
./include/bitcoin/node/block_arena.hpp:46:12: error: looser exception specification on overriding virtual function ‘virtual size_t libbitcoin::node::block_arena::detach() noexcept (false)’
46 | size_t detach() THROWS override;
| ^~~~~~
In file included from /usr/local/include/bitcoin/system/allocator.hpp:28,
from /usr/local/include/bitcoin/system.hpp:17,
from ./include/bitcoin/node/chasers/chaser_header.hpp:22,
from src/chasers/chaser_header.cpp:19:
/usr/local/include/bitcoin/system/arena.hpp:62:20: note: overridden function is ‘virtual size_t libbitcoin::arena::detach() noexcept’
62 | virtual size_t detach() NOEXCEPT = 0;
| ^~~~~~
In file included from ./include/bitcoin/node/block_memory.hpp:24,
from ./include/bitcoin/node/full_node.hpp:24,
from src/chasers/chaser_check.cpp:28:
./include/bitcoin/node/block_arena.hpp:46:12: error: looser exception specification on overriding virtual function ‘virtual size_t libbitcoin::node::block_arena::detach() noexcept (false)’
46 | size_t detach() THROWS override;
| ^~~~~~
In file included from /usr/local/include/bitcoin/system/allocator.hpp:28,
from /usr/local/include/bitcoin/system.hpp:17,
from /usr/local/include/bitcoin/network.hpp:17,
from ./include/bitcoin/node/chasers/chaser_check.hpp:24,
from src/chasers/chaser_check.cpp:19:
/usr/local/include/bitcoin/system/arena.hpp:62:20: note: overridden function is ‘virtual size_t libbitcoin::arena::detach() noexcept’
62 | virtual size_t detach() NOEXCEPT = 0;
| ^~~~~~
CXX src/chasers/libbitcoin_node_la-chaser_storage.lo
make: *** [Makefile:1633: console/bn-executor.o] Error 1
make: *** Waiting for unfinished jobs....
make: *** [Makefile:1402: src/libbitcoin_node_la-block_memory.lo] Error 1
make: *** [Makefile:1395: src/libbitcoin_node_la-block_arena.lo] Error 1
In file included from ./include/bitcoin/node/block_memory.hpp:24,
from ./include/bitcoin/node/full_node.hpp:24,
from src/chasers/chaser_snapshot.cpp:24:
./include/bitcoin/node/block_arena.hpp:46:12: error: looser exception specification on overriding virtual function ‘virtual size_t libbitcoin::node::block_arena::detach() noexcept (false)’
46 | size_t detach() THROWS override;
| ^~~~~~
In file included from /usr/local/include/bitcoin/system/allocator.hpp:28,
from /usr/local/include/bitcoin/system.hpp:17,
from ./include/bitcoin/node/chasers/chaser_snapshot.hpp:22,
from src/chasers/chaser_snapshot.cpp:19:
/usr/local/include/bitcoin/system/arena.hpp:62:20: note: overridden function is ‘virtual size_t libbitcoin::arena::detach() noexcept’
62 | virtual size_t detach() NOEXCEPT = 0;
| ^~~~~~
evoskuil23:05
After some refactoring and a new release build run with the debugger attached, I hit a break in
witness deserialization. There was a change to deserialization such that a byte vector (data_chunk) is allocated by the stream reader and returned as a raw pointer. This was immediately assigned to a shared_ptr, which will accept either a nullptr (shared_ptr.empty()) or a pointer to the data_chunk object. The pointer is then dereferenced, which is a segfault if the value was nullptr. So just needed to check the return value of the pointer or (bool)reader and exist the witness reader early.Friday, August 23, 2024
evoskuil00:36
The change was done to allow the vector to be created by the allocator instead of being returned as stack-allocated. The failure would only reproduce when an invalid block was received, and happened to attempt witness deserialization while the stream was invalid. This is pretty common, and there is no early termination of deserialization due to invalidity, as this would introduce a lot of additional conditions.
Saturday, August 24, 2024
Sunday, August 25, 2024
@pmienk if you are free in a day or two, I would like to have a small meet to set up my development environment. I think that can help me develop at a better pace and ofc testing
2 replies · pmienk, Shashank Shekhar Singh · last Aug 25, 2024
pmienk07:46
Otherwise, I'm happy to try and find a time. I can usually make myself available after 8pm Pacific Standard Time. Though it may be easier to provide instructions over text
evoskuil21:25
I encountered another minor issue with the memory allocator during testing. The current build is corrected, but the previous would infrequently fail due to the presumption that std::malloc was zero-filling, which it is not. Should be done with unit testing the allocator and integration today.
👍 2