Aug 6, 2024

It took some time to build up the infrastructure to reliably deserialize all aspects of a block into a custom allocator, however the amount of new code is very small, and the related changes to chain object deserialization are simple. Having done that it is now possible to destruct blocks on any thread without any memory deallocation cost whatsoever. The memory allocation cost is also close to zero. In comparison to a full chain sync using the default memory allocator, the new custom linear allocator is slightly faster even when blocks are deallocated on the allocating thread. However when blocks are projected to a common strand for caching, the increased cost is only about 10%, none of which is in deallocation - just posting the closures to a work queue. Previously, using the default memory allocator, the increased cost was totally prohibitive. The total run time went from about 60 minutes to at least 24 hours if not longer (never finished). So that aspect of the issue has been resolved. The cost of using the custom allocator is a fixed memory allocation for each network thread, equal to the largest expected block object size possible. Some measurements demonstrated that all but 1 of the 840k blocks have an object size under 12MB (wire serialization is limited to 4MB).