Nov 5, 2024

The database community is very split on mmap. Andy Pavlo is fighting a big battle to get DBMSes to not use mmap for a generic database. https://db.cs.cmu.edu/mmap-cidr2022/ The story about mangodb is illuminating. On the other hand, Eric has shown if you know the precise queries and you know the schema intimately, you can make it work. I would stay away from a generic mmap database, or at least let's discuss the issues listed in the paper below.

I would absolutely not recommend it for use in any general purpose database. mmap must page to get/set any amount of data. Data locality and access patterns become a significant consideration, pretty much dominating everything else. These are not generally encountered under you start operating under heavy stress, so it can look great until you get into thrashing and then it goes parabolic.

We have almost a decade of experience with this. Another major issue is fault tolerance and reliability. If you don't have an append only store, you can forget about fault tolerance, and reliability will be limited to detecting that the store is corrupt due to an incomplete flush on shutdown, but you will not be able to fix it (resync).

We tossed out the previous iteration and started over due to these issues. It cost a couple years at least. But the learning helped us understand how to achieve fault tolerance and reliability (with much stronger guarantees than BC), as well as achieving hot file system backup - which were all original goals of Amir's when he started with mmap. Finally, we have learned how to manage thrashing, but it's a minefield sometimes.

Thrashing is a complete non-issue with sufficient RAM. And ultimately the objective is to map nothing to SSD but to just maintain the store in RAM. I have a 256GB PC and a 768GB Mac so that I can test these. The 256GB was enough to hold the full chain when I started.

Then there's the issues of achieving (1) performant write/write thread safety (as well as read/write and read/read of course), (2) cross-file transactional commit for snapshotting and disk full scenarios (we handle disk full without the need for snapshot/recovery or restart), and (3) global read/write pointer invalidation when any given table requires a remap for expansion.

I do not think a general purpose database can come anywhere close the query performance we obtain, however solving these problems is some non-trivial specialized work.

I realize it may just be a curiosity for you, but BC doesn't have a store performance problem. That would be premature optimization. It most critically has an ordering problem.

thanks, helpful insights. and not really a curiosity, more was looking into leveldb replacements for a different problem i want to solve. after experimenting with mdbx a bit, im arriving at some of the same conclusions re: mmap for general purpose database.

im not entirely sure what you mean by this, assuming you mean ordering problem in the sense that blocks must be processed sequentially, effectively going through each state transition of the UTXO set to arrive at the chaintip ?

Yes. Concurrency is really about finding the places where unnecessary order is imposed on an algorithm. Sometimes this is low level, as in the way critical sections are imposed via mutex locking. Other times it's much higher level, like imposing a partial order on downloading and a total order on validation where neither are logically required. The result is the same however, underutilized resources.

There are of course tradeoffs. Concurrency imposes synchronization cost, which has to be measured against the benefit. For example, it is not required that txs be validated with their block, and it is not required that all previous blocks (or txs) be downloaded in order to validate a given block's txs. However tracking and notifying a tx (or block) that all of its required previous outputs have been downloaded is more costly than the partial ordering by block constraint.

Similarly fan-out by tx adds additional cost with no gain, if there are sufficient numbers of concurrent blocks to validate such that the CPU is fully occupied. This is almost always the case during initial and catch-up sync, but not at the very start and end of the sync. But still not worth that tiny advantage. On the other hand, it's materially faster to fan-out by tx when validating confirmability by block, given sequential execution by block.

I'm always looking for places where we've inadvertently imposed an ordering constraint where there might be a benefit that outweighs the synchronization cost. One case is that confirmability does not require validity, so that could run completely independently of validation (presently it chases validation). Another case is that confirmability can run concurrently by block within a candidate branch - but this may not be worth the synchronization cost (data-intensive).