Mar 16, 2024

The key design principle is based on Amir's append only design, however that suffered from hash table headers being co-located within the body files. The headers are random write access and cannot be recovered in case of a fault. Also memory maps must be flushed before they can be relied upon upon shutdown or for hot copy. This of course also required that indexation spanning multiple files was logically atomic. So to resolve these issues we split the headers and bodies, enforced true append only on the bodies, created a transactional commit system using a mutex-based transactor (only locked during backup, so not a performance consideration), created a method to pause channel reads (thereby suspending store writes) without dropping channels, and implemented flush to disk of all files after obtaining the exclusive transactor and before taking index snapshots.