Mar 17, 2024

Another interesting aspect of the memory map store that I forgot to mention is that, when a map for a given file needs to be made larger the operating system can remap it. In other words, invalidate all outstanding pointers into the map. This was another aspect that hadn't been dealt in the original implementation. This requires, on a per file basis, that each read/write pointer obtained from the store is an object that encapsulates a remap mutex. When the map allocation is exceeded for the given file, the store blocks until all shared locks are released and then holds exclusive access during the remap. This requires that all pointers are very short lived and cannot be passed into code that could be waiting on another pointer (deadlock). This was something that I implemented in the prototype, but it wasn't sufficiently safe. So in the current store reads and writes encapsulated within low level store primitives which manage memory access, passing higher level result sets back to the query implementations, and these providing the custom query interface that the store exposes publicly.