Oct 24, 2024

What is the purpose of a snapshot?

A snapshot is a flushed copy of the table heads, concurrent with the flush to disk of all table bodies. A snapshot looks exactly like the “head” subdirectory and rotates between a “primary” and “secondary” folder. This ensures that once there is a good snapshot, a failed future snapshot can always revert to the previous good one. The purpose is to allow the store to revert to the point where the snapshot was taken, in the case of a process fault (uncontrolled termination). I also use it in development to roll the store back to certain points (such as post-sync and pre-validate, by storing a snapshot once synced, though on big machines we sync and validate concurrently). The node knows for sure when a previous shutdown has not been completed, and when it starts up will automatically recover from the most recent snapshot. If there is no snapshot it will currently require a manual restart after clearing the store. This gives us full fault tolerance, which leveldb (and most general database systems) cannot actually provide. This is pretty extraordinary given that the store is fully write/write and read/write concurrent.

The other benefit of snapshots is that the store can be hot-backed up. The snapshot is written using file system atomicity, which ensures that if there is a snapshot directory it is good. So for example the store does not need to be stopped for backup to another drive by a backup system.

Automatically snapshotting 6 times during a sync adds about 5-10 minutes to the total time and ensures that recovery in a run that would take 60 minutes never loses more than 10 minutes.

Of course that can be a much more significant time savings on a slower machine/network.

Epic!