00 — Overview

Companion to 01-primitives, 02-tables, 03-storage and 04-queries.

1. Layers#

2. A relational store without a query engine#

The store is strictly relational, but every relation is resolved by one of two physical operations chosen when the table is designed:

• Navigation. A row holds the link of a row in another table. A link is that row's position (record ordinal or byte offset), so following it is a multiply and an add: no hash, no search, no lock beyond the remap lock. Links are the store's foreign keys (FK). Tables do not store their own primary keys; a row's PK is its position.

• Search. A hash head maps a key to the newest row with that key, and rows in a bucket chain to older rows. Search is how a natural key (NK) from outside the store becomes a link.

The 2021 design names the key kinds used throughout:

Invariant (Table-1). The only public entries to the archive are the two natural keys exposed by the chain: block hash and txid. Every other archive object is reached from one of them by navigation, or by a height index once confirmed. Optional entries are output script hash (address index) and wtxid (pool).

2.1 Deferred associations#

Two associations cannot be written when their parent is written, because the store is append-only and blocks and transactions arrive out of order:

• Header → transactions. Headers are archived first (headers-first sync), and a block's transactions arrive later. The association is a single txs row written once the header and all its transactions are archived. In 2021 this was a search; it is now an array map indexed directly by the header link, so it is navigation (tables/archives/txs.hpp:36-46).

• Output → spending input. An output can be archived before or after any input that spends it. The ins hash map is keyed by the spent point (NP), so spenders of an output are found by search on that point (tables/archives/ins.hpp).

2.2 Publication#

Bodies are append-only. A row is allocated, written, and only then made reachable: by a head push (search), an array or headmap cell (index), or a foreign key written into a row that is itself published later. A reader therefore never reaches a partly written row, without any read lock on the row (01 §4.1, 04 §2).

This one property gives both concurrency (readers and the single writer of each row do not coordinate) and recovery: a consistent state is a copy of the heads, since every body row beyond what the heads publish is unreachable (03 §6).

3. Table categories#

Details per table are in 02-tables.

4. Files#

Each table has a head file and, unless it is a headmap, one body file per record (store.hpp:346-359, tables/names.hpp).

<path>/
├── flush.lock                  present while open; left behind by a faulted close
├── process.lock                exclusive OS file lock
├── heads/                      the live heads
│   ├── archive_header.head
│   ├── index_candidate.head    headmap: the whole table
│   └── ...
├── primary/                    newest snapshot of heads/
├── secondary/                  previous snapshot (transient during rotation)
├── temporary/                  snapshot being written
├── archive_header.data
├── archive_ins_point.data      multi-record bodies: <name>_<record>.data
├── archive_ins_sequence.data
└── ...

Heads are a few percent of the store. Bodies are the bulk and are never copied (03 §6).

5. Optimizations since the 2021 design#

The implementation originally followed the design closely. Each difference below is an optimization made as the store evolved.

Cross-references#

• Node architecture — the chasers that drive these queries