Nov 4, 2024

I think this will be resolved through locality. During initial and catch-up sync, organization starts once there is a stronger candidate chain, and that evaluation (download/validation/confirmability) starts once there is a current header chain. This results in a cascade of single block organizations, since the next block in order is always a stronger chain than that previously confirmed. Consequently with concurrent validation/confirmation the same inputs/outputs/prevouts and indexes will already be paged in as a consequence of full block and prevout population. This makes navigating the links for confirmability almost free and should preclude paging. The only additional searches, reads and writes are to the strong_tx table, however this is a small table (tx scale with little data) that should not cause thrashing. So instead of running confirmation in series it makes a lot more sense to run it concurrently, and instead to run download and validation/confirmability in series.

As a consequence of concurrent block validation (partial order), to preserve locality this requires that confirmability follow closely from validation. However since confirmation is ordered and validation is partially ordered, this imposes order back onto validation. Yet as long as confirmability is significantly faster (which it is, absent thrashing, by about 4x on big box) then it will not significantly delay validation. This is a direct analog to the download concurrency window and the validator closely chasing downloads via higher thread priority. This can be most simply achieved by raising confirmability thread priority above validator.

But we really only want normal/high priorities in use. This works out since we have two scenarios. Either full concurrency with validate and confirm higher than download (where confirm may significantly lag validator) and download followed by validation/confirm where va look d is normal and confirm is high. This keeps confirm close to valid, and both independent of download.

That’s a reasonable bifurcation between the maximal and minimal hardware scenarios. Full concurrency on big RAM is probably around 6 hours, and sequentiality on the mini is prob 10-12 hours, with current optimization levels. In comparison the BC full validation time on the big Win is over 24 hours (probably even more on the monster Mac)… I haven’t tried it yet on the minis, but just download is 14 on win mini and 9.5 on Linux mini.