Oct 27, 2024

6 hours into this run, with 8 hours estimated and rate decreasing. This monster Mac pro run will end up taking longer than the Windows mini run.

15 hours into the monster mac pro Bitcoin Core sync and reporting 6 hours to go. Significantly worse than both the 15.3 hour windows mini sync, and the 9.5 hour Linux mini sync. All on a measured 2.3Gbps Internet, no script validation (assume valid) and resource unconstrained.

The lack of shani probably explains a big part of the difference between the big Mac and the Linux and Windows minis. However, BN does not even implement shani yet, and completes this in under an hour on a 32GB/16 thread windows mini, whereas this is a 728GB/32 thread machine. Both BC and BN are doing the same amount of hashing, because both skip script execution and are performing all other hashing (all headers, txs, with and without witness, merkle roots for tx and witness commitments). The difference is that BN is doing it on all threads, close to 100% on the mini, where BC looks more like a single-threaded app on this network and database aspect. As such these local optimizations like shani make a big difference. But this is really at the center of one of the main architectural distinctions. BC is at its heart a single threaded application that uses threads, and only really fans out for script validation. BN is only improved by shani and hash vectorization when it becomes CPU bound. This is why shani is deferred work - it can be dropped into the sha template with no change to any other code.

Since the theoretical min time is 38 minutes, and we are at 47-50, and generally CPU bound on the mini, I suspect that with shani we will see around a 10% reduction. The rest is a consequence of RAM flushing to disk, and this can only be entirely eliminated with full RAM (eg the big Mac). At that point we should be under 40 minutes, and cannot get past that without higher bandwidth.

10 Gbps Internet is already being deployed to consumers in the US. That will drop the theoretical limit to under 10 minutes, even with chain growth.

If you can download the full chain in 10 minutes, how long does it take to validate? Well once it's downloaded, the full chain can be validated concurrently, and it's all read. So as long as the SSD/RAM can keep up, it's limited only by CPU count. We already run validation at 100% CPU on 64 cores. While shani and other local optimizations can help, it's really just Moore's Law at this point.

The Core big Mac sync has reached block 850k at 18.1 hours. I was previously using 840k for benchmarking, but I'm updating it to 850k to keep it current and because we are using 50k sampling, so this will make accurate plotting easier.

For comparison with the previous two BC runs, the time to 840k was 17.35 hours.

Completed the BC monster Mac Pro (no shani) sync to 867,619 in 23.4 hours (default config, no tx index, some validation near the top).