Sep 2, 2019

Yup. Totally get that. It's just a to measure things, so we can know how things are improving etc. I was curious to know how much io was coming due to vm paging effort. I went for a 256 GB ram and 12 virtual cores wth nvme SSD machine. It looked interesting how there were no real page faults up to a certain amount of memory usage and then they go up quiet rapidly (kind of expected). But I'll share the results soon so we can look interpret them properly.

excellent, thanks.

Hey @evoskuil. Here's a first pass on the experiments I ran to see how disk and virtual memory was being used when running IBD. If nothing, hopefully this provides us a way to keep measuring as we keep improving the system. Let me know if should have done anything differently, I can have another go, or improve what we capture for the next time.

Blocks start to go from ~1kb to ~1mb as you get around 300,000, 3 orders of magnitude increase.

Yes. Of course! Thanks. It's not doing that bad then.

The increase in the reads is also as you expected.

Also note that resource usage is maximized in terms of RAM, CPU and network. So the more work there is to do, the more we use, limited by what the operating system will offer. The optimization is for speed.

Header download is not optimized, as it intends to allow slow peers in the interest of ensuring that fast peers don't always provide the chain.

It seems disk wasn't the bottle neck. Right?

Reading is clearly dominating performance, it's causing writes to decline as read cost slows down validation.

Am I reading correctly that you downloaded all blocks in less than 4 minutes with swap?

Yes. That was a bit mad.

That's the optimized part. As you can see, if we can conquer the read cost it can make a big difference.

I've implemented an in-memory cache (height-ordered circular buffer hash table) of downloads to preclude re-reading each block for validation. With enough memory this can contain the entire chain, reducing reads to much less than writes.

There is a cache in master,iirc. I left the default settings for it. Or is there a new one not in master?

The master cache is a dynamic blind cache of unspent outputs, which populates by height following validation, and is indexed by tx hash. It's useful on low memory machines, but will actually reuduce performance on high memory machines, since there's a population cost but no benefit.

The download cache can be very helpful, but not so great after a restart following a large set of downloads.

The worst case is where it's faster to download in order off the network than it is to read from the disk. That seems like an unlikely scenario, in which case, as long as the initial parallel download is fast enough to be negligible it would always be faster even to read after downloading.

That's the premise of the design, we'll see 🙂

The worry would be if the cache itself gets paged out to disk. Maybe we will have to consider mlock on cache and indexes? I am sure you thought about it, any reason we don't do it yet for things like indexes? Are they too large for small memory machines?

I don’t think we want to try and stop paging if the OS thinks it’s optimal. I think mlock could cause more significant problems. Given that indexes are more frequently accessed and small they should tend to be retained.

It’s a design principle to allow the OS to optimize resource usage, which is its function.

We just try and use all resources, without wasting any of course.