Bitcoin full node and query server

Feb 26, 2025

View in context in the libbitcoin Slack #general mirror.

evoskuil23:14

This is a sort-of related link: https://lwn.net/Articles/572911/

18 replies · pmienk, evoskuil, parsevalbtc · last Feb 27, 2025

pmienk00:15

Sounds awesome to know. So just want to ratchet dirty bytes down?

evoskuil00:31

sudo sysctl -w vm.dirty_ratio=80
sudo sysctl -w vm.dirty_background_ratio=40
sudo sysctl -w vm.dirty_expire_centisecs=6000

evoskuil00:32

Something like that. I'm having a long chat with Grok32 about it and related questions.

pmienk00:33

huh, those numbers are much larger than my defaults of 20 and 10. My reading was that moving down the ratio or absolute bytes would more frequently but more quickly flush.

evoskuil00:36

Right, on Windows it's 100% by default.

(edited)

evoskuil00:37

More frequent flushing isn't good. The node is constantly writing to disk whereas on Windows on the same hardware writes are infrequent.

evoskuil00:38

I believe that is making the performance difference.

pmienk00:39

ahh, well fingers crossed that makes the difference and it's a system configuration issue

evoskuil00:39

Grok3 ran the numbers and seems to agree. 😅

evoskuil00:39

But basically it's like operating on a machine with 6.4GiB RAM right now.

(edited)

pmienk00:41

seems sound logic to me

evoskuil00:41

I got on this path after watching the performance monitor, and very little resources being used, like nothing. Except it was constantly writing to disk.

evoskuil00:45

From Grok:

Thanks for sharing the full map::remap_(...) implementation—it's a beautifully ugly cross-platform beast

evoskuil00:45

😅 we are so cooked

evoskuil01:41

Here's an interesting related statement:

lower caps (Linux/macOS) flush earlier, slowing effective write rate (19MB/s vs. ~333MB/s)

evoskuil01:44

That's a 17.5x advantage for Windows on the same box over Linux. And the difference in times I see is very roughly about 37 minutes to 10 hours (16x).

evoskuil01:51

You're absolutely right to catch that nuance—my phrasing in "Why This Matters" might've implied that remaps directly slow the effective write rate, which isn't precise. Let me clarify: the remaps themselves aren't slowing the write rate directly, and they're not tied to the dirty page cap in terms of their frequency (they're driven by your 5%/15% growth logic). However, the interplay between remaps, the lower dirty page caps on macOS/Linux (6-12GiB/6.4GiB vs. Windows' 32GB/256GB), and synchronous I/O operations in your map::remap_(...) creates a bottleneck that throttles the effective write rate to 19MB/s (10 hours) versus Windows' 333MB/s (37 minutes). Let's disentangle this and explain how caps and remaps interact to cause your 17.5x slowdown, ensuring we align with your setup (690GiB, 32 files, 16 threads).

parsevalbtc10:06

Awesome, in the linked chat there is also a mention to journaling in ext4 filesystem that can have a role in the slow down.