Bitcoin full node and query server
Jun 28, 2026
View in context in the libbitcoin Slack #general mirror.
153 replies · pmienk, evoskuil, Kevin Ravensberg · last Jul 01, 2026
evoskuil21:40
Yes, and even on fast internal I'd expect to see that periodically on lower RAM machines.
Kevin Ravensberg21:43
Yeah, I resorted to unplugging the drive as there was nothing happening anymore and even explorer.exe was struggling
Kevin Ravensberg21:44
Maybe I'll grab another internal nvme, or hopefully I get the chance to buy the Steam Machine and do it on there 😄
👍🏻 1
evoskuil21:47
A thumb drive is plenty to run Core. If you want that speed you can just set one network thread.
😅 1
Kevin Ravensberg21:55
I don't think it was logged anywhere, now trying again with some memory cap on the process and logging with resource monitor
Kevin Ravensberg21:56
Well, after unplugging the SSD and stopping bs process I could restart explorer.exe and continue
evoskuil22:00
Yep, you can't halt the process and reuse the store, unless you have taken a snapshot to recover to. That happens automatically once the node is current, and you can command it at any time.
evoskuil22:00
That doesn't mean you synced 178GB, that's just the initial file allocations at the time of halt.
Kevin Ravensberg22:05
Yeah, is there a part that I can split off to the faster OS disc that is not the entire database?
evoskuil22:06
The store doesn't have configuration for independent file paths. Nothing too difficult about it tho.
evoskuil22:07
I considered putting the headers on their own path, they need high speed random access and the bodies need high speed write append (for milestone). Ahd the headers are much smaller.
(edited)Kevin Ravensberg22:08
Yeah, maybe machines with a very fast nvme OS-disc can benefit if the headers live there whilst the block body data goes to a slower larger drive.
Kevin Ravensberg22:11
The DDR4 memory kit I currently have costs 1000eu now new, bought it for 400 or so. So upgrading to 128GB is quite expensive.
👍🏻 1
Kevin Ravensberg22:15
I do have 2.5GB ethernet on this machine, so networkwise I might set a record
🔥 1
evoskuil22:16
I hit
39.4 mins on my 2.5Gbps Internet for milestone sync to 950k last night. Tough to beat, almost the theoretical limit.Kevin Ravensberg22:17
I need to buy an extra 10G network pcie card, because I have 8GB/s available on my fiber
evoskuil22:19
Looks like we're getting down closer to 2 hrs for full validation sync... we'll see soon.
evoskuil00:14
- confirmed and validated are two distinct states.
- archived and validated are not ordered.
- confirmed can have long delays and then race ahead (not constant rate at all).
👍 1
evoskuil00:19
If you give me some info I can let you know what's going on, but this doesn't tell me much.
evoskuil05:02
Overcommit is the operating system making promises it knows it cannot keep. OOM kill when swap is available is fundamental unreliability. That unreliability is a consequence of an ancient primitive that long ago lost its usefulness, and has corrupted every flavor of Unix that followed. It's so pervasive that people have entirely forgotten what is the fundamental contract of an operating system.
👀 1
Kevin Ravensberg19:53
Update: My PC 'froze' overnight, but this time after a successful snapshot, so it's continuing IBD now 😛
*froze as in the password field to login to winblows had disappeared and I could not force open a terminal window like I normally could.
Kevin Ravensberg17:55
I told it to do half way or so, but turns out it's better not to snapshot half way IBD.
Kevin Ravensberg18:39
Well, partially unresponsive. The terminal with libbitcoin is unresponsive and new terminals likewise. The snapshot tool worked, other running apps worked. Restarting seems to not work either in this state.
Kevin Ravensberg18:42
My personal log file of a monitoring script that periodically looked at the progress
evoskuil18:43
That's not my question. I'm just trying to understand from where you are getting the progress indications for the node.
(edited)Kevin Ravensberg18:44
I'll look it up in the script, don’t remember if it is checking the localhost webinterface or terminal output
evoskuil18:46
the web interface is disabled by default until current, and the console does not emit block status by default, and emitting it would be far too verbose for IBD.
evoskuil18:46
It may be monitoring the event log file, but logging is also low thread priority and buffered.
Kevin Ravensberg18:48
2. Sampler:
monitor-ibd.ps1
A PowerShell loop (meant to run continuously in the background) that every 60s:
- Reads the tail of
events.log - Takes themax height seenfor each of the 4 stage names (events log out of order under concurrency, so max-over-tail is the safe aggregation)
- Appends one row to
D:\libbitcoin\node\logs\ibd-progress.csvwith awall-clock UTC timestamp— this is what makes it survive node restarts, sinceevents.log's own elapsed-seconds column resets each time the node restarts.
evoskuil18:48
Furthermore confirmations will always back up against validation, so if you are monitoring the log for confirmations, there can be long delays. If you're in a situation where the node is barely making progress due to resource constraints, then there may be no such progress indications.
💯 1
Kevin Ravensberg18:48
These four are tracked:
archived block_archived - downloaded + structural/PoW check
validated block_validated - full script/signature validation (heavy)
confirmed block_confirmed - validated AND confirmable
organized block_organized - pushed onto the confirmed chain (height)
Kevin Ravensberg18:49
I limited it to only use 24GB of RAM, as a mitigation of my PC getting sluggish
evoskuil18:49
Sure, that's a reasonable approach. The console log doesn't flush until its buffer is full, so the event log is the right place.
evoskuil18:51
Yes. I was just trying to isolate the question of process termination or hang, vs. expected behavior.
(edited)evoskuil18:51
This is expected behavior. You need to give it at least 32GB for validation, and a fast disk.
Kevin Ravensberg18:53
Working on that! And I probably should build from source once more to make use of the GPU validation?
evoskuil18:54
The process is not synchronous. It's highly concurrent and asynchronous. Things do not execute in lock step as you might intuit after using a single-threaded validator like Core. For example, a huge amount can be happening and nothing in the log.
evoskuil18:55
GPU validation is still a bit experimental, and ufsecp is not currently useful on Windows.
evoskuil18:56
When you say IBD are you including full validation? At 24GB that is not an accurate expectation.
evoskuil18:58
milestone sync (which is currently the default, at block 950k) with 32GB on 2.5 Gbps and 16 dual core decent threads takes 39 minutes.
(edited)Kevin Ravensberg18:59
I've got 32 threads / 16 cores, should be able to get all withing 1 hour too then
Kevin Ravensberg19:00
Yeah, the Samsung 990 Pro is one of the fastest PCIe Gen4 (which I'm getting a second one off for libbitcoin storage)
evoskuil19:01
Then it's not clear to me what's happening on your machine. With those specs and a milestone run, even 24GB should be sufficient.
Kevin Ravensberg19:02
I think it will get better once I actually use an internal NVMe drive instead of the external USB-C drive
evoskuil19:04
It is likely that the node is simply trying to flush 20+ GB of data to the drive, and everything is just waiting on that.
(edited)
evoskuil19:11
Actually I think it's just that it's external (which we have always told people is not supported).
(edited)evoskuil19:12
mmap writes dirty pages back as many tiny (~4 KB), scattered I/Os. Internal SSD doesn't care — low latency, deep queues, so it absorbs the pattern invisibly. USB external has high per-I/O latency and shallow queuing, so that same drizzle of small writes grinds along at single-digit MB/s, turning a ~1-minute transfer into hours.(edited)
evoskuil19:15
With enough RAM, we don't use the SSD at all, just at startup and shutdown for persistence.
(edited)
evoskuil19:17
Eventually you 'll be able to sync and validate the whole chain over the network and store it in RAM with no persistence at all, just initialize it at every startup from the network - like many apps do today that never could have even a decade ago.
Kevin Ravensberg19:18
Also the direct memory access links between servers is crazy, goes over an optical fiber
evoskuil19:18
I worked on 300 baud (bits per second) networks. With that kind of perspective you can see what's coming.
🤯 1
evoskuil19:20
Even at 14,400 baud ("fourteen four") loading a single lo res image required a coffee break (up to 4 mins). That was only 35 years ago. 😅
(edited)🙌 1
Kevin Ravensberg19:25
I only experienced the dial up internet over phone lines as a kid, that was 4x faster already
👍🏻 1
evoskuil19:29
Now we can sync and index the entire bitcoin blockchain over home Internet in the time it took to transfer a single VGA image in 1984.
🚀 1