Bitcoin full node and query server
Jun 23, 2026
View in context in the libbitcoin Slack #general mirror.
josie15:35
up and running, might need to do some param tuning but so far seems to be chugging through. this is 1gbps , but im requesting a 10 gbps link for this box
214 replies · evoskuil, josie, Eric · last Jun 24, 2026
evoskuil15:36
What sustained speeds do you actually measure? I read you get about .5 on this setup.
(edited)evoskuil15:38
You should see a milestone sync running at full bandwidth limit once it hits block 300k or so. Validation runs cycle between Network and CPU limits.
josie15:44
still working my way around parsing the logs, but:
15:28:32 5,366,900 sigs 3s 1.79M sigs/s 15:30:45 1,071,533 sigs 0s ~1.07M sigs/s 15:30:55 2,315,127 sigs 0s ~2.32M sigs/s 15:31:06 4,558,730 sigs 1s 4.56M sigs/s 15:32:16 29,354,060 sigs 8s 3.67M sigs/s 15:33:31 36,497,618 sigs 10s 3.65M sigs/s 15:37:20 1,468,875 sigs 0s ~1.47M sigs/s 15:38:18 5,883,209 sigs 1s 5.88M sigs/s
🔥 1
josie16:08
heh yeah, just shut down and im trying to optimise my network config a bit and also give a bigger batch
evoskuil16:08
The secp batch rate on my machine is 360K sigs/sec, and you are almost 10x that at 3.33M sigs/sec.
evoskuil16:09
Great to see that it actually works. Now I have some motivation to work on it today, and to buy my new machine.
josie16:11
i dont really understand all the nobs in the network config, do you have a recommended one youve been using for your fast benchmarks ?
evoskuil16:12
In frigate are you passing the data to the GPU in horizontal rows, and are you using the 8 byte prefix and compressed keys?
evoskuil16:12
The defaults are pretty good for network. Give me the high level specs and I'll make some suggestions.
josie16:13
frigate uses duckdb as the embedded database. duckdb supports GPU acceleration and writing custom in database extensions
josie16:15
here is the relevant config:
[network] threads = 24 [node] threads = 24 maximum_concurrency = 300000 batch_signatures = 10000000 warn_dirty_background_ratio = 90 warn_dirty_ratio = 90
josie16:18
outbound:
[outbound] connections = 1500 connect_batch_size = 1 host_pool_capacity = 200000 seeding_timeout_seconds = 420
evoskuil16:18
I'd leave maximum_concurrency at the default of 500k. batch to 5M, ratios are just startup warnings... you need to make sure those settings are in effect on linux. matching threads to physical for node is a good start, it's for full validation and performs the block/prevout reads in parallel. With sigs no longer hitting the CPU this might go higher.
evoskuil16:19
batch size is too low, leave default of 5, since only 1 in 5 addresses is valid on average. otherwise you delay your reconnection time significantly, which increases download delays.
(edited)👍 1
evoskuil16:20
Host pool probably doesn't need to be increased, but also prob won't matter much. Just uses more ram and increases runtime circular buffer cost.
evoskuil16:21
If you aren't getting seeded in time, this is the right number to change. It won't actually delay startup, since once you have enough addresses the node starts while you finish seeding. It just delays stop if seeding fails. So if you need to increase this it's harmless.
evoskuil16:22
I leave network threads at 16 even on my 64 thread machine with 256GB RAM on my 2.5Gbps network. I did a bunch of trail runs and this was a sweet spot.
evoskuil16:25
node.delay_inbound = false native.bind = 127.0.0.1:8080 database.ecdsa_size = 10000000000 database.schnorr_size = 10000000000(edited)
evoskuil16:27
I also use this on big machines:
[database] header_rate = 15 txs_rate = 15 tx_rate = 15 point_rate = 15 ins_rate = 15 outs_rate = 15 input_rate = 15 output_rate = 15 candidate_rate = 15 confirmed_rate = 15 duplicate_rate = 15 prevout_rate = 15 strong_tx_rate = 15 validated_bk_rate = 15 validated_tx_rate = 15 filter_tx_rate = 15 filter_bk_rate = 15 address_rate = 15 ecdsa_rate = 15 schnorr_rate = 15
evoskuil16:27
The percentage table growth rates. Requires more disk overall due to up to 15% pad when synced.
(edited)evoskuil16:43
It polls but once the node is current it transitions to websocket push updates.
(edited)josie16:45
but in theory, unless i fucked up the code, you should be able to point sparrow at this API when finished and get a silent payments gpu scan
josie16:47
yeah it was faster on a previous run. im wondering if the batch size being so large is stalling out since early blocks dont have many sigs?
josie16:54
yeah was just looking into it looks like zfs was grinding to a halt for a second, not sure why
josie16:56
this box was formly a prod box for frigate + fulcrum, where we wanted to be able to quickly recover index data without resync, but for a bench test, ZFS is overkill and a slowdown. tomorrow ill rebuild the box with just good ol nvme ext4
josie16:58
page settings:
vm.dirty_ratio = 95 vm.dirty_background_ratio = 95 vm.dirty_bytes = 0 vm.dirty_background_bytes = 0 vm.dirty_expire_centisecs = 12000 vm.dirty_writeback_centisecs = 12000 SwapTotal = 0 SwapFree = 0
josie16:59
yeah i really just think that was a ZFS fluke , which should be solved by taking that out. zfs is adding CoW, compression, etc
josie17:02
[bitcoin] milestone = 0000000000000000000000000000000000000000000000000000000000000000:0 [database] address_rate = 15 candidate_rate = 15 confirmed_rate = 15 duplicate_rate = 15 ecdsa_rate = 15 ecdsa_size = 10000000000 filter_bk_rate = 15 filter_tx_rate = 15 header_rate = 15 input_rate = 15 ins_rate = 15 output_rate = 15 outs_rate = 15 path = /var/lib/libbitcoin-server/blockchain point_rate = 15 prevout_buckets = 1000000 prevout_rate = 15 prevout_size = 4000000000 schnorr_rate = 15 schnorr_size = 10000000000 silent_buckets = 1 silent_rate = 50 silent_size = 4000000000 silent_start_height = 709632 strong_tx_rate = 15 turbo = true tx_rate = 15 txs_rate = 15 validated_bk_rate = 15 validated_tx_rate = 15 [electrum] bind = 0.0.0.0:50001 maximum_history = 1000000 maximum_subscriptions = 1000000 self_bind = 136.243.9.246:50001 server_name = libbitcoin-gpu-test silent_payment_threads = 20 [inbound] bind = 0.0.0.0:8333 [log] path = /var/log/libbitcoin-server [native] bind = 136.243.9.246:8180 connections = 16 [network] threads = 20 [node] batch_signatures = 5000000 delay_inbound = false maximum_concurrency = 500000 threads = 14 warn_dirty_background_ratio = 90 warn_dirty_ratio = 90 [outbound] connect_batch_size = 5 connections = 100 host_pool_capacity = 10000 seeding_timeout_seconds = 420 [peer] enable_relay = false
josie17:06
i strongly suspect this is related to the higher GPU batch size , when i had it 1M, it validated from 0-400k in 30 mins
evoskuil17:08
I generally set milestone to genesis, so that it is found. I doubt this makes a difference however. Though otherwise it keeps looking for it.
evoskuil17:09
There is no batching at all in the first 295k blocks, so it wouldn't make any difference in that range.
evoskuil17:10
You have a 50k window, so it would still be looking at the same range regardless of the batch size.
evoskuil17:11
Confirmation not proceeding makes sense, as this is not coupled to other modes. If you got 295k fast and only waiting on confirmation, that's normal. I'm not used to watching this in the browser.
josie17:13
here is what im seeing in the logs that makes me think theres a race:
block_confirmed max=295001
block_organized max=295001
block_validated max=363037
but 295002 is archived, from the logs
josie17:15
because the GPU tables drained, but confirmation is still stuck at 2950001, despite us having 295002, the node kept downloading and eventually bricked
evoskuil17:19
because the GPU tables drained, but confirmation is still stuck at 2950001
confirmation is loosely coupled to download/validation, it does not delay them, and if CPU is at 100% (common in full validation) then it will be deferred. Also during high CPU logging will be deferred.
(edited)evoskuil17:19
despite us having 295002, the node kept downloading
So this is perfectly normal.
(edited)josie17:24
youre right, the server getting killed seems unrelated, systemd sigkilled it after telling it to stop, which was likely initiated by another nix service on the box for mem usage
👍🏻 1
evoskuil17:33
I haven't implemented handling for all stop signals on linux/mac, just <ctrl-c>. logoff/shutdown signal handling for windows is implemented, but service (daemon) registration isn't. These are some release punchlist items.
evoskuil17:36
So if it was killed on this run but not the previous runs, it likely means it was running quite a bit faster. The OS panics when the memory acquisition rate is high.
(edited)josie17:42
i think i may have found the issue. perhaps race condition isnt the best way to frame it but it does look a deadlock can happen. https://github.com/libbitcoin/libbitcoin-node/blob/75fd039ba9a51951e1542ef78a97c26167ae63bd/src/chasers/chaser_validate_batch.cpp#L52 the comment says batched_ is only a sort optimisation, but process_valids() uses batched_ afaict as the only source of blocks to mark valid and then clears it
josie17:43
it seems a block whose rows were purged but whose link missed process_valids() remains unvalidated, so get_validated_fork() stops at that height and the confirm chaser never advances beyond it, which would match what i saw with the stuck block at 295002
evoskuil17:44
It's redundant with the results from the batching output, but if we use the batch output results for positive and negative, we need to merge the results from the ecdsa and schnorr tables. On order to prevent duplicated validated events and store updates, that requires a merge/sort of the two result sets.
evoskuil17:44
So we reduce the results to negative only, and use this for the success set (which is always 100% of the results).
evoskuil17:45
evoskuil17:46
This is all done in the batch verification critical section:
https://github.com/libbitcoin/libbitcoin-node/blob/75fd039ba9a51951e1542ef78a97c26167ae63bd/src/chasers/chaser_validate_batch.cpp#L86
which prevents any batch accumulation.
evoskuil17:47
I've run this quite a few times over the whole chain and haven't seen any issue with it.
evoskuil17:49
I think you are assuming that the confirmation chaser is hung, but there's really no reason to believe that's the case.
josie17:50
i feel like im not explaining myself well. the problem i suspect is this:
{
std::shared_lock lock{ mutex_, std::defer_lock };
if (capturing) lock.lock();
if ((ec = block.connect(ctx, capture)))
return ec;
batched = capture.batched;
faulted = capture.faulted;
}
this code acquires a lock but releases it before the block link is added in
batched_ . it seems that the chaser can acquire a lock after the rows are written but before the blocks push_batch() handler has ran. so it verifies and purges the rows but only links in batched_ are marked as validjosie17:51
that doesnt seem to match what the comment says and implies to me the signature row can be visible before batched_ is aware of it , but the final decision is made off of what batched_ is aware of
josie17:52
its still unclear to me why the batch size would make a difference here, because i definitely did not see this with 1M batch size
evoskuil17:53
batched_ is protected by the strand. Any pending update to batched is added after the critical section completes, because it is also stranded.josie17:53
tldr; process_batch() can verify and purge a block’s signature rows before that block’s delayed push_batch() adds its link to batched_, so process_valids() never marks the block valid.
josie17:54
this leaves the block stuck unvalidated with no batch rows left to reprocess, so confirmation stalls at that height even though later batches keep completing
evoskuil17:54
This of course means that batched_ can be updated after it's accumulated sigs are verified. This is fine.
evoskuil17:55
It may be a problem in the very last ("residual") batches of the run, but you didn't encounter that.
evoskuil17:56
The critical section prevents updates to the tables during evaluation, since otherwise we would have to truncate the table above the writes.
evoskuil17:56
The batch is not coupled to the tables, it's a list of all blocks that have been accumulated. They don't have to process at the same time.
josie17:58
i think we are talking past each other, i still see an issue: what youre saying is only true if batched is not part of the commit part , but it that doesnt seem to be true
evoskuil17:58
It's only essential that the tables are processed before the batch records that accumulated them, so that invalidation preceeds validation.
josie17:59
// set_block_valid(batched_ excluding ecdsa/schnorr failures)
// ------------------------------------------------------------------------
if (!process_valids(residual))
{
fault(error::batch4);
return;
}evoskuil17:59
I understand what you are saying, and it's a good question. But I don't think you are following how message flow is is being processed in the strand.
josie17:59
and the callee:
for (const auto& link: batched_)
{
if (link == header_link::terminal)
continue;
size_t height{};
if (!query.get_height(height, link) ||
!query.set_block_valid(link))
return false;
notify_block(system::error::success, height, link, false);
}
batched_.clear();josie18:00
but at least from what i do understand, this looks like a smoking gun to me and would explain the stalled block i just saw
evoskuil18:01
The process was terminated, which means very high resource consumption. That means confirmations were not happening.
(edited)evoskuil18:02
Confirmation only proceeds in the CPU gaps, once validations are contiguous, and the queries are fairly expensive. So it is common for there to be significant delays.
(edited)evoskuil18:03
Also such activity depends on the proactor pump delivering messages, which affects logging, and eventing as well. So under heavy load logging can be significantly delayed.
josie18:04
i remain unconvinced. will try to reproduce with a test and either humbly admit defeat or gloat in victory 🍺
👍🏻 1
evoskuil18:15
Think of it this way... there is a continuous flow of values into a table, and into a vector.
The table writes are concurrent by block and at the granularity of tx outputs. The vector writes are sequential (stranded) by block (input would be way too much to message-buffer, blocks is easy/slow). The vector is only submitted to the message pump after the table writes for that block are committed.
The critical section only makes write and verify/purge mutually-exclusive (and since verify is extremely rare, there is no meaningful contention).
The strand will deliver batched messages to the vector at any time, but cannot do so during verify, because both are on the same strand. So batched cannot be increased during verify, and is purged at the end of verify.
So a latent block id making its way through the message pump may land on batched after the table(s) that contained the records for that same block are purged. This doesn't matter. The tables only carry verification failure, so if the block failed, it's a reset.
This latent block id will land once verification completes and has purged both the tables and the batched_ vector. This is fine. The block will get marked as valid on the next verification.
The final (residual) batches ensure that both the tables and the batched_ vector are processed in full before batching gives way to non-batching.
(edited)evoskuil18:20
The most challenging part of this was that last step... specifically because of the issue you are raising.
josie19:00
haha racing this race (but actually stepped away for the evening , dinner ain't gonna cook itself). but i'm going to pick this up first thing tomorrow will bring up the server without ZFS
👍🏻 1
evoskuil19:05
I updated the batch post-processing to handle the valids in a parallel loop, so should be a little faster.
E
Eric00:12
The FreeBSD Forums
I'm finally reading the Second Edition of "The Design and Implementation of the FreeBSD Operating System" and found this passsage:
> The ZFS filesystem integrated from OpenSolaris is the one exception to the integrated buffer cache. ZFS has its own set of memory that it manages by itself. Files...
😯 1😵💫 1
josie10:31
@evoskuil alright alright, gotta eat humble pie. after reading your explanation carefully and reading the code, i agree that a later batch can recover a link that was placed after its rows were purged. so this was not a stall, and was instead libbitcoin and ZFS fighting for memory. i also had a leftover watchdog on the box that would kill the service using a lot of memory (this was a leftover from when i was running fulcrum and frigate on the server and didnt want things to get locked up)
👍🏻 1
josie10:43
i am glad i chased it down, tho, because i do think i have something. i kept looking at the code and something didn't sit right. so here is what i believe can happen:
imagine block
a has a bad sig. when validating `a` the bad signature row is written into the batch table under the batch mutex. But a’s block id is not added to batched_ at that moment, rather it is posted later to the strand via push_batch().
before push_batch() runs, block b finishes and triggers batch verification. verifier sees a’s bad signature row, correctly marks a unconfirmable, and purges the table. but because a’s id is not yet in batched_, the invalid link cleanup cannot mark a’s entry as terminal.
now a’s delayed push_batch() finally runs and appends a to batched_. On the next batch, process_valids() treats every non-terminal link in batched_ as valid, so a can be marked valid even though it was already marked unconfirmable for an invalid signature.
relevant code paths (not using line links because i also have some local changes):
// chaser_validate_parallel.cpp
{
std::shared_lock lock{ mutex_, std::defer_lock };
if (capturing) lock.lock();
if ((ec = block.connect(ctx, capture)))
return ec;
batched = capture.batched;
faulted = capture.faulted;
}
^ signature rows are written during block.connect(), but the block link is not added to batched_ here.
// chaser_validate.cpp
if (batched)
{
++batch_backlog_;
POST(push_batch, link, height);
return;
}
^ link posted later
// chaser_validate_batch.cpp batched_.push_back(link); process_batch(is_maximum());
^ link can arrive after another batch processes rows
// chaser_validate_batch.cpp
for (const auto& link: invalids)
{
if (!query.get_height(height, link) ||
!query.set_block_unconfirmable(link))
return false;
notify_block(system::error::invalid_signature, height, link, false);
}
std::ranges::replace_if(batched_, [&](const auto& link) NOEXCEPT
{
return contains(invalids, link);
}, header_link::terminal);
^ this only removes invalid links that are already present in batched_
// chaser_validate_batch.cpp
for (const auto& link: batched_)
{
if (link == header_link::terminal)
continue;
if (!query.get_height(height, link) ||
!query.set_block_valid(link))
return false;
notify_block(system::error::success, height, link, false);
}
^ and lastly later marks every non terminal batched_ link valid, even if that link arrived late and the block had already been marked unconfirmable.
(edited)josie10:54
here is the test i used to try and reproduce this:
BOOST_AUTO_TEST_CASE(chaser_validate__process_batch__late_invalid_batched_link__does_not_become_valid)
{
configuration config{ chain::selection::mainnet };
config.database.path = TEST_DIRECTORY;
config.node.batch_signatures = 1;
const auto events = [](auto, auto) NOEXCEPT {};
store store_{ config.database };
query query_{ store_ };
BOOST_REQUIRE(test::clear(TEST_DIRECTORY));
BOOST_REQUIRE_EQUAL(store_.create(events), database::error::success);
BOOST_REQUIRE(query_.initialize(config.bitcoin.genesis_block));
const chain::block invalid_block{ block1_data, true };
const chain::block trigger_block{ block2_data, true };
const chain::block valid_block{ block3_data, true };
BOOST_REQUIRE(query_.set(invalid_block, database::context{ 0, 1, 0 }, false, false));
BOOST_REQUIRE(query_.set(trigger_block, database::context{ 0, 2, 0 }, false, false));
BOOST_REQUIRE(query_.set(valid_block, database::context{ 0, 3, 0 }, false, false));
const auto invalid_link = query_.to_header(invalid_block.hash());
const auto trigger_link = query_.to_header(trigger_block.hash());
const auto valid_link = query_.to_header(valid_block.hash());
BOOST_REQUIRE(query_.push_candidate(invalid_link));
BOOST_REQUIRE(query_.push_candidate(trigger_link));
BOOST_REQUIRE(query_.push_candidate(valid_link));
// invalid_link has a bad sig row, trigger_link has a good sig row.
BOOST_REQUIRE(query_.set_signature(sighash_bad, ecdsa_compressed,
ecdsa_signature, 0, invalid_link));
BOOST_REQUIRE(query_.set_signature(ecdsa_sighash, ecdsa_compressed,
ecdsa_signature, 0, trigger_link));
BOOST_REQUIRE_EQUAL(query_.ecdsa_records(), 2u);
code invalid_state{};
code trigger_state{};
code valid_state{};
{
const network::logger log{};
full_node node{ query_, config, log };
batch_test_chaser chaser{ node };
// trigger_link push_batch() processes both table rows before
// invalid_link delayed push_batch() has appended invalid_link to batched_
chaser.complete_batched(trigger_link, 2u);
chaser.drain();
BOOST_REQUIRE_EQUAL(query_.ecdsa_records(), 0u);
BOOST_REQUIRE_EQUAL(query_.get_block_state(invalid_link),
database::error::block_unconfirmable);
BOOST_REQUIRE_EQUAL(query_.get_block_state(trigger_link),
database::error::block_valid);
// invalid_link arrives late into batched_ after its bad row was purged
chaser.complete_batched(invalid_link, 1u);
chaser.drain();
// later good batch causes process_valids() to iterate the stale
// invalid_link and call set_block_valid() on it.
BOOST_REQUIRE(query_.set_signature(ecdsa_sighash, ecdsa_compressed,
ecdsa_signature, 0, valid_link));
chaser.complete_batched(valid_link, 3u);
chaser.drain();
invalid_state = query_.get_block_state(invalid_link);
trigger_state = query_.get_block_state(trigger_link);
valid_state = query_.get_block_state(valid_link);
}
BOOST_REQUIRE(!store_.close(events));
// this fails on upstream: invalid_state becomes block_valid.
BOOST_CHECK_EQUAL(invalid_state, database::error::block_unconfirmable);
BOOST_CHECK_EQUAL(trigger_state, database::error::block_valid);
BOOST_CHECK_EQUAL(valid_state, database::error::block_valid);
} (edited)josie10:56
seems the simple enough to fix, tho, i tried this:
diff --git a/src/chasers/chaser_validate_batch.cpp b/src/chasers/chaser_validate_batch.cpp
index a269929f..9b8cfda7 100644
--- a/src/chasers/chaser_validate_batch.cpp
+++ b/src/chasers/chaser_validate_batch.cpp
@@ -210,8 +210,13 @@ bool chaser_validate::process_valids(bool residual) NOEXCEPT
continue;
size_t height{};
- if (!query.get_height(height, link) ||
- !query.set_block_valid(link))
+ if (!query.get_height(height, link))
+ return false;
+
+ if (query.get_block_state(link) == database::error::block_unconfirmable)
+ continue;
+
+ if (!query.set_block_valid(link))
return false;
notify_block(system::error::success, height, link, false);josie11:03
could still be off base tho! the tricky thing about trying to repro behaviour like this in the test is its too easy to do things in a way that doesnt match how the code actually runs, so would appreciate eyes on this
josie11:47
another interesting insight: the GPU i have seems to degrade at higher batch sizes, e.g., 60-70M and it was 3.51M sigs/sec, but at 300M it was down to 2.3 - 2.5M sigs/sec. this seems dependent on the backlog of inflight blocks coming in, so not really something i can tune for. working on a patch that would make it much tighter to the actual GPU batch size set by the user , because sometimes too large of a batch size can make the GPU hang
josie12:20
i think i got a decent way to control batch size without locking everything up, im running that now and its looking like it will finish full validation in < 2.5 hours
evoskuil13:18
The engine should be controlling the batch size that the GPU sees. What we send gets chunked up internally. So it really should not matter if the batch we send is large.
(edited)josie13:19
that doesnt seem to be the case, or rather, the work that the engine needs to do is creating a slowdown.
josie13:19
once i got into the later blocks i was seeing slowdowns in gpu processing correlating with larger sig batches being sent in the logs
evoskuil13:22
That would be a bridge or engine problem. All we pass to the bridge is a pointer and row count.
evoskuil13:22
evoskuil13:23
The issue you described above is correct, I mentioned this yesterday and commented it here.
evoskuil13:25
The resolution you show above is correct but not ideal, as that creates a hashmap search for every confirming block.
evoskuil13:26
1 million hashmap queries on the block validated table is not catastrophic, but ideally we want to avoid that when we can.
evoskuil13:30
Tighter control over the sig accumulation would be nice, as long as it doesn't come at the cost of increased contention. Generally the size of the table shouldn't matter except as it may increase paging and mremap costs. If you are seeing significant perf reduction at larger batch sizes, in the verification, take a look at whether that's coming from paging. If it's not then it should be manageable in the bridge/engine.
josie15:07
ah nice! i missed the bug explanation in the convo yesterday, but admittedly wasn't reading as carefully as i could have been. sadly i was hoping to have a sip of the celebratory beer , but doesnt count if the bug is known 😕
evoskuil15:18
Well the discussion led to me seeing it, so partial credit. The state search isn't a good call in this situation mainly because it would impact the hot path whereas the problem it would resolve only hits the ice cold path (~never happens).
evoskuil15:20
i think i got a decent way to control batch size without locking everything up
Lemee know what the idea is and I'll give some feedback.
(edited)👍 1
evoskuil15:25
//...We also don't want to read block state before // every write of valid state.
This was a ref to your proposed solution. So good insight and valid solution, for a very complex interaction. 👍🏻
❤️ 1
josie15:28
to the preposed solution i had for batch sizing, (afk so keeping it short):
target reached -> seal -> drain admitted writers/links -> verify/purge -> reopen
josie15:29
which obviously comes with some performance hits, im running with it now to see if i noticed anything and will keep tinkering. i think it will be super small tho
josie15:33
altho i am noticing some IO contention / paging which makes me wonder if ive got the machine set up correctly / appropriately sized (only 62GB). will let this finish and run again tomorrow
evoskuil15:38
You will definitely have a significant amount of paging. The chain is 800GB and you have 62GB RAM. All of the blocks are written (as in milestone) and then read (with prevouts) for validation. With enough RAM this goes away entirely, but otherwise it's the tradeoff that allows for concurrent validation.
evoskuil15:39
But WRT batch verification the question is really just whether the large batch slowdown is the result of paging, or is it from the engine.
josie15:39
oh yeah, on that im not sure. now that i see the paging here, im guessing its probably the result of paging, and perhaps the engine is fine. as you said, you're just passing a pointer
evoskuil15:40
which obviously comes with some performance hits
Yes, blocking the validator is a significant performance hit. In fact I'm planning to remove the critical section once I've got a mmap top prune feature added to the tables.
evoskuil15:41
The batch tables can continue to be written while batch verifying, the only reason I lock during verify is so that we can purge the entire table.
evoskuil15:41
but by controlling the batch size, i am consistently seeing 5M/s which is what i wanted
Sure, that's easy to achieve, but the lock isn't good.
evoskuil15:42
mmap doesn't provide a simple way to purge the front of a map, which is what we need for this (sort of circular buffer).
(edited)evoskuil15:43
The size of the batch shouldn't be too important. The purpose of the configured value is to just set a minimum.
evoskuil15:44
If a lower average tends to outperform, then I would just reduce the configured minimum.
josie15:45
i had considered that , but i mostly wrote this as a quick patch to ensure something else wasnt going on with my gpu
evoskuil15:45
I work hard to keep the implementation lock-free, so even though this lock is cheap, it does suspend all validation while the batch verify is running, which given that verify runs on GPU and validation runs on CPU is not ideal.
(edited)evoskuil15:47
also i set batch_size, i want it to be exactly that haha
Eventual consistency is a necessary design principle for efficient concurrent code.
(edited)evoskuil15:49
WRT the hashmap query... you will notice that we use query quite liberally. Often I'll use a query as opposed to moving around a captured value that we already have...
evoskuil15:50
That's because most queries are direct navigation to a memory address that is 99.999% likely already paged in.
evoskuil15:51
So it's important to make this distinction WRT what is a hashmap query. If the query operates over a table with a large body then it can get very costly, as it potentially results in paging across the full body, as the conflict resolution linked lists are in the body.
(edited)evoskuil15:52
One of the big no-no's is reading before writing. This was initially evaluated, such as for txs. The cost is prohibitive.
josie15:54
oh looking at the the design i posted above again, the lock has nothing to do with batch table size control, rather that was for the wedged block scenario i saw in the first place. it was happening again, subsequent batches were not clearing it out. as i understand it, your argument is residuals would have eventually cleared it out, but then i started noticing io pressure (mmap), and thought id try this run and see if that fixed the problem
evoskuil15:54
We do however have a feature in the hashmaps that allows us to determine with very high probability if there is a conflict (potential existence), using a distributed bloom filter in the hashmap header. It's just integrated into the base hashmap heads, but isn't sufficiently allocated to be useful except for the points table.
(edited)josie15:54
however, (while not seeing the wedged block), i am still running into the io pressure which leads me to believe the machine is just under resourced and the wedged block wasnt really the issue
evoskuil15:55
"residuals" in the code refers only to the tail end (< configured size) of the entire run.
(edited)evoskuil16:00
The behavior associated with the valid-after-invalid issue would not cause a delay, it's no different than if that one block had been validated in the next batch. Since the batches aren't ordered, it's not consequential.
evoskuil16:02
The efficient resolution of that issue is to create a sidecar vector of invalids, as a member of the chaser. Then just push any invalid, strand protected, and check that list before setting valid. That makes the hot path check a free test against an empty vector.
evoskuil16:03
> oh i think they are separate things (valid after invalid vs delay)
They are, the delay is the natural course given how default priorities are set.
(edited)evoskuil16:04
That sidecar could be cleared at a coalescence point, but really not worth doing.
(edited)evoskuil16:07
Note that I see significant degradation in ultrafast CPU verification when the machine is stressed. It goes from 75% to 200% of secp256k1. This is not a paging issue, as it happens with individual sigs on otherwise identical runs and when there is over 100GB of free RAM. But I haven't measured GPU.
(edited)evoskuil16:09
I've assumed it's the Windows path, or it's limited to CPU, but maybe not. But it's not limited to the batching scneario.
(edited)josie16:10
secp256k1 does everything on the stack , no heap or memory allocation. i wonder if that could be a different with ultrafast and making a difference on a stressed machine ?
evoskuil16:11
It could be the batch horizontal-vertical rewrite getting disproportionately more costly, but we'll have that removed shortly.
(edited)evoskuil16:13
Yeah, in the unbatched run it can't be tied to batch size... I'm just referring to the direct drop-in replacement for
secp256k1.evoskuil16:14
It gets through millions of sigs and is beating secp by -25%, and then falls behind to complete at about +50-100%.
(edited)evoskuil16:18
bool chaser_validate::process_invalids(const header_links& invalids) NOEXCEPT
{
BC_ASSERT(stranded());
auto& query = archive();
for (const auto& link: invalids)
{
size_t height{};
if (!query.get_height(height, link) ||
!query.set_block_unconfirmable(link))
return false;
invalids_.push_back(link);
notify_block(system::error::invalid_signature, height, link, false);
}
// Set all invalid links in batched_ to terminal (to be skipped).
if (!invalids_.empty())
{
std::ranges::replace_if(batched_, [&](const auto& link) NOEXCEPT
{
return contains(invalids_, link);
}, header_link::terminal);
}
return true;
} (edited)