Nov 4, 2024

yeah, what you've explained here, especially windows vs ranges is what i assumed was going on, but i wanted to keep the delving post simple, hence the example with ranges. i need to reread and digest the rest of your answers, but so far i think the only misconception i had was assuming that confirmability was also happening under the milestone. regarding threat model, i agree there is no material difference (that i can see) and the primary reason we dont skip this in core is because we have to delete the spent outputs to maintain a UTXO set

as a side note, not that im married to the analogy, but i think the windows actually makes this closer to a mapreduce problem, where confirmability is the "shuffle" phase (but i agree it doesnt perfectly map and dont want to try to jam a square peg in a round hole)

Yeah, BC does it because it needs to populate the utxo store in order, not because there is some difference in security. BN populates outputs to the store and confirmedness associations during download when under milestone. So we do the same we just benefit from parallelism.

I don't think that BC can't take advantage of this type of parallelism while retaining the utxo store, as the utxo store requires block-ordered population.

I've run tests with the download window reduced to 1024 blocks (like BC) to see if this type of benefit can be achieved in such a small window, allowing blocks to be memory buffered. But in this configuration BN slows dramatically to essentially BC speed.

So we must impose this order in a chaser that follows validation or download, both of which require re-reading through large portions of the chain.

BC could do that as well, chasing validation and creating the utxo store. But this means populating a large and redundant table. We just populate the associations that provide the essential information provided by the utxo store. This is a costly model in terms of proving confirmability, but it should not be any cheaper to navigate the same associations to build the redundant store. This is also more costly for us because we store unconfirmed/conflicts in the same space, but that also provides other very significant advantages, such as an append only store and having everything already validated and written when a compact block arrives.

agree with everything here. if we forget BN/BC for a second, my intuition is it will be hard to have a universal model that is performant for IBD (a data processing problem), vs updating chaintip/serving blocks to peers which is more of a high performance/realtime service problem

I saw that earlier, and made some comments on it here. The first observation is that serving up blocks to peers is not a performance critical operation, either for the node or its peers. Secondly, BN reads, parses, and serializes the entire chain (and does everything else for a milestone run, which is a lot, including all the hashing - not required on read) in about 50 minutes, and would be closer to 40 minutes with sufficient RAM, on a network connection that produces a theoretical limit of 38 minutes to transfer the entire chain as a single file. Reading should therefore be much faster than the network can handle. Thirdly, there are a number of significant advantages in getting to completion with a compact block. There's no reason to assume that the means of verifying confirmability will have a greater cost than a utxo model, or that the other material advantages won't be dominant (for example, BN does not have to create or serialize a block).

my instinct is to disagree here, but ill need to give it some more thought.

strongly agree here! i dont want to assume anything. im mostly going on what im familiar with while also trying to upskill on understanding the architecture you are building here, hence trying to be clear these are intuitions

(intuitions informed by more familiarity with BC's architecture, admittedly)