Oct 30, 2024

This shows a very consistent time difference (lag) between the buffering of each job and the completion of validation of the net block. The buffering progression is initially ordered from 1-850k, but the validator is asynchronous, so completion is out of order within what is buffered. Initially with very small blocks, the validator is fast enough to preclude any backlog (note that the x axis is not time-based but block height-based, so the first 1/5th of the graph completes in one minute). The backlog starts accumulating until it hits the arbitrary 1024 * 5000 tx limit (configured). Then it is cycling back to zero delay before reaccumulating a backlog. This is because the validator owns the CPU while it works off all of the jobs, and its completion handlers don't execute until all the backlogged jobs are complete. This implies some amount of waste as the CPU drops off the peg momentarily in order to get the backlog restarted. However without a limit the backlog would include a job for every block. The tx count that imposes the limit is read directly from the store, not from the deserialized block, since of course the limit is imposed prior to the read/validate job. The waste can be seen infrequently in the perf monitor as narrow CPU slices that drop off the 100% peg. Ideally we would want to be always buffering jobs while validating, but we need a limit and that's hard to measure if the stranded job completion handler invocation is blocked by the ongoing work.