Apr 16, 2025

There has been one, long-standing, mysterious behavior in the node. You may have heard me talk of it in past meetings. It seemed to come and go randomly, unaffected by my attempts to isolate it, and narrowly focused around segwit txs. Specifically, witness data would be missing at script validation time (causing validation failure). It wasn't clear whether the data was supposed to be there or incorrectly expected, or whether it was read and lost, or never read, from either the wire or the store. But sparse reproduction and many possibilities made it very hard to diagnose. So finally I put in diagnostics to trace out the segwit tx count for each block. Given that this count is driven off of witness data, this shows what I'm getting off the wire. Then when the validation failure occurred, I also wrote out the count. That was a couple weeks ago. Last night it happened again. The failing blocks were read from the wire without witness data, and all failed for lack of witness data, and were read from the store without witness data. So either there was an intermittent failure to read the data off the wire, or another mystery. But given that there was just this isolated and contiguous segment of blocks with missing data it finally dawned on me what was happening. There is a peer that is sending non-segwit serializations despite negotiation for witness data serializations. This doesn't get caught by the witness commitment check, because it's optional without witness data. The requirement for witness data can only be determined by investigating the script patterns (which is what precipitates the later validation failure). Fortunately most blocks above segwit have witness data, so this additional check is very rarely required - only when segwit is active and only for blocks with no witness data (which is already checked during the commitment validation). Without this the block gets archived and later during validation (and not at all during milestone) the scripts are validated - resulting in an invalidation of the archived block, and subsequent stall. This manifests similarly to a block hash malleability failure.