That 1 outlier block is over 28MB deserialized. Turns out that's because it has 500,003 witness elements. It also happens to be the block that caused both LND and btcd to stall. Since each witness element is a smart pointer to byte vector, this more than doubles the typical object ratio. Not a problem for us, just need to get the max sizing correct.
(edited)