Feb 28, 2023

I recently removed the channel registration vector, which was required for both stop notification and message broadcast, and replaced it with a de-subscribable (new) stop subscriber. This was a half-step toward creating a broadcaster, to which channels subscribe for both stop notification and messages broadcast from outside the node or across channels. This cleaned up the control inversion of the node maintaining a channel vector, but it left for future work the task of creating the broadcaster. Today I laid a bunch of the groundwork for the broadcaster. This an analog to the message subscriber (pump), which protocols use to obtain messages incoming from their channel's peer. So I normalized a few things in the pump implementation with the anticipated broadcaster. This also included doing some longstanding work in hash optimization. Bitcoin p2p performs a double hash on each message and graphs 4 bytes as a checksum, placed into the header. This can be bypassed when reading messages, but is necessary for outgoing - in case another node is validating. For empty messages this is a fixed value, so free. For some messages, the hash is required after reading, and generally available when writing (to the net). This applies most obviously to blocks/headers and txs. Obtaining the hash from a read is most optimally done via the read buffer. Previously we would reserialize the tx just to get its hash, which is a huge expense. Also, when we get a block/header/tx from the store - we have the hash. So I worked it out a while back that message objects would have the ability to ferry their identity hash and/or checksum, so that we could eliminate all of the redundant and hugely expensive hashing. I just stubbed in the simple remaining changes to eliminate all redundant hashing, and to ferry the hash and/or checksum as applicable for certain message, on a message-by-message type basis. The consequence is that we will never compute hashes on hash/block/tx objects retrieved from the database for sending on the wire, never serialize an object off the wire to obtain its hash, either for identity or checksum validation, never compute a hash redundantly (for both checksum validation and identity), and never compute a hash/checksum for a message that is read and then forwarded. I contemplated storing checksums in the database for certain message types, but the storage cost is large and there is no advantage for header/block/tx objects, as the checksum is a cast of the database natural key. There really aren't any other p2p message types for which it could be stored.