Mar 13, 2024

Just finished the concurrent download load balancer. As we currently run a configured number of channels and distribute downloads to each in a configured set size, it is easy for channels to become starved for work and then just idle until others are done. This results in all but one channel idling toward completion. I resolved this in prototyping by selecting the slowest channel with work for termination, splitting its work into two sets, and notifying channels of new work. Dropping the channel has a small cost, but the channel cannot be retained as that would results in a significant level of duplicate block downloads, or very inefficient single block requests. Since channels are recovered very quickly (earlier work from last year) the cost is trivial.