Stuck lookups involving block requests that never time out

[michaelsproul]

Description

We are investigating a pattern of stuck block lookups characterised by the log:

WARN Notify the devs a sync lookup is stuck ...

The root cause seems to be that we never receive a response for some BlocksByRoot requests, not even a timeout. This has been observed with Caplin peers.

Often this type of stuck lookup does not pose any risk to the node's sync status, as the block is imported via HTTP or gossip in the meantime. This may be incidental (and lucky), or related to the root cause in some way that we have not yet determined. It might be prudent to remove lookups which are obsoleted by blocks imported via HTTP, as we do for gossip.

Version

Lighthouse v6.0.1

[AgeManning]

For this particular issue, I think #6711 fixes it