Dynamic NAT detection

[AgeManning]

Description

We can determine if lighthouse has correctly forwarded its ports via a number of mechanisms.

• Discovery: Discovery has its own in-built system, which as far as I can tell works fine and we report this through our /nat api and in our metrics. It functions via the discv5 PING RPC coupled with counting the number of inbound connections.
• Libp2p: For libp2p, we use UPnP (which sets up routes in a local router), may use libp2p autonat in the future (which does a discv5-like ping to see if external ports are open), or what we currently use, is simply count the number of incoming connections.

The logic being, that if we get inbound connections, it typically means that we are externally contactable. There is the caveat, that if we connect to a peer, and that peer disconnects, then immediately dials and reconnects to us, then we register it as an incomming peer, and this would be a false positive.

In principle, we should get a stream of incoming peers, if we are contactable AND discovery is contactable. If libp2p is contactable but discovery is not, then we wont be advertising our ENR and will not see incoming connections. In this case, we would incorrectly report libp2p ports as being closed, because our discovery ports are.

I think we want a simple message for the users, to abstract away all of this complexity, and simply give a yes/no for their configuration. Our /nat api more or less does this. It just spits out this (https://github.com/sigp/lighthouse/blob/stable/common/system_health/src/lib.rs#L221)

We could integrate UPnP and autonat in deciding whether libp2p nat is open or not. They have some failure modes as discussed here: #6677.

I think the simplest approach is just to use the number of incoming peers, like we are doing here: https://github.com/sigp/lighthouse/blob/stable/beacon_node/lighthouse_network/src/peer_manager/network_behaviour.rs#L215

I think it might be better to have some threshold, say 3 incoming peers before we deem our NAT correctly configured, to reduce the case of false positives.

Also, when peers disconnect, I think we should check that if we are below this threshold then we set the NAT state to be false. So that if the nat configuration changes during runtime we can dynamically identify the change. This would involve setting the metric to 0 if incoming peers is less than the threshold. We could handle it here: https://github.com/sigp/lighthouse/blob/stable/beacon_node/lighthouse_network/src/peer_manager/network_behaviour.rs#L290

We can check number of incoming peers via our peerdb: See for example this read: https://github.com/sigp/lighthouse/blob/stable/beacon_node/lighthouse_network/src/peer_manager/network_behaviour.rs#L304

Open to other suggestions tho