[BUG] Overwhelming Bandwidth Traffic

[BorisJov]

1. Summary - It has been noticed in several namespaces that the las2peer bootstrap deployments are using objectively too much traffic then they should, by several orders of magnitude. Something like 3-30MB/s are used in both receiving and sending, even when there are no services attached. When mobsos-data-processing is connected, then the bandwidth usage jumps even higher, even up to 60MB/s at times.
2. Bug Details
   1. What? - Something is happening that is generating a lot of unnecessary traffic.
   2. Where? - Unknown.
   3. When?/How often? - Whenever a las2peer bootstrap is used.
   4. How?/Current state - A scenario where just the rwthacis/las2peer:v1.0.1 and rwthacis/monitored-geth-client:latest pods were run. In about one minute, the bandwith achieved about 1 MB/s and soon reached ~3 MB/s. This state lasted some time and after ~10 mins I noticed it was much lower (<1KB/s). When I started the mobsos-data-processing pod, the bandwidth jumped very quickly (<15s) to 30MB/s and after 10mins it fluctuated between 25-45MB/s. The bandwidth was mainly used up by the las2peer-bootstrap pod (>99.99%) while the other mentioned pods were running normally (<2KB/s in general, see image below). After the mobsos pod was shut down, the used bandwidth did not seem to drop at all (remained around 25-30MB/s).

3. Fixed state - An explanation for this behaviour should be found and the excess messages being sent and received should be eliminated.

[BorisJov]

Additional tests have been made to determine the source of the problem. First, only the moodle-data-proxy was started and bootstrapped to it, the learning-locker service. The traffic spiked, indicative of this traffic issue. Thus we determined that the problem was indeed a las2peer one.

The next step was to clone the las2peer template project, create two nodes, one bootstrapped to the other, and see how the traffic was. Using ifconfig no major spikes were initially detected. However, once pastry configurations were placed - the ones stating an external address for the host node, a noticeable increase in traffic was noted. This leads us to conclude that this issue is indeed a las2peer one, most probably related to pastry.