Inter-node Communication Buffer warning

[bdaoudtdc]

Describe the bug

We have a 3 node cluster.
We see this message in the logs
[warning] <0.646.0> rabbit_sysmon_handler busy_dist_port <0.1287.0> [{name,delegate_management_1},{initial_call,{delegate,init,1}},{gen_server2,process_next_msg,1},{message_queue_len,0}] {#Port<0.874>,unknown}

According to the docs ..

Inter-node traffic between a pair of nodes uses a TCP connection with a buffer known as the inter-node communication buffer. Its size is 128 MB by default. This is a reasonable default for most workloads.

We believe we have a very light workload , however this message came up few times so we increased the buffer to 512 MB , but we still see the message.

To Reproduce

It happens randomly ..

Version and environment information

RabbitMQ: [3.8.12]
RabbitMQ Cluster Operator: [1.7.0 ]
Kubernetes: [e.g. 1.17.13]

[ansd]

Converting this to a discussion.

[ansd]

Relevant docs in https://www.rabbitmq.com/networking.html#distribution-port-buffer-limit and http://erlang.org/doc/man/erl.html#+zdbbl.

@bdaoudtdc our top suspects are:

1. Large message payloads, or
2. High network latency, or
3. Low throughput (packet loss or shedding perhaps)

Thanks @gerhard @lhoguin.

[gerhard]

To continue this conversation, this will help: https://grafana.com/grafana/dashboards/11352

This is normal in a 3-node cluster running RabbitMQ 3.8 with low to moderate load: http://grafana.gke.rabbitmq.com/d/d-SFCCmZz/erlang-distribution?orgId=1&from=now-24h&to=now