Swallowed exceptions in heartbeat thread causing hangs

[mb-syss]

We are occasionally seeing the error below (2.0.1), and afterwards messaging hangs. We have some reconnection logic in place, but in that situation no exception seems to reach the user code at all.

From what I understand browsing the code, if the heartbeat check runs at the right (or rather: wrong) time _check_for_exceptions will remove the exception from the queue and throw it in the heartbeat thread without handling. Also the connection/channel state does not seem to be updated. Therefore user code will not notice the closed connection/dead IO thread and hang indefinitely.

[2021-08-27 12:36:40,207] [rabbitpy.io:CRITICAL] In on_error: 'Connection reset'
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.8/threading.py", line 1254, in run
    self.function(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/rabbitpy/heartbeat.py", line 59, in _maybe_send
    self._channel0.send_heartbeat()
  File "/usr/local/lib/python3.8/site-packages/rabbitpy/channel0.py", line 140, in send_heartbeat
    self.write_frame(heartbeat.Heartbeat())
  File "/usr/local/lib/python3.8/site-packages/rabbitpy/base.py", line 251, in write_frame
    if self._can_write():
  File "/usr/local/lib/python3.8/site-packages/rabbitpy/base.py", line 285, in _can_write
    self._check_for_exceptions()
  File "/usr/local/lib/python3.8/site-packages/rabbitpy/base.py", line 313, in _check_for_exceptions
    raise exception
rabbitpy.exceptions.ConnectionResetException: Connection was reset at socket level

[matteogrolla]

Hi, we have a similar issue on 2.0.1 under python 3.10.
It's actually very bad because it hangs our producer till manual intervention.
Is there any solution or workaround?
thanks

{ "name": "rabbitpy.io", "levelname": "CRITICAL", "message": "In on_error: ConnectionResetError(104, 'Connessione interrotta dal corrispondente')", "time": "2022-09-15T08:23:57.861858" }

Exception in thread Thread-1996:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.10/threading.py", line 1358, in run
self.function(*self.args, **self.kwargs)
File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.10/site-packages/rabbitpy/heartbeat.py", line 59, in _maybe_send
self._channel0.send_heartbeat()
File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.10/site-packages/rabbitpy/channel0.py", line 140, in send_heartbeat
self.write_frame(heartbeat.Heartbeat())
File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.10/site-packages/rabbitpy/base.py", line 251, in write_frame
if self._can_write():
File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.10/site-packages/rabbitpy/base.py", line 285, in _can_write
self._check_for_exceptions()
File "/root/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.10/site-packages/rabbitpy/base.py", line 313, in _check_for_exceptions
raise exception
rabbitpy.exceptions.ConnectionResetException: Connection was reset at socket level