Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exception when running on slow to connect hosts #1220

Open
julienlavergne opened this issue Oct 15, 2024 · 2 comments
Open

Exception when running on slow to connect hosts #1220

julienlavergne opened this issue Oct 15, 2024 · 2 comments
Labels
dependency issue Issues in dependencies that affect pyinfra.

Comments

@julienlavergne
Copy link
Contributor

julienlavergne commented Oct 15, 2024 •
edited
Loading

Describe the bug

A exception is raise when running some tasks on 13 hosts. I do not get the error when running only on 4 hosts, or only 1 hosts. I suspect the error shows more often as the number of hosts increases.

PyInfra v3.1.1

I noticed it happends with machines on which the Connecting to hosts part takes a bit of time (1-3 seconds). When I use hosts that connects in less than ~1 sec, there is not issue.

In some occasions, I only get the Key Error _print_spinner Exception, but then the process is able to continue without facing any other Exception.

EDIT: I confirm that I can trigger the issue when using only 2 hosts that are slow to connect to.

To Reproduce

I was not able to create a minimal reproducible example.
My set of files always reproduces the issue. But when I reduce it, the error disappear. So it may have something to do with a particular timing.

  • Operation code & usage
  • Target system information
  • Example using the @docker connector (helps isolate the problem)

Expected behavior

No exception

Meta

  • Include output of pyinfra --support.

    If you are having issues with pyinfra or wish to make feature requests, please
    check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues .
    When adding an issue, be sure to include the following:

    System: Linux
      Platform: Linux-4.18.0-553.22.1.el8_10.x86_64-x86_64-with-glibc2.28
      Release: 4.18.0-553.22.1.el8_10.x86_64
      Machine: x86_64
    pyinfra: v3.1.1
      black: v24.10.0
      black: v24.10.0
      click: v8.1.7
      configparser: v7.1.0
      distro: v1.9.0
      gevent: v24.10.2
      jinja2: v3.1.4
      packaging: v24.1
      paramiko: v3.5.0
      python-dateutil: v2.9.0.post0
      pywinrm: v0.5.0
      setuptools: v70.1.0
      typeguard: v4.3.0
      typing-extensions: v4.12.2
      wheel: v0.43.0
    Executable: <hidden path>/.tox/pyinfra/bin/pyinfra
    Python: 3.11.8 (CPython, GCC 12.3.0)
  • How was pyinfra installed (source/pip)?
pip install
  • Include pyinfra-debug.log (if one was created)
  • Consider including output with -vv and --debug.
aiodns==3.2.0,aiohappyeyeballs==2.4.3,aiohttp==3.10.10,aiosignal==1.3.1,attrs==24.2.0,bcrypt==4.2.0,black==24.10.0,Brotli==1.1.0,certifi==2024.8.30,cffi==1.17.1,charset-normalizer==3.4.0,click==8.1.7,configparser==7.1.0,cryptography==43.0.1,distro==1.9.0,frozenlist==1.4.1,gevent==24.10.2,greenlet==3.1.1,hvac==2.3.0,idna==3.10,Jinja2==3.1.4,MarkupSafe==3.0.1,multidict==6.1.0,mypy-extensions==1.0.0,packaging==24.1,paramiko==3.5.0,pathspec==0.12.1,pip==24.1,platformdirs==4.3.6,propcache==0.2.0,pycares==4.4.0,pycparser==2.22,pyinfra==3.1.1,PyNaCl==1.5.0,pyspnego==0.11.1,python-dateutil==2.9.0.post0,pywinrm==0.5.0,requests==2.32.3,requests_ntlm==1.3.0,setuptools==70.1.0,six==1.16.0,typeguard==4.3.0,typing_extensions==4.12.2,urllib3==2.2.3,wheel==0.43.0,xmltodict==0.14.1,yarl==1.15.2,zope.event==5.0,zope.interface==7.1.0

--> Loading config...
--> Loading inventory...
--> Connecting to hosts...
    No host key for host08 found in known_hosts
    No host key for host07 found in known_hosts
    No host key for host03 found in known_hosts
    No host key for host08 found in known_hosts
    No host key for host04 found in known_hosts
    No host key for host09 found in known_hosts
    No host key for host11 found in known_hosts
    No host key for host07 found in known_hosts
    No host key for host05 found in known_hosts
    No host key for host10 found in known_hosts
    No host key for host01 found in known_hosts
    No host key for host06 found in known_hosts
    No host key for host02 found in known_hosts
    [host08] Connected
    [host07] Connected
    [host03]       Connected
    [host04]       Connected
    [host11]       Connected
    [host09]       Connected
    [host08]       Connected
    [host05]       Connected
    [host10]       Connected
    [host07]       Connected
    [host06]       Connected
    [host01]       Connected
    [host02]       Connected

--> Preparing operations...
--> Preparing Operations...
    Loading: my_task.py
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError

    [host07] Command socket/SSH error: SSHException('Channel is not open',)
    [host07] Error: could not load fact: server.LinuxDistribution 
    [host08] Command socket/SSH error: SSHException('Channel is not open',)
    [host08] Error: could not load fact: server.LinuxDistribution 

--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 176, in do_something
    if linux_distribution["major"] <= 8:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<=' not supported between instances of 'NoneType' and 'int'

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1049, in _bootstrap_inner
    self._delete()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1081, in _delete
    del _active[get_ident()]
        ~~~~~~~^^^^^^^^^^^^^
KeyError: 140737278873728
2024-10-15T03:16:08Z <Greenlet at 0x7ffff3839080: <bound method Thread._bootstrap of <paramiko.Transport at 0xf3847e10 (unconnected)>>> failed with KeyError

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1049, in _bootstrap_inner
    self._delete()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1081, in _delete
    del _active[get_ident()]
        ~~~~~~~^^^^^^^^^^^^^
KeyError: 140737278871008
2024-10-15T03:16:08Z <Greenlet at 0x7ffff38385e0: <bound method Thread._bootstrap of <paramiko.Transport at 0xf37dc490 (unconnected)>>> failed with KeyError


--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1125, in open_channel
    raise e
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 2201, in run
    ptype, m = self.packetizer.read_message()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 496, in read_message
    header = self.read_all(self.__block_size_in, check_rekey=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 324, in read_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError


--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
    self._send_user_message(m)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
    self._send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
    self.packetizer.send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
    self.write_all(out)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError


--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
    self._send_user_message(m)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
    self._send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
    self.packetizer.send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
    self.write_all(out)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError


--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
    self._send_user_message(m)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
    self._send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
    self.packetizer.send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
    self.write_all(out)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError


--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
    self._send_user_message(m)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
    self._send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
    self.packetizer.send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
    self.write_all(out)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError


--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
    self._send_user_message(m)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
    self._send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
    self.packetizer.send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
    self.write_all(out)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError


--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
    self._send_user_message(m)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
    self._send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
    self.packetizer.send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
    self.write_all(out)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>


--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/tools/python-3.11.8/lib/python3.11/threading.py", line 1037, in _bootstrap_inner
    del _limbo[self]
        ~~~~~~^^^^^^
KeyError: <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
2024-10-15T03:16:08Z <Greenlet at 0x7ffff383ae80: <bound method Thread._bootstrap of <Thread(Thread-41 (_print_spinner), stopped daemon 140737278881408)>>> failed with KeyError


--> Disconnecting from hosts...
--> An exception occurred in: my_task.py:

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "my_task.py", line 20, in <module>
    local.include("tasks/task1.py")
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/local.py", line 37, in include
    exec_file(filename)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 71, in exec_file
    raise UnexpectedExternalError(e, filename)
Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
    self._send_user_message(m)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
    self._send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
    self.packetizer.send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
    self.write_all(out)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

pyinfra_cli.exceptions.UnexpectedExternalError: <exception str() failed>

Traceback (most recent call last):
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra_cli/util.py", line 65, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "<hidden_path>/tasks/task1.py", line 24, in <module>
    do_something(
  File "<hidden_path>/utils/task2.py", line 174, in do_something
    linux_distribution = host.get_fact(LinuxDistribution)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 353, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 320, in get_host_fact
    return get_fact(state, host, cls, args=args, kwargs=kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 188, in get_fact
    return _get_fact(
           ^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/facts.py", line 253, in _get_fact
    status, output = host.run_shell_command(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/api/host.py", line 422, in run_shell_command
    return self.connector.run_shell_command(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 338, in run_shell_command
    return_code, combined_output = execute_command_with_sudo_retry(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/util.py", line 199, in execute_command_with_sudo_retry
    return_code, output = execute_command()
                          ^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/pyinfra/connectors/ssh.py", line 316, in execute_command
    stdin_buffer, stdout_buffer, stderr_buffer = self.client.exec_command(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/client.py", line 560, in exec_command
    chan = self._transport.open_session(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 994, in open_session
    return self.open_channel(
           ^^^^^^^^^^^^^^^^^^
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1117, in open_channel
    self._send_user_message(m)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1988, in _send_user_message
    self._send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/transport.py", line 1964, in _send_message
    self.packetizer.send_message(data)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 468, in send_message
    self.write_all(out)
  File "/opt/tools/python-3.11.8/lib/python3.11/site-packages/paramiko/packet.py", line 382, in write_all
    raise EOFError()
EOFError
@julienlavergne julienlavergne changed the title Exception when running on several hosts Exception when running on slow to connect hosts Oct 15, 2024
@julienlavergne
Copy link
Contributor Author

I pinpointed the issue with an upgrade of gevent dependency. Version 24.10.2 has the issue, but version 24.2.1 does not. Could you freeze that dependency while figuring out what is wrong ?

@Fizzadar Fizzadar added the dependency issue Issues in dependencies that affect pyinfra. label Nov 17, 2024
@Fizzadar
Copy link
Member

Hrm that's a big jump in versions, we should try pin down the exact version with the issue and ideally an issue on gevent's side to track against before disabling.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
dependency issue Issues in dependencies that affect pyinfra.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Fizzadar @julienlavergne