Mitogen uses "ansible_ssh_common_args/ansible_ssh_extra_args" of main group to connect to "delegated_to:" hosts

[koef]

Migrating from Ansible 2.9.25/Mitogen 0.2.10rc1 to Ansible 5.4.0/Mitogen 0.3.2 I've detected that new version of Mitogen uses ansible_ssh_common_args/ansible_ssh_extra_args of main group which specified in "hosts:" parameter of play, to connect to "delegated_to:" hosts.

I'm sure it's a bug.

$ pip freeze
ansible==5.4.0
ansible-core==2.12.3
cffi==1.15.0
cryptography==36.0.1
Jinja2==3.0.3
MarkupSafe==2.1.0
mitogen==0.3.2
packaging==21.3
pycparser==2.21
pyparsing==3.0.7
PyYAML==6.0
resolvelib==0.5.4

$ python --version
Python 3.9.7

$ ansible-config dump --only-changed
DEFAULT_STDOUT_CALLBACK(/Users/koef/sources/ansible/mitogen_issue/ansible.cfg) = yaml
DEFAULT_STRATEGY_PLUGIN_PATH(env: ANSIBLE_STRATEGY_PLUGINS) = ['/Users/koef/sources/ansible/mitogen_issue/.venv/lib/python3.9/site-packages/ansible_mitogen/plugins/strategy']
HOST_KEY_CHECKING(/Users/koef/sources/ansible/mitogen_issue/ansible.cfg) = False
RETRY_FILES_ENABLED(/Users/koef/sources/ansible/mitogen_issue/ansible.cfg) = False

Ansible controller: macOS Monterey (12.1)

Target hosts:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.4 LTS
Release:	20.04
Codename:	focal
$ python3 --version
Python 3.8.10

Inventory file:

group1:
  hosts:
    test-mitogen-1:
      ansible_host: 10.74.196.172
      ansible_user: cloud-user
      ansible_password: "somepass"
      ansible_ssh_common_args: "-o ForwardX11=no"
group2:
  hosts:
    test-mitogen-2:
      ansible_host: 10.74.196.221
      ansible_user: cloud-user

Example play:

---
- name: Test play
  hosts: group1
  gather_facts: false
  strategy: mitogen_linear
  tasks:
    - name: Get hostnames of group1 hosts
      ansible.builtin.command: "hostname"
      register: test_var1

    - name: Print hostnames of group1 hosts
      ansible.builtin.debug:
        var: test_var1.stdout

    - name: Run 'hostname' on delegated host
      ansible.builtin.command: "hostname"
      register: test_var2
      delegate_to: test-mitogen-2

    - name: Print output of 'hostname' on delegated host
      ansible.builtin.debug:
        var: test_var2.stdout

ansible.cfg:

[defaults]
host_key_checking = False
retry_files_enabled = False
stdout_callback = yaml

[ssh_connection]
pipelining = True

Output:

$ ansible-playbook -vvv -i inventory/hosts.yaml test.yaml
ansible-playbook [core 2.11.6]
  config file = /Users/koef/sources/ansible/mitogen_issue/ansible.cfg
  configured module search path = ['/Users/koef/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/4.7.0_1/libexec/lib/python3.10/site-packages/ansible
  ansible collection location = /Users/koef/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.10.0 (default, Oct 13 2021, 06:45:00) [Clang 13.0.0 (clang-1300.0.29.3)]
  jinja version = 3.0.2
  libyaml = True
Using /Users/koef/sources/ansible/mitogen_issue/ansible.cfg as config file
host_list declined parsing /Users/koef/sources/ansible/mitogen_issue/inventory/hosts.yaml as it did not pass its verify_file() method
script declined parsing /Users/koef/sources/ansible/mitogen_issue/inventory/hosts.yaml as it did not pass its verify_file() method
Parsed /Users/koef/sources/ansible/mitogen_issue/inventory/hosts.yaml inventory source with yaml plugin
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: test.yaml ****************************************************************************************************************************************************************************************
1 plays in test.yaml

PLAY [Test play] *******************************************************************************************************************************************************************************************
[top  2247] 12:35:15.938684 D ansible_mitogen.process: inherited open file limits: soft=256 hard=(infinity)
[top  2247] 12:35:15.938865 D ansible_mitogen.process: raised soft open file limit from 256 to 524288
[mux  2250] 12:35:15.956225 D mitogen.service: Pool(19f0, size=32, th='MainThread'): initialized
[mux  2250] 12:35:15.958489 D ansible_mitogen.process: Service pool configured: size=32
META: ran handlers

TASK [Get hostnames of group1 hosts] ***********************************************************************************************************************************************************************
task path: /Users/koef/sources/ansible/mitogen_issue/test.yaml:7
<test-mitogen-1> Attempting python interpreter discovery
[task 2251] 12:35:16.011207 D ansible_mitogen.mixins: _low_level_execute_command("echo PLATFORM; uname; echo FOUND; command -v '/usr/bin/python'; command -v 'python3.9'; command -v 'python3.8'; command -v 'python3.7'; command -v 'python3.6'; command -v 'python3.5'; command -v 'python2.7'; command -v 'python2.6'; command -v '/usr/libexec/platform-python'; command -v '/usr/bin/python3'; command -v 'python'; echo ENDFOUND", in_data=<class 'NoneType'>, exe=None, dir=None)
[task 2251] 12:35:16.013083 D ansible_mitogen.process: will use multiplexer 0 (/var/folders/w2/qzjccyhx1kvg9hnbhr_sm8zc0000gn/T/mitogen_unix_2o3fw9dm.sock) to connect to "test-mitogen-1"
[task 2251] 12:35:16.013546 D mitogen.unix: client: connecting to /var/folders/w2/qzjccyhx1kvg9hnbhr_sm8zc0000gn/T/mitogen_unix_2o3fw9dm.sock
[task 2251] 12:35:16.014402 D mitogen.unix: client: local ID is 1, remote is 0
[mux  2250] 12:35:16.014514 D mitogen.unix: listener: accepted connection from PID 2251: unix_client.2251
[mux  2250] 12:35:16.019576 D mitogen.parent: creating connection to context 2 using mitogen.ssh
[mux  2250] 12:35:16.137164 D mitogen.parent: command line for Connection(None): ssh -o "LogLevel ERROR" -l cloud-user -o "Compression yes" -o "ServerAliveInterval 30" -o "ServerAliveCountMax 10" -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -o "GlobalKnownHostsFile /dev/null" -C -o ControlMaster=auto -o ControlPersist=60s -o ForwardX11=no 10.74.196.172 /usr/bin/python -c "'import codecs,os,sys;_=codecs.decode;exec(_(_(\"eNqVkl1PwjAUhu/5Fd6dNo7RVQ1xcYkIKCbykQUlBpdlH502jHbpgIm/3o4ZGHhhvOvT9z3nvFuPa8wcmZsZzxjCDWUUNeLJmYZEqgXCdqM8x+uMImJYhOADu0adlFatiqNU5gy5dVB1mNWh0FANXItgqcfPiec40AtUwQUEIq5L1Jvb1LsB63qn5NvcZJ8sWq+CMGW6qrXOVSvkopVtVx9SgH3iODVQsw3/Hq/vKaHWPsGGqZxLMbcvysqypa1rmNhwpW9h8jodjEf+U+d51B30e/69Ox76M7czmfRd8BywYPc39v6O+/BCtHAcvfJoTNGxYBzjOaAlX8l3JuyFZMntMIjupFw0J0o2qZnKKEhtSq8IBtzQHQvFVwxZBgwfp2NCyJsAnSSSsf5c3Og6Pio3IZYZE/r9QYWATcWCGFltckmwAV88052SzDn4ZgYUIZTLkWQ/A7q7c/XgJ+7iL/fvlNZxyv0qUfwNXJXkfQ==\".encode(),\"base64\"),\"zip\"))'"
[mux  2250] 12:35:16.144025 D mitogen.parent: child for Connection(None) started: pid:2252 stdin:110 stdout:110 stderr:108

...

TASK [Run 'hostname' on delegated host] ********************************************************************************************************************************************************************
task path: /Users/koef/sources/ansible/mitogen_issue/test.yaml:15
[mux  2250] 12:35:23.404564 D mitogen.[ssh.10.74.196.172]: Dispatcher: dispatching (None, 'mitogen.core', 'Dispatcher', 'forget_chain', ('MacBook-Pro-2.local-2251-112d51600-320f1486d7',), Kwargs({}))
[mux  2250] 12:35:23.405152 D mitogen.[ssh.10.74.196.172]: Dispatcher: Message(1004, 1, 0, 101, 0, b'\x80\x02(NX\x0c\x00\x00\x00mitogen.coreq\x00X\n\x00\x00\x00Dispatcherq\x01X\x0c\x00\x00\x00forge'..150) -> None
[task 2256] 12:35:23.421815 D ansible_mitogen.process: will use multiplexer 0 (/var/folders/w2/qzjccyhx1kvg9hnbhr_sm8zc0000gn/T/mitogen_unix_2o3fw9dm.sock) to connect to "test-mitogen-1"
[task 2256] 12:35:23.422700 D mitogen.unix: client: connecting to /var/folders/w2/qzjccyhx1kvg9hnbhr_sm8zc0000gn/T/mitogen_unix_2o3fw9dm.sock
[task 2256] 12:35:23.423608 D mitogen.unix: client: local ID is 2005, remote is 0
[mux  2250] 12:35:23.423617 D mitogen.unix: listener: accepted connection from PID 2256: unix_client.2256
[mux  2250] 12:35:23.425396 D mitogen.parent: creating connection to context 2006 using mitogen.ssh
[mux  2250] 12:35:23.427310 D mitogen.parent: command line for Connection(None): ssh -o "LogLevel ERROR" -l cloud-user -o "Compression yes" -o "ServerAliveInterval 30" -o "ServerAliveCountMax 10" -o "BatchMode yes" -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -o "GlobalKnownHostsFile /dev/null" -C -o ControlMaster=auto -o ControlPersist=60s -o ForwardX11=no 10.74.196.221 /usr/bin/python3 -c "'import codecs,os,sys;_=codecs.decode;exec(_(_(\"eNqVkl1PwjAUhu/5Fd6dNo7RVQ1xcYkIKCbykQUlBpdlH502jHbpgIm/3o4ZGHhhvOvT9z3nvFuPa8wcmZsZzxjCDWUUNeLJmYZEqgXCdqM8x+uMImJYhOADu0adlFatiqNU5gy5dVB1mNWh0FANXItgqcfPiec40AtUwQUEIq5L1Jvb1LsB63qn5NvcZJ8sWq+CMGW6qrXOVSvkopVtVx9SgH3iODVQsw3/Hq/vKaHWPsGGqZxLMbcvysqypa1rmNhwpW9h8jodjEf+U+d51B30e/69Ox76M7czmfRd8BywYPc39v6O+/BCtHAcvfJoTNGxYBzjOaAlX8l3JuyFZMntMIjupFw0J0o2qZnKKEhtSq8IBtzQHQvFVwxZBgwfp2NCyJsAnSSSsf5c3Og6Pio3IZYZE/r9QYWATcWCGFltckmxAV88052SzDn4ZgYUIZTLkWQ/A7q7c/XgJ+7iL/fvlNZxyv0qUfwNXbnkfw==\".encode(),\"base64\"),\"zip\"))'"

...

As you can see from the text above, the parameter "-o ForwardX11=no" is set to connect to the host of "group1" (task "Get hostnames of group1 hosts"), which is correct, and to connect to "delegated_to" the host "test-mitogen-2" (task "Run 'hostname' on delegated host"), which is not the expected behavior.

• Is your version of Ansible patched in any way?
  No

• Are you running with any custom modules, or module_utils loaded?
  No

• Have you tried the latest master version from Git?
  Yes, with the same result (commit id 5b505f5)

[upekkha]

This could be a duplicate of issue #902. Maybe you can try the proposed fix #903.

[aisbergg]

Maybe you can try the proposed fix #903.

It did fix a similiar issue with ssh args for a delegated_to host for me.