Ansible playbook error: "Connection timed out during banner exchange"

Question

I am trying to connect to my bastion host and then tunnel that ssh connection to connect to a jenkins host but keep running into the connection timed out error.

The error message in full is:

fatal: [<bastion-public-ip>]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: Connection timed out during banner exchange",
    "unreachable": true
}

my inventory file:

[bastion]
#ip for bastion host
<bastion-public-ip>

[bastion:vars]
ansible_ssh_user=ubuntu
ansible_python_interpreter=/usr/bin/python3
ansible_ssh_common_args='-o StrictHostKeyChecking=no'
ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -q [email protected]"'
ansible_ssh_private_key_file=~/.ssh/bastion.pem

[jenkins]
#ip of EC2 instance
10.0.1.49

[jenkins:vars]
ansible_user=ubuntu
ansible_python_interpreter=/usr/bin/python3
ansible_ssh_common_args='-o StrictHostKeyChecking=no'
[email protected]'
ansible_ssh_common_args='-o StrictHostKeyChecking=no'
ansible_ssh_private_key_file=~/j-key.pem

And my playbook file is:

- hosts: bastion
  become: true

- hosts: jenkins
  # run as the root user on the remote machine
  become: true
  # list of tasks to run on the remote machine
  tasks:
  - name: 'update system'
    apt: name=aptitude state=latest update_cache=yes force_apt_get=yes

  - name: 'install docker'
    script: docker.sh

  - name: 'install docker-compose'
    script: docker-compose.sh

  - name: 'install Jenkins'
    script: jenkins.sh

My ansible version is: ansible 2.9.13

EDIT: Wanted to add the rules for the security group my bastion server is in

Port 22 is open to TCP traffic from everywhere currently

EDIT 2: -vvv flag output

ansible-playbook 2.9.13
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/qa/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/qa/.local/lib/python3.8/site-packages/ansible
  executable location = /home/qa/.local/bin/ansible-playbook
  python version = 3.8.5 (default, Jul 28 2020, 12:59:40) [GCC 9.3.0]
Using /etc/ansible/ansible.cfg as config file
host_list declined parsing /home/qa/Projects/Final-Project/ansible/inventory as it did not pass its verify_file() method
script declined parsing /home/qa/Projects/Final-Project/ansible/inventory as it did not pass its verify_file() method
auto declined parsing /home/qa/Projects/Final-Project/ansible/inventory as it did not pass its verify_file() method
Parsed /home/qa/Projects/Final-Project/ansible/inventory inventory source with ini plugin

PLAYBOOK: test-playbook.yml ******************************************************************************************************
2 plays in test-playbook.yml

PLAY [bastion] *******************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************
task path: /home/qa/Projects/Final-Project/ansible/test-playbook.yml:1
<bastion_public_ip> ESTABLISH SSH CONNECTION FOR USER: ubuntu
<bastion_public_ip> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/home/qa/.ssh/bastion.pem"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ubuntu"' -o ConnectTimeout=25 -o 'ProxyCommand=ssh -W %h:%p -q [email protected]' -o ControlPath=/home/qa/.ansible/cp/13196dcaf6 bastion_public_ip '/bin/sh -c '"'"'echo ~ubuntu && sleep 0'"'"''
<bastion_public_ip> (255, b'', b'Connection timed out during banner exchange\r\n')
fatal: [bastion_public_ip]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: Connection timed out during banner exchange",
    "unreachable": true
}

PLAY RECAP ***********************************************************************************************************************
bastion_public_ip                  : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

Solution

You defined multiple ansible_ssh_common_args vars per host. That does'nt work. Ansible will use only one of it (I asume the last). Also you try to reach the bastion host via 10.1.0.49 (but you do not mention that in your question). I think, the second ansible_ssh_common_args in the hostvars of bastion is wrong.

You could have this vars...

[bastions:vars]
ansible_ssh_common_args='-o StrictHostKeyChecking=no'
...

[jenkins:vars]
ansible_ssh_common_args: '-o StrictHostKeyChecking=no -o ProxyCommand="ssh -o StrictHostKeyChecking=no -W [%h]:%p -q [email protected]"'

Also - just a notice - ignoring the host key is not very "secure" - especially when you use a bastion host (which is more or less a man-in-the-middle which you need to trust).