Ssh – Ansible Playbook running locally but unable to ssh to host via AWX

SUMMARY

I have ansible and ansible AWX installed on same AWS EC2 instance and I am trying to deploy build on another app AWS EC2 instance. I am able to run the playbook via ansible-playbook command,however, get Failed to connect to the host via ssh when using AWX

• AWX inventory has just that host added along with ansible_user :
  xxxxx which is the user I want to deploy with
• Ansible/AWX host has the xxxxx user setup with password less access to the target host which also has the same user (public keys copied)
• I also tried creating this user in aws_task and aws_web containers and copied the public keys to target host

• Running the ssh command AWS runs while gathering information fails

  ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=xxxxx -o ConnectTimeout=10 -o ControlPath=/tmp/awx_108_dKOwxm/cp/61735cd9d3 hostname '/bin/sh -c '"'"'echo ~xxxxx && sleep 0'"'"''

Output:

debug3: muxserver_listen: temporary control path /tmp/awx_72_VGFOPt/cp/61735cd9d3.16cjnTwr0hSE1mXc
bind: No such file or directory
unix_listener: cannot bind to path: /tmp/awx_72_VGFOPt/cp/61735cd9d3.16cjnTwr0hSE1mXc

• Running the command after leaving some configuration options – ControlPersist,ControlMaster,ssh works –

  ssh -vvv -C -o StrictHostKeyChecking=no -o 'IdentityFile="/home/xxxxx/.ssh/id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=xxxxx -o ConnectTimeout=10 hostname

Questions

1. Is this a permission issue while creating the ControlPath? The aws_web and aws_task container both have /tmp folder with 777 permission
2. Which configuration (ansible.cfg) does AWX pick – from the aws_task/aws_web containers or from the host on which it is deployed. I have also tried to set ssh_args = -C (removing the ControlMaster and ControlPersist) at all the three places in ansible.cfg and restarted the AWS containers, without any luck
3. Why it is not working with Control Params

Additional Information

Ansible Version

ansible 2.7.1
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Jul 13 2018, 13:06:57) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

Configuration

ANSIBLE_SSH_ARGS(/etc/ansible/ansible.cfg) = -C

OS / ENVIRONMENT

Ansible and AWX host OS -> CentOS Linux release 7.5.1804 (Core)

Target host OS -> CentOS Linux release 7.5.1804 (Core)

AWX Version: 2.1.0

AWX installed via -> https://github.com/ansible/awx/blob/devel/INSTALL.md#docker-or-docker-compose

STEPS TO REPRODUCE

1) Created Project in AWX with SCM type git.

2) Created inventory in AWX adding the host with variable ansible_user: xxxxx

3) Created job template specifying job type run and selecting project and inventory created in 1 and 2. Provided the extra variable build: in the template as required in the playbook

PlayBook

https://github.com/sandeepdharembra/playbooks/blob/master/installnodejs.yml

ACTUAL RESULTS while running the job template

ansible-playbook 2.7.1
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/var/lib/awx/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.5 (default, Jul 13 2018, 13:06:57) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
Parsed /tmp/awx_108_dKOwxm/tmpo_yCMg inventory source with script plugin
Loading callback plugin awx_display of type stdout, v2.0 from /usr/lib/python2.7/site-packages/awx/lib/awx_display_callback/module.pyc
1 plays in installnodejs.yml
PLAY [all] *********************************************************************
META: ran handlers
TASK [Created new build directory] *********************************************
task path: /var/lib/awx/projects/_6__festaging/installnodejs.yml:10
<hostname> ESTABLISH SSH CONNECTION FOR USER: xxxxx
<hostname> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=xxxxx -o ConnectTimeout=10 -o ControlPath=/tmp/awx_108_dKOwxm/cp/61735cd9d3 hostname '/bin/sh -c '"'"'echo ~xxxxx && sleep 0'"'"''
<hostname> (255, '', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket "/tmp/awx_108_dKOwxm/cp/61735cd9d3" does not exist\r\ndebug2: resolving "hostname" port 22\r\ndebug2: ssh_connect_direct: needpriv 0\r\ndebug1: Connecting to hostname [hostname] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug3: timeout: 9997 ms remain after connect\r\ndebug1: permanently_set_uid: 0/0\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /root/.ssh/id_rsa type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /root/.ssh/id_rsa-cert type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /root/.ssh/id_dsa type -1\r\ndeb…
fatal: [hostname]: UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh: OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket \"/tmp/awx_108_dKOwxm/cp/61735cd9d3\" does not exist\r\ndebug2: resolving \"hostname\" port 22\r\ndebug2: ssh_connect_direct: needpriv 0\r\ndebug1: Connecting to hostname [172.16.5.91] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug3: timeout: 9997 ms remain after connect\r\ndebug1: permanently_set_uid: 0/0\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /root/.ssh/id_rsa type -1\r\ndebug1: key_load_public: No such file or directory\r\ndebug1: identity file /root/.ssh/id_rsa-cert ty…
PLAY RECAP *********************************************************************
hostname   : ok=0    changed=0    unreachable=1    failed=0