Ubuntu – Supervisorctl error on Ansible: [Errno 111] Connection refused: file: /usr/lib/python2.7/socket.py

It seems there are some problems with supervisor floating around: https://stackoverflow.com/questions/18859063/supervisor-socket-error-issue

I tried the solution by essentially adding the "config" file path to every supervisorctl call in Ansible, but to no avail. Can someone please help me figure out what could be going on?

I'm running this script on an Ubuntu 14.04 target, and it has a lot of config before this which is working (nothing to do with supervisor though).

— RELEVANT DATA —

Here is the full output:

TASK: [web | Create the app Supervisor config file] ***************************
 ESTABLISH CONNECTION FOR USER: ubuntu
 EXEC ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 52.34.2.144 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1449725067.46-180800778030180 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1449725067.46-180800778030180 && echo $HOME/.ansible/tmp/ansible-tmp-1449725067.46-180800778030180'
 EXEC ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 52.34.2.144 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=xkzueyvbanrookgwiqyphaqsifzyzvfc] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-xkzueyvbanrookgwiqyphaqsifzyzvfc; rc=flag; [ -r /etc/supervisor/conf.d//diffview.conf ] || rc=2; [ -f /etc/supervisor/conf.d//diffview.conf ] || rc=1; [ -d /etc/supervisor/conf.d//diffview.conf ] && rc=3; python -V 2>/dev/null || rc=4; [ x"$rc" != "xflag" ] && echo "${rc} "/etc/supervisor/conf.d//diffview.conf && exit 0; (python -c '"'"'"'"'"'"'"'"'import hashlib; BLOCKSIZE = 65536; hasher = hashlib.sha1(); afile = open("'"'"'"'"'"'"'"'"'/etc/supervisor/conf.d//diffview.conf'"'"'"'"'"'"'"'"'", "rb") buf = afile.read(BLOCKSIZE) while len(buf) > 0: hasher.update(buf) buf = afile.read(BLOCKSIZE) afile.close() print(hasher.hexdigest())'"'"'"'"'"'"'"'"' 2>/dev/null) || (python -c '"'"'"'"'"'"'"'"'import sha; BLOCKSIZE = 65536; hasher = sha.sha(); afile = open("'"'"'"'"'"'"'"'"'/etc/supervisor/conf.d//diffview.conf'"'"'"'"'"'"'"'"'", "rb") buf = afile.read(BLOCKSIZE) while len(buf) > 0: hasher.update(buf) buf = afile.read(BLOCKSIZE) afile.close() print(hasher.hexdigest())'"'"'"'"'"'"'"'"' 2>/dev/null) || (echo '"'"'"'"'"'"'"'"'0 '"'"'"'"'"'"'"'"'/etc/supervisor/conf.d//diffview.conf)'"'"''
 PUT /tmp/tmpu2Kd5e TO /home/ubuntu/.ansible/tmp/ansible-tmp-1449725067.46-180800778030180/file
 EXEC ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 52.34.2.144 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=fouavmbioyywhqluctyfcxtlzjpmcgsm] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-fouavmbioyywhqluctyfcxtlzjpmcgsm; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/ubuntu/.ansible/tmp/ansible-tmp-1449725067.46-180800778030180/file; rm -rf /home/ubuntu/.ansible/tmp/ansible-tmp-1449725067.46-180800778030180/ >/dev/null 2>&1'"'"''
ok: [52.34.2.144] => {"changed": false, "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/supervisor/conf.d//diffview.conf", "size": 186, "state": "file", "uid": 0}

TASK: [web | Re-read the application Supervisor config file] ******************
 ESTABLISH CONNECTION FOR USER: ubuntu
 REMOTE_MODULE supervisorctl name=diffview state=present config=/etc/supervisor/conf.d//diffview.conf
 EXEC ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 52.34.2.144 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1449725069.32-154486085091195 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1449725069.32-154486085091195 && echo $HOME/.ansible/tmp/ansible-tmp-1449725069.32-154486085091195'
 PUT /tmp/tmpARfHNt TO /home/ubuntu/.ansible/tmp/ansible-tmp-1449725069.32-154486085091195/supervisorctl
 EXEC ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 52.34.2.144 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=ypkutdogmxrxixvmalzccgdxqrgnfses] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-ypkutdogmxrxixvmalzccgdxqrgnfses; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/ubuntu/.ansible/tmp/ansible-tmp-1449725069.32-154486085091195/supervisorctl; rm -rf /home/ubuntu/.ansible/tmp/ansible-tmp-1449725069.32-154486085091195/ >/dev/null 2>&1'"'"''
failed: [52.34.2.144] => {"cmd": "/usr/bin/supervisorctl -c /etc/supervisor/conf.d//diffview.conf reread", "failed": true, "rc": 2}
stdout: error: , [Errno 111] Connection refused: file: /usr/lib/python2.7/socket.py line: 571


FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/home/vagrant/site.retry

52.34.2.144                : ok=40   changed=2    unreachable=0    failed=1
localhost                  : ok=4    changed=0    unreachable=0    failed=0

HEre's my playbook's relevant yml:

---

#location/other info of conf file: http://supervisord.org/configuration.html

#UBUNTU-specific per the page above
- name: Create the app Supervisor config file
  template: src=app_supervisor_config.j2
            dest="{{ supervisor_app_conf }}"
            backup=yes

- name: Re-read the application Supervisor config file
  supervisorctl: name={{ application_name }} state=present config={{ supervisor_app_conf }}

- name: Restart application Supervisor
  supervisorctl: name={{ application_name }} state=restarted config={{ supervisor_app_conf }}

- name: Create the algo Supervisor config file
  template: src=algo_supervisor_config.j2
            dest="{{ supervisor_algo_conf }}"
            backup=yes

- name: Re-read the algo Supervisor config files
  supervisorctl: name={{ algo_app_name }} state=present config={{ supervisor_algo_conf }}

- name: Restart {{ algo_app_name }} Supervisor
  supervisorctl: name={{ algo_app_name }} state=restarted config={{ supervisor_algo_conf }}