I'm trying to create an AMI (based on ubuntu; ami-2d39803a) that I can reuse for a project with the a set of components installed. I'm using packer to achieve this with a shell script that gets executed by a provisioner:
#/bin/bash
# salt
sudo add-apt-repository --yes ppa:saltstack/salt
sudo apt-get update
sudo apt-get install --yes salt-api salt-cloud salt-master salt-minion salt-ssh salt-syndic
# run on startup
sudo update-rc.d salt-master defaults
# salt hostname for minions
sudo bash -v -c "echo 127.0.0.1 salt >> /etc/hosts"
# get docker-formula and move it to /srv/salt
sudo mkdir /tmp/docker-formula
sudo git clone https://github.com/saltstack-formulas/docker-formula /tmp/docker-formula/.
sudo mkdir -p /srv/salt
sudo cp -vr /tmp/docker-formula/docker /srv/salt/docker/
# top.sls
sudo cp -v /ops/config/top.sls /srv/salt/
# accept all minions (minions try to connect to master every 30 seconds)
sudo sleep 30
sudo salt-key -A --yes
# apply to minions
sudo salt '*' -v -t 60 state.apply
# add user to docker group
sudo usermod -aG docker $USER
Here is the odd part. Running this works just fine when I create a new instance and execute the shell script commands one by one. However, when I run packer build I get the following:
......
aws-us-east-1-ubuntu-base: The following keys are going to be accepted:
aws-us-east-1-ubuntu-base: Unaccepted Keys:
aws-us-east-1-ubuntu-base: ip-172-30-2-245.ec2.internal
aws-us-east-1-ubuntu-base: Key for minion ip-172-30-2-245.ec2.internal accepted.
aws-us-east-1-ubuntu-base: Executing job with jid 20160913191722659701
aws-us-east-1-ubuntu-base: -------------------------------------------
aws-us-east-1-ubuntu-base:
aws-us-east-1-ubuntu-base: ip-172-30-2-245.ec2.internal:
aws-us-east-1-ubuntu-base: Minion did not return. [No response]
aws-us-east-1-ubuntu-base: usermod: group 'docker' does not exist
==> aws-us-east-1-ubuntu-base: Terminating the source AWS instance...
==> aws-us-east-1-ubuntu-base: No AMIs to cleanup
==> aws-us-east-1-ubuntu-base: Deleting temporary security group...
==> aws-us-east-1-ubuntu-base: Deleting temporary keypair...
Build 'aws-us-east-1-ubuntu-base' errored: Script exited with non-zero exit status: 6
It's sudo salt '*' -v -t 60 state.apply failing with:
Minion did not return. [No response]
The only distinction between the two scenarios is that packer is executing the bash script from a file while creating an instance is not executing a file. I can't see how it has to do with the communication between salt master and its minions though.
Any thoughts?
Best Answer
After a dozen terminated instances and another dozen of tricks, I have figured it out. Apparently salt master is a somehow of a slacker:
Notice the delays between my (intentional) sleeps. These delays were causing all sort of issues since most of the commands in my shell script are executed instantly.
Another point is the minion receiving the job
__mine_intervalas soon as its key is accepted. I'm not sure why the minion could not just queue the job I asked it for, instead it kept responding withMinion did not return. [No response]. Another sleep solved this bit.Here is my working script: (uncomment the tails if you want to see salt logs)