I'm experiencing a strange issue with a fabric script I'm using to bootstrap a server on EC2.
I launch a stock Ubuntu 12.04 AMI (ami-3d4ff254), wait for it to start, then proceed with:
with settings(host_string="ubuntu@%s" % i.dns_name,
connection_attempts=30):
sudo('apt-get -qy update')
sudo('apt-get -qy install --no-install-recommends mdadm') # don't install postfix
#etc...
The apt-get update appears to run fine and gives no errors, however (2/3 of the time or so) installing mdadm throws a "no installation candidate" error.
When I ssh into the server and run apt-get install mdadm I get the same error. Running apt-get update by hand, then the package installs fine.
Any ideas on what might be happening, or ideas for debugging?
EDIT: It turns out by adding a sleep(10) after the SSH server comes up and before trying apt-get update it works fine. This seems to be an issue with the configuration of apt on this AMI combined with perhaps some strangeness on Amazon's end, or DNS issues.
Here are my entries in /etc/apt/sources.list:
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise main
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise main
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise-updates main
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise-updates main
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise universe
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise universe
deb http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise-updates universe
deb-src http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ precise-updates universe
deb http://security.ubuntu.com/ubuntu precise-security main
deb-src http://security.ubuntu.com/ubuntu precise-security main
deb http://security.ubuntu.com/ubuntu precise-security universe
deb-src http://security.ubuntu.com/ubuntu precise-security universe
Here is the output from the apt-get update that screws up my mdadm install:
Ign http://security.ubuntu.com precise-security InRelease
Get:1 http://security.ubuntu.com precise-security Release.gpg [198 B]
Ign http://archive.ubuntu.com precise InRelease
Get:2 http://security.ubuntu.com precise-security Release [49.6 kB]
Ign http://archive.ubuntu.com precise-updates InRelease
Hit http://archive.ubuntu.com precise Release.gpg
Get:3 http://archive.ubuntu.com precise-updates Release.gpg [198 B]
Hit http://archive.ubuntu.com precise Release
Get:4 http://security.ubuntu.com precise-security/main amd64 Packages [177 kB]
Get:5 http://archive.ubuntu.com precise-updates Release [49.6 kB]
Hit http://archive.ubuntu.com precise/main amd64 Packages
Hit http://archive.ubuntu.com precise/restricted amd64 Packages
Hit http://archive.ubuntu.com precise/universe amd64 Packages
Hit http://archive.ubuntu.com precise/multiverse amd64 Packages
Hit http://archive.ubuntu.com precise/main i386 Packages
Hit http://archive.ubuntu.com precise/restricted i386 Packages
Hit http://archive.ubuntu.com precise/universe i386 Packages
Hit http://archive.ubuntu.com precise/multiverse i386 Packages
Get:6 http://security.ubuntu.com precise-security/restricted amd64 Packages [3,969 B]
Hit http://archive.ubuntu.com precise/main TranslationIndex
Get:7 http://security.ubuntu.com precise-security/universe amd64 Packages [48.6 kB]
Hit http://archive.ubuntu.com precise/multiverse TranslationIndex
Get:8 http://security.ubuntu.com precise-security/multiverse amd64 Packages [2,188 B]
Get:9 http://security.ubuntu.com precise-security/main i386 Packages [183 kB]
Hit http://archive.ubuntu.com precise/restricted TranslationIndex
Get:10 http://archive.ubuntu.com precise/universe TranslationIndex [2,922 B]
Get:11 http://archive.ubuntu.com precise-updates/main amd64 Packages [405 kB]
Get:12 http://security.ubuntu.com precise-security/restricted i386 Packages [3,968 B]
Get:13 http://security.ubuntu.com precise-security/universe i386 Packages [48.7 kB]
Get:14 http://security.ubuntu.com precise-security/multiverse i386 Packages [2,357 B]
Get:15 http://security.ubuntu.com precise-security/main TranslationIndex [73 B]
Get:16 http://security.ubuntu.com precise-security/multiverse TranslationIndex [71 B]
Get:17 http://security.ubuntu.com precise-security/restricted TranslationIndex [71 B]
Get:18 http://security.ubuntu.com precise-security/universe TranslationIndex [73 B]
Get:19 http://archive.ubuntu.com precise-updates/restricted amd64 Packages [8,215 B]
Get:20 http://archive.ubuntu.com precise-updates/universe amd64 Packages [149 kB]
Get:21 http://security.ubuntu.com precise-security/main Translation-en [86.5 kB]
Get:22 http://archive.ubuntu.com precise-updates/multiverse amd64 Packages [8,948 B]
Hit http://security.ubuntu.com precise-security/multiverse Translation-en
Get:23 http://archive.ubuntu.com precise-updates/main i386 Packages [410 kB]
Hit http://security.ubuntu.com precise-security/restricted Translation-en
Get:24 http://security.ubuntu.com precise-security/universe Translation-en [30.3 kB]
Get:25 http://archive.ubuntu.com precise-updates/restricted i386 Packages [8,218 B]
Get:26 http://archive.ubuntu.com precise-updates/universe i386 Packages [149 kB]
Get:27 http://archive.ubuntu.com precise-updates/multiverse i386 Packages [9,930 B]
Get:28 http://archive.ubuntu.com precise-updates/main TranslationIndex [3,564 B]
Get:29 http://archive.ubuntu.com precise-updates/multiverse TranslationIndex [2,605 B]
Get:30 http://archive.ubuntu.com precise-updates/restricted TranslationIndex [2,461 B]
Get:31 http://archive.ubuntu.com precise-updates/universe TranslationIndex [2,850 B]
Get:32 http://archive.ubuntu.com precise/main Translation-en [726 kB]
Get:33 http://archive.ubuntu.com precise/multiverse Translation-en [93.4 kB]
Get:34 http://archive.ubuntu.com precise/restricted Translation-en [2,395 B]
Get:35 http://archive.ubuntu.com precise/universe Translation-en [3,341 kB]
Get:36 http://archive.ubuntu.com precise-updates/main Translation-en [202 kB]
Get:37 http://archive.ubuntu.com precise-updates/multiverse Translation-en [5,606 B]
Get:38 http://archive.ubuntu.com precise-updates/restricted Translation-en [2,048 B]
Get:39 http://archive.ubuntu.com precise-updates/universe Translation-en [87.4 kB]
Ign http://archive.ubuntu.com precise/main Translation-en_US
Ign http://archive.ubuntu.com precise/multiverse Translation-en_US
Ign http://archive.ubuntu.com precise/restricted Translation-en_US
Fetched 6,308 kB in 8s (786 kB/s)
Reading package lists...
And here's the output for the one that works (i.e. after waiting a few seconds before doing the apt-get update):
Ign http://us-east-1.ec2.archive.ubuntu.com precise InRelease
Ign http://us-east-1.ec2.archive.ubuntu.com precise-updates InRelease
Get:1 http://us-east-1.ec2.archive.ubuntu.com precise Release.gpg [198 B]
Get:2 http://us-east-1.ec2.archive.ubuntu.com precise-updates Release.gpg [198 B]
Get:3 http://us-east-1.ec2.archive.ubuntu.com precise Release [49.6 kB]
Ign http://security.ubuntu.com precise-security InRelease
Get:4 http://us-east-1.ec2.archive.ubuntu.com precise-updates Release [49.6 kB]
Get:5 http://security.ubuntu.com precise-security Release.gpg [198 B]
Get:6 http://us-east-1.ec2.archive.ubuntu.com precise/main Sources [934 kB]
Get:7 http://security.ubuntu.com precise-security Release [49.6 kB]
Get:8 http://us-east-1.ec2.archive.ubuntu.com precise/universe Sources [5,019 kB]
Get:9 http://us-east-1.ec2.archive.ubuntu.com precise/main amd64 Packages [1,273 kB]
Get:10 http://security.ubuntu.com precise-security/main Sources [50.1 kB]
Get:11 http://us-east-1.ec2.archive.ubuntu.com precise/universe amd64 Packages [4,786 kB]
Get:12 http://security.ubuntu.com precise-security/universe Sources [14.5 kB]
Get:13 http://security.ubuntu.com precise-security/main amd64 Packages [177 kB]
Get:14 http://us-east-1.ec2.archive.ubuntu.com precise/main i386 Packages [1,274 kB]
Get:15 http://security.ubuntu.com precise-security/universe amd64 Packages [48.6 kB]
Get:16 http://us-east-1.ec2.archive.ubuntu.com precise/universe i386 Packages [4,796 kB]
Get:17 http://security.ubuntu.com precise-security/main i386 Packages [183 kB]
Get:18 http://us-east-1.ec2.archive.ubuntu.com precise/main TranslationIndex [3,706 B]
Get:19 http://us-east-1.ec2.archive.ubuntu.com precise/universe TranslationIndex [2,922 B]
Get:20 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main Sources [178 kB]
Get:21 http://security.ubuntu.com precise-security/universe i386 Packages [48.7 kB]
Get:22 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe Sources [60.2 kB]
Get:23 http://security.ubuntu.com precise-security/main TranslationIndex [73 B]
Get:24 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main amd64 Packages [405 kB]
Get:25 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe amd64 Packages [149 kB]
Get:26 http://security.ubuntu.com precise-security/universe TranslationIndex [73 B]
Get:27 http://security.ubuntu.com precise-security/main Translation-en [86.5 kB]
Get:28 http://security.ubuntu.com precise-security/universe Translation-en [30.3 kB]
Get:29 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main i386 Packages [410 kB]
Get:30 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe i386 Packages [149 kB]
Get:31 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main TranslationIndex [3,564 B]
Get:32 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe TranslationIndex [2,850 B]
Get:33 http://us-east-1.ec2.archive.ubuntu.com precise/main Translation-en [726 kB]
Get:34 http://us-east-1.ec2.archive.ubuntu.com precise/universe Translation-en [3,341 kB]
Get:35 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main Translation-en [202 kB]
Get:36 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe Translation-en [87.4 kB]
Fetched 24.6 MB in 12s (1,976 kB/s)
Reading package lists...
I don't know why the former is full of more Hits and Igns.
Best Answer
Without some more in-depth testing (which I'm not going to do right now) it's hard to say exactly what is going on, but the general outline is:
When you first start up an Ubuntu cloud instance on EC2, a script runs which customizes it a bit for the EC2 environment. One of the things that happens is that your
/etc/apt/sources.listis rewritten to point to Ubuntu mirrors which are themselves hosted in Amazon S3 (which means you pay no bandwidth costs for install/update software).My guess is that something is subtly different between the Amazon archives and the general archives which causes the strangeness you're seeing; with a bit more testing and poking around you'll probably find out exactly what it is.
My suggestion to make your deployment script work would be to simply wait until all of Ubuntu's initialization stuff is complete before you run your own script.