Centos – Amazon EC2 instance won’t boot: Kernel panic – not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

amazon ec2amazon-ebsbootcentoscentos6

My instance was running for years and suddenly stopped responding Jun 1st. I tried to reboot it, but it would not boot. It gave errors in the system log: https://pastebin.com/rSxr1kLs

Linux version 2.6.32-642.11.1.el6.x86_64 (mockbuild@c1bm.rdu2.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC) ) #1 SMP Fri Nov 18 19:25:05 UTC 2016
Kernel command line: root=/dev/xvde ro LANG=en_US.UTF-8 KEYTABLE=us
VFS: Cannot open root device "xvde" or unknown-block(0,0)
Please append a correct "root=" boot option; here are the available partitions:
Kernel panic – not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

I tried to detach the EBS volume and re-attach it as /dev/sda1 according to the documentation: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstances.html#FilesystemKernel

However, it gave an error Error attaching volume: Invalid value '/dev/sda1' for unixDevice. Attachment point /dev/sda1 is already in use and I was unable to attach it. I re-attached it as /dev/sda but it still won't boot and it still gives the error in the system log.

I was able to launch a new instance into the exact same availability zone and attached my EBS volume as /dev/sdf. It shows up inside the instance as /dev/xvdj. I mounted it with mount /dev/xvdj /xvdj. I can see the grub.conf file:

[root@ip-172-31-4-249 grub]# cat /xvdj/boot/grub/grub.conf
default=0
timeout=1

title CentOS (2.6.32-642.11.1.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-642.11.1.el6.x86_64 root=/dev/xvde ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
title CentOS (2.6.32-504.30.3.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-504.30.3.el6.x86_64 root=/dev/xvde ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
        initrd /boot/initramfs-2.6.32-504.30.3.el6.x86_64.img
title CentOS (2.6.32-504.3.3.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-504.3.3.el6.x86_64 root=/dev/xvde ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
        initrd /boot/initramfs-2.6.32-504.3.3.el6.x86_64.img
title CentOS (2.6.32-504.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-504.el6.x86_64 root=/dev/xvde ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
        initrd /boot/initramfs-2.6.32-504.el6.x86_64.img
title CentOS (2.6.32-431.29.2.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-431.29.2.el6.x86_64 root=/dev/xvde ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
        initrd /boot/initramfs-2.6.32-431.29.2.el6.x86_64.img
title CentOS (2.6.32-431.23.3.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-431.23.3.el6.x86_64 root=/dev/xvde ro crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
        initrd /boot/initramfs-2.6.32-431.23.3.el6.x86_64.img

This compares to the grub.conf of the running instance:

[root@ip-172-31-4-249 grub]# cat /boot/grub/grub.conf
default=0
timeout=1

title CentOS-6-x86_64-20130527-03 2.6.32-358.6.2.el6.x86_64
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-358.6.2.el6.x86_64 root=/dev/xvde ro
        initrd /boot/initramfs-2.6.32-358.6.2.el6.x86_64.img

Does it matter that it doesn't have initrd line in the first option?

I tried to mount the EBS volume to the new instance with /dev/sda, but it still wouldn't boot with the same error Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0).

CentOS 6

Best Answer

I created a new instance by going to Images > AMIs > Private Images > Selecting the image the instance was started from > Launch. I launched in exactly the same availability zone, not just US or region, but the 2a, 2b, 2c must match as well. I stopped the new instance. I disconnected the EBS volume from the old instance. I re-attached the EBS volume to the new instance at /dev/sdf. I started the new instance. The EBS volume shows up inside the instance as /dev/xvdj so I mounted it with mkdir /xvdj; mount /dev/xvdj /xvdj. I edited /xvdj/boot/grub/grub.conf and changed default=0 to default=1. I saved the file, stopped the new instance, re-attached the EBS volume to the old instance and it started. I ran yum update in the old instance and double-checked /boot/grub/grub.conf and double-checked that it would reboot.

I also found this regarding updates to CentOS kernel: grub.conf missing initrd path after kernel update

I noticed after I ran yum update I now had 2 entries in grub.conf without initrd. Running # yum reinstall kernel.x86_64 works to fix that.

Related Topic