Linux – Custom Centos5 AMI kernel panic

amazon ec2amazon-amicentoskernel-paniclinux

I'm trying to build a Custom Centos 5 AMI on Amazon and during boot it kernel panics.
I've spent almost a week on it and I can't figure out what's wrong.

I've followed the following recipe:
http://blog.maclawran.ca/gotchas-porting-centos-58-from-virtualbox-to

I've also tried other similar recipes on the internet with the same results. The only things I changed from the recipe above was:

  • Used the latest Centos ISO/repos (5.9)
  • Have a single partition (/boot inside /)
  • Root partition using ext4 instead of ext3

Steps were pretty much as follows:

  • Install Centos on Virtual Box
  • Convert .vdi disk to .img
  • Upload disk image to AWS
  • Copy image to a volume
  • Make a few changes to initrd, fstab, grub.conf
  • snapshot volume
  • create AMI
  • Launch new instance

EC2 system log

Xen Minimal OS!
  start_info: 0xac4000(VA)
    nr_pages: 0x26700
  shared_inf: 0xbfa11000(MA)
     pt_base: 0xac7000(VA)
nr_pt_frames: 0x9
    mfn_list: 0x990000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line: root=/dev/sda1 ro 4
  stack:      0x94f860-0x96f860
MM: Init
      _text: 0x0(VA)
     _etext: 0x5ffbd(VA)
   _erodata: 0x78000(VA)
     _edata: 0x80ae0(VA)
stack start: 0x94f860(VA)
       _end: 0x98fe68(VA)
  start_pfn: ad3
    max_pfn: 26700
Mapping memory range 0xc00000 - 0x26700000
setting 0x0-0x78000 readonly
skipped 0x1000
MM: Initialise page allocator for c01000(c01000)-26700000(26700000)
MM: done
Demand map pfns at 26701000-2026701000.
Heap resides at 2026702000-4026702000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x26701000.
Initialising scheduler
Thread "Idle": pointer: 0x2026702010, stack: 0x26640000
Initialising xenbus
Thread "xenstore": pointer: 0x20267027c0, stack: 0x26650000
Dummy main: start_info=0x96f960
Thread "main": pointer: 0x2026702f70, stack: 0x26660000
"main" "root=/dev/sda1" "ro" "4" 
vbd 2049 is hd0
******************* BLKFRONT for device/vbd/2049 **********


backend at /local/domain/0/backend/vbd/2087/2049
Failed to read /local/domain/0/backend/vbd/2087/2049/feature-barrier.
Failed to read /local/domain/0/backend/vbd/2087/2049/feature-flush-cache.
16777216 sectors of 512 bytes
**************************

Press `ESC' to enter the menu... 1   
Press `ESC' to enter the menu... 0   
    [H
    [J  Booting 'CentOS (2.6.18-348.2.1.el5)'



root (hd0,0)

 Filesystem type is ext2fs, partition type 0x83

kernel /boot/vmlinuz-2.6.18-348.2.1.el5xen ro root=LABEL=/

initrd /boot/initrd-2.6.18-348.2.1.el5xenec2.img



xc_dom_probe_bzimage_kernel: kernel is not a bzImage
close blk: backend at /local/domain/0/backend/vbd/2087/2049
Bootdata ok (command line is ro root=LABEL=/)

Linux version 2.6.18-348.2.1.el5xen (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-54)) #1 SMP Tue Mar 5 17:05:33 EST 2013

BIOS-provided physical RAM map:

 Xen: 0000000000000000 - 0000000026f00000 (usable)

No mptable found.

Built 1 zonelists.  Total pages: 155531

Kernel command line: ro root=LABEL=/

Initializing CPU#0

PID hash table entries: 4096 (order: 12, 32768 bytes)

Xen reported: 2666.746 MHz processor.

Console: colour dummy device 80x25

Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)

Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)

Software IO TLB disabled

Memory: 602796k/637952k available (2559k kernel code, 26616k reserved, 1762k data, 196k init)

Calibrating delay using timer specific routine.. 6688.32 BogoMIPS (lpj=13376641)

Security Framework v1.0.0 initialized

SELinux:  Initializing.

selinux_register_security:  Registering secondary module capability

Capability LSM initialized as secondary

Mount-cache hash table entries: 256

CPU: L1 I cache: 32K, L1 D cache: 32K

CPU: L2 cache: 6144K

CPU: Physical Processor ID: 0

CPU: Processor Core ID: 3

(SMP-)alternatives turned off

Brought up 1 CPUs

checking if image is initramfs... it is

Grant table initialized

NET: Registered protocol family 16

Brought up 1 CPUs

PCI: setting up Xen PCI frontend stub

ACPI: Interpreter disabled.

Linux Plug and Play Support v0.97 (c) Adam Belay

pnp: PnP ACPI: disabled

xen_mem: Initialising balloon driver.

usbcore: registered new driver usbfs

usbcore: registered new driver hub

PCI: System does not support PCI

PCI: System does not support PCI

NetLabel: Initializing

NetLabel:  domain hash size = 128

NetLabel:  protocols = UNLABELED CIPSOv4

NetLabel:  unlabeled traffic allowed by default

NET: Registered protocol family 2

IP route cache hash table entries: 32768 (order: 6, 262144 bytes)

TCP established hash table entries: 131072 (order: 9, 2097152 bytes)

TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)

TCP: Hash tables configured (established 131072 bind 65536)

TCP reno registered

audit: initializing netlink socket (disabled)

type=2000 audit(1363277397.008:1): initialized

VFS: Disk quotas dquot_6.5.1

Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

Initializing Cryptographic API

alg: No test for crc32c (crc32c-generic)

ksign: Installing public key data

Loading keyring

- Added public key 1A2C6C3E680FB539

- User ID: CentOS (Kernel Module GPG key)

io scheduler noop registered

io scheduler anticipatory registered

io scheduler deadline registered

io scheduler cfq registered (default)

pci_hotplug: PCI Hot Plug PCI Core version: 0.5

rtc: IRQ 8 is not free.

Non-volatile memory driver v1.2

Linux agpgart interface v0.101 (c) Dave Jones

brd: module loaded

Xen virtual console successfully installed as xvc0

Bootdata ok (command line is ro root=LABEL=/)

Linux version 2.6.18-348.2.1.el5xen (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-54)) #1 SMP Tue Mar 5 17:05:33 EST 2013

BIOS-provided physical RAM map:

 Xen: 0000000000000000 - 0000000026f00000 (usable)

No mptable found.

Built 1 zonelists.  Total pages: 155531

Kernel command line: ro root=LABEL=/

Initializing CPU#0

PID hash table entries: 4096 (order: 12, 32768 bytes)

Xen reported: 2666.746 MHz processor.

Console: colour dummy device 80x25

Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)

Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)

Software IO TLB disabled

Memory: 602796k/637952k available (2559k kernel code, 26616k reserved, 1762k data, 196k init)

Calibrating delay using timer specific routine.. 6688.32 BogoMIPS (lpj=13376641)

Security Framework v1.0.0 initialized

SELinux:  Initializing.

selinux_register_security:  Registering secondary module capability

Capability LSM initialized as secondary

Mount-cache hash table entries: 256

CPU: L1 I cache: 32K, L1 D cache: 32K

CPU: L2 cache: 6144K

CPU: Physical Processor ID: 0

CPU: Processor Core ID: 3

(SMP-)alternatives turned off

Brought up 1 CPUs

checking if image is initramfs... it is

Grant table initialized

NET: Registered protocol family 16

Brought up 1 CPUs

PCI: setting up Xen PCI frontend stub

ACPI: Interpreter disabled.

Linux Plug and Play Support v0.97 (c) Adam Belay

pnp: PnP ACPI: disabled

xen_mem: Initialising balloon driver.

usbcore: registered new driver usbfs

usbcore: registered new driver hub

PCI: System does not support PCI

PCI: System does not support PCI

NetLabel: Initializing

NetLabel:  domain hash size = 128

NetLabel:  protocols = UNLABELED CIPSOv4

NetLabel:  unlabeled traffic allowed by default

NET: Registered protocol family 2

IP route cache hash table entries: 32768 (order: 6, 262144 bytes)

TCP established hash table entries: 131072 (order: 9, 2097152 bytes)

TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)

TCP: Hash tables configured (established 131072 bind 65536)

TCP reno registered

audit: initializing netlink socket (disabled)

type=2000 audit(1363277397.008:1): initialized

VFS: Disk quotas dquot_6.5.1

Dquot-cache hash table entries: 512 (order 0, 4096 bytes)

Initializing Cryptographic API

alg: No test for crc32c (crc32c-generic)

ksign: Installing public key data

Loading keyring

- Added public key 1A2C6C3E680FB539

- User ID: CentOS (Kernel Module GPG key)

io scheduler noop registered

io scheduler anticipatory registered

io scheduler deadline registered

io scheduler cfq registered (default)

pci_hotplug: PCI Hot Plug PCI Core version: 0.5

rtc: IRQ 8 is not free.

Non-volatile memory driver v1.2

Linux agpgart interface v0.101 (c) Dave Jones

brd: module loaded

Xen virtual console successfully installed as xvc0

Event-channel device installed.

Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2

ide: Assuming 50MHz system bus speed for PIO modes; override with idebus=xx

ide-floppy driver 0.99.newide

usbcore: registered new driver hiddev

usbcore: registered new driver usbhid

drivers/usb/input/hid-core.c: v2.6:USB HID core driver

PNP: No PS/2 controller found. Probing ports directly.

i8042.c: No controller found.

mice: PS/2 mouse device common for all mice

md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27

md: bitmap version 4.39

TCP bic registered

Initializing IPsec netlink socket

NET: Registered protocol family 1

NET: Registered protocol family 17

XENBUS: Device with no driver: device/vbd/2049

XENBUS: Device with no driver: device/vif/0

XENBUS: Device with no driver: device/console/0

Initalizing network drop monitor service

Write protecting the kernel read-only data: 506k

Red Hat nash version 5.1.19.6 starting
Mounting proc filesystem
Mounting sysfs filesystem
Creating /dev
Creating initial device nodes
Setting up hotplug.
Creating block device nodes.
Loading ehci-hcd.ko module
Loading ohci-hcd.ko module
Loading uhci-hcd.ko module
USB Universal Host Controller Interface driver v3.0

Loading scsi_mod.ko module
SCSI subsystem initialized

Loading sd_mod.ko module
Loading libata.ko module
Loading ata_piix.ko module
Loading ahci.ko module
Loading dm-mem-cache.ko module
Loading dm-mod.ko module
device-mapper: uevent: version 1.0.3

device-mapper: ioctl: 4.11.6-ioctl (2011-02-18) initialised: dm-devel@redhat.com

Loading dm-log.ko module
Loading dm-region_hash.ko module
Loading dm-message.ko module
Loading dm-raid45.ko module
device-mapper: dm-raid45: initialized v0.2594l

Loading xenblk.ko module
Registering block device major 8

register_blkdev: cannot get major 8 for sd

xen_blk: can't get major 8 with name sd

vbd vbd-2049: 19 xlvbd_add at /local/domain/0/backend/vbd/2087/2049

Loading xennet.ko module
netfront: Initialising virtual ethernet driver.

netfront: device eth0 has copying receive path.

Waiting for driver initialization.
Scanning and configuring dmraid supported devices
Creating root device.
mkrootdev: expected fs options
Mounting root filesystem.
mount: missing mount point
Setting up other filesystems.
Setting up new root fs
setuproot: moving /dev failed: No such file or directory
no fstab.sys, mounting internal defaults
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
switchroot: mount failed: No such file or directory
Kernel panic - not syncing: Attempted to kill init!

/boot/grub/grub.conf 

default=0
timeout=2
hiddenmenu

title CentOS (2.6.18-348.2.1.el5)
   root (hd0,0)
   kernel /boot/vmlinuz-2.6.18-348.2.1.el5xen ro root=LABEL=/
   initrd /boot/initrd-2.6.18-348.2.1.el5xenec2.img

/etc/fstab 

LABEL=/                 /                       ext4    defaults        1 1
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0

AMI settings:

Root Device:      /dev/sda1
Kernel ID:        aki-8b655dff
Architecture:     x86_64
Root Device Type: ebs
Image Size:       8 GiB
Block Devices:    /dev/sda1=snap-4dfffb65:8:true:standard

Instance settings:

Root Device:   sda1
Block Devices: sda1

init script inside /boot/initrd-2.6.18-348.2.1.el5xenec2.img 

#!/bin/nash

mount -t proc /proc /proc
setquiet
echo Mounting proc filesystem
echo Mounting sysfs filesystem
mount -t sysfs /sys /sys
echo Creating /dev
mount -o mode=0755 -t tmpfs /dev /dev
mkdir /dev/pts
mount -t devpts -o gid=5,mode=620 /dev/pts /dev/pts
mkdir /dev/shm
mkdir /dev/mapper
echo Creating initial device nodes
mknod /dev/null c 1 3
mknod /dev/zero c 1 5
mknod /dev/urandom c 1 9
mknod /dev/systty c 4 0
mknod /dev/tty c 5 0
mknod /dev/console c 5 1
mknod /dev/ptmx c 5 2
mknod /dev/rtc c 10 135
mknod /dev/tty0 c 4 0
mknod /dev/tty1 c 4 1
mknod /dev/tty2 c 4 2
mknod /dev/tty3 c 4 3
mknod /dev/tty4 c 4 4
mknod /dev/tty5 c 4 5
mknod /dev/tty6 c 4 6
mknod /dev/tty7 c 4 7
mknod /dev/tty8 c 4 8
mknod /dev/tty9 c 4 9
mknod /dev/tty10 c 4 10
mknod /dev/tty11 c 4 11
mknod /dev/tty12 c 4 12
mknod /dev/ttyS0 c 4 64
mknod /dev/ttyS1 c 4 65
mknod /dev/ttyS2 c 4 66
mknod /dev/ttyS3 c 4 67
echo Setting up hotplug.
hotplug
echo Creating block device nodes.
mkblkdevs
echo "Loading ehci-hcd.ko module"
insmod /lib/ehci-hcd.ko 
echo "Loading ohci-hcd.ko module"
insmod /lib/ohci-hcd.ko 
echo "Loading uhci-hcd.ko module"
insmod /lib/uhci-hcd.ko 
mount -t usbfs /proc/bus/usb /proc/bus/usb
echo "Loading scsi_mod.ko module"
insmod /lib/scsi_mod.ko 
echo "Loading sd_mod.ko module"
insmod /lib/sd_mod.ko 
echo "Loading libata.ko module"
insmod /lib/libata.ko 
echo "Loading ata_piix.ko module"
insmod /lib/ata_piix.ko 
echo "Loading ahci.ko module"
insmod /lib/ahci.ko 
echo "Loading dm-mem-cache.ko module"
insmod /lib/dm-mem-cache.ko 
echo "Loading dm-mod.ko module"
insmod /lib/dm-mod.ko 
echo "Loading dm-log.ko module"
insmod /lib/dm-log.ko 
echo "Loading dm-region_hash.ko module"
insmod /lib/dm-region_hash.ko 
echo "Loading dm-message.ko module"
insmod /lib/dm-message.ko 
echo "Loading dm-raid45.ko module"
insmod /lib/dm-raid45.ko 
echo "Loading xenblk.ko module"
insmod /lib/xenblk.ko 
echo "Loading xennet.ko module"
insmod /lib/xennet.ko 
echo Waiting for driver initialization.
stabilized --hash --interval 1000 /proc/scsi/scsi
mkblkdevs
echo Scanning and configuring dmraid supported devices
echo Creating root device.
mkrootdev -t  -o defaults,ro 
echo Mounting root filesystem.
mount /sysroot
echo Setting up other filesystems.
setuproot
echo Switching to new root and running init.
switchroot

any help is greatly appreciated

Gustavo

Best Answer

Between these two points is your clue

mkrootdev: expected fs options
...
switchroot: mount failed: No such file or directory
Kernel panic - not syncing: Attempted to kill init!

The bootloader has loaded the kernel into memory and decompressed the initrd. The init process has started and has attempted to mount the permanent storage (ie: the root filesystem), at which point the initramfs is unmounted and boot continues from the permanent storage.

Your initrd is not able to mount the root filesystem, so init dies, hence your kernel panic.

I would try separating / and /boot into separate partitions, and use ext3. I don't know much about Amazon or VirtualBox, but I do know Red Hat will not support ext4 root filesystems on EL5, so it's not an intended or tested use case.

Related Topic