FreeBSD 10 won’t boot to ZFS root after power failure

bootbsdfreebsdzfs

While installing new hardware today, I involuntarily disconnected the power to one of our servers. This server is running FreeBSD 10 with Root-on-ZFS as offered by the installer.
It has a total of 36 disks, distributed into two RAID-Z2 which belong to the same pool (called zroot).

After switching the server back on, it will now no longer boot. Before the boot menu shows up the following messages are printed:

Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf

Despite these messages, the system continues to boot the kernel until it stops at a mountfrom> prompt and I can't get it to continue from there. If I enter zfs:zroot/ROOT/default, it just says unknown filesystem.

I can however boot from a USB stick, import the zpool and I can read both /boot/defaults/loader.conf and /boot/loader.conf. In fact, it appears that the zpool is perfectly fine.

I have then tried to reinstall the bootcode:

gpart bootcode -b /tmp/zroot/boot/pmbr -p /boot/gptzfsboot -i 1 da0

I have tried to recreate the zpool cache file:

zpool set cachefile=/tmp/zroot/boot/zfs/zpool.cache zroot

I have tried to manually tell the bootloader to load the ZFS module.

All of these attempts still resulted in the same behaviour and I am now completely stuck on what I could possibly do to make the system boot again.

Edit:

Output of zpool list:

$ zpool list
NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
zroot  97.8T  28.0T  69.7T    28%  1.33x  ONLINE  /tmp/zfs

Currently the system is booted from a USB stick, thus I could not import the pool to /

Contents of /boot/loader.conf:

$ cat /boot/loader.conf
zfs_load="YES"
ipmi_load="YES"

Edit 2:

$ zpool status
  pool: zroot
 state: ONLINE
  scan:
config:

    NAME                                            STATE     READ WRITE CKSUM
    zroot                                           ONLINE       0     0     0
      raidz2-0                                      ONLINE       0     0     0
        gptid/f8c57b3a-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/f97b7e8b-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/fa3c41d9-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/faf62101-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/fbb19e1b-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/fc6b75db-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/fd26cd36-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/fddb4b8e-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/fe9a55f6-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/ff582110-083e-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/001713d1-083f-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/00d90b6c-083f-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/0192be91-083f-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/023ea058-083f-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/02fb8ee4-083f-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/03ab78ec-083f-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/04632542-083f-11e4-b11b-002590e745f4  ONLINE       0     0     0
        gptid/052144fd-083f-11e4-b11b-002590e745f4  ONLINE       0     0     0
      raidz2-1                                      ONLINE       0     0     0
        da18                                        ONLINE       0     0     0
        da19                                        ONLINE       0     0     0
        da20                                        ONLINE       0     0     0
        da21                                        ONLINE       0     0     0
        da22                                        ONLINE       0     0     0
        da23                                        ONLINE       0     0     0
        da24                                        ONLINE       0     0     0
        da25                                        ONLINE       0     0     0
        da26                                        ONLINE       0     0     0
        da27                                        ONLINE       0     0     0
        da28                                        ONLINE       0     0     0
        da29                                        ONLINE       0     0     0
        da30                                        ONLINE       0     0     0
        da31                                        ONLINE       0     0     0
        da32                                        ONLINE       0     0     0
        da33                                        ONLINE       0     0     0
        da34                                        ONLINE       0     0     0
        da35                                        ONLINE       0     0     0

errors: No known data errors

Best Answer

Ok, never mind what I said in my other answer. You won't be able to boot into single-user mode if you are having problems with ZFS partitions anyway. What's funny is after I posted my answer, I had a problem with power again and I couldn't boot into my system, just like your problem.

So here is what I did to get back in.

I used a FreeBSD 10.0-RELEASE Live CD (it was a bootonly I think -- to minimize download)

  1. Boot into the CD and choose "Live CD" from the options. (log in as root)

  2. Execute the following commands to attach partitions using geli; you would have to enter your passphrase if you have a root-on-ZFS full disk encryption setup

     mkdir /tmp/bootpool
 zpool import -f bootpool
 zfs set mountpoint=/tmp/bootpool bootpool
 zfs mount -a
 cp /tmp/bootpool/boot/encryption.key /tmp/
 zfs umount -a
 zfs set mountpoint=/bootpool bootpool
 zpool export bootpool
 geli attach -k /tmp/encryption.key /dev/ada0p4
 zpool import -f -R /mnt zroot

I had to zpool import with -f flag because the power down problem didn't export the partition in the right way.

After executing these commands I rebooted and the system went online again.

Though, I had another problem with bootpool not loading correctly after boot.

The /boot folder was empty, it was also a symbolic link to the /bootpool/boot directory. I also found out that when importing the zfs bootpool partition it created /bootpool/boot/boot so the issue was the 2 boot folders under /bootpool.

So I did this after reboot to load the bootpool manually.

      zpool export bootpool
      zpool import -f bootpool
      (cd to root and remove the old symbolic link `boot`)
      cd /
      rm boot
      (notice the double `boot` directory issue)
      ln -sf bootpool/boot/boot/

That's it and the /boot symbolic link worked again and I could load the missing kernel modules, for example kldload linux or anything else.

Hope that helps, and if anybody knows a clean fix for the bootpool problem above please share it with me.

Thanks,

Draco

References:

Related Topic