ZFS Pool Recovery – Fix ‘Last Used by Another System’ Error

data-recoveryfreebsdtruenaszfs

tl;dr: I am running a FreeBSD 11.2 NAS server with ZFS on a server with 16GB ECC memory. I've discovered that my main pool has disappeared. ZFS can see the pool, but reports fault ZFS-8000-EY and that it cannot import 'tank': pool may be in use from other system, it was last accessed by nasserver (hostid: 0xaaaaaaaa) on Sat Apr 13 04:16:22 2019, however this pool has never been used by another system and nasserver is the current host. I am afraid my pool is corrupt. What should I do to attempt to recover it? I have backups, but they are out of date since I have to do them manually. I have not tried to run zpool import -f tank yet, since I want to be careful to avoid making anything worse.

Full question:

I am running a FreeBSD 11.2 NAS server with ZFS. I returned from vacation to find the server up, but the NAS volumes were inaccessible. When I ssh'd into the box, I found that my man pool tank was missing:

root@nasserver:/home/root # zpool status   pool: zroot state: ONLINE status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.   scan: scrub repaired 0 in 0h0m with 0 errors on Sat Mar 23 03:01:44 2019 config:


        NAME                                  STATE     READ WRITE CKSUM
        zroot                                 ONLINE       0     0     0
          mirror-0                            ONLINE       0     0     0
            diskid/DISK-XXXXXXXXXXXXXXXXXXX1  ONLINE       0     0     0
            diskid/DISK-XXXXXXXXXXXXXXXXXXX2  ONLINE       0     0     0


errors: No known data errors

The drives appear to be available, and I tried to import them, but ZFS reports that it cannot import 'tank': pool may be in use from other system, it was last accessed by nasserver (hostid: 0xaaaaaaaa) on Sat Apr 13 04:16:22 2019. This is odd, because I have never used these disks on another system and the hostname reported is the hostname of the machine they're attached to:

root@nasserver:/home/root # zpool import
   pool: tank
     id: 12345678901234567901
  state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: http://illumos.org/msg/ZFS-8000-EY
config:


        tank                    ONLINE
          raidz1-0              ONLINE
            gpt/tank-1          ONLINE
            gpt/tank-2          ONLINE
            gpt/tank-3          ONLINE
            gpt/tank-4          ONLINE


root@nasserver:/home/root # zpool import tank
cannot import 'tank': pool may be in use from other system, it was last accessed by nasserver (hostid: 0xaaaaaaaa) on Sat Apr 13 04:16:22 2019
use '-f' to import anyway

I'm afraid my pool may be corrupt. I have backups, but they are somewhat out of date. What should I do to recover my pool?

Best Answer

zpool import -f tank, just like it says in the error dialog.

This is a fairly uncommon bug, but I've seen it happen before (maybe once or twice on 100+ machines over the last ten years); if something appears to have changed regarding the hostname and possibly some other magic, zfs won't auto-import the pool out of (possibly an excess of) caution that this might not really be the machine that pool belongs to. Use -f to override that caution, and poof, here's your pool (and it should auto-import fine the next go-round).

This is probably most likely to happen after a crash or power outage that leaves the system down without the pool properly exported on the way down.

Related Topic