How to move a Solaris Container from one zfs filesystem to another on the same server

solaris

This seems like a simple thing to do, if it would only work as documented. However, it doesn't work thanks to some bug in the zoneadm move process. Go figure. Firstly let me show what the trivial documentation says :

http://docs.oracle.com/cd/E19455-01/817-1592/gcgnc/index.html

Well that says the command to move a non-global zone from one place to another all on the same server is just

zoneadm -z zonename move /some/new/pathname

That looks pretty darn trivial … however if the non-global zone has any contents in it where a directory has special ( not entirely trivial ascii ) characters in the filenames then the process blows up. Please allow me to demonstrate :

Step one : shut down the zone. Ensured it was halted thus :

# zlogin -C -e\@ z_001
[Connected to zone 'z_001' console]

node002 console login: root
Password: 
Last login: Sun Jul 21 05:12:17 on console
Jul 25 17:50:06 node002 login: ROOT LOGIN /dev/console
Oracle Corporation      SunOS 5.10      Generic Patch   January 2005
node002-sparc-SunOS5.10 # shutdown -g0 -y -i0    

Shutdown started.    Thu Jul 25 17:51:06 GMT 2013

Changing to init state 0 - please wait
Broadcast Message from root (console) on node002 Thu Jul 25 17:51:07...
THE SYSTEM node002 IS BEING SHUT DOWN NOW ! ! !
Log off now or risk your files being damaged

node002-sparc-SunOS5.10 # showmount: node002: RPC: Program not registered
svc.startd: The system is coming down.  Please wait.
svc.startd: 70 system services are now being stopped.
syslogd: /dev/sysmsg: I/O error
svc.startd: The system is down.

[NOTICE: Zone halted]
@.
[Connection to zone 'z_001' console closed]
#

Great .. now comes the zoneadm move :

# zoneadm -z z_001 move /zones/z_001
Moving across file-systems; copying zonepath /zone/z_001...zoneadm: zone 'z_001': 'copy' failed with exit code 5.

The copy failed.
More information can be found in /var/log/zoneAAAcwaOwX

Cleaning up zonepath /zones/z_001...

#

Failed? Let's see why :

# cat  /var/log/zoneAAAcwaOwX
cpio: Error with fstatat() of "", errno 2, No such file or directory
cpio: Error with fstatat() of "root/usr/local/build/automake-1.13_SunOS5.10_sparcv9/t/instspc.dir/a", errno 2, No such file or directory
cpio: Cannot open directory b, errno 2, No such file or directory
cpio: Error with fstatat() of "root/usr/local/build/automake-1.13_SunOS5.10_sparcv9/t/instspc.dir/a", errno 2, No such file or directory
cpio: Error with fstatat() of "b", errno 2, No such file or directory
62026208 blocks
5 error(s)
#

Failed because cpio ( the thing being used by zoneadm move ) can not handle special characters. If one downloads the GNU automake source tarball and does a configure
and then make with a make check you will see piles of special characters inside the testsuite area. This will cause cpio on Solaris to blow up.

So I hope there is some Solaris guru that can confirm two things for me :

1) the only zone config file I need to edit is in /etc/zones
2) I can edit that one file myself and do the move of the zone 
    filesystem myself also.

Am I correct ?

EDIT —- this is the directory and the characters that cpio can not handle correctly :

# ls -lapb /zone/z_001/root/usr/local/build/automake-1.13_SunOS5.10_sparcv9/t/instspc.dir
total 1398
drwxr-xr-x   2 gnuism  other          3 Dec 31  2012 `/
drwxr-xr-x   2 gnuism  other          2 Dec 31  2012 \012/
drwxr-xr-x  64 gnuism  other         74 Dec 31  2012 ./
drwxr-xr-x  42 gnuism  other       4045 Dec 31  2012 ../
drwxr-xr-x   2 gnuism  other          3 Dec 31  2012 '/
drwxr-xr-x   2 gnuism  other          3 Dec 31  2012 "/
drwxr-xr-x   2 gnuism  other          3 Dec 31  2012 $/
drwxr-xr-x   2 gnuism  other          3 Dec 31  2012 \/
drwxr-xr-x   2 gnuism  other          3 Dec 31  2012 &/
drwxr-xr-x   2 gnuism  other          3 Dec 31  2012 #/
drwxr-xr-x   2 gnuism  other          3 Dec 31  2012 a\012b/
-rw-r--r--   1 gnuism  other      38455 Dec 31  2012 aclocal.m4
lrwxrwxrwx   1 gnuism  other         59 Dec 31  2012 ar-lib -> /usr/local/build/automake-1.13_SunOS5.10_sparcv9/lib/ar-lib
drwxr-xr-x   2 gnuism  other          7 Dec 31  2012 autom4te.cache/
drwxr-xr-x   3 gnuism  other          6 Dec 31  2012 build-a_lf_b/
.
.
.
drwxr-xr-x   2 gnuism  other          2 Dec 31  2012 dest-tab/
drwxr-xr-x   2 gnuism  other          2 Dec 31  2012 dest-tilde/
-rwxr-xr-x   1 gnuism  other      13997 Dec 31  2012 install-sh
-rw-r--r--   1 gnuism  other       1395 Dec 31  2012 Makefile.am
-rw-r--r--   1 gnuism  other      39656 Dec 31  2012 Makefile.in
-rwxr-xr-x   1 gnuism  other       6873 Dec 31  2012 missing
-rw-r--r--   1 gnuism  other         49 Dec 31  2012 source.c
drwxr-xr-x   2 gnuism  other          8 Dec 31  2012 sub/
drwxr-xr-x   2 gnuism  other          3 Dec 31  2012 @&t@/
#

A note from the OP: too bad the OpenSolaris project was killed by Oracle as we
could easly have seen how zoneadm makes the call to use cpio and changed it to
use something far far batter for the task. I solved this problem by doing the
move myself and used "star" or POSIX Standard TAR from Jörg Schilling to do the
zonemove.

Solution : 

    1 - Manually do the copy with a better tool from the source to dest. 

        1.1 - get sources for star from ftp://ftp.berlios.de/pub/schily/
        1.2 - extract them
        1.3 - cd into the extracted star sources and run /usr/xpg4/bin/make
        1.4 - once that ends just run /usr/xpg4/bin/make install 

        you now have star and it can handle anything that lives on a filesystem

        1.5 - do a copy from source to dest path with star 

            /opt/schily/bin/star -copy -p -acl -sparse -dump -C source . dest

    2 - cd to /etc/zones and edit the zonefile xml to use the new zonepath

    3 - edit the "index" file in /etc/zones to use the new zonepath

Special Detail to note : 

    1 - if you edit the zone XML file but do not edit the zonepath in the 
        index file you will get a warning message when you try to boot the zone : 

        # zoneadm -z z_001 boot 
        The XML repository has zonepath '/z/001',
        but the index file has zonepath '/zone/z_001'.
        These must match, so fix the incorrect entry.

Best Answer

Alternative: Copy via ZFS instead then change the zone path.

Using your parameters:

Zone name: z_001
Old zpool name: zone
New zpool name: zones
Zone lives on ZFS filesystem zone/z_001

We have the following procedure:

Shut down the zone.
Take a snapshot of its ZFS filesystem: zfs snapshot zone/z_001@premove
Transfer the snapshot: zfs send zone/z_001@premove | zfs recv zones/z_001
Detach the zone: zoneadm -z z_001 detach
Change the path: zonecfg -z z_001 set zonepath=/zones/z_001
Reattach the zone: zoneadm -z z_001 attach
Boot the zone.

Steps 4-6 change /etc/zones/index for you.

You don't have to shut down the zone first; you can take and send the snapshot, then shut it down, and take and send a second snapshot which will be incremental and much smaller and quicker: zfs send -i zone/z_001@premove zone/z_001@snap2 | zfs recv zones/z_001

You'll end up with a snapshot on the destination zpool too, which you can keep or delete as you will.

Related Solutions

How to clone ZFS ACLs from one file to another

Instead of using ls -lV you can use ls -lv which can be fed in to a script to convert it to a sequence of chmod commands to duplicate the ACLs.

E.g. if the ACl looks like this:

$ ls -lv file1
     0:owner@::deny
     1:owner@:read_data/write_data/append_data/write_xattr/execute
         /write_attributes/write_acl/write_owner:allow
     2:group@:read_data/write_data/append_data:deny
     3:group@:execute:allow
     4:everyone@:read_data/write_data/append_data/write_xattr
        /write_attributes/write_acl/write_owner:deny
     5:everyone@:read_xattr/execute/read_attributes/read_acl/synchronize:allow

It should be turned in to the following sequence of chmod commands:

chmod A0=owner@::deny file2
chmod A1=owner@:read_data/write_data/append_data/write_xattr/execute/write_attributes/write_acl/write_owner:allow file2
chmod A2=group@:read_data/write_data/append_data:deny file2
chmod A3=group@:execute:allow file2
chmod A4=everyone@:read_data/write_data/append_data/write_xattr/write_attributes/write_acl/write_owner:deny file2
chmod A5=everyone@:read_xattr/execute/read_attributes/read_acl/synchronize:allow file2

I recently found myself in a situation where a script like the one described above would have been useful, so here's a little Bash script I made (can also be sourced to the shell and run as a function) to print the list of chmod commands necessary to copy the ZFS ACLs from one file to another:

#!/bin/bash

acl_as_chmods () {
# print list of chmod commands to copy the ACL entries from '$1' to '$2'
 [[ -a "$1" ]] 2>/dev/null || {
   echo "Valid reference file required." >&2
   return 1
 }
 ls -vd "$1" | {
   read # first line is not ACL information; bypass
   read ACL_entry
     echo -n "chmod A=${ACL_entry#*:}"
   # if no target file was specified as '$2', use 'TARGET' variable at runtime
   while read ACL_entry || { echo " ${2-\$TARGET}"; false; }
   do
     [[ "$ACL_entry" == [0-9]*:* ]] && \
       echo -en " ${2-\$TARGET}\nchmod A${ACL_entry%%:*}+${ACL_entry#*:}" || \
       echo -n "$ACL_entry"
   done
 }
}

## run as script or source function to shell?
__acl_as_chmods () {
 [[ "${FUNCNAME[1]}" == "source" ]] || acl_as_chmods "$@"
}

__acl_as_chmods "$@"

Here are a couple example usages and their output for file1 from above:

~$ ./acl_as_chmods.sh file1 file2
chmod A=owner@::deny file2
chmod A1+owner@:read_data/write_data/append_data/write_xattr/execute/write_attributes/write_acl/write_owner:allow file2
chmod A2+group@:read_data/write_data/append_data:deny file2
chmod A3+group@:execute:allow file2
chmod A4+everyone@:read_data/write_data/append_data/write_xattr/write_attributes/write_acl/write_owner:deny file2
chmod A5+everyone@:read_xattr/execute/read_attributes/read_acl/synchronize:allow file2

~$ source acl_as_chmods.sh
~$ acl_as_chmods file1
chmod A=owner@::deny $TARGET
chmod A1+owner@:read_data/write_data/append_data/write_xattr/execute/write_attributes/write_acl/write_owner:allow $TARGET
chmod A2+group@:read_data/write_data/append_data:deny $TARGET
chmod A3+group@:execute:allow $TARGET
chmod A4+everyone@:read_data/write_data/append_data/write_xattr/write_attributes/write_acl/write_owner:deny $TARGET
chmod A5+everyone@:read_xattr/execute/read_attributes/read_acl/synchronize:allow $TARGET

If we like, we can even evaluate these chmods directly, if both files are accessible on this host and we wish to copy the ACLs from file1 to file2 immediately:

~$ ls -Vd file*  #BEFORE
-rwx--x--x   1 user user       0 Jun 19 04:12 file1
            owner@:--------------:------:deny
            owner@:rwxp---A-W-Co-:------:allow
            group@:rw-p----------:------:deny
            group@:--x-----------:------:allow
         everyone@:rw-p---A-W-Co-:------:deny
         everyone@:--x---a-R-c--s:------:allow
---x------+  1 user user       0 Jun 19 04:12 file2
            owner@:--x-----------:------:allow

~$ eval "$(acl_as_chmods file1 file2)"

~$ ls -Vd file*  #AFTER 
-rwx--x--x   1 user user       0 Jun 19 04:12 file1
            owner@:--------------:------:deny
            owner@:rwxp---A-W-Co-:------:allow
            group@:rw-p----------:------:deny
            group@:--x-----------:------:allow
         everyone@:rw-p---A-W-Co-:------:deny
         everyone@:--x---a-R-c--s:------:allow
-rwx--x--x   1 user user       0 Jun 19 04:12 file2
            owner@:--------------:------:deny
            owner@:rwxp---A-W-Co-:------:allow
            group@:rw-p----------:------:deny
            group@:--x-----------:------:allow
         everyone@:rw-p---A-W-Co-:------:deny
         everyone@:--x---a-R-c--s:------:allow

Moving a ZFS Filesystem Between Pools

The question about disabling services essentially depends on the services themselves:

Are they actively writing to the file system you plan to move ?
Are they storing persistent data you want to keep.

In any case, sending a ZFS file system might take a long time. It is possible to minimize service unavailability by keeping them online most of the time that way:

create a snapshot
send that snapshot the way previously suggested but while keeping all services active
when the fs is received on the other pool, disable the critical services bounds to that file system. Make sure the new file system on the destination pool is not modified as changes will be discarded later anyway.
create a second snapshot (e.g. snapshot2)

send that second snapshot incrementally, that will be much quicker that the previous transfer. e.g.:

zfs send -i rpool/filesystem@snapshot rpool/filesystem@snapshot2 \
    | zfs receive -F destinationpool/filesystem

when done, move the filesystem mount point from the old dataset to the new one. e.g.:
```
zfs set mountpoint=/application/directory.old rpool/filesystem
zfs set mountpoint=/application/directory destination/filesystem
```
You need to make sure no process is bound to /application/directory (e.g.: accessing files or having it as its current directory) to achieve that.
re-enable the service(s) and you are done.

Best Answer

Related Solutions

How to clone ZFS ACLs from one file to another

Moving a ZFS Filesystem Between Pools

Related Topic