Linux – ZFS snapshot send incremental

centos7linuxzfszfsonlinux

I have been happily using zfsnap with CentOS and ZFSonLinux in a scripted manner similar to how it is described on the web site, but have run into an issue.
Unfortunately backups did not run a few days ago, and now I am stuck in an odd situation. I have a series of incremental backups on my backup ZFS server from my main ZFS server and some newer snapshots on the main server. Unfortunately due to the missed backup and automatic destruction of older snapshots on the main server there is no longer a common snapshot between the two systems to perform an incremental send with.
i.e. on backup there is:

backuppool/x@snap2 
backuppool/x@snap3
backuppool/x@snap4

and on the main zfs server there is:

mainpool/x@snap6 
mainpool/x@snap7
mainpool/x@snap8

Is there any way to get them in sync again or do I have to destroy everything on backup and perform a full(there is not enough room for two full backups)?
I tried specifying a send referencing the name of snap4 on the remote backup server from the main server, but as the main server does not have snap 4 it cannot calculate the incremental. I also tried to send snap4 from the backup to the main server, but it appears that the only way to do this would be to do it as a full send(not enough room), and I would need to wipe out existing snapshots.
The deleting of all your backup history if you get snapshots mismatched seems pretty inconvenient, is there any solution in this scenario?

Best Answer

As you have no common baseline snapshot in your setup, you will not be able to perform an incremental zfs send any longer. You would need to destroy the dataset in backuppool:

zfs destroy -r backuppool/x

and re-send the most recent snapshot to backuppool

zfs send mainpool/x@6 | [transfer magic] | zfs recv backuppool/x

The deleting of all your backup history if you get snapshots mismatched seems pretty inconvenient, is there any solution in this scenario?

The "solution" is simply to not delete source snapshots unless those snapshots have been verified to have transmitted successfully (i.e. show up in the remote's zfs list -tall output). I have no idea if zfsnap will support you with this, but even if it won't, you could use a shell script performing this kind of a check before letting zfsnap delete anything.

Related Solutions

Best compression for ZFS send/recv

It sounds like you've tried all of the best compression mechanisms and are still being limited by the line speed. Assuming running a faster line is out of the question, have you considered just running the backups less frequently so that they have more time to run?

Short of that, is there some kind of way to lower the amount of data being written? Without knowing your application stack its hard to say how, but just doing things like making sure apps are overwriting existing files instead of creating new ones might help. And making sure you arent saving backups of temp/cache files that you wont need.

Why is `zfs list -t snapshot` orders of magnitude slower than `ls .zfs/snapshot`

Snapshot operations are a function of the number of snapshots you have, RAM, disk performance and drive space. This would be a general ZFS issue, not something unique to the Linux variant.

The better question is: Why you have 1797 snapshots of a zvol? This is definitely more than recommended and makes me wonder what else is happening on the system.

People say "ZFS snapshots are free", but that's not always true.

While ZFS snaps don't have an impact on production performance, the high number you have clearly require disk accesses to enumerate.

Disk access time > RAM access time, hence the order of magnitude difference.

strace output. Note the time per syscall and imagine how poorly it would scale with the number of snapshots in your filesystem.

# strace -c ls /ppro/.zfs/snapshot

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  0.00    0.000000           0        10           read
  0.00    0.000000           0        17           write
  0.00    0.000000           0        12           open
  0.00    0.000000           0        14           close
  0.00    0.000000           0         1           stat
  0.00    0.000000           0        12           fstat
  0.00    0.000000           0        28           mmap
  0.00    0.000000           0        16           mprotect
  0.00    0.000000           0         3           munmap
  0.00    0.000000           0         3           brk
  0.00    0.000000           0         2           rt_sigaction
  0.00    0.000000           0         1           rt_sigprocmask
  0.00    0.000000           0         2           ioctl
  0.00    0.000000           0         1         1 access
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         1           fcntl
  0.00    0.000000           0         2           getdents
  0.00    0.000000           0         1           getrlimit
  0.00    0.000000           0         1           statfs
  0.00    0.000000           0         1           arch_prctl
  0.00    0.000000           0         2         1 futex
  0.00    0.000000           0         1           set_tid_address
  0.00    0.000000           0         1           set_robust_list
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000                   133         2 total

versus

# strace -c zfs list -t snapshot

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.003637          60        61         7 ioctl
  0.00    0.000000           0        12           read
  0.00    0.000000           0        50           write
  0.00    0.000000           0        19           open
  0.00    0.000000           0        19           close
  0.00    0.000000           0        15           fstat
  0.00    0.000000           0        37           mmap
  0.00    0.000000           0        19           mprotect
  0.00    0.000000           0         1           munmap
  0.00    0.000000           0         4           brk
  0.00    0.000000           0         2           rt_sigaction
  0.00    0.000000           0         1           rt_sigprocmask
  0.00    0.000000           0         3         1 access
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         1           getrlimit
  0.00    0.000000           0         1           arch_prctl
  0.00    0.000000           0         2         1 futex
  0.00    0.000000           0         1           set_tid_address
  0.00    0.000000           0         1           set_robust_list
------ ----------- ----------- --------- --------- ----------------
100.00    0.003637                   250         9 total

Best Answer

Related Solutions

Best compression for ZFS send/recv

Why is `zfs list -t snapshot` orders of magnitude slower than `ls .zfs/snapshot`

Related Topic