Make consistent copy of maildir

backupdata-consistencymaildir

Assuming I'm using Dovecot and it's maildir format to save and access mails on a server. How do I avoid race conditions while copying a maildir?

I did read some tutorial about backups and using maildir but it did no see anyone writing about this. They just use cp or rsync to copy the directory to another place. Is it impossible that maildir become an inconsistent state by copying or do I need some kind of locking?

EDIT: I want to make regular backups but I don't think that it realy metters for the question. I know that I could stop the mail server (Dovecot and Postfix) but I think it should be possible without doing this. As far as i know maildir support concurrent access by different applications.

Best Answer

I just read the documentation for Maildir in Dovecot and few other documents about Maildir and Maildir++. I hope that I did not miss anything important.

Maildir is designed to work without locks. Most required operations are atomic on modern file systems. This means that you do not have to care much about race conditions like inconsistent read. But there are still some issues if you want to backup a Maildir while the mail-server is running.

Issues

  • Backing up and restoring tmp/ is useless. Each Maildir contains the directories new/, cur/ and tmp/. Directory tmp/ contains mails which are being written to disk at the moment. They are moved to new/ when they are written successfully. This means that files in tmp/ may not be complete yet. Even if the file is complete, the process which was writing the file is not running anymore after restoring the backup. This means if such file is restored, it will never be added to the mailbox and it may never be removed.

  • It may be sensible to exclude dovecot-uidlist.lock from backup. Dovecot uses an extension of Maildir called Maildir++. This extension need locking. Acquiring the lock for reading is not required by the extension but it may be sensible to exclude the lock-file from backup.

  • Acquire dovecot-uidlist.lock or use snapshot of filesystem. Basically, you can just copy the directory but it is possible to miss some mails by race conditions. The reason is that listing and copying the content of a directory (recursively) is not atomic. This means that it is possible to miss an email while creating the backup when a user is changing a tag (e.g. seen/unseen) or moving the mail. To handle such situations, Dovecot acquires the lock-file (same as above) for every action. If you are using Dovecot, you can solve the problem by acquiring the lock before creating the backup. This can be done with /usr/lib/dovecot/maildirlock. Another possibility is to use snapshots of your filesystem. Since creating a snapshot is atomic, the problem does not occur on snapshots.

As a quick summary: If you want to make a backup while Dovecot is running, you should first acquire the lock-file dovecot-uidlist.lock or create a snapshot of you filesystem. Then, you can copy the Maildir. It may be sensible to exclude tmp/ and dovecot-uidlist.lock from your backup.

Related Topic