Convert UTF-8 NFD Filenames to UTF-8 NFC with rsync or afpd

afpmac-osxrsynctruenasutf-8

I have a home file server running FreeNAS 8. A few days ago I used rsync to upload my entire iTunes library from Mac so that I could load my library over the network instead of off a slow USB drive. This mostly worked, and iTunes runs much better now, but I'm running into issues accessing any songs that have non-ascii characters in it (I first noticed the problem when loading Queensrÿche tracks). The files would show up in the Finder, but any attempt to access them made them vanish until I reconnected to the server.

After some research I found out this is because OSX uses a different UTF character order from Linux. OSX filesystems use Unicode Normalization Form D (NFD), where linux uses Form C (NFC). Rsync doesn't convert these forms when it performs the copy from my mac to the server, now when iTunes tries to access a file with a special character over the network, the files on the server have the wrong encoding and afpd reports they don't exist.

What is the best way to address this problem? Is it possible to make rsync perform the unicode conversion while uploading the base library to the server? Can I configure afpd to transmit/receive filenames in NFD format? Is there an easy solution to change the filenames on the server? I found some stuff about a program named convmv, but I don't know if I can run that on FreeNAS.

Best Answer

Note: If you are using version 3.0.0 or newer of rsync, the --iconv option as mentioned in the other answers is clearly the superior solution.

Something that should work is rsyncing between the source directory and the mounted remote file system (SMB, NFS, AFP), which rsync will just treat as local file system.

However, I do not know how well this works in practice, and you have to work around different issues, for example the delta-transfer algorithm won’t be used by default (since source and destination are “local”) (maybe --no-whole-file will work?), you have to check,e.g., that SMB effectively preserves modification times, etc.

Related Topic