Doveadm pw may generate invalid password if a special charactare is used

Did you configure mail to access your mail using POP3/IMAP? If not, it will directly access your mbox.

Have a look at your maildir dovecot is using: mail_location = maildir:/home/vmail/%d/%n/Maildir and also the directory ~/mbox (default mbox directory).

Try to monitor /var/log/mail.log, without further information from there we won't be able to fix that problem.

glibc 2.7 and higher do support sha512, but only in a salted way. If you do have salted SHA512-hashes, using "crypt" should instantly work.

If you do have unsalted sha512-hashes: please do avoid them - seriously. There are pre-calculated databases available, so looking up the hash from those databases will give you instantly either the cleartext-password or at least something which will work for a successful login. Calculating hashes using current GPUs is extremely fast as well, so about any unsalted way needs to be avoided.

For example, https://www.crackstation.net/ gives you both a nice webinterface for looking up both millions of unsalted passwords from their hash and some education regarding "salted vs. unsaltes hashes", choosing correct (randomized) salts and "good" hash algorithms.

As a real-life-example, millions of unhashed password-hashes from linkedin and lastfm did leak onto the internet and it did take only a few days to find matching cleartext-passwords for most of them using pre-calculated hash databases or rainbow tables. Using salted hashes, it's more likely to take months or years to find some cleartext-data matching your hashes.

If you do understand the risks of unsalted hashes and still absolutely do need to implement unsalted sha512 via MySQL, you can do so by supplying a custom skript to pure-authd (it's fairly trivial doing so). See http://download.pureftpd.org/pure-ftpd/doc/README.Authentication-Modules for more information on this.