The first thing to do in this situation is to use the -v option to ssh, so you can see what types of authentication is tried and what the result is. Does that help enlighten the situation?
In your update to your question, you mention "on another local Ubuntu". Have you copied over the ssh private key to the other machine?
AFAIK, there is no way to bypass the keyfile permission check with ssh or ssh-add (and you can't trick it with named pipe or such). Besides, you do not actually want to trick ssh, but just to be able to use your key files.
Indeed, TrueCrypt volume is supposed to keep your data private, so mounting the volumes as world-readable (default behaviour of TrueCrypt) is not really optimum. If you're using a FAT-formatted volume, you really should ajust the mount options, as Dan Carley suggested.
Although mount options aren't yet correctly supported by TrueCrypt for OS X (even if you launch TC using the command line interface and the mount options from the man page - already tried), OS X does support mount option defaults based on the volume name.
You need to know your user id (usually 501 if you are the first/only user of the computer). You can get it with "id -u".
Let' say you volume name is "PRIVATE" (volume names are in capitals), and your uid is 501, all you have to do is adding this line to /etc/fstab :
LABEL=PRIVATE none msdos -u=501,-m=700
You need to be root to create/edit this file (it is not present in default OSX install) :
sudo vim /etc/fstab
Next time you mount the volume, it'll have permission 700 and owner id 501.
This also works with USB drives (which are usually formatted in FAT, too).
Best Answer
Restore from backup or rebuild the server.