Slow git clone and fetch

How about this:

cd ~/.ssh
echo "Host MYSERVER.com" > config
echo "Port 3031" >> config
cd
git clone git@MYSERVER.com:gitosis-admin.git

From Gitosis' point of view, your name is just the name, in keydir, of the public key that you authenticated with. No configuration on your local machine matters, except as it affects what public key you use, and the string at the end of the key doesn't matter -- just the filename.

You've given your public key two names, so when you authenticate with that key it is undefined which one of them it finds when it looks for a name for that key. When you changed some other things in Gitosis, presumably it happened to change the arbitrary choice of which name it found.

(Specifically, I believe Gitosis generates its own authorized_keys to contain all the keys in keydir, one per line, with options to tell sshd to only let the user run Gitosis, only in a controlled manner, and telling Gitosis the name of the key. If multiple lines have the same key, I'm not sure which one sshd ends up using -- maybe the first, maybe the last, maybe it's arbitrary. The order in which Gitosis writes them may also be arbitrary. But really the details of how the arbitrary choice happens are beside the point, because nobody guarantees they won't change in the next update to sshd or gitosis.)

You should pick which name you want to use to refer to yourself in the Gitosis config, and keep in keydir only the copy of your key under that name.