Ssh – linux ssh passphrase not working, but sftp works

I set up ssh with keys and it was working fine for a while. Ive just had a bit of a clean up and now some seriously weird things are happening, disabling terminal access from my server

As I say SFTP is working, asks me for my passphrase, enter passphrase, then get access

When I use ssh over terminal it asks for passphrase, i enter the same passphrase and it says permission denied?

I am connecting to ubuntu server with a centos client.

Ive also tried using windows putty as the client, but again it denies my passphrase.

How come it works on my centos client through sftp? is the passphrase cached or something?

I use the same passphrase for everything, so I am wondering why terminal access is completely ignoring my passphrase.

I have access to all the files, so is there any way I can just turn on passwords in ssh conf and then reboot the server to restart ssh (since i cannot reboot it from terminal)

nautlius asks me for keyring and i enter my passphrase and it works for sftp, how come terminal access has stopped working in the same way?

Any help is much much much appreciated.

Cheers

ke