Linux – How to automate GPG file decryption over SSH

I would like to use GnuPG to decrypt short messages that are stored on a
remote host (running Linux), i.e.:

1. ssh [<user>@]<host>
2. gpg -d <file-to-decrypt>
3. interact with gpg-agent and/or just type in the password
4. close SSH connection

but in a more automated way. I just want to type a single command and the script should do the rest (except for (interactive) input of the password), i.e.:

1. remote-gpg [<user>@]<host> <file-to-decrypt>
2. [query for password without echoing it back in plaintext]
3. [dump the decrypted text on stdout] AND close the SSH connection

My main challenge is merging the "ssh" and "gpg" step.

I am looking for a simple and effective way to achieve this:

• with as little dependencies on the client side (ideally, just ssh +
  core utilities)
• without messing a lot with sockets and pipes on the server side
• with no assumption about whether gpg-agent is running on the server

So, solutions such as writing a GNU expect script are out of question.

Motivation: A typical use case would be using a phone (running Android) to retrieve an encrypted message (e.g., a password) from the remote server. Closing the connection upon information delivery is desired because you might forget logging out before your phone gets lost/stolen, and the thief cannot retrieve additional information without guessing the password (since password is queried every time). Finally, removing your phone's SSH key from server's authorized_keys file is all it takes to prevent a security breach.

By the way, would there be any additional security risks with that approach (as compared to the 4-step manual process)?