Make scp prompt me for a username and password

That should absolutely match hidden files. The / at the end of the source says "every file under this directory". Nevertheless, testing and research bear you out. This is stupid behavior.

The "answer" is to append a dot to the end of the source:

scp -rp src/. user@server:dest/

The real answer is to use rsync.

How have you configured the server to allow your key? By putting your public key in the .ssh/authorized_keys on the server? If that is the case, it should work. Are you using the same usernames (both remote and local) when you use ssh/scp as when you use rsync?

Oh, and btw, this question should be asked at serverfault.com instead I think.