Ubuntu – Git clone over SSH access denied only from specific server

gitgitlabsshUbuntu

I have an Ubuntu 12.04 server inside my office network that is being used to host our git repos. The server is running Gitlab 7.1.1, and has SSH on a non-standard port.

It works perfectly for everyone inside and outside of the office network with one exception. The exception is a managed server with a hosting company, so I only have limited access to it.

When I try to clone the repo on the non-standard port, it times out. I assume this is because the outgoing port is blocked by the hosting company. To solve that, I setup port forwarding on my office router to forward incoming traffic on port 22, to my non-standard port on the git server. I tested by cloning from a Digial Ocean VPS without specifying the port, and it worked fine.

But the problematic server now gives the following error:

Access denied.
fatal: The remote end hung up unexpectedly

The git server is registering an SSH connection being made, and accepting the public key, as shown in /var/log/auth.log:

Jan 20 15:09:07 gitlab sshd[3043]: Accepted publickey for git from 10.0.1.254 port 60771 ssh2
Jan 20 15:09:07 gitlab sshd[3043]: pam_unix(sshd:session): session opened for user git by (uid=0)
Jan 20 15:09:09 gitlab sshd[3162]: Received disconnect from 10.0.1.254: 11: disconnected by user
Jan 20 15:09:09 gitlab sshd[3043]: pam_unix(sshd:session): session closed for user git

That looks no different to any other auth request in the log, so I have no idea why the clone is failing?

I setup a test repo on Github, and the problematic server clones fine over SSH. It just has a problem with repos from my office server over SSH for some reason.

It's also worth noting that the problematic server can clone over HTTP with a username and password fine, it's only over SSH that is a problem.

Any ideas what the issue is here?

PS, I'm more a coder than server admin, I am getting involved in this to try improve my company's deployment process, so much of this is new to me

Best Answer

I have resolved this.

The problem was the public key for the problematic server had already been added as a "deploy key" to Gitlab for earlier unrelated testing. It was then deleted as a deploy key, but for some reason the key persisted in the Gitlab DB. Gitlab then let me re-add the same key to a test user without complaining that it already existed elsewhere in the DB. But when trying to authenticate with that key, Gitlab would look up the key, and get the first "orphaned" key, so it would only give anonymous access which obviously wouldn't allow the clone.

To solve the issue, I found the orphaned key id, and removed it with the git-shell:

./bin/gitlab-keys rm-key key-21

Where 21 was the ID of the orphaned key.

Everything is now working as expected.

Related Solutions

Windows – Setting up Git over SSH on Windows using a non-standard port

I have a similar workflow but use gitosis to manage the repo on my VPS.

I use the msysgit bash with the windows path and plink as you suggest above but in addition I use pagent to manage the SSH key.

To access my repos this is the order I take,

Open Pagent and access my key (you only enter the passphrase here once if you have one and even if you do it remembers it for the whole session).
Then open Git bash and add/commit/push/pull/clone as normal.

One caveat I will note is that I had to convert the key I created with the Git bash using Putty Gen (available on the same link above) so that Pagent could use it.

Ubuntu – Loggin in ssh server: Permission denied, please try again

Are you certain that the user account you're attempting to access is correctly configured? If you log in as root on the system, can you su to the user account?

# su - username

What do you see in your logs after a failed connection attempt? On many systems, sshd will log to something /var/log/secure or /var/log/auth.log. Also, I note that you have PasswordAuthentication enabled but ChallengeResponseAuthentication disabled. Do you see the same behavior if you enable ChallengeResponseAuthentication?

Here are some general diagnostic steps to use when you have ssh problems:

Enable verbose diagnostics in ssh:
```
ssh -v host.example.com
```
This will cause the client to output a variety of diagnostic messages as it negotiates the connection. This will often provide a clue to the problem.
Run the server in debug mode.

On your server, stop sshd, then run it from the command line like this:
```
/usr/sbin/sshd -d
```
This will produce verbose debug logging on stderr that will very often contain useful information.

If neither of these helps you figure out what's going on, would you add the output to your question?

Best Answer

Related Solutions

Windows – Setting up Git over SSH on Windows using a non-standard port

Ubuntu – Loggin in ssh server: Permission denied, please try again

Related Topic