Ssh – Unable to git clone over SSH from Bitbucket from certain servers

bitbucketconnectiongitssh

We have a Bitbucket Server instance hosted in AWS. From some other AWS servers (in another region), git clone over SSH fails with

ssh: connect to host (hostname) port 7999: Connection refused

However, other servers in AWS (in the same region as the Bitbucket server) can successfully clone over SSH, using the same URL.

Other information:

  • Bitbucket is definitely listening on port 7999:

    $ sudo netstat -tnlp | grep :7999

tcp6 0 0 :::7999 :::* LISTEN 20707/java

    (process 20707 is the main Bitbucket process)

  • Bitbucket is running behind Apache as a reverse proxy to provide SSL.

  • tcptraceroute and mtr on port 7999 from the instance that can't clone to Bitbucket on port 7999 successfully connect.
  • The public key for the keypair used by the instance that can't connect is added to the repository's access keys with read access on Bitbucket.
  • In the AWS security groups, the cloning instance's group allows all outbound connections, and the Bitbucket server's groups allow connections from anywhere on port 7999.

  • Results from a tcpdump on port 7999 on the cloning instance:

    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

16:56:53.348387 IP (tos 0x0, ttl 64, id 61715, offset 0, flags [DF], proto TCP (6), length 60)
(cloning server's hostname).38606 > (bitbucket server's hostname).irdmi2: Flags [S], cksum 0x2799 (incorrect -> 0xb1db), seq 3675985409, win 26883, options [mss 8961,sackOK,TS val 1512892178 ecr 0,nop,wscale 7], length 0

16:56:53.489908 IP (tos 0x0, ttl 252, id 37586, offset 0, flags [none], proto TCP (6), length 40)
(bitbucket server's hostname).irdmi2 > (cloning server's hostname).38606: Flags [R.], cksum 0x24e4 (correct), seq 1472002966, ack 3675985410, win 26883, length 0

  • Results from ssh'ing from the cloning server to Bitbucket:

     $ sudo ssh -vvv -p 7999 ssh://git@stash.tddevops.com
 OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
 debug1: Reading configuration data /root/.ssh/config
 debug1: Reading configuration data /etc/ssh/ssh_config
 debug1: /etc/ssh/ssh_config line 56: Applying options for *
 debug2: ssh_connect: needpriv 0
 debug1: Connecting to stash.tddevops.com [172.24.16.201] port 7999.
 debug1: connect to address 172.24.16.201 port 7999: Connection refused
 ssh: connect to host stash.tddevops.com port 7999: Connection refused

  • tcpdump on the Bitbucket server shows no data when a clone is attempted.

Best Answer

From https://serverfault.com/a/288493/442063:

You see the "incorrect" checksums due to a feature called TCP checksum offloading. The checksum fields for outgoing TCP packets are not pre-calculated by the operating system but instead set to 0 and left for calculation by the NIC processor. The Wireshark FAQ has a more detailed explanation.

That might help, but your problem sounds like a firewall issue. It couldn't hurt to make sure you do not have an internal firewall running on this hosted AWS server. Without knowing what OS you are running, I can't really speculate on what to look for. Many Linux operating systems use iptables. For example, you can see here how to turn off and disable the firewall for Oracle Linux or Redhat Linux.

Related Topic