Centos – SVN Server not responding

apache-2.2centosiptablessvn

I've been bashing my head against a wall with this one all day and I would greatly appreciate a few more eyes on the problem at hand.

We have an in-house SVN Server that contains all live and development code for our website. Our live server can connect to this and get updates from the repository.

This was all working fine until we migrated the SVN Server from a physical machine to a vSphere VM. Now, for some reason that continues to fathom me, we can no longer connect to the SVN Server.

The SVN Server runs CentOS 6.2, Apache and SVN 1.7.2. SELinux is well and trully disabled and the problem remains when iptables is stopped.

Our production server does run an older version of CentOS and SVN but the same system worked previously so I don't think that this is the issue.

Of note, if I have iptables enabled, using service iptables status, I can see a single packet coming in and being accepted but the production server simply hangs on any svn command. If I give up waiting and do a CTRL-C to break the process I get a "could not connect to server".

To me it appears to be something to do with the SVN Server rejecting external connections but I have no idea how this would happen.

Any thoughts on what I can try from here?

Thanks, Rob

Edit: Network topology
Production server sits externally to our in-house SVN server. Our IPCop (?) firewall allows connections from it (and it alone) on port 80 and passes the connection to the SVN Server. The hardware is all pretty decent and I don't doubt that its doing its job correctly, especially as iptables is seeing the new connections.

subversion.conf (in /etc/httpd/conf.d)

LoadModule dav_svn_module     modules/mod_dav_svn.so
<Location /repos>
   DAV svn
   SVNPath /var/svn/repos

   <LimitExcept PROPFIND OPTIONS REPORT>
      AuthType Basic
      AuthName "SVN Server"
      AuthUserFile /var/svn/svn-auth
      Require valid-user
   </LimitExcept>
</Location>

ifconfig

eth0      Link encap:Ethernet  HWaddr 00:0C:29:5F:C8:3A
          inet addr:172.16.0.14  Bcast:172.16.0.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe5f:c83a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:32317 errors:0 dropped:0 overruns:0 frame:0
          TX packets:632 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2544036 (2.4 MiB)  TX bytes:143207 (139.8 KiB)

netstat -lntp

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      1484/mysqld
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      1135/rpcbind
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1351/sshd
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      1230/cupsd
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      1575/master
tcp        0      0 0.0.0.0:58401               0.0.0.0:*                   LISTEN      1153/rpc.statd
tcp        0      0 0.0.0.0:5672                0.0.0.0:*                   LISTEN      1626/qpidd
tcp        0      0 :::139                      :::*                        LISTEN      1678/smbd
tcp        0      0 :::111                      :::*                        LISTEN      1135/rpcbind
tcp        0      0 :::80                       :::*                        LISTEN      1615/httpd
tcp        0      0 :::22                       :::*                        LISTEN      1351/sshd
tcp        0      0 ::1:631                     :::*                        LISTEN      1230/cupsd
tcp        0      0 ::1:25                      :::*                        LISTEN      1575/master
tcp        0      0 :::445                      :::*                        LISTEN      1678/smbd
tcp        0      0 :::56799                    :::*                        LISTEN      1153/rpc.statd

iptables –list -v -n (when iptables is stopped)

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

iptables –list -v -n (when iptables is running, after one attempted svn connection)

Chain INPUT (policy ACCEPT 68 packets, 6561 bytes)
 pkts bytes target     prot opt in     out     source               destination
   19  1304 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
    1    60 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:80
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:80
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW udp dpt:80

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 17 packets, 1612 bytes)
 pkts bytes target     prot opt in     out     source               destination

tcpdump

17:08:18.455114 IP 'production server'.43255 > 'svn server'.local.http: Flags [S], seq 3200354543, win 5840, options [mss 1380,sackOK,TS val 2011458346 ecr 0,nop,wscale 7], length 0
17:08:18.455169 IP 'svn server'.local.http > 'production server'.43255: Flags [S.], seq 629885453, ack 3200354544, win 14480, options [mss 1460,sackOK,TS val 816478 ecr 2011449346,nop,wscale 7], length 0
17:08:19.655317 IP 'svn server'.local.http > 'production server'k.43255: Flags [S.], seq 629885453, ack 3200354544, win 14480, options [mss 1460,sackOK,TS val 817679 ecr 2011449346,nop,wscale 7], length 0

Best Answer

(I think you should update your question with full details of your SVN configuration from httpd.conf, and also ifconfig output, and netstat -lntp to some pastebin)

However if you do the following...

[root@workstation001 ~]# iptables --flush             <---  flush rules
[root@workstation001 ~]# iptables --list -n -v
Chain INPUT (policy ACCEPT 17 packets, 2792 bytes)    <---  is default policy ACCEPT?
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 18 packets, 1004 bytes)
 pkts bytes target     prot opt in     out     source               destination 


[root@workstation001 ~]# setenforce 0
setenforce: SELinux is disabled               <---  f**k off selinux

And you do this on both the guest and VM host and you still cannot connect, then the problem is not iptables, or selinux.

So you can proceed to check each stage of the problem like so....

check you can resolve the svn server hostname;

[root@workstation001 ~]# dig svn.somehost.co.uk +short
www.somehost.co.uk.
209.135.17.202       <---  resolved!!

Can you ping the IP?

ping 209.135.17.202
PING 209.135.17.202 (209.135.17.202) 56(84) bytes of data.
64 bytes from 209.135.17.202: icmp_req=1 ttl=50 time=132 ms <---  reply!
....

(assuming your svn is exposed using mod_svn_dav apache module...)

Can you connect to this server on port 80;

[root@workstation001 ~]# telnet svn.liepaper.co.uk 80
Trying 209.135.17.202...
Connected to svn.liepaper.co.uk.          <---  connected!
Escape character is '^]'.
^]
telnet> 

Can you connect to the httpd server and get the server string... (you can use curl, netcat, telnet, or whatever your TCP connect tool of choice here)

[root@workstation001 ~]# wget -O- --server-response http://svn.liepaper.co.uk
--2012-03-20 16:16:06--  http://svn.liepaper.co.uk/
Resolving svn.liepaper.co.uk... 209.135.17.202
Connecting to svn.liepaper.co.uk|209.135.17.202|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 401 Authorization Required
  Date: Tue, 20 Mar 2012 16:16:06 GMT
  Server: Apache                    <---  server header value
  WWW-Authenticate: Basic realm="Authorization Realm"
  Content-Length: 401
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html; charset=iso-8859-1
Authorization failed.

then try the username/password like so;

wget -O- --server-response --user=mysvnuser --password=mokeypoke http://svn.liepaper.co.uk/projects/admin/

should include something like;

HTTP/1.1 200 OK

and then try the command line svn tools to see if they make the connection;

[root@workstation001 ~]# svn list --username=mysvnuser --password=mokeypoke http://svn.liepaper.co.uk/projects/admin/
mystupidcode/
anotherwasteofcomputer/
cantprogrammefortoffee/

Basically if you get here svn is working, its the client tools or the proxy that are misconfigured....etc...

One thing to consider is that proxies mess with headers, and also that Windows 7 doesn't like Basic Auth, so you might need to implement digest authentication, or some self signed SSL to bypass either your VM, or some proxy, or windoze network work for bef**kering your svn connections. (dont try any of this until you know what the problem is...)


also, all the hits on the REJECT rule suggest that you might not even being trying to connect to SVN on port 80 anyway.

 8    26840 2356K REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Stick in another rule with a LOG line before that to dump those packets to syslog;

itpables  -A INPUT 8 -j LOG --log-prefix "LOGDROP: "

change the "8" to whatever the correct rule number for your INPUT chain.... then look in /var/log for kern.log or whatever log file your syslog is configured to send iptables logging to. (it should be obvious because its going to get massive quickly)