I've been bashing my head against a wall with this one all day and I would greatly appreciate a few more eyes on the problem at hand.
We have an in-house SVN Server that contains all live and development code for our website. Our live server can connect to this and get updates from the repository.
This was all working fine until we migrated the SVN Server from a physical machine to a vSphere VM. Now, for some reason that continues to fathom me, we can no longer connect to the SVN Server.
The SVN Server runs CentOS 6.2, Apache and SVN 1.7.2. SELinux is well and trully disabled and the problem remains when iptables is stopped.
Our production server does run an older version of CentOS and SVN but the same system worked previously so I don't think that this is the issue.
Of note, if I have iptables enabled, using service iptables status, I can see a single packet coming in and being accepted but the production server simply hangs on any svn command. If I give up waiting and do a CTRL-C to break the process I get a "could not connect to server".
To me it appears to be something to do with the SVN Server rejecting external connections but I have no idea how this would happen.
Any thoughts on what I can try from here?
Thanks, Rob
Edit: Network topology
Production server sits externally to our in-house SVN server. Our IPCop (?) firewall allows connections from it (and it alone) on port 80 and passes the connection to the SVN Server. The hardware is all pretty decent and I don't doubt that its doing its job correctly, especially as iptables is seeing the new connections.
subversion.conf (in /etc/httpd/conf.d)
LoadModule dav_svn_module modules/mod_dav_svn.so
<Location /repos>
DAV svn
SVNPath /var/svn/repos
<LimitExcept PROPFIND OPTIONS REPORT>
AuthType Basic
AuthName "SVN Server"
AuthUserFile /var/svn/svn-auth
Require valid-user
</LimitExcept>
</Location>
ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:5F:C8:3A
inet addr:172.16.0.14 Bcast:172.16.0.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe5f:c83a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:32317 errors:0 dropped:0 overruns:0 frame:0
TX packets:632 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2544036 (2.4 MiB) TX bytes:143207 (139.8 KiB)
netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 1484/mysqld
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1135/rpcbind
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1351/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1230/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1575/master
tcp 0 0 0.0.0.0:58401 0.0.0.0:* LISTEN 1153/rpc.statd
tcp 0 0 0.0.0.0:5672 0.0.0.0:* LISTEN 1626/qpidd
tcp 0 0 :::139 :::* LISTEN 1678/smbd
tcp 0 0 :::111 :::* LISTEN 1135/rpcbind
tcp 0 0 :::80 :::* LISTEN 1615/httpd
tcp 0 0 :::22 :::* LISTEN 1351/sshd
tcp 0 0 ::1:631 :::* LISTEN 1230/cupsd
tcp 0 0 ::1:25 :::* LISTEN 1575/master
tcp 0 0 :::445 :::* LISTEN 1678/smbd
tcp 0 0 :::56799 :::* LISTEN 1153/rpc.statd
iptables –list -v -n (when iptables is stopped)
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
iptables –list -v -n (when iptables is running, after one attempted svn connection)
Chain INPUT (policy ACCEPT 68 packets, 6561 bytes)
pkts bytes target prot opt in out source destination
19 1304 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
1 60 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:80
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 17 packets, 1612 bytes)
pkts bytes target prot opt in out source destination
tcpdump
17:08:18.455114 IP 'production server'.43255 > 'svn server'.local.http: Flags [S], seq 3200354543, win 5840, options [mss 1380,sackOK,TS val 2011458346 ecr 0,nop,wscale 7], length 0
17:08:18.455169 IP 'svn server'.local.http > 'production server'.43255: Flags [S.], seq 629885453, ack 3200354544, win 14480, options [mss 1460,sackOK,TS val 816478 ecr 2011449346,nop,wscale 7], length 0
17:08:19.655317 IP 'svn server'.local.http > 'production server'k.43255: Flags [S.], seq 629885453, ack 3200354544, win 14480, options [mss 1460,sackOK,TS val 817679 ecr 2011449346,nop,wscale 7], length 0
Best Answer
(I think you should update your question with full details of your SVN configuration from httpd.conf, and also
ifconfig
output, andnetstat -lntp
to some pastebin)However if you do the following...
And you do this on both the guest and VM host and you still cannot connect, then the problem is not iptables, or selinux.
So you can proceed to check each stage of the problem like so....
check you can resolve the svn server hostname;
Can you ping the IP?
(assuming your svn is exposed using mod_svn_dav apache module...)
Can you connect to this server on port 80;
Can you connect to the httpd server and get the server string... (you can use curl, netcat, telnet, or whatever your TCP connect tool of choice here)
then try the username/password like so;
should include something like;
HTTP/1.1 200 OK
and then try the command line svn tools to see if they make the connection;
Basically if you get here svn is working, its the client tools or the proxy that are misconfigured....etc...
One thing to consider is that proxies mess with headers, and also that Windows 7 doesn't like Basic Auth, so you might need to implement digest authentication, or some self signed SSL to bypass either your VM, or some proxy, or windoze network work for bef**kering your svn connections. (dont try any of this until you know what the problem is...)
also, all the hits on the REJECT rule suggest that you might not even being trying to connect to SVN on port 80 anyway.
Stick in another rule with a LOG line before that to dump those packets to syslog;
change the "8" to whatever the correct rule number for your INPUT chain.... then look in /var/log for kern.log or whatever log file your syslog is configured to send iptables logging to. (it should be obvious because its going to get massive quickly)