Ldap – Tortoise SVN: Initial connect timeout

apache-2.4ldapsvntimeouttortoisesvn

I have a problem when connecting Tortoise SVN to my SVN server. The first connect seems to run into a timeout (SVN hangs about 20 seconds) and the second (automatic) try works instantly. I.e. every update, commit, log takes a looong time and drives everyone crazy…

The same SVN action from a Linux commandline client works instantly. Access via web browser works also without any delay.

The SVN setup is as follows:
The SVN server runs CentOS 7, Apache 2.4.6 and mod_dav_svn 1.7.14. Access via SSL is configured. Authentication is done via LDAP on a Windows Server 2012 R2.
The client (Windows 7 x64) runs Tortoise 1.7.15 (x64).

This is the relevant Apache configuration (obfuscated…):

# Reduce LDAP cache to 30 seconds
LDAPCacheTTL 30 
LDAPOpCacheTTL 30

<Location /svn>
  DAV svn
  SVNParentPath /var/svnrepo
  SVNListParentPath on
  AuthName "My Repository"
  AuthType basic
  AuthBasicProvider ldap
  AuthLDAPURL "ldap://dc.mydomain.local/OU=Group of Users,OU=MyOU,DC=MyDomain,DC=local?sAMAccountName?sub?(objectClass=*)" NONE

  AuthLDAPBindDN "CN=ServiceUser,OU=Group of Users,OU=MyOU,DC=MyDomain,DC=local"
  AuthLDAPBindPassword "VERYSECRETPASSWORD"

  # Require ldap group via Microsoft rule "LDAP_MATCHING_RULE_IN_CHAIN"
  Require ldap-filter memberof:1.2.840.113556.1.4.1941:=CN=SVN-Group,OU=Group of DomainLocalGroups,OU=MyOU,DC=MyDomain,DC=local
</Location>

The Apache log (ssl_error_log) starts like this when initiating SVN activity:

[Wed Aug 26 07:48:14.560025 2015] [ssl:info] [pid 2310] [client 192.168.1.155:49316] AH01964: Connection to child 0 established (server svnserver.mydomain.local:443)
[Wed Aug 26 07:48:14.560563 2015] [ssl:debug] [pid 2310] ssl_engine_kernel.c(1878): [client 192.168.1.155:49316] AH02043: SSL virtual host for servername svnserver.mydomain.local found

Now SVN seems to hang (timestamp at second 14) – Apache would kill the connection a bit later when module reqtimeout is enabled (I disabled it in this scenario). About 20 seconds later (timestamp at second 36), activity continues:

[Wed Aug 26 07:48:36.102214 2015] [ssl:debug] [pid 2310] ssl_engine_kernel.c(224): [client 192.168.1.155:49316] AH02034: Subsequent (No.2) HTTPS request received for child 0 (server svnserver.mydomain.local:443)
[Wed Aug 26 07:48:36.102267 2015] [authz_core:debug] [pid 2310] mod_authz_core.c(809): [client 192.168.1.155:49316] AH01626: authorization result of Require ldap-filter memberof:1.2.840.113556.1.4.1941:=CN=SVN-Group,OU=Group of DomainLocalGroups,OU=MyOU,DC=MyDomain,DC=local: denied (no authenticated user yet)
[Wed Aug 26 07:48:36.102275 2015] [authz_core:debug] [pid 2310] mod_authz_core.c(809): [client 192.168.1.155:49316] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)
[Wed Aug 26 07:48:36.102334 2015] [authnz_ldap:debug] [pid 2310] mod_authnz_ldap.c(501): [client 192.168.1.155:49316] AH01691: auth_ldap authenticate: using URL ldap://dc.mydomain.local/OU=Group of Users,OU=MyOU,DC=MyDomain,DC=local?sAMAccountName?sub?(objectClass=*)
[Wed Aug 26 07:48:36.147960 2015] [ldap:debug] [pid 2310] util_ldap.c(372): AH01278: LDAP: Setting referrals to On.
[Wed Aug 26 07:48:36.154921 2015] [authnz_ldap:debug] [pid 2310] mod_authnz_ldap.c(593): [client 192.168.1.155:49316] AH01697: auth_ldap authenticate: accepting john.doe

The ssl_access_log and ssl_request_log starts at second 36, nothing is logged from second 14

For me, it looks like Tortoise SVN starts connection either without credentials or not using https (?) which leads to a timeout. The second connection seems to be using https but with the wrong credentials and finally Tortoise connects using the right credentials and authentication succeeds.

Any ideas? What happens between second 14 and second 36 and how can I prevent it from happening? 😉

Thanks,
Florian

Update: I found a solution (although I'm not really sure what the reason is…):
Everytime I start Tortoise SVN (or the Windows svn commandline client), I noticed activity on the firewall: the domain controller was trying to reach several servers, e.g. c.root-servers.net which seem to be VERISIGN servers to check the SSL certificate or to find new root certificates etc (?). This was blocked by the firewall so the DC took its time to go through its list of alternative servers.

We are using self-signed certificates, therefore my first try was to inject our CA certificate into Windows' certificate management via group policy. This worked (at least SVN did not complain about an unknown CA after deleting all auth information). Nevertheless, the 20-sec-delay is still there (and the DC is still trying to reach Verisign).

Eventually I tried the local SVN configuration option "ssl-authority-files" to statically pass the CA certificate et voilà: the delay is gone!

I.e. this leads to 20 sec delay:

svn list myrepo

and this not

svn list --config-option servers:global:ssl-authority-files=C:\temp\mycertificate.crt myrepo

It's still a bit sad that this workaround needs configuration on every client, but at least it is a solution…

I am not sure what Windows is doing when verifying the certificate, maybe it is checking for revocation or something and waiting for timeout (because it cannot reach Verisign and friends). No idea.

Best Answer

Issue: After invoking SVN on the command line on a firewalled server, nothing visible happens for 15 seconds, then the program quits with the following error:

svn: E170013: Unable to connect to a repository at URL 'SVN.REPOSITORY.REDACTED'

svn: E730054: Error running context: An existing connection was forcibly closed by the remote host.

Investigation: Internet research on the above errors did not uncover any pertinent information.

Process Tracing (procmon) showed a connection attempt to an Akamai (cloud services) server after the SSL/TLS handshake to the SVN Server. The hostname for the server was not shown in Process tracing. Reverse DNS lookup showed a184-51-112-88.deploy.static.akamaitechnologies.com or a184-51-112-80.deploy.static.akamaitechnologies.com as the hostname, and the IP was either 184.51.112.88 or 184.51.112.80 (2 entries in DNS cache).

Packet capture tool (MMA) showed a connection attempt to the hostname ctldl.windowsupdate.com after the SSL/TLS Handshake to the SVN server.

The windows Crypto API was attempting to connect to Windows Update to retrieve Certificate revocation information (CRL – certificate revocation list). The default timeout for CRL retrieval is 15 seconds. The timeout for authentication on the server is 10 seconds; as 15 is greater than 10, this fails.

Resolution: Internet research uncovered the following: (also see picture at bottom)

Solution 1: Decrease CRL timeout Group Policy -> Computer Config ->Windows Settings -> Security Settings -> Public Key Policies -> Certificate Path Validation Settings -> Network Retrieval – see picture below.

https://subversion.open.collab.net/ds/viewMessage.do?dsForumId=4&dsMessageId=470698

support.microsoft.com/en-us/kb/2625048

blogs.technet.com/b/exchange/archive/2010/05/14/3409948.aspx

Solution 2: Open firewall for CRL traffic

support.microsoft.com/en-us/kb/2677070

Solution 3: SVN command line flags (untested)

serverfault.com/questions/716845/tortoise-svn-initial-connect-timeout - alternate svn command line flag solution.

Additional Information: Debugging this issue was particularly difficult. SVN 1.8 disabled support for the Neon HTTP RA (repository access) library in favor of the Serf library which removed client debug logging. [1] In addition, the SVN error code returned did not match the string given in svn_error_codes.h [2] Also, SVN Error codes cannot be mapped back to their ENUM label easily, this case SVN error code E170013 maps to SVN_ERR_RA_CANNOT_CREATE_SESSION.

  1. stackoverflow.com/questions/8416989/is-it-possible-to-get-svn-client-debug-output
  2. people.apache.org/~brane/svndocs/capi/svn__error__codes_8h.html#ac8784565366c15a28d456c4997963660a044e5248bb3a652768e5eb3105d6f28f
  3. code.google.com/archive/p/serf/issues/172

Suggested SVN Changes:

  1. Enable Verbosity on the command like for all operations

  2. Add error ENUM name to stderr

  3. Add config flag for Serf Library debug logging.