Jenkins occasionally fails to clone/fetch

gitgitlabJenkins

I am running code hosted on Gitlab on regular basis in Jenkins. From time to time, Jenkins GIT plugin failes to fetch the remote repository. It works fine when I just run it again.

Any idea? I have notifications for failed builds and it drives me crazy.

Output:

Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on vps6 (worker) in workspace /home/worker/workspace/foo
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git@gitlab.com:bar/foo.git # timeout=10
Fetching upstream changes from git@gitlab.com:bar/foo.git
 > git --version # timeout=10
using GIT_SSH to set credentials 
 > git fetch --tags --progress git@gitlab.com:bar/foo.git +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git@gitlab.com:bar/foo.git
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
    at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
    at hudson.scm.SCM.checkout(SCM.java:504)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
    at hudson.model.Run.execute(Run.java:1727)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress git@gitlab.com:bar/foo.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: ssh_exchange_identification: Connection closed by remote host
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2002)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1721)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:405)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
    at hudson.remoting.UserRequest.perform(UserRequest.java:210)
    at hudson.remoting.UserRequest.perform(UserRequest.java:53)
    at hudson.remoting.Request$2.run(Request.java:364)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to vps6
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1737)
        at hudson.remoting.UserResponse.retrieve(UserRequest.java:313)
        at hudson.remoting.Channel.call(Channel.java:952)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
        at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
        at com.sun.proxy.$Proxy52.execute(Unknown Source)
        at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:886)
        at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
        at hudson.scm.SCM.checkout(SCM.java:504)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
        at hudson.model.Run.execute(Run.java:1727)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
Finished: FAILURE

Best Answer

I found the solution here: https://secure.phabricator.com/T11469

In my case the issue was due to the "MaxStartups" directive in sshd, which specifies the maximum number of concurrent unauthenticated connections to the SSH daemon (see man sshd_config for details).

When many Jenkins jobs are running concurrently, this may results in many concurrent unauthenticated connections to the SSH daemon, in which case sshd will start dropping connections. You can set the "MaxStartups" directive in /etc/ssh/sshd_config to increase number of concurrent unauthenticated connections at which sshds will start dropping connections (defaults to 10).

Related Topic