Jenkins occasionally fails to clone/fetch

Question

I am running code hosted on Gitlab on regular basis in Jenkins. From time to time, Jenkins GIT plugin failes to fetch the remote repository. It works fine when I just run it again.

Any idea? I have notifications for failed builds and it drives me crazy.

Output:

Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on vps6 (worker) in workspace /home/worker/workspace/foo
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url [email protected]:bar/foo.git # timeout=10
Fetching upstream changes from [email protected]:bar/foo.git
 > git --version # timeout=10
using GIT_SSH to set credentials 
 > git fetch --tags --progress [email protected]:bar/foo.git +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from [email protected]:bar/foo.git
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
    at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
    at hudson.scm.SCM.checkout(SCM.java:504)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
    at hudson.model.Run.execute(Run.java:1727)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress [email protected]:bar/foo.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: ssh_exchange_identification: Connection closed by remote host
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2002)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1721)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:405)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
    at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
    at hudson.remoting.UserRequest.perform(UserRequest.java:210)
    at hudson.remoting.UserRequest.perform(UserRequest.java:53)
    at hudson.remoting.Request$2.run(Request.java:364)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to vps6
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1737)
        at hudson.remoting.UserResponse.retrieve(UserRequest.java:313)
        at hudson.remoting.Channel.call(Channel.java:952)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
        at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
        at com.sun.proxy.$Proxy52.execute(Unknown Source)
        at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:886)
        at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
        at hudson.scm.SCM.checkout(SCM.java:504)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
        at hudson.model.Run.execute(Run.java:1727)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
Finished: FAILURE

Solution

I found the solution here: https://secure.phabricator.com/T11469

In my case the issue was due to the "MaxStartups" directive in sshd, which specifies the maximum number of concurrent unauthenticated connections to the SSH daemon (see man sshd_config for details).

When many Jenkins jobs are running concurrently, this may results in many concurrent unauthenticated connections to the SSH daemon, in which case sshd will start dropping connections. You can set the "MaxStartups" directive in /etc/ssh/sshd_config to increase number of concurrent unauthenticated connections at which sshds will start dropping connections (defaults to 10).

Comments

[c4f4t0r]

I had the same problem and it was solved with Serrano solution