Why is Jenkins failing when fetching from git, while the command line isn't?

Question

All of my Jenkins builds are failing at the git fetch line.

It's failing at git fetch --tags --progress [email protected]:ethenwilson/whentoact.git

Started by user anonymous
Building in workspace /Users/ethen/.jenkins/workspace/Build NikNik
 > git rev-parse --is-inside-work-tree
Fetching changes from the remote Git repository
 > git config remote.origin.url [email protected]:ethenwilson/whentoact.git
Fetching upstream changes from [email protected]:ethenwilson/whentoact.git
 > git --version
using GIT_SSH to set credentials NikNik BitBucket SSH Key
 > git fetch --tags --progress [email protected]:ethenwilson/whentoact.git +refs/heads/*:refs/remotes/origin/*
FATAL: Failed to fetch from [email protected]:ethenwilson/whentoact.git
hudson.plugins.git.GitException: Failed to fetch from [email protected]:ethenwilson/whentoact.git
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:622)
    at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:854)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:879)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:624)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:530)
    at hudson.model.Run.execute(Run.java:1732)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:88)
    at hudson.model.Executor.run(Executor.java:234)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress [email protected]:ethenwilson/whentoact.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: remote: Counting objects: 2682, done.[K
remote: Compressing objects:   0% (1/1399)   [K
remote: Compressing objects:   1% (14/1399)   [K
...
remote: Compressing objects:  99% (1398/1399)   [K
remote: Compressing objects: 100% (1399/1399)   [K
remote: Compressing objects: 100% (1399/1399), done.[K
Receiving objects:   0% (1/2682)   
Receiving objects:   1% (27/2682)   
...  
Receiving objects:  78% (2092/2682), 4.07 MiB | 1.59 MiB/s   
Corrupted MAC on input.
Disconnecting: Packet corrupt
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1325)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1186)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:257)
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:620)
    ... 10 more

When I run git fetch --tags --progress [email protected]:ethenwilson/whentoact.git from the command line, it works fine, which means my SSH keys must be working.

I'm connecting to BitBucket with Jenkins with SSH verification. Jenkins gets the key from the file it's located (the default one), so I know that Jenkins is using the same key as I am when I run from the command line.

I'm using the latest build of the BitBucket and Git plugins for Jenkins. My installed Git on my Mac is version 1.8.5.2 (Apple Git-48).

My jenkins start command is nohup java -jar ~/jenkins.war --httpPort=8081 --ajp13Port=8010 > /tmp/jenkins.log 2>&1 &.

What's going wrong?

EDIT: I was wrong, I had accidentally hit an option to have the SSH Key be in the wrong place when I did that. Now, using @borrrden's suggestion, it still gives the same error. **EDIT: As @borrrden suggested, I changed my start command to nohup java -Dorg.jenkinsci.plugins.gitclient.Git.useCLI=true -jar ~/Downloads/jenkins.war --httpPort=8081 --ajp13Port=8010 > /tmp/jenkins.log 2>&1 &, and now I get a different crash:

Started by user anonymous
Building in workspace /Users/ethen/.jenkins/workspace/Build NikNik
 > git rev-parse --is-inside-work-tree
Fetching changes from the remote Git repository
 > git config remote.origin.url [email protected]:ethenwilson/whentoact.git
Fetching upstream changes from [email protected]:ethenwilson/whentoact.git
 > git --version
using GIT_SSH to set credentials NikNik BitBucket SSH Key
 > git fetch --tags --progress [email protected]:ethenwilson/whentoact.git +refs/heads/*:refs/remotes/origin/*
FATAL: Failed to fetch from [email protected]:ethenwilson/whentoact.git
hudson.plugins.git.GitException: Failed to fetch from [email protected]:ethenwilson/whentoact.git
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:622)
    at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:854)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:879)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:624)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:530)
    at hudson.model.Run.execute(Run.java:1732)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:88)
    at hudson.model.Executor.run(Executor.java:234)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress [email protected]:ethenwilson/whentoact.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1406)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1194)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:265)
    at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:620)
    ... 10 more

Solution 1

1) Go to job configuration

2) Go to the "Source Code Management" section

3) Additional behaviors > add

4) Select "Wipe out repository and force clone"

This will delete and re-clone only the workspace which is for your job. If you'd like to confirm before deleting, then I suggest echoing out the $WORKSPACE variable via a batch/bash command build step.

Also, this makes the build much slower, so I suggest removing it after one build.

Solution 2

I had this problem as well and was only able to solve it by deleting the workspace of the problematic repository on our master Jenkins server.

I think the problem was that there was a connection-error (like @gbjbaanb said) in a few of the builds (our Bitbucket crashed). This left the workspace on master in a corrupt state, and because Jenkins tries to use cached workspaces where it can, this caused every following build to fail as well.

Solution 3

For me, this was hitting the 10 minute default timeout for the git-client plugin.

Solved by setting an advanced clone behaviour on the job and upping the timeout.

In the job configuration page under the Git plugin section, there is a drop-down list "Add". Within that dropdown list there is a selection "Advanced clone behaviours". When you add the advanced clone behaviors, you'll see a field for "Timeout (in minutes) for clone and fetch operation".

If you add additional behaviours before the operation you can up the timeout for clone and checkout - which has translated to a higher timeout value in my console

• Advanced Checkout behaviours
• Advanced Clone behaviours

Putting any value in timeout overrides the default.

Knowledge gained from JENKINS-20445.

Solution 4

Seems like a network error:

Receiving objects: 78% (2092/2682), 4.07 MiB | 1.59 MiB/s

Corrupted MAC on input.

Disconnecting: Packet corrupt

fatal: The remote end hung up unexpectedly

fatal: early EOF

fatal: index-pack failed

suggests the network broke at 78% of the way through.

Seems to be a common problem.

Solution 5

This issue is probably caused by a timeout check in place while fetching. You can increase it by following the advice mentioned below.

In the job configuration page under the Git plugin section, there is a drop-down list "Add". Within that dropdown list there is a selection "Advanced clone behaviours". When you add the advanced clone behaviors, you'll see a field for "Timeout (in minutes) for clone and fetch operation".

View more solutions

Comments

[Lyserty]

I have the same issue on my Jenkins instance on Docker. When I try to clone repo with the same SSH key from the same workspace in the terminal it works perfectly but running Jenkins job it trowing error Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress [email protected]:SomeName/repo-name.git +refs/heads/*:refs/remotes/origin/*" returned status code 128: All suggestions below doesn't work :(

[Eliza Wilson]

What can I do to fix it then? I've tested this to happen about fifty times in a row and every build had this, so this is not a one off thing.

[gbjbaanb]

@EthenA.Wilson see edit, and search for 'git fatal: early EOF' for some suggested ways to fix it.

[borrrden]

If that's the case then why is he able to clone from the command line with no problem?

[gbjbaanb]

@borrrden you'll have to ask him - sometimes users say "it always works for me" when they mean "I tried it once". Maybe its just fluke. Maybe his network is QoS based on user. I don't know - I do know what 'remote end hung up' means though.

[Eliza Wilson]

@borrrden I tried to use the command line to do git fetch around a dozen times over the course of trying to figure out this problem, and it worked every time.

[Max]

Yup! Depending on the version of the git plugin, the option might also be called "Lightweight checkout" - which you want to disable, then re-run, and then re-enable.