Git fetch hangs on git-upload-pack

Question

Seemingly at random, our local repos can no longer fetch from our Bonobo server. It happens with our biggest remote repo (about 4GB) to different people and at different times, when we try to fetch locally. At first it was just every few months, but now it's increasing in frequency and happened to a lot of local repos at the same time today. For now we are solving it by moving a working .git folder around.

It rapidly prints some POST git-upload-pack (gzip X to Y bytes), then hangs for half an hour to an hour.

If I delete a pack file from the objects dir, it complains about missing things then starts fetching correctly, but only for that git-upload-pack.

I've tried repack and gc in various ways to no effect. I tried upgrading git on both server and client from 1.8.4 to 1.9.

cloneing had the same problem, but upgrading Bonobo solved that, even though it was the same version of git. A freshly cloned repo yesterday was working but today it has the same problem.

Of interest may be that we have a lot of dll's and pdb's in it. It's been in use for a year now and was imported from an SVN repo with git-svn.

This is the trace

$ GIT_TRACE=1 git fetch -v
trace: built-in: git 'fetch' '-v'
trace: run_command: 'git-remote-https' 'origin' 'https://xxx
/yyy.git'
trace: run_command: '"C:/Program Files (x86)/GitExtensions/GitCredentialWinStore
/git-credential-winstore.exe" get'
trace: run_command: '"C:/Program Files (x86)/GitExtensions/GitCredentialWinStore
/git-credential-winstore.exe" store'
trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--in
clude-tag' '--thin' 'https://xxx/yyy.git/'
trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--i
nclude-tag' '--thin' 'https://xxx/yyy.git/'
POST git-upload-pack (gzip 2057 to 1096 bytes)
POST git-upload-pack (gzip 2307 to 1222 bytes)
POST git-upload-pack (gzip 3657 to 1914 bytes)
POST git-upload-pack (gzip 6207 to 3192 bytes)
POST git-upload-pack (gzip 12607 to 6374 bytes)

Googling shows some people had this problem but nothing mentioned (upgrade bonobo, etc) worked.

Solution 1

git uses curl internally, so use the following config settings to debug:

Client Configuration:

git config --global http.postBuffer 524288000

Client Environment Variables:

• GIT_CURL_VERBOSE=1
• GIT_HTTP_MAX_REQUESTS=16

Server Configuration:

• increase <requestLimits maxAllowedContentLength=[desired size]> in web.config; the size could be 1073741824
• increase <httpRuntime maxRequestLength=[desired size]> in web.config; try the value 1024000

In addition, the following settings can automatically abort slow transfers:

• GIT_HTTP_LOW_SPEED_TIME
• GIT_HTTP_LOW_SPEED_LIMIT

If the HTTP transfer speed is less than GIT_HTTP_LOW_SPEED_LIMIT for longer than GIT_HTTP_LOW_SPEED_TIME, the transfer is aborted.

References

• Git - Environment Variables
• Git - git-config Documentation
• Git source - http.c
• How can I debug git/git-shell related problems?
• Git - How to use .netrc file on windows to save user and password
• Bonobo Git Server - FAQ
• Bonobo Git Server - Changelog
• Rate limits
• Limits in AWS CodeCommit
• Scaling Bitbucket Server
• Bitbucket Server config properties

Solution 2

I know this is an old one, but i had a similar issue and wanted to share my experience. I never really got to the cause, but after updating to the latest Git (on the server - even though bonobo ships with its own version), checking permissions, commands like git fsck upgrading Bonobo and even adding a ton of extra logging to the code, this is what did it for me.

Bonobo creates a bare repo on the server, and i could sucessfully clone directly from it on the server, and could fetch / pull etc. This prompted me to try to replicate the repo and replace it with a copy. So, in a temp folder:

git clone [path_to_Bonobo repo] temp_repo

Then moved the Bonobo git repo out of the repository root folder, entered the Bonobo root folder and:

git clone --mirror [path_to_temp_repo] [original_repo_name]

I also did a:

git fetch --prune

git push --prune [path_to_temp_repo] +refs/remotes/origin/*:refs/heads/* +refs/tags/*:refs/tags/*

After this, Bonobo happily started fetching pulling and pushing again.