gitlab: git clone https with large repos fails

Question

When trying to clone a large repo (~700MB) over https, git fails with:

c:\git-projects>git clone https://git.mycompany.de/fs.git
Cloning into 'fs'...
Username for 'https://git.mycompany.de': mwlo
Password for 'https://[email protected]':
efrror: RPC failed; result=22, HTTP code = 500
atal: The remote end hung up unexpectedly

clone over ssh works:

c:\git-projects>git clone [email protected]:fs.git
Cloning into 'fs'...
remote: Counting objects: 144564, done.
remote: Compressing objects: 100% (30842/30842), done.
remote: Total 144564 (delta 95360), reused 143746 (delta 94542)
Receiving objects: 100% (144564/144564), 601.34 MiB | 1.33 MiB/s, done.
Resolving deltas: 100% (95360/95360), done.
Checking out files: 100% (4649/4649), done.

Cloning smaller repositories with https also works:

c:\git-projects>git clone https://git.mycompany.de/git-test.git
Cloning into 'git-test'...
remote: Counting objects: 135, done.
remote: Compressing objects: 100% (129/129), done.
remote: Total 135 (delta 68), reused 0 (delta 0)
Receiving objects: 100% (135/135), 18.77 KiB | 0 bytes/s, done.
Resolving deltas: 100% (68/68), done.

I have already tweaked some parameters but without success:

/etc/nginx/nginx.conf
worker_processes  2; # have two cpu's
keepalive_timeout  120;
client_max_body_size 3072m;

/home/git/gitlab/config/gitlab.yml
## Git settings
  # CAUTION!
  # Use the default values unless you really know what you are doing
  git:
    bin_path: /usr/bin/git
    # Max size of a git object (e.g. a commit), in bytes
    # This value can be increased if you have very large commits
    max_size: 3221225472 # 3072.megabytes
    # Git timeout to read a commit, in seconds
    timeout: 120

We would like to use git clone https, as the visual studio tools for git still not have implemented ssh.

On the server are two processes, CPU load goes to 100% after a while, then the processes are terminated.

git pack-objects --revs --all --stdout --progress --delta-base-offset 

Regards, Marco

---

System information
System:         Debian 6.0.7
Current User:   root
Using RVM:      no
Ruby Version:   1.9.3p392
Gem Version:    1.8.23
Bundler Version:1.3.5
Rake Version:   10.0.4

GitLab information
Version:        5.3.0
Revision:       148eade
Directory:      /home/git/gitlab
DB Adapter:     mysql2
URL:            https://git.mycompany.de
HTTP Clone URL: https://git.mycompany.de/some-project.git
SSH Clone URL:  [email protected]:some-project.git
Using LDAP:     yes
Using Omniauth: no

GitLab Shell
Version:        1.4.0
Repositories:   /home/git/repositories/
Hooks:          /home/git/gitlab-shell/hooks/
Git:            /usr/bin/git

Solution 1

This is reported in issue 3079: https cloning requires a large amount of resources (CPU, but mainly memory) on the GitLab server, and currently (GitLab 5.x) large repos.

Even GitLab 6.0 comes with commits like 7ecebdd, mentioning timeouts when cloning large repo.

I haven't tested with GitLab 6, though (which should release tomorrow).

Solution 2

Consider implementing a chunking plugin for nginx, such as HttpChunkinModule.

I personally haven't deployed the above but have a client with similar issues.

In their case, the problem is on the client side, where we need to instruct devs to use the following tweak to their local git config:

git config http.postBuffer 524288000 #Set to 500MB

The above will simply allow larger git-related http requests on the client side (we've got plenty of memory on the server side).

Comments

[mawl]

I will update to 6.0 these days, let's see if it brings a fix.

[mawl]

6.0.1 having the same issues. Now it runs with unicorn not puma and error code is 502.

[VonC]

@mawl I have seen that switch (stackoverflow.com/a/18398991/6309). But I didn't see a 6.0.1. I follow the commits for the upcoming 6.1.

[mawl]

Version 6.0 is 6.0.1 (Take a look on the help page after updating)

[spuder]

The HttpChunkinModule is no longer needed if you are using nginx > 1.3.8 wiki.nginx.org/HttpChunkinModule

[senthilkumar]

I had the same error when pushing to a https server, this fix solved the problem, thanks.

[maestromani]

@Vonc, Im having the same issue in 6.2.3. How do we resolve this ?

[VonC]

@maestromani did you try with a more recent version of GitLab first?

[maestromani]

@Vonc, Since this is a production server need to plan that activity :-(. If i didn't get any solution then i have too.