502 Bad Gateway from Nginx for large GitLab fork

Question

I'm running GitLab 6.0.0 through Nginx and can fork small repositories, but when I try to fork a large repository (2GB) I see a "502 Bad Gateway" page after about one minute.

/var/log/nginx/gitlab_error.log shows:

2013/08/29 12:21:33 [error] 25098#0: *221 upstream prematurely closed connection while reading response header from upstream, 
  client: 12.34.56.78, 
  server: myserver, 
  request: "POST /mygroup/myproject/fork HTTP/1.1", 
  upstream: "http://unix:/home/git/gitlab/tmp/sockets/gitlab.socket:/mygroup/myproject/fork", 
  host: "myserver", 
  referrer: "http://myserver/mygroup/myproject/fork"

Solution

Issue 1527 suggests a memory issue (and that memory requirement in mentioned in the doc).

It can also be because of an initial timeout:

I found this error would also occur because the unicorn workers would sometimes take 33 seconds to start and they're configured to timeout after 30 seconds.

You can modify unicorn config file /home/git/gitlab/config/unicorn.rb :

timeout 300

In your NGiNX config, you can also add:

proxy_connect_timeout 300;
proxy_read_timeout 300;

If you have a /etc/nginx/fastcgi_params file with your NGiNX, you can add:

fastcgi_connect_timeout 60;
fastcgi_send_timeout 180;
fastcgi_read_timeout 180;
fastcgi_buffer_size 156 16k;
fastcgi_buffers 256 16k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_intercept_errors on;
fastcgi_max_temp_file_size 0

fastcgi_pass            unix:/tmp/fpm.sock;

---

Note that after a 502, and after making the fixes mentioned above, it is a good idea to clear the browser cache before trying again to access gitlab.

Comments

[Nicholas Albion]

Yes I do have a /etc/nginx/fastcgi_params file, but every line is in the format: "fastcgi_param PARAM_IN_CAPS $value". Should the fastcgi lines you list above be in the unicorn.rb file?

[VonC]

@NicholasAlbion no, those lines shouldn't be (to my knowledge) in the unicorn.rb file. The most important for me was the timeout 300 directive, giving time to the web server to wait for the initial compilation occurring during the very first query.

[Nicholas Albion]

I switched from unicorn to puma and it's working better now. gist.github.com/tkretschmer-rb/5248495

[VonC]

@NicholasAlbion interesting: quite the opposite of stackoverflow.com/a/18398991/6309, then?

[solusipse]

This was the only solution that worked in my case. Thank you!

[Paul Calabro]

@NicholasAlbion: Wow! That actually worked! Thanks for posting that! I've been scratching my head for hours... :)