Gitlab Configuration Issues:: NGINX Unicorn Port Conflict

Question

I have managed to partly setup Gitlab on a Linux CentOS server with Apache, Git, PHP, PostGreSQL and MySQL. I am running the Chef Cookbook version. I got the rpm from here. I wanted to use it to manage my Git repo better and more visually and this seemed to be a good choice. But now I run into issues getting it to work.

Just to make it really work and update all files I decided to rerun the configuration using gitlab-ctl reconfigure. Second run did work:

Chef Client finished, 4 resources updated
gitlab Reconfigured!

See full log

The hoster had already put NGINX on 8080 not get into an argument with Apache running on port 80 where we have a LAMP project running. But now Ruby's Unicorn Web Server seems to be conflicting with NGINX. I have worked with NGINX a little bit, not much and this is my first stab at Gitlab. Anyways this is what I figured out with the help of my hoster.

When I log into testserver.domain.net and pass the following command:

netstat -ln |grep 8080 I see

tcp        0      0 127.0.0.1:8080              0.0.0.0:*                   LISTEN      

So something is running on 8080 According to my hoster it should run on 0.0.0.0:8080. And when we check what is running on that port we see

netstat -tupln |grep 8080
tcp        0      0 127.0.0.1:8080              0.0.0.0:*                   LISTEN      21627/unicorn maste 

When we check the process id 21627, we see

cat /proc/21627/cmdline 
unicorn master -E production -c /var/opt/gitlab/gitlab-rails/etc/unicorn.rb /opt/gitlab/embedded/service/gitlab-rails/config.ru

That is a Ruby process, and not a NGINX process.

So NGINX seems to be conflicting with Unicorn.

And when we check the logs of nginx we see that nginx cannot get going because of this:

tail -f /var/log/gitlab/nginx/error.log 
2014/07/28 09:43:10 [emerg] 23122#0: bind() to 0.0.0.0:8080 failed (98: Address already in use)
2014/07/28 09:43:10 [emerg] 23122#0: still could not bind()
2014/07/28 09:43:12 [emerg] 23123#0: bind() to 0.0.0.0:8080 failed (98: Address already in use)
2014/07/28 09:43:12 [emerg] 23123#0: bind() to 0.0.0.0:8080 failed (98: Address already in use)
2014/07/28 09:43:12 [emerg] 23123#0: bind() to 0.0.0.0:8080 failed (98: Address already in use)
2014/07/28 09:43:12 [emerg] 23123#0: bind() to 0.0.0.0:8080 failed (98: Address already in use)
2014/07/28 09:43:12 [emerg] 23123#0: bind() to 0.0.0.0:8080 failed (98: Address already in use)
2014/07/28 09:43:12 [emerg] 23123#0: still could not bind()

I googled Unicorn.rb and found this link. I also read that:

Unicorn is an HTTP server for Rack applications designed to only serve fast clients on low-latency, high-bandwidth connections and take advantage of features in Unix/Unix-like kernels. Slow clients should only be served by placing a reverse proxy capable of fully buffering both the the request and response in between Unicorn and slow clients.

When I check the file /var/opt/gitlab/gitlab-rails/etc/unicorn.rb I do see it uses 8080. Issue is that it seems Unicorn should work together with NGINX so perhaps I should not change the port.

What step should I take to make Gitlab work? Can Gitlab work without Unicorn? I would think not. Should I then pick another port for it or perhaps for NGINX?

Solution 1

You should override the ports in /etc/gitlab/gitlab.rb. Don't mess with /var/opt/gitlab/... because any manual configuration will be lost after a reconfigure. In particular read on Setting the NGINX listen port. In the downloads page there is a sentence: For troubleshooting and configuration options please see the Omnibus GitLab readme. I wonder if this is not seen by people :/ If not we should make it more clear.

Solution 2

To add to Axil's comments. I had to change it in /var/opt/gitlab/gitlab-rails/etc/unicorn.rb file as well. A restart after that worked, I did not lose my configs. The unicorn master was not picking my changes from /etc/gitlab/gitlab.rb file even after multiple restarts.

Solution 3

You should update the /etc/gitlab/gitlab.rb file.

Make sure when you modify the file that you are uncommenting the line that you are changing. It's using chef, so if the gitlab.rb file is configured properly when you run sudo gitlab-ctl reconfigure; it will update the corresponding file properly. Then sudo gitlab-ctl restart to restart the services.

sudo lsof -Pni |grep <port number> is your friend when determining port conflicts

Solution 4

The documentation suggests setting 'nginx['listen_port'] = 8080' but when I did this unicorn failed to bind port 8080. I set the nginx port to 8888 instead and it worked. This suggests that in order to set the nginx port to 8080, it is necessary to change unicorn from default port of 8080 to something else, but I didn't explore that possibility. It would be nice if the example for setting the nginx port was a setting that would work in the default configuration, which mine is, as well as I know - I only installed it this morning.

View more solutions

Comments

[VijayKumar]

This just helped me as well. Changing anything on /etc/gitlab.gitlab.rb does not help, changing /var/opt/gitlab/gitlab-rails/etc/unicorn.rb did. I wonder why the port is hardcoded as 8080

[axil]

Restarting will work (gitlab-ctl restart), but when you update to a new version reconfigure is run (gitlab-ctl reconfigure), so any changes will be lost. There is new documentation in docs.gitlab.com/omnibus/settings/…

[Edenshaw]

Although Axil's answer is correct, yours was the one that shed more light on this issue for me. I have to make two modifications to the gitlab.rb file: gitlab_workhorse['auth_backend'] = "http://localhost:8080" and this one: unicorn['port'] = 8080. Then execute sudo gitlab-ctl reconfigure and finally sudo gitlab-ctl restart. Thanks a lot!