What "access rights" could be blocking access to a gitlab repository?

Question

I'm trying to setup gitlab (6.5.1) on a fresh clean server. Everything appears to work, but git is unable to push to any project. Following the commands from the newly created project page and pushing to the remote via ssh gives:

$ git push -u origin master
fatal: Could not read from remote repository.

Please make sure you have the correct access
rights and the repository exists.

This seems to be a fairly common problem. Unfortunately it seems to have a number of potential causes and none of them seem to match. From issue 3424 on an old release and various other sources online I've seen and checked the following suggestions:

• Leftover ssh keys

  This is a clean setup with no leftovers. My key got added to the authorized keys file properly and is the only one listed.

• Running ssh with debug logging shows errors related to Ruby environment vars.

  Mine comes up clean. SSH debug shows a successful connection. Everything about the authentication handshaking is normal, then this is the end of the output:

  debug1: Sending command: git-receive-pack 'username/reponame.git'
  debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
  debug1: client_input_channel_req: channel 0 rtype [email protected] reply 0
  debug1: channel 0: free: client-session, nchannels 1
  debug1: fd 0 clearing O_NONBLOCK
  debug1: fd 1 clearing O_NONBLOCK
  

• Problems with the gitlab-shell environment.

  Unlike many others with the same error message above, my gitlab-shell check script returns a clean bill of health:

  % sudo -u gitlab -H ~gitlab/gitlab-shell/bin/check   
  Check GitLab API access: OK
  Check directories and files: 
          /var/lib/gitlab/repositories: OK
          /var/lib/gitlab/.ssh/authorized_keys: OK
  Test redis-cli executable: redis-cli 2.8.5
  Send ping to redis server: PONG
  

• Restart {unicorn,sidekiq,redis}

  The reports that restarting one or more services clears this up don't seem to apply here. This isn't an intermittent problem that releading a daemon fixes.

• The repo isn't getting physically created

  But it is. The first time every time, the bare git repo in ~gitlab/repositories/username/reponame.git gets created every time and seems to have the correct permissions.

• Gitlab-shell can't talk to the API server because A) DNS problems, B) wrong ip/port/interface binding C) not having/having a trailing slash.

  The check script says the API access is fine.

  I'm not running nginx, so the default ip binding issue related to that is n/a.

  I've tried both *:8080 and 127.0.0.1:8080 for the listen value in unicorn.yml.

  Beyond that, I have tried various iterations of localhost, 127.0.0.1 and the fully qualified domain name (which is DNS resolving fine) with and without trailing slashes in shell.yml to no avail. I have also tried wiring this directly to the unicorn server on port 8080 instead of the Apache SSL / proxy host on port 80. Nothing seems to make any difference. My certificate is not self signed and are working fine for browsers, but I tried setting self_signed_cert: true anyway. Nothing.

• The reported git paths are wrong, add fully qualified path from gitlab user home.

  This seems like a legit suggestion if the gitlab-shell isn't doing some monkey business to correct this already, but I tried changing git remote add origin gitlab@server:username/reponame.git to ``git remote add origin gitlab@server:repositories/username/reponame.git` to no avail. Same error.

This seems to be the litany of solutions suggested, but none of them seem right. Note I am able to push over http. The login prompt accepts my ldap username and password and accepts a push. This is only a problem trying to use SSH. Testing just the ssh login part with ssh -T gitlab@server works fine.

What else could be causing this error?

How does one go about debugging such an issue in gitlab? There does not seem to be anything relevant at all in ~gitlab/gitlab-shell/gitlab-shell.log. Where would a more informative error message be found?

Solution

I'm pretty sure you have a configuration problem between SSH and the system because of this SSH debug message :

client_input_channel_req: channel 0 rtype [email protected] reply 0

You receive this message immediately after a successful authentication and no message from bash which means no program was launched after login.

Watch in your passwd file if you have the right settings for the gitlab user :

gitlab:x:1011:1012:GitLab,,,:/path/to/gitlab:/bin/bash

Verify bash has no weird thing in configurations files such as

• bash.bashrc
• .profile
• .bashrc

Then go up to the uper level : Gitlab-shell Verify the /path/to/gitlab/.ssh/authorized_keys has the configuration below :

command="/path/to/gitlab/gitlab-shell/bin/gitlab-shell key-2",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa A...

with /path/to/gitlab/gitlab-shell/bin/gitlab-shell owned by the gitlab user and executable.

You can be sure gitlab-shell is fully operational by launching the command :

# /path/to/gitlab-shell/bin/gitlab-shell
Welcome to GitLab, Anonymous!

If the remote login is actually working and properly connected to the gitlab-shell, you should get the same welcome message (but matched to the user whose ssh key you used to login) before it dumps you out if you try to login remotely.

$ ssh gitlab@server
Welcome to GitLab, <your user's full name>!
Connection to <server> closed.

No message here probably indicates ssh isn't connecting you to gitlab at all.

Finally, checkout your gitlab-shell configuration (config.yml) and verify if :

http_settings:
    # trailing slash is important
    gitlab_url: "https://remote_server/"
    ca_file: /path/to/webserver/certificate.crt

and eventually :

    self_signed_cert: false

Comments

[DrGkill]

What's the output of : "ssh -vv gitlab@server" ??

[Caleb]

@DrGkill Nothing interesting (to my eyes), see for yourself.

[DrGkill]

See in your sshd_config if gitlab is allowed for external login. I would say 2 possible culprit : SSH or gitlab-shell. What does auth.log says when gitlab is connecting ?

[Caleb]

@DrGkill I've checked that (and if it wasn't allowed external login by sshd or pam the logs I showed you above would not have shown a successful authentication notice). SSH is connecting just fine. The server side logs show a successful authentication, that a session was opened, some systemd stuff about setting up the user env, then "received disconnect from <remote ip>" and some more stuff about cleaning up the session.

[Caleb]

Doh. <head-desk> You were on the right track all along. The gitlab user was being assigned /bin/false. Either the Arch package doesn't set the shell right when adding that user or I broke it somewhere in the process of implementing ldap authentication on this system. Thanks for all the effort.

[Huygens]

I had the same problem as Caleb, and I also found by myself that to make it work one need to set a shell instead of /bin/{false,true,nologin}. But actually the user is created without shell on purpose during installation (see: gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/…) sudo adduser --disabled-login --gecos 'GitLab' git. So it ought to work without a shell. I'm still looking for another solution.