GitLab - Does not accept pushes via SSH [asks for SSH password], works with HTTP?

Question

I have looked at multiple solutions for this problem, including: https://serverfault.com/questions/515395/gitlab-unable-to-push-via-ssh Gitlab not working with SSH Keys http://community.bitnami.com/t/unable-to-make-a-ssh-connection-to-gitlab-but-http-push-and-clone-are-working/14939

---

Okay, so...I just installed GitLab today. I followed this manual. When I got to creating the user, "git" was in use so I decided to use "gitlab". If this was a bad decision, I can go back and try to see what is using "git" and why and if I can stop that and give it to GitLab.

Basically, when I push GitLab, I'm trying to use the SSH address: [email protected]:user/repo. When I use "git push origin master" (with origin pointing at the aforementioned address), I am asked for the user "gitlab"'s password to the SSH server. If I enter it, GitLab then tells me that "user/repo" doesn't appear to be a repository and then it exits.

The output of "ssh -vvvT" is:

C:\Users\Jake>ssh -vvvT [email protected]
OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007
debug2: ssh_connect: needpriv 0
debug1: Connecting to git.redacted.com [redacted] port 22.
debug1: Connection established.
debug1: identity file /c/Users/Jake/.ssh/identity type -1
debug3: Not a RSA1 key file /c/Users/Jake/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'Proc-Type:'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'DEK-Info:'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /c/Users/Jake/.ssh/id_rsa type 1
debug3: Not a RSA1 key file /c/Users/Jake/.ssh/id_dsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'Proc-Type:'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'DEK-Info:'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /c/Users/Jake/.ssh/id_dsa type 2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubu
ntu-2ubuntu2
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.6
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-g
roup-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour1
28,arcfour256,arcfour,aes192-cbc,aes256-cbc,[email protected],aes128-c
tr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour1
28,arcfour256,arcfour,aes192-cbc,aes256-cbc,[email protected],aes128-c
tr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@open
ssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@open
ssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: [email protected],ecdh-sha2-nistp256,ecdh-
sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hel
lman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour12
8,[email protected],[email protected],[email protected],ae
s128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndae
[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour12
8,[email protected],[email protected],[email protected],ae
s128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndae
[email protected]
debug2: kex_parse_kexinit: [email protected],[email protected],um
[email protected],[email protected],[email protected],hma
[email protected],[email protected],hmac-sha1-96-etm@opens
sh.com,[email protected],hmac-md5,hmac-sha1,[email protected],umac-1
[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh
.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: [email protected],[email protected],um
[email protected],[email protected],[email protected],hma
[email protected],[email protected],hmac-sha1-96-etm@opens
sh.com,[email protected],hmac-md5,hmac-sha1,[email protected],umac-1
[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh
.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_init: found hmac-md5
debug1: kex: server->client aes128-cbc hmac-md5 none
debug2: mac_init: found hmac-md5
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 119/256
debug2: bits set: 524/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: check_host_in_hostfile: filename /c/Users/Jake/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug3: check_host_in_hostfile: filename /c/Users/Jake/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug1: Host 'git.majornoob.com' is known and matches the RSA host key.
debug1: Found key in /c/Users/Jake/.ssh/known_hosts:1
debug2: bits set: 515/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /c/Users/Jake/.ssh/identity (0x0)
debug2: key: /c/Users/Jake/.ssh/id_rsa (0xa01b360)
debug2: key: /c/Users/Jake/.ssh/id_dsa (0xa01b378)
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /c/Users/Jake/.ssh/identity
debug3: no such identity: /c/Users/Jake/.ssh/identity
debug1: Offering public key: /c/Users/Jake/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: /c/Users/Jake/.ssh/id_dsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
[email protected]'s password:

C:\Users\Jake>

This appears to have trouble reading my SSH key(s)? I'm not sure what is happening.

I have tried two different SSH keys with two different encryption types using the following commands:

ssh-keygen -t rsa -C "[email protected]"
ssh-keygen -t dsa -C "[email protected]"

Here is the output if I attempt to push using SSH protocol:

C:\Users\Jake\Documents\Development\test>git push origin master
[email protected]'s password:
fatal: 'jake/test.git' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

C:\Users\Jake\Documents\Development\test>

Why is that even asking for the SSH password? I want it to use keys.

The keys are on the GitLab installation, here is some proof of that:

I'm not sure where to go from here or what other information I can provide to make diagnosing my issue easier. Please help me out! I will oblige to any question you ask that may make fixing this easier on you.

Thank you in advance!

---

edit: When I use "ssh -vvvT [email protected]" and I actually enter the password for the user, it shows me the Ubuntu MOTD. I'm fairly 200% sure this shouldn't happen.

Solution 1

You are right. the MOTD should not be displayed to any user. Gitlab has its own built-in shell that handles users, but that shell is only run when a user logs in with their private key, which isn't working for you. Because logging in as git with a password brings you to a prompt, you should delete the password for git:

sudo passwd -d git

And then disable logging in with bash by running sudo vim /etc/passwd, find the git user, and at the end where it says /bin/bash, replace it with /bin/nologin.

As for the keys, make sure your client computer can use keys. Try sshing to localhost. It should ask for a password. Log out, add the client's ssh key to itself with:

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys #This should not require root

Then try logging in to localhost again. If it doesn't ask for a password, you know your client is set up correctly.

If your client is fine, try debugging the server. BEFORE you do this, delete your keys from the screenshot. Then, make sure Gitlab has full access to the /home/git/.ssh directory by running sudo chmod -R git /home/git/.ssh. Re-add your keys and then run sudo cat /home/git/.ssh/authorized_keys and see if your keys are there.

If this doesn't work, check out your gitlab logs by running tailf /home/git/gitlab/{LOG}.log and then replace {LOG} with unicorn, application, production, or githost, then try adding your keys.

REMEMBER to delete your keys before doing anything. Keys are stored in the database and are supposed to be stored in /home/git/.ssh/authorized_keys. If they are in the database, but not in the file, you could have problems.

Solution 2

My situation

Server: Ubuntu 14.04.1 Updating to last package versions (sudo apt-get update && sudo apt-get dist-upgrade). Gitlab: 6.9.0 (gitlab version doesn't matter really).

All worked perfectly before upgrading and rebooting the server, with correct ssh keys and all the related stuff. But after upgrading packages, all users where prevented for a normal operation, but asking for the password of the user "git".

What have I done?

After several hours, I did find what really happened: new permission checkings on /home/git directory when running git commands.

Steps to check if it serves for your case:

1. tail -f /var/log/auth.log
2. Try to do a git operation (like git remote update).
3. Check log file. A line like this appeared: Authentication refused: bad ownership or modes for directory /home/git.
4. Ensure git propietary of /home/git: sudo chown git:git /home/git.
5. Ensure directory permissions: sudo chmod 755 /home/git.
6. Place into git repository and do git remote update without any problem.

That worked for me!!!

Comments

[simpleauthority]

Thank you for your reply!! I reinstalled GitLab on a Debian 7 server instead of Ubuntu 14.04. Same little problem... I've removed git's password, and set it's bash to nologin. I'm not sure what you mean by "ssh to localhost"?

[Alex Berry]

he means literally, at a bash prompt, ssh localhost, or from another computer ssh my_hostname

[simpleauthority]

Ok....thank you both. I'm not sure what the hell I did/didn't do, but I switched it to nologin and in desperation, switched it back to /bin/bash. Now it works. Erm..So, I really have no clue what did or didn't work. I will accept this answer anyway.