Getting GitLab CI to clone private repositories

Question

I have GitLab & GitLab CI set up to host and test some of my private repos. For my composer modules under this system, I have Satis set up to resolve my private packages.

Obviously these private packages require an ssh key to clone them, and I have this working in the terminal - I can run composer install and get these packages, so long as I have the key added with ssh-add in the shell.

However, when running my tests in GitLab CI, if a project has any of these dependencies the tests will not complete as my GitLab instance needs authentication to get the deps (obviously), and the test fails saying Host key verification failed.

My question is how do I set this up so that when the runner runs the test it can authenticate to gitlab without a password? I have tried putting a password-less ssh-key in my runners ~/.ssh folder, however the build wont even add the key, "eval ssh-agent -s" followed by ssh-add seems to fail saying the agent isn't running...

Solution 1

I'm posting this as an answer since others weren't completely clear and/or detailed IMHO

Starting from GitLab 8.12+, assuming the submodule repo is in the same server as the one requesting it, you can now:

1. Set up the repo with git submodules as usual (git submodule add git@somewhere:folder/mysubmodule.git)

2. Modify your .gitmodules file as follows

    [submodule "mysubmodule"]
      path = mysubmodule
      url = ../../group/mysubmodule.git
   

   where ../../group/mysubmodule.git is a relative path from your repository to the submodule's one.

3. Add the following lines to gitlab-ci.yml

    variables:
      GIT_SUBMODULE_STRATEGY: recursive
   

   to instruct the runner to fetch all submodules before the build.

Caveat: if your runner seems to ignore the GIT_SUBMODULE_STRATEGY directive, you should probably consider updating it.

(source: https://docs.gitlab.com/ce/ci/git_submodules.html)

Solution 2

See also other solutions:

• git submodule permission (see Marco A.'s answer)
• job token and override repo in git config (see a544jh's answer)

---

Here a full howto with SSH keys:

General Design

• generating a pair of SSH keys
• adding the private one as a secure environment variable of your project
• making the private one available to your test scripts on GitLab-CI
• adding the public one as a deploy key on each of your private dependencies

Generating a pair of public and private SSH keys

Generate a pair of public and private SSH keys without passphrase:

ssh-keygen -b 4096 -C "<name of your project>" -N "" -f /tmp/name_of_your_project.key

Adding the private SSH key to your project

You need to add the key as a secure environment variable to your project as following:

• browse https://<gitlab_host>/<group>/<project_name>/variables
• click on "Add a variable"
• fill the text field Key with SSH_PRIVATE_KEY
• fill the text field Value with the private SSH key itself
• click on "Save changes"

Exposing the private SSH key to your test scripts

In order to make your private key available to your test scripts you need to add the following to your .gitlab-ci.yml file:

before_script:
  # install ssh-agent
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
  # run ssh-agent
  - eval $(ssh-agent -s)
  # add ssh key stored in SSH_PRIVATE_KEY variable to the agent store
  - ssh-add <(echo "$SSH_PRIVATE_KEY")
  # disable host key checking (NOTE: makes you susceptible to man-in-the-middle attacks)
  # WARNING: use only in docker container, if you use it with shell you will overwrite your user's ssh config
  - mkdir -p ~/.ssh
  - echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config

Code Snippet comes from GitLab documentation

Adding the public SSH key as a deploy key to all your private dependencies

You need to register the public SSH key as deploy key to all your private dependencies as following:

• browse https://<gitlab_host>/<group>/<dependency_name>/deploy_keys
• click on "New deploy key"
• fill the text field Title with the name of your project
• fill the text field Key with the public SSH key itself
• click on "Create deploy key"

Solution 3

If you don't want to fiddle around with ssh keys or submodules, you can override the repo in git's configuration to authenticate with the job token instead (in gitlab-ci.yml):

before_script:
  - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.example.com/group/repo.git".insteadOf [email protected]:group/repo.git

Solution 4

The currently accepted answer embeds Gitlab-specific requirements into my .gitmodules file. This forces a specific directory layout for local development and would complicate moving to another version control platform.

Instead, I followed the advice in Juddling's answer. Here's a more complete answer.

My .gitmodules files has the following contents:

[submodule "myproject"]
    url = [email protected]:mygroup/myproject.git

In my gitlab-ci.yml I have the following:

build:
  stage: build
  before_script:
    - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@git.myhost.com/".insteadOf "[email protected]:"
    - git submodule sync && git submodule update --init

The trailing / and : are critical in the git config line, since we are mapping from SSH authentication to HTTPS. This tripped me up for a while with "Illegal port number" errors.

I like this solution because it embeds the Gitlab-specific requirements in a Gitlab-specific file, which is ignored by everything else.

Solution 5

I used deploy tokens to solve this issue, as setting up SSH keys for a test runner seems a little long winded.

git clone http://<username>:<deploy_token>@gitlab.example.com/tanuki/awesome_project.git

The deploy tokens are per project and are read only.

View more solutions