How to make GitLab Runner in Docker see a custom CA Root certificate

Question

I have installed and configured:

1. an on-premises GitLab Omnibus on ServerA running on HTTPS
2. an on-premises GitLab-Runner installed as Docker Service in ServerB

ServerA certificate is generated by a custom CA Root

The Configuration

I've have put the CA Root Certificate on ServerB:

/srv/gitlab-runner/config/certs/ca.crt

Installed the Runner on ServerB as described in Run GitLab Runner in a container - Docker image installation and configuration:

docker run -d --name gitlab-runner --restart always \
           -v /srv/gitlab-runner/config:/etc/gitlab-runner \
           -v /var/run/docker.sock:/var/run/docker.sock \
           gitlab/gitlab-runner:latest

Registered the Runner as described in Registering Runners - One-line registration command:

docker run --rm -t -i 
            -v /srv/gitlab-runner/config:/etc/gitlab-runner 
           --name gitlab-docker-runner gitlab/gitlab-runner register \
           --non-interactive \
           --executor "docker" \
           --docker-image alpine:latest \
           --url "https://MY_PRIVATE_REPO_URL_HERE/" \
           --registration-token "MY_PRIVATE_TOKEN_HERE" \
           --description "MyDockerServer-Runner" \
           --tag-list "TAG_1,TAG_2,TAG_3" \
           --run-untagged \
           --locked="false"

This command gave the following output:

Updating CA certificates...
Runtime platform arch=amd64 os=linux pid=5 revision=cf91d5e1 version=11.4.2
Running in system-mode.

Registering runner... succeeded runner=8UtcUXCY
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

I checked with

$ docker exec -it gitlab-runner bash 

and once in the container with

$ awk -v cmd='openssl x509 -noout -subject' '
/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt

and the custom CA root is correctly there.

The Problem

When running Gitlab-Runner from GitLab-CI, the pipeline fails miserably telling me that:

$ git clone https://gitlab-ci-token:${CI_BUILD_TOKEN}@ServerA/foo/bar/My-Project.wiki.git

Cloning into 'My-Project.wiki'...

fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@ServerA/foo/bar/My-Project.wiki.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

ERROR: Job failed: exit code 1

It does not recognize the Issuer (my custom CA Root), but according to The self-signed certificates or custom Certification Authorities, point n.1, it should out-of-the-box:

Default: GitLab Runner reads system certificate store and verifies the GitLab server against the CA’s stored in system.

I've then tried the solution from point n.3, editing

/srv/gitlab-runner/config/config.toml:

and adding:

[[runners]]
tls-ca-file = "/srv/gitlab-runner/config/certs/ca.crt"

But it still doesn't work.

How can I make Gitlab Runner read the CA Root certificate?

Solution 1

While I've still not got why it doesn't work out-of-the-box, I've found the Egg of Columbus:

Gitlab-Runner configuration:

[[runners]]
  name = "MyDockerServer-Runner"
  url = "https://MY_PRIVATE_REPO_URL_HERE/"
  token = "MY_TOKEN_HERE"
  executor = "docker"
  ...
  [runners.docker]
    image = "ubuntu:latest"

  # The trick is the following:
    volumes = ["/cache","/srv/gitlab-runner/config:/etc/gitlab-runner"]
    ...

Gitlab-ci.yml pipeline:

MyJob:
    image: ubuntu:latest

    script:
      - awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt
      - git clone https://gitlab-ci-token:${CI_BUILD_TOKEN}@ServerA/foo/bar/My-Project.wiki.git
      - wget -O foo.png https://ServerA/foo/bar/foo.png 

    before_script:
      - apt-get update -y >/dev/null
      - apt-get install -y apt-utils dialog >/dev/null
      - apt-get install -y git >/dev/null
      - apt-get install -y wget >/dev/null

    # The trick is the following:
      - cp /etc/gitlab-runner/certs/ca.crt /usr/local/share/ca-certificates/ca.crt
      - update-ca-certificates

That's it:

• Mount the volume once (per Docker executor)
• Update the CA certificates once (per job)

And everything will work as expected: git clone, wget https, etc...

A great workaround, until someone at GitLab will fix it or explain me where I'm wrong (be my guest!)

Solution 2

You have two options:

Ignore SSL verification

Put this at the top of your .gitlab-ci.yml:

variables:
  GIT_SSL_NO_VERIFY: "1"

Point GitLab-Runner to the proper certificate

As outlined in the official documentation, you can use the tls-*-file options to setup your certificate, e.g.:

[[runners]]
  ...
  tls-ca-file = "/etc/gitlab-runner/ssl/ca-bundle.crt"
  [runners.docker]
  ...

As the documentation states, "this file will be read every time when runner tries to access the GitLab server."

Other options include tls-cert-file to define the certificate to be used if needed.

Solution 3

Not sure it's the best approach, but at least it worked for me. You can create a customized gitlab runner image and add your root CA inside:

├── Dockerfile
└── myca.crt

# Dockerfile
FROM gitlab/gitlab-runner:latest
COPY myca.crt /usr/local/share/ca-certificates
RUN update-ca-certificates

Build it:

docker build -t custom-gitlab-runner .

And rerun all your commands, just remember to use this new image name.

Off-topic, but related and might be useful

Dockerized gitlab-runner seem to also ignore entries in your /etc/hosts, so if you have launched Gitlab on a custom domain, e.g. https://gitlab.local.net, you need to pass the values from /etc/hosts when launching/registering gitlab runner:

docker run -d --name gitlab-runner --restart always \
       --add-host="gitlab.local.net:192.168.1.100" \
       ...

If you want to launch docker:dind (docker in docker service) container to build docker images, you also need to set these values inside /srv/gitlab-runner/config/config.toml:

[[runners]]
  url = "https://gitlab.local.net/"
  executor = "docker"
  pre_clone_script = "echo '192.168.1.100 gitlab.local.net registry.local.net' >> /etc/hosts"
  ...

Solution 4

From the output you provided i think that the certificate might be OK but you are lacking the CRL file : server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

The CRL file is used to verify that even if the certificate is valid is hasn't been revoked by the CA owner. You shoudl then need to :

1) Generate a CRL file based on your CA:

openssl ca -gencrl -keyfile ca.key -cert ca.crt -out crl.pem

source: https://blog.didierstevens.com/2013/05/08/howto-make-your-own-cert-and-revocation-list-with-openssl/

2) Instruct the runner to use it :

[[runners]]
  ...
  tls-ca-file = "/etc/gitlab-runner/ssl/ca-bundle.crt"
  crl-file = "/etc/gitlab-runner/ssl/ca.crl"

3) Of course setting GIT_SSL_NO_VERIFY will work but you will be more sensitive to man-in-the-middle attacks

View more solutions

Comments

[Philipp Ludwig]

@AndreaLigios check the logging output of the gitlab-runner process; maybe you could incorporate more information about your setup - which of the several methods of installing the runner did you chose, etc.

[Andrea Ligios]

You mean /srv/gitlab-runner/config/certs I guess. It's not working, in either way... this thing is driving me crazy. I can clearly see that the Gitlab-Runner installed as Docker Service is accepting the CA root and updating its certificates; logging it and checking, it's there. The problem is that the Runner spawned by GitLab-CI is not. My configuration is the default one for a containerized GitLab-Runner, if I make it print the CAs from gitlab-ci, the custom CA is not there :/

[Andrea Ligios]

I've re-edited the question adding all the details and the links to the docs. tls-ca-file from your answer should be in [[runners]] section, not in [runners.docker] section, but it doesn't work the same :(

[Philipp Ludwig]

@AndreaLigios Yeah, I now think the certificate is only for the communication between the runner and gitlab for the jobs, but not for the git checkout - I guess you will have to live with GIT_SSL_NO_VERIFY or find a way to tell git about your certificate.

[Andrea Ligios]

But, according to the docs you posted, it should!: Git cloning The runner injects missing certificates to build CA chain to build containers. This allows the git clone and artifacts to work with servers that do not use publicly trusted certificates. This approach is secure, but makes the runner a single point of trust. Help and feedback

[Andrea Ligios]

I suspect there's something missing in the documentation, which needs some tuning in the config.toml (eg what value should tls_verify have, what priviledge_mode, etc...), tls-ca-file alone IMHO isn't enough. And that without mentioning that according to that page, everything should work without doing anything :/

[Philipp Ludwig]

Yes, that's how I understood it as well. Maybe it's a bug in gitlab-runner.

[Andrea Ligios]

Thank you for your answer, I'll try it ASAP (though I'm not convinced yet CRL is mandatory). As said in the other answer, according to the docs, tls-ca-file should stay in [[runners]] not in [runners.docker] :)

[Andrea Ligios]

Also, why DER format for output? This says DER is not supported for certificates, so I guess it's not also for CRL: docs.gitlab.com/runner/configuration/tls-self-signed.html

[webofmars]

@AndreaLigios I cut and pasted your config above. If the runners.docker section is not the right place to put the tls-ca-file please feel free to edit the question to avoid subsequent mistakes :-) I edited the answer accordingly

[webofmars]

@AndreaLigios : you are right about DER format. I edited the answer accordingly

[Andrea Ligios]

You copypasted the other answer, not my question. According to the docs it shoud be [[runners]] ... tls-ca-file = "" ... [docker.runners]. Check it out :)

[webofmars]

You are right @AndreaLigios my bad i edited it anyway.

[Andrea Ligios]

Even if your solution didn't work, I've awarded your answer because you've spent some time on it and I hate wasting stuff. Enjoy.

[Andrea Ligios]

Thank you for your valuable answer! I'll try it ASAP

[Newteq Developer]

I've followed the parts where you say "the trick is the following" and it seems to work perfectly now! :) I'm just wondering, is it really necessary to specify the volumes too? What is the purpose of it and what does that portion actually do?

[Andrea Ligios]

You are running a pipeline with a Docker Runner, which runs within a Docker container. To copy the certificate file under /usr/local/share/ca-certificates, you need to make the certificate file available in the container. To do that, you map a local path (/srv/gitlab-runner/config in my case) to a container path (/etc/gitlab-runner), so that when you do cp /etc/gitlab-runner/certs/ca.crt ... inside the container, you're under the hood retrieving the file from the host path /srv/gitlab-runner/config/certs. Without mounting the volume, the cp would not find anything to copy

[Andrea Ligios]

@NewteqDeveloper, this is a small article I've written which covers this Docker concepts in an easy way: baeldung.com/docker-compose .

[Pedro Pombeiro]

@AndreaLigios not sure why this solves your issue, it looks like the Runner image is already doing the same in the init script. The only difference seems to be that you're not passing the --fresh flag to update-ca-certificates. Any ideas?

[Pedro Pombeiro]

I should specify that it's the Runner Helper image which is doing that, whereas what you're doing is running the same logic in the Runner container (not the Helper one). Normally it's the helper image which takes care of doing common operations such as Git fetches, etc. So yeah, if you need to do the same sort of operations (or wget) in your build script, then you'd need to mount/install your custom CA.

[paoloyx]

@PedroPombeiro hi, same exact situation in my homelab setup: * Gitlab 12.9.3 on ServerA with self-signed certificate from a custom CA root * Gitlab-runner 12.9.0 on ServerB with CA root copied to /srv/gitlab-runner/config/certs/ca.crt. I'm using a vanilla Autodevops pipeline, in a unmodified project generated using Spring Template, and keeps me failing with Error response from daemon: Get https://gitlab.[domain]:4567/v2/:x509: certificate signed by unknown authority I can docker login to that address with Gitlab credentials from ServerB.

[Pedro Pombeiro]

@paoloyx I've created a merge request in GitLab Runner to improve the documentation in regards to self-signed certs, please let me know if that helps.