Docker in Docker for gitlab: Client sent an http request to https server / failed to remove network

Question

I am trying ta

I am trying to get a docker in docker configuration for my gitlab instance running but I just can’t get it working.

Here is what I want to do:

1. Start a „docker in docker“ image
2. Start a gitlab runner in another docker image
3. Use docker in docker from gitlab CI.

All of that is running under Ubuntu 18.04. Here are the commands

1. Create a Network

   sudo docker network create gitlab-runner-net

To start docker in docker:

sudo docker run --privileged --name gitlab-dind -d \
    --network gitlab-runner-net  --network-alias gitlab-runner-net  \
    -e DOCKER_TLS_CERTDIR=/certs \
    -v docker-certs-ca:/certs/ca \
    -v docker-certs-client:/certs/client \
     -v /var/lib/docker \
     docker:19.03.13-dind   --storage-driver=overlay2 

And the for the runner

  sudo docker run -d  --name gitlab-runner  --restart always  --network gitlab-runner-net  -v /srv/gitlab-runner/config.toml:/etc/gitlab-runner/config.toml  -e DOCKER_TLS_CERTDIR=/certs -v docker-certs-client:/certs/client:ro -e DOCKER_HOST=tcp://gitlab-dind:2376  gitlab/gitlab-runner:alpine

And here is the config.toml

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlab-did"
  url = „cleaned“
  token = „cleaned
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    host = "tcp://gitlab-dind:2376"
    tls_verify = false
    image = "docker:19.03.13"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/certs"]
    shm_size = 0

The containers spin up fine and the gitlab runner registers. But then I use the following .gitlab-ci.yml

image: docker:19.03.12 services:

• docker:19.03.12-dind

before_script:

• docker info

build: stage: build script: - docker build -t my-docker-image . - docker run my-docker-image /script/to/run/tests

And the result is

> Running with gitlab-runner 13.4.1 (e95f89a0)   on gitlab-did FPGoD8Ms
> Preparing the "docker" executor 00:09 ERROR: Failed to remove network
> for build ERROR: Preparation failed: Error response from daemon:
> Client sent an HTTP request to an HTTPS server. (docker.go:985:0s)
> Will be retried in 3s ... ERROR: Failed to remove network for build
> ERROR: Preparation failed: Error response from daemon: Client sent an
> HTTP request to an HTTPS server. (docker.go:985:0s) Will be retried in
> 3s ... ERROR: Failed to remove network for build ERROR: Preparation
> failed: Error response from daemon: Client sent an HTTP request to an
> HTTPS server. (docker.go:985:0s) Will be retried in 3s ... ERROR: Job
> failed (system failure): Error response from daemon: Client sent an
> HTTP request to an HTTPS server. (docker.go:985:0s)

I am trying to fix this for days now. I played around with so many settings that I just lost overview.

Does anybody please have a suggestion?

Solution 1

I had a similar use case (Jenkins CI), and ran into the same problem. I was able to work around it by not using docker in docker at all. Instead, I mounted /var/run/docker.sock into the docker container (i.e -v /var/run/docker.sock:/var/run/docker.sock). The URL for the docker daemon becomes unix:///var/run/docker.sock.

This suggestion came from https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/ which is referred to directly from the official docker in docker image on dockerhub.

The only thing I had to alter on the container was to make sure my user was part of the docker group and that the docker group had the same gid on both the host and container.

Solution 2

Disable TLS by setting env var DOCKER_TLS_CERTDIR=""

Source: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4501

Comments

[Doug]

This doesn't fix the real issue and reduces security.