Gitlab CI error during connect: Get http://docker:2375/v1.40/containers/json?all=1: dial tcp: lookup docker on xx.xx.xx.x:xx: no such host

Question

I've recently created a new repository in gitlab.com, started a new Ubuntu instance in digitalocean, installed docker and gitlab-runner in the ubuntu instance. Also, did gitlab-runner register and passed the correct host and token from gitlab.com ci/cd settings. Pushed a branch, and the expected pipeline job runs but fails, presenting the error:

error during connect: Get http://docker:2375/v1.40/containers/json?all=1: dial tcp: lookup docker on 67.207.67.2:53: no such host

Just followed the basic steps and expected it to work perfectly. By looking for the error when google search, there's no info.

The config.toml file, which is created automatically on gitlab-runner register:

concurrent = 1
check_interval = 0

[[runners]]
  name = "digital ocean runner"
  url = "https://gitlab.com/"
  token = "xxxxxxxxxxxxxxxxx"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "ruby:2.1"
    privileged = true
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]

The gitlab ci file:

image: docker:latest
services:
  - docker:dind
stages:
  - test

test-build:
  stage: test
  script:
    - echo "Fine!"
    - docker info
  tags:
    - docker

Steps to reproduce

open a gitlab.com account
Install Ubuntu latest
Install gitlab-runner
do `register` and should have the config.toml above

What causes this issue?

The error log for one one of the tests, not for the job above:

Running with gitlab-runner 12.3.0 (a8a019e0)
  on foobar laptop szWcjfZg
Using Docker executor with image foobar/docker-bash ...
Starting service docker:dind ...
Authenticating with credentials from /Users/foobar/.docker/config.json
Pulling docker image docker:dind ...
Using docker image sha256:5768e15eefd175c1ba6969b616cfe827152556c5fe691b9258cb57d1a5c37e9d for docker:dind ...
Waiting for services to be up and running...

*** WARNING: Service runner-szWcjfZg-project-14670943-concurrent-0-docker-0 probably didn't start properly.

Health check error:
service "runner-szWcjfZg-project-14670943-concurrent-0-docker-0-wait-for-service" timeout

Health check container logs:


Service container logs:
2019-10-05T23:18:52.128774700Z Generating RSA private key, 4196 bit long modulus (2 primes)
2019-10-05T23:18:53.209639200Z .......................................................................................................................................................................................++++
2019-10-05T23:18:53.694383300Z .......................................................................................++++
2019-10-05T23:18:53.694784400Z e is 65537 (0x010001)
2019-10-05T23:18:53.710661300Z Generating RSA private key, 4196 bit long modulus (2 primes)
2019-10-05T23:18:53.789938500Z ............++++
2019-10-05T23:18:54.926850200Z ..............................................................................................................................................................................................................++++
2019-10-05T23:18:54.927064600Z e is 65537 (0x010001)
2019-10-05T23:18:54.953296700Z Signature ok
2019-10-05T23:18:54.953354500Z subject=CN = docker:dind server
2019-10-05T23:18:54.953422500Z Getting CA Private Key
2019-10-05T23:18:54.967240700Z /certs/server/cert.pem: OK
2019-10-05T23:18:54.970126300Z Generating RSA private key, 4196 bit long modulus (2 primes)
2019-10-05T23:18:55.244959900Z .................................................++++
2019-10-05T23:18:55.317443900Z ...........++++
2019-10-05T23:18:55.317858100Z e is 65537 (0x010001)
2019-10-05T23:18:55.339564700Z Signature ok
2019-10-05T23:18:55.339581700Z subject=CN = docker:dind client
2019-10-05T23:18:55.339671300Z Getting CA Private Key
2019-10-05T23:18:55.356469400Z /certs/client/cert.pem: OK
2019-10-05T23:18:55.359632000Z mount: permission denied (are you root?)
2019-10-05T23:18:55.359866500Z Could not mount /sys/kernel/security.
2019-10-05T23:18:55.359910700Z AppArmor detection and --privileged mode might break.
2019-10-05T23:18:55.361037300Z mount: permission denied (are you root?)

*********

Authenticating with credentials from /Users/foobar/.docker/config.json
Pulling docker image foobar/docker-bash ...
Using docker image sha256:2ecf555077c9d0a7df0cc45188094a84fbbae4ce4c28cac37a652e9765bf3157 for foobar/docker-bash ...
Authenticating with credentials from /Users/foobar/.docker/config.json
Running on runner-szWcjfZg-project-14670943-concurrent-0 via Foobar-MacBook-Pr.local...
Authenticating with credentials from /Users/foobar/.docker/config.json
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/foobar/foobar-project/.git/
Checking out 4f9789a7 as review-ci-test...

Skipping Git submodules setup
Authenticating with credentials from /Users/foobar/.docker/config.json
Authenticating with credentials from /Users/foobar/.docker/config.json
Authenticating with credentials from /Users/foobar/.docker/config.json
$ echo "INFO - Clear exited containers" # collapsed multi-line command
INFO - Clear exited containers
INFO - List all services running...
error during connect: Get http://docker:2375/v1.40/containers/json?all=1: dial tcp: lookup docker on 192.168.65.1:53: no such host
Authenticating with credentials from /Users/foobar/.docker/config.json
ERROR: Job failed: exit code 1

Solution 1

I might have skipped the optional installation of docker in my Ubuntu instance, so if the gitlab runner is going to use Docker, remember to install it:

curl -sSL https://get.docker.com/ | sh

I got back to this issue and seem to have partially fixed it by checking it the gitlab runner is running

gitlab-runner status

systemctl is-enabled gitlab-runner

systemctl is-enabled docker

I do have a different issue to solve, but it doesn't seem to be related. That is:

error during connect: Post http://docker:2375/v1.40/images/ [...] dial tcp: lookup docker on xx.xxx.xx.x:xx: no such host

Then I found the solution, there's a problem in Gitlab CI that is reported here ( https://gitlab.com/gitlab-org/gitlab-runner/issues/4566#note_199261985 ); And there are a lot of ways to fix this but my solution follows:

1) Set the .gitlab-ci.yml services to use an older dind version:

services:
  - docker:18.09.7-dind

2) Check the toml file /etc/gitlab-runner/config.toml, set the DOCKER_HOST just in case, to point to the right place, make sure it runs in privileged mode. My working version is:

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "xxxxxx xxxxxxxx"
  url = "https://gitlab.com/"
  token = "xxxxxxxxxx"
  executor = "docker"
  pre_build_script = "export DOCKER_HOST=tcp://docker:2375"
  [runners.custom_build_dir]
  [runners.docker]
    tls_cert_path = ""
    tls_verify = false
    image = "alpine:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

Hope this helps someone else in the future!

Solution 2

I want to share the solution!

just add image = docker:stable and privileged = true

and that's all! enjoy!

Comments

[Admin]

Not sure what's the point of downvoting and not contributing. The issue above is a known issue and gitlab ci haven't provided a fix yet. Thank you so much for the down vote, that's the way to build a great community.

[danday74]

thanks so much - u r a genius sir :) I just had to use docker:18.09.7-dind and privileged = true

[danday74]

I worked out how to use docker:dind (not an older version) - my answer is on SO here - stackoverflow.com/a/65855484/1205871

[Phil D.]

I'm not entirely sure what the significance of step #2 is (time for some reading) but it worked for me. Legend!

[Erica Xu]

privileged = true was all it took more than the GitLab documentation said. Thank you :-)