docker build with nvidia runtime

Question

I have a GPU application that does unit-testing during the image building stage. With Docker 19.03, one can specify nvidia runtime with docker run --gpus all but I also need access to the gpus for docker build because I do unit-testing. How can I achieve this goal?

For older version of docker that use nvidia-docker2 it was not possible to specifiy runtime during build stage, BUT you can set the default runtime to be nvidia, and docker build works fine that way. Can I do that in Docker 19.03 that doesn't need nvidia-docker anymore? If so, how?

Solution 1

You need use nvidia-container-runtime as explained in docs: "It is also the only way to have GPU access during docker build".

Steps for Ubuntu:

1. Install nvidia-container-runtime:

   sudo apt-get install nvidia-container-runtime

2. Edit/create the /etc/docker/daemon.json with content:

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
         } 
    },
    "default-runtime": "nvidia" 
}

3. Restart docker daemon:

   sudo systemctl restart docker

4. Build your image (now GPU available during build):

   docker build -t my_image_name:latest .

Solution 2

A "solution" I found is to first run a base image with the host nvidia drivers mounted on it

docker run -it --rm --gpus ubuntu

And then build my app within the container manually and commit the resulting image. This is not ideal and it would be best to have access to nvidia-smi during the build phase.

Comments

[shortcipher3]

This response seems to be relevant to docker run not docker build

[Anton Ganichev]

This is due to a syntax error (extra comma in 6-th line). Fixed now.

[RunOrVeith]

FYI if you need this because you want to compile custom kernels with pytorch, you can use the nvidia development base image (e.g. nvidia/cuda:11.0-cudnn8-devel-ubuntu18.04) and set ENV TORCH_CUDA_ARCH_LIST=Turing , then you can also build them without having a GPU available during build

[juanpa.arrivillaga]

So, literally the only way to get build to use the nvidia runtime (or I guess, any other runtime) is to set it as the default? Geez. How is this not an option?