How to run podman from inside a container?

Question

I want to run podman as a container to run CI/CD pipelines. However, I keep getting this error from the podman container:

$ podman info
ERRO[0000] 'overlay' is not supported over overlayfs
Error: could not get runtime: 'overlay' is not supported over overlayfs: backing file system is unsupported for this graph driver

I am using the Jenkins Kubernetes plugin to write CI/CD pipelines that run as containers within a Kubernetes cluster. I've been successful at writing pipelines that use a Docker-in-Docker container to run docker build and docker push commands.

However, running a Docker client and a Docker Daemon inside a container makes the CI/CD environment very bloated, hard to configure, and just not ideal to work with. So I figured I could use podman to build Docker images from Dockerfiles without using a fat Docker daemon.

The problem is that podman is so new that I have not seen anyone attempt this before, nor I am enough of a podman expert to properly execute this.

So, using the podman installation instructions for Ubuntu I created the following Dockerfile:

FROM ubuntu:16.04

RUN apt-get update -qq \
    && apt-get install -qq -y software-properties-common uidmap \
    && add-apt-repository -y ppa:projectatomic/ppa \
    && apt-get update -qq \
    && apt-get -qq -y install podman

# To keep it running
CMD tail -f /dev/null

So I built the image and ran it as follows:

# Build
docker build -t podman:ubuntu-16.04 .

# Run
docker run --name podman -d podman:ubuntu-16.04

Then when running this command on the running container, I get an error:

$ docker exec -ti podman bash -c "podman info"

ERRO[0000] 'overlay' is not supported over overlayfs
Error: could not get runtime: 'overlay' is not supported over overlayfs: backing file system is unsupported for this graph driver

I install podman on an Ubuntu 16.04 machine I had and ran the same podman info command I got the expected results:

host:
  BuildahVersion: 1.8-dev
  Conmon:
    package: 'conmon: /usr/libexec/crio/conmon'
    path: /usr/libexec/crio/conmon
    version: 'conmon version , commit: '
  Distribution:
    distribution: ubuntu
    version: "16.04"
  MemFree: 2275770368
  MemTotal: 4142137344
  OCIRuntime:
    package: 'cri-o-runc: /usr/lib/cri-o-runc/sbin/runc'
    path: /usr/lib/cri-o-runc/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 2146758656
  SwapTotal: 2146758656
  arch: amd64
  cpus: 2
  hostname: jumpbox-4b3620b3
  kernel: 4.4.0-141-generic
  os: linux
  rootless: false
  uptime: 222h 46m 33.48s (Approximately 9.25 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 15
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Does anyone know how I can fix this error and get podman working from a container?

Solution 1

Your Dockerfile should install iptables as well:

FROM ubuntu:16.04

RUN apt-get update -qq \
    && apt-get install -qq -y software-properties-common uidmap \
    && add-apt-repository -y ppa:projectatomic/ppa \
    && apt-get update -qq \
    && apt-get -qq -y install podman \
    && apt-get install -y iptables

# To keep it running
CMD tail -f /dev/null

Then run the command with:

docker run -ti --rm podman:test bash -c "podman --storage-driver=vfs info"

This should give you the response you expect.

Solution 2

The suggestion from mihai succeeds for info but as soon as I try, for example, run --rm docker.io/library/hello-world I get an error:

error creating network namespace for container …: mount --make-rshared /var/run/netns failed: "operation not permitted"
failed to mount shm tmpfs "/var/lib/containers/storage/vfs-containers/…/userdata/shm": operation not permitted

I only managed to solve this by setting a non-root user for the image and then running the container in privileged mode, which defeats the purpose of the exercise since DinD could already do this:

FROM ubuntu:18.04

RUN apt-get update -qq \
    && apt-get install -qq -y software-properties-common uidmap \
    && add-apt-repository -y ppa:projectatomic/ppa \
    && apt-get update -qq \
    && apt-get -qq -y install podman \
    && apt-get install -y iptables

RUN adduser --disabled-login --gecos test test

USER test

ENTRYPOINT ["podman", "--storage-driver=vfs"]
CMD ["info"]

used as

docker build -t podman:test .
docker run --rm --privileged podman:test run --rm docker.io/library/hello-world

Solution 3

I tried this myself with a more permissive config (--privileged=true), with storage volumes mounted from the host and also with iptables installed in the container and was able to run it (i.e sudo apt-get install iptables).

$ podman run -it --rm -v /var/run/containers/storage:/var/run/containers/storage -v /var/lib/containers/storage:/var/lib/containers/storage --storage-driver=overlay --privileged=true  mine bash
root@e275668d7c36:/# apt-get install -y -qq iptables
...
root@e275668d7c36:/# podman info
host:
  BuildahVersion: 1.8-dev
  Conmon:
    package: 'conmon: /usr/libexec/crio/conmon'
    path: /usr/libexec/crio/conmon
    version: 'conmon version , commit: '
  Distribution:
    distribution: ubuntu
    version: "16.04"
  MemFree: 71659520
  MemTotal: 482099200
  OCIRuntime:
    package: 'cri-o-runc: /usr/lib/cri-o-runc/sbin/runc'
    path: /usr/lib/cri-o-runc/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 2
  hostname: e275668d7c36
  kernel: 4.15.0-1035-aws
  os: linux
  rootless: false
  uptime: 315h 17m 53s (Approximately 13.12 days)
insecure registries:
  registries: []
registries:
  registries: []
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 2
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 4
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

If you'd like to use docker you can use the --privileged flag too.

Keep in mind that there are other tools specifically designed to build containers and some of them without privileged mode:

• Kaniko
• img
• Buildkit
• Buildah (Companion to Podman)
• Bazel (With it's container build module)
• Knative container build templates

Comments

[Vasili Angapov]

It also should be noted that storage driver VFS has significantly lower performance and eats more space. More here: docs.docker.com/storage/storagedriver/vfs-driver

[Mihai]

True but u would rather have a slower pipeline than running it as root

[Eldad Assis]

Following this example on Mac produces an error with the output of podman info: ERRO[0000] unable to write system event: "write unixgram @00006->/run/systemd/journal/socket: sendmsg: no such file or directory". When I actually try to run a container podman --storage-driver=vfs run docker.io/hello-world I get more errors.

[el-davo]

@EldadAssis I had the same issue. Adding this flag seems to fix it. --cgroup-manager=cgroupfs. It even seems to work without --privileged

[Thomas Suedbroecker]

It does't work for me ... where can I find a simple example to "run podman from inside a container" ?