Setting build args in docker-compose.yml using env_file

Question

I'm trying to use Docker and Docker Compose to create a containerized app. I have a PubNub account, which allows me to use different API keys for different environments (dev, test, prod). To help me build images for this, I am trying to use build args set with an env_file.

It's not working.

WARNING: The PUB_KEY variable is not set. Defaulting to a blank string.
WARNING: The SUB_KEY variable is not set. Defaulting to a blank string.

Questions:

• What mistake am I making in setting the build args?
• How do I fix it?
• Is this a good way to set ENV variables for the containers scan and flask?

At the very bottom is an IntelliJ IDE screenshot, or the text code is just below.

Here is the docker-compose.yml content:

version: '3.6'

services:

  scan:
    env_file:
      - sample.env
    build:
      context: .
      dockerfile: Dockerfile
      args:
        pub_key: $PUB_KEY
        sub_key: $SUB_KEY
      target: scan
    image: bt-beacon/scan:v1

  flask:
    env_file:
      - sample.env
    build:
      context: .
      dockerfile: Dockerfile
      args:
        pub_key: $PUB_KEY
        sub_key: $SUB_KEY
      target: flask
    image: bt-beacon/flask:v1
    ports:
      - "5000:5000"

And the Dockerfile:

# --- BASE NODE ---
FROM python:3.6-jessie as base
ARG pub_key
ARG sub_key

RUN test -n "$pub_key"
RUN test -n "$sub_key"

# --- SCAN NODE ---
FROM base as scan

ENV PUB_KEY=$pub_key
ENV SUB_KEY=$sub_key

COPY app/requirements.scan.txt /

RUN apt-get update
RUN apt-get -y install bluetooth bluez bluez-hcidump python-bluez python-numpy python3-dev libbluetooth-dev libcap2-bin
RUN pip install -r /requirements.scan.txt
RUN setcap 'cap_net_raw,cap_net_admin+eip' $(readlink -f $(which python))

COPY app/src /app
WORKDIR /app

CMD ["./scan.py", "$pub_key", "$sub_key"]


# -- FLASK APP ---
FROM base as flask

ENV SUB_KEY=$sub_key

COPY app/requirements.flask.txt /
COPY app/src /app

RUN pip install -r /requirements.flask.txt

WORKDIR /app

EXPOSE 5000

CMD ["flask", "run"]

Finally, sample.env:

# PubNub app keys here
PUB_KEY=xyz1
SUB_KEY=xyz2

Solution

env_file can only set environment variables inside a service container. Variables from env_file cannot be injected into docker-compose.yml itself.

You have such options (described there in detail):

• inject these variables into the shell, from which you run docker-compose up
• create .env file containing these variables (syntax identical to your sample.env)

Personally I would separate image building process and container launching process (take away image building responsibility from docker-compose to external script, then building process can be configured easily).

Comments

[PANDA Stack]

Thank you - it sounds like the answer is "what you want is not possible right now." To be clear, your preference would be to use a script to call build and push to a Docker repo, then use docker-compose.yml to pull existing images?

[Andrii Maletskyi]

If you want fast, simple solution, just remove env_file section from compose. and rename sample.env to .env, and everything should work

[PANDA Stack]

I see - so there is only one file that can work here, and it is named .env only. Thank you.

[PANDA Stack]

I renamed the file to `.env' and Docker deployed "successfully." But the keys are not in the container environment! Any ideas?

[Andrii Maletskyi]

I can't reproduce it. In my docker image keys are effectively set up by ENV statement. You should have SUB_KEY in 'flask' container, and both keys in 'scan' container

[Andrii Maletskyi]

debug it with docker-compose exec scan env. Do you see keys in command output?

[PANDA Stack]

Empty! I'm so confused!! PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin‌​:/usr/bin:/sbin:/bin HOSTNAME=ef7e949d79a7 TERM=xterm LANG=C.UTF-8 GPG_KEY=0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D PYTHON_VERSION=3.6.5 PYTHON_PIP_VERSION=10.0.1 SUB_KEY= HOME=/root

[PANDA Stack]

It is getting the build arg in the Dockerfile, but that appears to be a blank string? Or perhaps I am not assigning correctly...

[Andrii Maletskyi]

ARG statement must be present in every build stage, not only in the base

[PANDA Stack]

Interesting. I see that you are right, and there is some consternation over this decision. github.com/moby/moby/issues/34129