how do you manage secret values with docker-compose v3.1?

Question

Version 3.1 of the docker-compose.yml specification introduces support for secrets.

I tried this:

version: '3.1'

services:
  a: 
    image: tutum/hello-world
  secret: 
    password: the_password
  b:
    image: tutum/hello-world

$ docker-compose up returns:

Unsupported config option for services.secret: 'password'

How can we use the secrets feature in practice?

Solution 1

You can read the corresponding section from the official documentation.

To use secrets you need to add two things into your docker-compose.yml file. First, a top-level secrets: block that defines all of the secrets. Then, another secrets: block under each service that specifies which secrets the service should receive.

As an example, create the two types of secrets that Docker will understand: external secrets and file secrets.

1. Create an 'external' secret using docker secret create

First thing: to use secrets with Docker, the node you are on must be part of a swarm.

$ docker swarm init

Next, create an 'external' secret:

$ echo "This is an external secret" | docker secret create my_external_secret -

(Make sure to include the final dash, -. It's easy to miss.)

2. Write another secret into a file

$ echo "This is a file secret." > my_file_secret.txt

3. Create a docker-compose.yml file that uses both secrets

Now that both types of secrets are created, here is the docker-compose.yml file that will read both of those and write them to the web service:

version: '3.1'

services:
  web:
    image: nginxdemos/hello
    secrets:                    # secrets block only for 'web' service
     - my_external_secret
     - my_file_secret

secrets:                        # top level secrets block
  my_external_secret:
    external: true
  my_file_secret:
    file: my_file_secret.txt

Docker can read secrets either from its own database (e.g. secrets made with docker secret create) or from a file. The above shows both examples.

4. Deploy your test stack

Deploy the stack using:

$ docker stack deploy --compose-file=docker-compose.yml secret_test

This will create one instance of the web service, named secret_test_web.

5. Verify that the container created by the service has both secrets

Use docker exec -ti [container] /bin/sh to verify that the secrets exist.

(Note: in the below docker exec command, the m2jgac... portion will be different on your machine. Run docker ps to find your container name.)

$ docker exec -ti secret_test_web.1.m2jgacogzsiaqhgq1z0yrwekd /bin/sh

# Now inside secret_test_web; secrets are contained in /run/secrets/
root@secret_test_web:~$ cd /run/secrets/

root@secret_test_web:/run/secrets$ ls
my_external_secret  my_file_secret

root@secret_test_web:/run/secrets$ cat my_external_secret
This is an external secret

root@secret_test_web:/run/secrets$ cat my_file_secret
This is a file secret.

If all is well, the two secrets we created in steps 1 and 2 should be inside the web container that was created when we deployed our stack.

Solution 2

Given you have a service myapp and a secrets file secrets.yml:

Create a compose file:

version: '3.1'

services:
  myapp:
    build: .
    secrets:
      secrets_yaml

Provision a secret using this command:

docker secret create secrets_yaml secrets.yml

Deploy your service using this command:

docker deploy --compose-file docker-compose.yml myappstack

Now your app can access the secret file at /run/secrets/secrets_yaml. You can either hardcode this path in your application or create a symbolic link.

---

The different question

This answer is probably to the question "how do you provision your secrets to your docker swarm cluster".

The original question "how do you manage secret values with docker compose" implies that the docker-compose file contains secret values. It doesn't.

There's a different question: "Where do you store the canonical source of the secrets.yml file". This is up to you. You can store it in your head, print on a sheet of paper, use a password manager, use a dedicated secrets application/database. Heck, you can even use a git repository if it's safely secured itself. Of course, never store it inside the system you're securing with it :)

I would recommend vault. To store a secret:

# create a temporary secret file
cat secrets.yml | vault write secret/myappsecrets -

To retrieve a secret and put it into your docker swarm:

vault read -field=value secret/myappsecrets | docker secret create secrets_yaml -

Of course, you can use docker cluster itself as a single source of truth for you secrets, but if your docker cluster breaks, you'd lost your secrets. So make sure to have a backup elsewhere.

---

The question nobody asked

The third question (that nobody asked) is how to provision secrets to developers' machines. It might be needed when there's an external service which is impossible to mock locally or a large database which is impossible to copy.

Again, docker has nothing to do with it (yet). It doesn't have access control lists which specify which developers have access to which secrets. Nor does it have any authentication mechanism.

The ideal solution appears to be this:

• A developer opens some web application.
• Authenticates using some single sign on mechanism.
• Copies some long list of docker secret create commands and executes them in the terminal.

We have yet to see if such an application pops up.

Solution 3

You can also specify secrets stored locally in a file using file: key in secrets object. Then you don't have to docker secret create them yourself, Compose / docker stack deploy will do it for you.

version: '3.1'

secrets:
  password:
    file: ./password

services:
  password_consumer:
    image: alpine
    secrets:
      - password

Reference: Compose file version 3 reference: Secrets

Solution 4

One question was raised here in the comments, why should I initialize a swarm if I only need secrets? And my answer is that secrets is created for the swarm, where you have more than one node and you want to manage and share secrets in a secure way. But if you have one node, this will not (almost) add any extra security if someone can access your host machine where you have the one node swarm, as secrets can be retrieved from the running containers, or directly on the host if the secret is created from a file, like a private key.

Check this blog: https://www.docker.com/blog/docker-secrets-management/

And read the comments: "Thank you very much for the introductory article. The steps are mentioned to view the contents of secrets in container will not work when the redis container is created on a worker node."

Solution 5

Is that the exact indentation of your docker-compose.yml file? I think secret secrets should be nested under a (i.e. one of the services), not directly under services section.

View more solutions

Comments

[Eric]

yes, that was the exact indentation. I tried nesting the secret dictionary under a (and also at the same level as services) and got the same result.

[Eric]

I get this result when I run the first command you specify: Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. I'm confused! Do I need to start a swarm in order to use docker-compose with secrets?

[Eric]

docker secret create seems to require that there be a pre-existing swarm? do I need to create one?

[Mike Hearn]

Oops, yes you do. The docker stack deploy command is part of the Swarm engine. I'll add a line in Step 1 to indicate that.

[Vanuan]

@Eric So you're running this as a developer? I'm afraid Docker doesn't have support for that use case yet. But yeah, you could create a docker swarm consisting of only your developer machine. That's out of scope of this question.

[Eric]

I don't understand the reasoning behind that. Mine and many other use cases don't require a swarm, but do require secrets. Why make us use a swarm if we just want secrets?

[Vanuan]

@Eric If you don't require swarm, you could just mount the secret file from your local machine. Why do you need docker secrets?

[Eric]

@Vanuan because I need secrets to start containers on my remote machine, not just my local machine. for example, the official owncloud image has a docker-compose.yml that asks you to write down the MySQL password as an environment variable, which is bad practice. I thought docker secrets would solve that?

[Vanuan]

@Eric Yes. But secrets still need to be stored somewhere. In case of docker secrets they're stored in a cluster storage. If you don't use a cluster, you can just create a file and mount it. You don't need the docker secrets feature.

[demaniak]

Hm, documentation makes it look like you should create a secret from a file, just like you create from simple string, but as demonstrated here, the secret file must be available to the stack deploy command. Am I not understanding something about the file option?

[demaniak]

Nm, it just hit me: external secrets (file or just string) is like external networks, and the other option lets the deploy command create it on the fly for the stack, like a default-created overaly network. nice.

[Bret Fisher]

Secrets are usable in docker-compose w/o swarm as of 1.11 if you use file not external, but note they are not secure, because compose isn't a production tool, and like said here, there's no place to store them encrypted without swarm raft db. If you define file-based secrets properly in compose file, docker-compose will bind-mount the file into /run/secrets to emulate what swarm does for easier developer workflow when working locally. If you need a secure storage on single-node server then you can init a single-node swarm and it works fine.

[Bret Fisher]

As of Feb 8th docker-compose 1.11 supports file-based secrets in compose files for local dev. See my comment above on chosen answer for details :)

[Vanuan]

@BretFisher but what's the point if it's essentially the same as specifying using volume's ./file_based_secret:/run/secrets/my_secret ?

[Bret Fisher]

@Vanuan Right, there's no functional difference in container. It's about seamless workflow, and limiting the need for multiple compopse files.

[Jamie Jackson]

It's instructive to run this with docker-compose, as you get a couple of warnings, about the external secret. In the container, you can see the file-based secret in /run/secrets, but not the external one.

[Partha]

the keyword is secret only docs.docker.com/engine/reference/commandline/secret_inspect

[Jinna Balu]

How to define multiple variables in the password file?

[Kumar Saurabh]

@BretFisher: What do you mean by "compose isn't a production tool"? The docker documentation has a section dedicated to "Use Compose in production" if someone is using a single server. docs.docker.com/compose/production

[Yunnosch]

Please make more obvious how this answers the question at the top of the page (instead of a question asked in the comments).

[Eric Aya]

This looks more like a comment to another answer rather than a new answer.

[sam-6174]

In case it's not obvious to anyone else, the docker-compose secrets are not accessible to the docker build. See this