Docker Compose - How to store database data?

Question

I am new to docker and developing a project using docker compose. From the documentation I have learned that I should be using data only containers to keep data persistant but I am unable to do so using docker-compose. Whenever I do docker-compose down it removes the the data from db but by doing docker-compose stop the data is not removed. May be this is because that I am not creating named data volume and docker-compose down hardly removes all the containers. So I tried naming the container but it threw me errors. Please have a look at my yml file:

version: '2'
services: 
   data_container:
     build: ./data
     #volumes:
     #  - dataVolume:/data
  db:
    build: ./db
    ports:
      - "5445:5432"
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_DB=postgres
    #   - PGDATA=/var/lib/postgresql/data/pgdata
    volumes_from:
     # - container:db_bus
       - data_container
  geoserver:
    build: ./geoserver
    depends_on:
      - db
    ports:
      - "8004:8080"
    volumes:
      - ./geoserver/data:/opt/geoserverdata_dir

  web:
    build: ./web
    volumes:
      - ./web:/code
    ports:
      - "8000:8000"
    depends_on:
      - db
    command: python manage.py runserver 0.0.0.0:8000

  nginx:
    build: ./nginx
    ports:
      - "83:80"
    depends_on:
      - web

The Docker file for the data_container is:

FROM stackbrew/busybox:latest
MAINTAINER Tom Offermann <[email protected]>

# Create data directory
RUN mkdir /data

# Create /data volume
VOLUME /data

I tried this but by doing docker-compose down, the data is lost. I tried naming the data_container as you can see the commented line, it threw me this error:

ERROR: Named volume "dataVolume:/data:rw" is used in service "data_container" but no declaration was found in the volumes section.

So right now what I am doing is I created a stand alone data only named container and put that in the volumes_from value of the db. It worked fine and didn't remove any data even after doing docker-compose down.

My queries:

• What is the best approach to make containers that can store database's data using the docker-compose and to use them properly ?

• My conscious is not agreeing with me on approach that I have opted, the one by creating a stand alone data container. Any thoughts?

Solution 1

docker-compose down

does the following

Stops containers and removes containers, networks, volumes, and images created by up

So the behaviour you are experiencing is expected.

Use docker-compose stop to shutdown containers created with the docker-compose file but not remove their volumes.

Secondly you don't need the data-container pattern in version 2 of docker compose. So remove that and just use

  db:
    ...
    volumes:
       - /var/lib/postgresql/data

Solution 2

docker-compose down stops containers but also removes them (with everything: networks, ...).

Use docker-compose stop instead.

I think the best approach to make containers that can store database's data with docker-compose is to use named volumes:

version: '2'
services:

  db: #https://hub.docker.com/_/mysql/
    image: mysql
    volumes:
      - "wp-db:/var/lib/mysql:rw"
    env_file:
      - "./conf/db/mysql.env"

volumes:
  wp-db: {}

Here, it will create a named volume called "wp-db" (if it doesn't exist) and mount it in /var/lib/mysql (in read-write mode, the default). This is where the database stores its data (for the mysql image).

If the named volume already exists, it will be used without creating it.

When starting, the mysql image look if there are databases in /var/lib/mysql (your volume) in order to use them.

You can have more information with the docker-compose file reference here: https://docs.docker.com/compose/compose-file/#/volumes-volume-driver

Solution 3

To store database data make sure your docker-compose.yml will look like if you want to use Dockerfile

version: '3.1'

services:
  php:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - 80:80
    volumes:
      - ./src:/var/www/html/
  db:
    image: mysql
    command: --default-authentication-plugin=mysql_native_password
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: example
    volumes:
      - mysql-data:/var/lib/mysql

  adminer:
    image: adminer
    restart: always
    ports:
      - 8080:8080
volumes:
  mysql-data:

your docker-compose.yml will looks like if you want to use your image instead of Dockerfile

version: '3.1'   

services:
  php:
    image: php:7.4-apache
    ports:
      - 80:80
    volumes:
      - ./src:/var/www/html/
  db:
    image: mysql
    command: --default-authentication-plugin=mysql_native_password
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: example
    volumes:
      - mysql-data:/var/lib/mysql

  adminer:
    image: adminer
    restart: always
    ports:
      - 8080:8080
volumes:

if you want to store or preserve data of mysql then must remember to add two lines in your docker-compose.yml

volumes:
  - mysql-data:/var/lib/mysql

and

volumes:
  mysql-data:

after that use this command

docker-compose up -d

now your data will persistent and will not be deleted even after using this command

docker-compose down

extra:- but if you want to delete all data then you will use

docker-compose down -v

to verify or check database data list by using this command

docker volume ls

DRIVER              VOLUME NAME
local               35c819179d883cf8a4355ae2ce391844fcaa534cb71dc9a3fd5c6a4ed862b0d4
local               133db2cc48919575fc35457d104cb126b1e7eb3792b8e69249c1cfd20826aac4
local               483d7b8fe09d9e96b483295c6e7e4a9d58443b2321e0862818159ba8cf0e1d39
local               725aa19ad0e864688788576c5f46e1f62dfc8cdf154f243d68fa186da04bc5ec
local               de265ce8fc271fc0ae49850650f9d3bf0492b6f58162698c26fce35694e6231c
local               phphelloworld_mysql-data

Comments

[ldg]

Is there a reason you don't want to use a named volume for your data, like volumes: - /data/mydata:/var/lib/postgresql/data/pgdata which is the preferred method over data container volumes.

[Salman Ghauri]

How can I backup the data or make it portable by using this approach?

[lukeaus]

Its already on the docker host in a volume. You need to access that volume with a tool that can dump the data. e.g. docker-compose exec POSTGRES_CONTAINER pg_dump | bzip2 > path/on/host/mydb-dump-`date "+%Y%m%d%H%M%S"`.sql.bz2

[Salman Ghauri]

This helped a lot. There is an error while I tried to back up the data that is: pg_dump: [archiver (db)] connection to database "root" failed: FATAL: role "root" does not exist Can you help me with this?

[lukeaus]

@SalmanGhauri probably heading off topic for this - try posting a new question if you continue to have problems

[Salman Ghauri]

This issue was resolved by using named volumes. If you use named volumes doing docker-compose down will not destroy the data.