Docker executing cp or mv command in Dockerfile, but changes does not show up in an image

Question

I am quite new to programming and docker. I did come across such a problem, which I couldn't find an answer to.

I created my custom docker image with no problems and then I wanted to use this image as my base for further changes. So I created my Dockerfile, which looks like (see below) and when I use sudo docker build -t my-name . it executes with no problems, however when i run the image (sudo docker run -it my-name /bin/bash) I see no changes in the directory list (directory /root/new_files/ does not exist).

I seem to think this populated somehow from my first build on which my this image is based as I did these commands many times before with no problems.

Any advice, what could have caused Docker executing the program, but changes not seen in the image itself.

My Docker file:

FROM plu_build_1:latest

ENV BASEDIR=/root

WORKDIR /root

RUN cp -a $BASEDIR/TEMPLATE/ $BASEDIR/DEMO/

COPY DEMO/parameters.DEMO $BASEDIR/DEMO/

COPY DEMO/config.DEMO $BASEDIR/DEMO/

ENV PATH="${BASEDIR}/bin:${PATH}"

VOLUME ["/root/DEMO/LOG/"]
CMD ["bash"]

I want to note, that instead of cp -a I tried 'mv' command ... with no luck also seems that creating links with 'ln' does not work.

However if after build I enter image and run the same command inside running image it works fine, that means I can run 'cp -a $BASEDIR/TEMPLATE/ $BASEDIR/DEMO/' and it works.

So, --no-cache didn't help. Below is the base image Dockerfile:

base image Dockerfile:

FROM fedora:25
RUN yum -y update \
  && yum -y install file gcc gcc-gfortran gcc-c++ glibc.i686 libgcc.i686 libpng-devel jasper jasper-devel hostname m4 make perl \ 
  tar tcsh time wget which zlib zlib-devel openssh-clients openssh-server net-tools \
  netcdf-fortran libpng15 iproute-tc tcp_wrappers-libs sendmail procmail psmisc procps-ng mailx  findutils ImageMagick \
  perl-CPAN ncl netcdf libpng libjpeg-turbo which patch vim less bzip2 \
  && yum clean all

RUN yum -y install netcdf-openmpi-devel.x86_64 netcdf-fortran-openmpi-devel.x86_64 netcdf-fortran-openmpi.x86_64 hdf5-openmpi.x86_64 openmpi.x86_64 openmpi-devel.x86_64 \
  && yum clean all

COPY files.tgz /root
COPY files-bin.tgz /root
COPY rings.tgz /root
# extract all and link all files

RUN tar -xvzf files.tgz \
&& tar -xvzf files-bin.tgz \
&& tar -xvzf rings.tgz \
&& rm files*.tgz \
&& rm rings.tgz 

WORKDIR /root/bin
COPY prog-cmake-linux.tar /root/bin
COPY files-cmake-linux.tar /root/bin
RUN tar xf prog-cmake-linux.tar \
 && tar xf files-cmake-linux.tar \
 && rm prog-cmake* \
 && rm files-cmake* \
 && rm -rdf /root/bin/test/ \
 && rm -rdf /root/bin/main/ \
 && rm -rdf /root/bin/*grid/ \
 && mkdir /wrf/netcdf_links \
 && ln -sf /usr/lib64/openmpi/lib /root/netcdf_links/lib \
 && ln -sf /usr/include/openmpi-x86_64 /root/netcdf_links/include 

RUN (echo y;echo o conf prerequisites_policy follow;echo o conf commit) | cpan   && cpan install Proc/Background.pm \
  && ln -s libnetcdff.so.6 /lib64/libnetcdff.so.5 \
  && ln -s libnetcdf.so.11 /lib64/libnetcdf.so.7

RUN echo export LDFLAGS="-lm" >> /etc/bashrc \
 && echo export NETCDF=/root/netcdf_links >> /etc/bashrc \
 && echo export JASPERINC=/usr/include/jasper/ >> /etc/bashrc \
 && echo export JASPERLIB=/usr/lib64/ >> /etc/bashrc \
 && echo export LD_LIBRARY_PATH="/usr/lib64/openmpi/lib" >> /etc/bashrc \
 && echo export PATH="/usr/lib64/openmpi/bin:$PATH" >> /etc/bashrc \
 && echo setenv LDFLAGS "-lm" >> /etc/csh.cshrc \
 && echo setenv NETCDF "/root/netcdf_links" >> /etc/csh.cshrc \
 && echo setenv JASPERINC "/usr/include/jasper/" >> /etc/csh.cshrc \
 && echo setenv JASPERLIB "/usr/lib64/" >> /etc/csh.cshrc \
 && echo setenv LD_LIBRARY_PATH "/usr/lib64/openmpi/lib" >> /etc/csh.cshrc \
 && echo setenv PATH "/usr/lib64/openmpi/bin:$PATH" >> /etc/csh.cshrc \
 && echo export BASEDIR=$BASEDIR >> /etc/bashrc \
 && echo export PATH+=:\$BASEDIR/bin >> /etc/bashrc

ENV LD_LIBRARY_PATH /usr/lib64/openmpi/lib
ENV PATH  /usr/lib64/openmpi/bin:$PATH

# set up ssh configuration
COPY ssh_config /root/.ssh/config
RUN mkdir -p /root/.openmpi
COPY default-mca-params.conf /root/.openmpi/mca-params.conf

RUN mkdir -p /var/run/sshd \
    && ssh-keygen -A \
    && sed -i 's/#PermitRootLogin yes/PermitRootLogin yes/g' /etc/ssh/sshd_config \
    && sed -i 's/#RSAAuthentication yes/RSAAuthentication yes/g' /etc/ssh/sshd_config \
    && sed -i 's/#PubkeyAuthentication yes/PubkeyAuthentication yes/g' /etc/ssh/sshd_config \
    && ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' \
    && chmod 600 /root/.ssh/config \
    && chmod 700 /root/.ssh \
    && cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
#
WORKDIR /root
VOLUME /root

Solution

The immediate cause of your problems is that, once you declare a directory a VOLUME in a Dockerfile, you can never make changes in that directory tree ever again. In particular, since your base image Dockerfile ends with

VOLUME /root

then a couple of steps later

FROM plu_build_1:latest  # inherits that VOLUME
RUN cp -a /root/TEMPLATE/ /root/DEMO/

is a no-op because nothing in /root can ever be changed ever again.

The very short answer here is to never put VOLUME in a Dockerfile at all. It maybe makes sense for things like database servers that have a single directory tree that you almost always want to outlive a single container if you can, but that's an exception, and it has some confusing side effects (like this).

Looking at this Dockerfile more broadly, it looks like a full-blown development environment more than a self-contained reusable image. (It contains two network servers, two compiler stacks and a third interpreted language runtime, and an out-of-the-mainstream interactive shell; it goes out of its way to configure both shells' dotfiles, when many typical Docker paths run no shells at all; it contains an easily extracted ssh key that gives root permissions to something.) You might consider whether a stack built on Vagrant, a full-blown VM, and a more modular configuration management system like Ansible is a better match for what you're trying to build.

Comments

[Marius P]

I will try this out, but as I mentioned I tried changing commands from cp to mv but it didn't work. Also adding ln -sf lines to link anything just to check if it is a problem with cp or mv, but seems that it is for all of them. COPY in the mean time works fine (add files from host to image)

[Marius P]

didn't help.. Posting base image Dockerfile above

[Marius P]

thanks a lot for advice! VOLUME was the problem in my base Docker image. I will scout your offered options for my build as well! thank you