How to mimic '--volumes-from' in Kubernetes

Question

I'm looking for a pattern that allows to share volumes between two containers running on the same pod in Kubernetes.

My use case is: I have a Ruby on Rails application running inside a docker container. The docker image contains static assets in /app/<app-name>/public directory, and I need to access those assets from the nginx container running alongside in the same pod.

In 'vanilla' docker I would have used --volumes-from flag to share this directory:

docker run --name app -v /app/<app-dir>/public <app-image>
docker run --volumes-from app nginx

After reading this doc: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/volumes.md I tried this (only relevant entries presented):

spec:
  containers:
    - image: <app-image>
      name: <app-name>
      volumeMounts:
        - mountPath: /app/<app-name>/public
          name: assets
    - image: nginx
      name: nginx
      volumeMounts:
        - mountPath: /var/www/html
          name: assets
          readOnly: true
    volumes:
      - name: assets
        hostPath:
          path: /tmp/assets

But:

• Even though /tmp/assets on the node exists, it's empty
• /app/<app-name>/public inside the app container is also empty

As a workaround I'm gonna try to populate the shared directory when the application container is up (simply cp /app/<app-name>/public/* to shared directory), but I really dislike this idea.

Question: how to mimic --volumes-from in Kubernetes, or if there is no direct counterpart, how can I share files from one container to other running in the same pod ?

apiVersion: v1beta3

Client Version: version.Info{Major:"0", Minor:"17", GitVersion:"v0.17.0", GitCommit:"82f8bdac06ddfacf493a9ed0fedc85f5ea62ebd5", GitTreeState:"clean"}
Server Version: version.Info{Major:"0", Minor:"17", GitVersion:"v0.17.0", GitCommit:"82f8bdac06ddfacf493a9ed0fedc85f5ea62ebd5", GitTreeState:"clean"}

Solution 1

[update-2016-8] In latest Kubernetes release, you can use a very nice feature named init-container to replace the postStart part in my answer below, which will make sure the container order.

apiVersion: v1
kind: Pod
metadata:
  name: javaweb-2
spec:
  initContainers:
  - name: war
    image: resouer/sample:v2
    command: ["cp", "/sample.war", "/app"]
    volumeMounts:
    - mountPath: /app
      name: app-volume
  containers:
  - name: tomcat
    image: resouer/mytomcat:7.0
    command: ["sh","-c","/root/apache-tomcat-7.0.42-v2/bin/start.sh"]
    volumeMounts:
    - mountPath: /root/apache-tomcat-7.0.42-v2/webapps
      name: app-volume
    ports:
    - containerPort: 8080
      hostPort: 8001
  volumes:
  - name: app-volume
    emptyDir: {}

NOTE: initContainer is still a beta feature so the work version of this yaml is actually like: http://kubernetes.io/docs/user-guide/production-pods/#handling-initialization, please notice the pod.beta.kubernetes.io/init-containers part.

---original answer begin---

Actually, you can. You need to use container life cycle handler to control what files/dirs you want to share with other containers. Like:

---
apiVersion: v1
kind: Pod
metadata:
    name: server
spec:
    restartPolicy: OnFailure
    containers:
    - image: resouer/sample:v2
      name: war
      lifecycle:
        postStart:
          exec:
            command:
              - "cp"
              - "/sample.war"
              - "/app"
      volumeMounts:
      - mountPath: /app
        name: hostv1 
    - name: peer
      image: busybox
      command: ["tail", "-f", "/dev/null"]
      volumeMounts:
      - name: hostv2
        mountPath: /app/sample.war
    volumes:
    - name: hostv1
      hostPath:
          path: /tmp
    - name: hostv2
      hostPath:
          path: /tmp/sample.war

Please check my gist for more details:

https://gist.github.com/resouer/378bcdaef1d9601ed6aa

And of course you can use emptyDir. Thus, war container can share its /sample.war to peer container without mess peer's /app directory.

If we can tolerate /app been overridden, it will be much simpler:

---
apiVersion: v1
kind: Pod
metadata:
  name: javaweb-2
spec:
  restartPolicy: OnFailure
  containers:
  - image: resouer/sample:v2
    name: war
    lifecycle:
      postStart:
        exec:
          command:
            - "cp"
            - "/sample.war"
            - "/app"
    volumeMounts:
    - mountPath: /app
      name: app-volume
  - image: resouer/mytomcat:7.0
    name: tomcat
    command: ["sh","-c","/root/apache-tomcat-7.0.42-v2/bin/start.sh"]
    volumeMounts:
    - mountPath: /root/apache-tomcat-7.0.42-v2/webapps
      name: app-volume
    ports:
    - containerPort: 8080
      hostPort: 8001 
  volumes:
  - name: app-volume
    emptyDir: {}

Solution 2

The answer is - for now - you can't. Here's a couple of discussion threads from the Kubernetes issues:

• https://github.com/GoogleCloudPlatform/kubernetes/issues/6120
• https://github.com/GoogleCloudPlatform/kubernetes/issues/831

However, may I suggest that you have an alternate design that might work better?

1. If your assets are locked at the point of the container going live, you could use something like gitRepo volume which would copy it to an emptyDir at the point of going live, and would mean you wouldn't have to move the content around at all, just download it directly to the shared directory.
2. If your assets are locked at the point of the container being built, it's probably best to copy them in at that point, using the Docker COPY command.
3. If you really want to stick with the way you're doing it, you would have to copy the content to the emptyDir volume, which is designed for exactly what you're looking for (minus the lack of having to copy it in).

NFS[1] volumes also could solve your problem, but may be overly complex.

Additionally, I'd recommend that these two services exist in different pods, so you can scale each separately. You can create a service endpoint to communicate between them if you need to.

[1] https://github.com/GoogleCloudPlatform/kubernetes/blob/master/examples/nfs/nfs-web-pod.yaml

Solution 3

Further update from the future:

There is now a FlexVol plugin for Docker volumes: https://github.com/dims/docker-flexvol

At the time of writing, FlexVol is still an alpha feature though, so caveat emptor.

Solution 4

Kubernetes has its own volume types and these are most used volume type:

1. emptyDir
2. secret
3. gitRepo
4. hostPath (similar to --volumes-from)
5. config Maps
6. persistent storage (storage disks provided by cloud platforms)

You can find more about kubernets volumes here -https://kubernetes.io/docs/concepts/storage/volumes/

an example of hostpath volume :

apiVersion: v1
kind: Pod
metadata:
  name: test-pd
spec:
  containers:
  - image: k8s.gcr.io/test-webserver
    name: test-container
    volumeMounts:
    - mountPath: /test-pd
      name: test-volume
  volumes:
  - name: test-volume
    hostPath:
      # directory location on host
      path: /data
      # this field is optional
      type: Directory

hostpath will mount host/node directory to container directory.Multiple containers inside a pod can use different or same volumes.You need to mention it in each container. hostPath volumes are independent of pod lifecycle but it create tight coupling between node and pod , you should avoid using hostPath.

View more solutions

Comments

[cthulhu]

The answer from @aronchick was valuable, but your answer is best for my use case, so I accept it, thank you.

[Alex]

I see some problems here: 1. What happens when two Pods run on the same node? 2. What will happen in a rolling update? When two different pod version run on the same node, they overwrite each others source.

[harryz]

@Alex I just use hostDir for example, you can use emptyDIr, then there will be no overwrite problem. See my gist I posted.

[Zhao Li]

Thanks for the example config; I'm able to have the container move the application package to the persistent volume. But how do you keep Openshift/Kubernetes from thinking the container crashed because it finished copying and exited? It seems like I'm not following the solution 100%.

[Zhao Li]

Thanks in advance for your help! @harryz

[harryz]

You need to add a restartPolicy="OnFailure" to that Pod. But I strongly suggest use initContainer now because it will ensure the container order. btw, your volume does not to be PV, emptyDir is enough

[think01]

I'm using Kubernetes 1.4 and initContainers is not recognized as a correct field. Have I to activate some beta/alpha feature for it to work? In the official documentation I cannot find it, I found this instead: kubernetes.io/docs/user-guide/production-pods/…

[harryz]

Yes, its a beta feature, you should use "pod.beta.kubernetes.io/init-containers" as the doc said.

[Freedom_Ben]

This seems potentially problematic if you will ever have more than one instance of a pod running on a given host. You can do that of course but it's an important limitation to be aware of. I have not proven this concern valid, but thought I'd share in case others are evaluating options. If I'm wrong, please let me know.

[Freedom_Ben]

This comes with a big downside in that you can't deploy new images for the containers independently, and you can't build a generic image for reuse and have it vary it's behavior. For example I built an nginx image to serve static assets for various rails apps. I'll have to build a different version of the nginx version for each rails app and deploy them together. That's a super unfortunate limitation, nevertheless, for now, this does look like the best solution (and it's very clever :-) )