Kubernetes Unable to mount volumes for pod

37,518

Solution 1

A persistent volume is just the declaration of availability of some storage inside your kubernetes cluster. There is no binding with your pod at this stage.

Since your pod is deployed through a StatefulSet, there should be in your cluster one or more PersistentVolumeClaims which are the objects that connect a pod with a PersistentVolume.

In order to manually bind a PV with a PVC you need to edit your PVC by adding the following in its spec section:

volumeName: "<your persistent volume name>"

Here an explanation on how this process works: https://docs.openshift.org/latest/dev_guide/persistent_volumes.html#persistent-volumes-volumes-and-claim-prebinding

Solution 2

My case is an edge case, and I doubt that you will reach it. However, I will describe it because, it cost me a lot of grey hairs - and maybe it will save yours.

This same error occurred for me despite PV and PVC being binned. Pod was constantly in ContainerCreating stare, yet kubectl get events throw the error asked in this question.

$ kubectl get pv
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
sewage-db   5Ti        RWO            Retain           Bound    global-sewage/sewage-db   nfs                     3h40m

$kubectl get pvc -n global-sewage 
NAME        STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS   AGE
sewage-db   Bound    sewage-db   5Ti        RWO            nfs            3h39m

After rebooting the server it turned out that, one of 32GiB RAM physical memory was corrupted. Removing the memory fixed the issue.

37,518

Author by

Philip Kirkbride

Updated on July 09, 2022

Comments

Philip Kirkbride almost 2 years
I'm trying to setup a volume to use with Mongo on k8s.

I use kubectl create -f pv.yaml to create the volume.

pv.yaml:
```
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pvvolume
  labels:
    type: local
spec:
  storageClassName: standard
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/nfs"
  claimRef:
    kind: PersistentVolumeClaim
    namespace: default
    name: pvvolume
```
I then deploy this StatefulSet that has pods making PVCs to this volume.

My volume seems to have been created without problem, I'm expecting it to just use the storage of the host node.

When I try to deploy I get the following error:

Unable to mount volumes for pod "mongo-0_default(2735bc71-5201-11e8-804f-02dffec55fd2)": timeout expired waiting for volumes to attach/mount for pod "default"/"mongo-0". list of unattached/unmounted volumes=[mongo-persistent-storage]

Have a missed a step in setting up my persistent volume?
Philip Kirkbride almost 6 years

The volumeName is set, but the PVC is still pending.
whites11 almost 6 years

Is the storage size on your PVC less or equal to 10Gb?
Philip Kirkbride almost 6 years

According to the yaml of the Stateful set it should be 2Gi. When I look in the k8s UI it shows - for capacity gateway.ipfs.io/ipfs/…
whites11 almost 6 years

Just to be sure try and get the PVC's yaml and check
Philip Kirkbride almost 6 years

Seems to be 2 in the yaml gist.github.com/kirkins/47b33b4f86e44f1b8f533bef40e3ddd1
whites11 almost 6 years

the volumeName seems wrong to me. In the gist it is pvvolume while the actual name is pv-volume with a dash. Is it a typo?
Philip Kirkbride almost 6 years

Good catch, I updated the question. I pasted the old version of the PV, I ended up removing the - because it seemed to cause a problem. These are the changes I made since the old version stackoverflow.com/posts/50218042/revisions
whites11 almost 6 years

Another thing I see is the claimRef. It is referencing to a PVC called pvvolume. Nothing wrong per se, but it doesn't look like a PVC name as created by a statefulset. Is your mongo pvc really named pvvolume?
Philip Kirkbride almost 6 years

I just deleted everything and re-generated based on the yaml in my question, and the StatefulSet yaml I linked. The PVC yaml generated seems to be the exact same as the gist. Maybe this is a clue to what went wrong?
whites11 almost 6 years

Sorry I must have missed something. Could you post the output of kubectl get pvc on your namespace please?
Philip Kirkbride almost 6 years

mongo-persistent-storage-mongo-0 Pending pvvolume 0 fast 1h
Philip Kirkbride almost 6 years

wait ... that isn't right. If I just deleted everything and regenerated the setup the PVC shouldn't be 1 hour old.
whites11 almost 6 years

The pvc have to be deleted manually. They don't get deleted if you delete the statefulset that generated them.
whites11 almost 6 years

Anyway error is in the PV IMHO, the claimRef section should have name: mongo-persistent-storage-mongo-0 rather than name: pvvolume
Philip Kirkbride almost 6 years

hmm ok I changed that and now the PVC is successful, there we go it is working!
whites11 almost 6 years

Great! Happy journey with k8s