Permission denied with Openshift PersistentStorage on GlusterFS

Question

I'm trying to use OpenShift with PersistentStorage on a GusterFS cluster.

I'm starting one of the default templates : mysql-persistent

I've installed a GlusterFS cluster and create a volume gv_mysql_01

I've added the glusterfs endpoints in openshift :

oc get endpoints
NAME                ENDPOINTS                                 AGE
glusterfs-cluster   10.100.134.26:24007,10.100.134.28:24007   1h

I've the create the PersistentVolume on openshift :

cat gluster-mysql_01-storage.yaml 
apiVersion: "v1"
kind: "PersistentVolume"
metadata:
  name: "mysql" 
spec:
  capacity:
    storage: "512Mi" 
  accessModes:
    - "ReadWriteOnce"
  glusterfs: 
    endpoints: "glusterfs-cluster" 
    path: "gv_mysql_01" 
    readOnly: false
  persistentVolumeReclaimPolicy: "Recycle"

oc create -f gluster-mysql_01-storage.yaml

The PersistentStorage is bind to the container :

oc get pv
NAME      LABELS    CAPACITY   ACCESSMODES   STATUS    CLAIM        REASON    AGE
mysql     <none>    512Mi      RWO           Bound     test/mysql             53m

oc get pvc
NAME      LABELS                               STATUS    VOLUME    CAPACITY   ACCESSMODES   AGE
mysql     template=mysql-persistent-template   Bound     mysql     512Mi      RWO           1h

On on host of the cluster the volume is mount :

10.100.134.26:gv_mysql_01 on /var/lib/origin/openshift.local.volumes/pods/c111c480-8ec7-11e5-8405-0a57f8bdd6b3/volumes/kubernetes.io~glusterfs/mysql type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

but in the container logs :

docker logs b8cd5bb3b0be
Running mysql_install_db ...
mkdir: cannot create directory '/var/lib/mysql/data': Permission denied
chmod: cannot access '/var/lib/mysql/data/mysql': Permission denied
mkdir: cannot create directory '/var/lib/mysql/data': Permission denied
chmod: cannot access '/var/lib/mysql/data/test': Permission denied
151119 14:30:20 [Warning] Can't create test file /var/lib/mysql/data/mysql-1-q2yxh.lower-test
151119 14:30:20 [Warning] Can't create test file /var/lib/mysql/data/mysql-1-q2yxh.lower-test
/opt/rh/mysql55/root/usr/libexec/mysqld: Can't change dir to '/var/lib/mysql/data/' (Errcode: 13)
151119 14:30:20 [ERROR] Aborting

I've try to change owner,group,rights of /var/lib/origin/openshift.local.volumes/pods/c111c480-8ec7-11e5-8405-0a57f8bdd6b3/volumes/kubernetes.io~glusterfs/mysql with the uid:gid of the mysql user in the container, but it still doesn't work.

Am I missing something ?

Solution 1

I was having the same error and I was able to get it working by changing the default scc restricted policy strategy. Since you are running the openshift mysql image that runs as a specific user (27), you need to change the restricted scc to allow the id to run

[root@ose1 ceph]# oc edit scc restricted

then edit the file (with vi) and change the fsGroup value from RunAsAny to MustRunAs

    fsGroup:
        type: MustRunAs

You can see this:

[root@ose1 ceph]# oc get scc restricted
NAME         PRIV      CAPS      HOSTDIR   SELINUX     RUNASUSER        FSGROUP     SUPGROUP   PRIORITY
restricted   false     []        false     MustRunAs   MustRunAsRange   MustRunAs   RunAsAny   <none>

You also need to change the default UID range in the namespace you are using:

[root@ose1 ceph]# oc edit ns default  (sub your namespace for 'default')

openshift.io/sa.scc.uid-range: 25/10000

I made start at 25 as an example, but this will allow the openshift mysql image to run it's user of 27

Solution 2

Try these SELinux settings

setsebool -P virt_use_fusefs 1
setsebool -P virt_sandbox_use_fusefs 1

Comments

[mick]

the fsGroup is already MustRunAs. I've change the uid-range but I still get the permission denied. For the moment the only solution I get is to disable selinux, and chown 26:26 the mysql glusterfs mountpoint, and chmod 777. Without all of that MySQL doesn't start.

[screeley]

@mick - I'm trying to reproduce your problem, but am not able to, I did notice that you are using id 26. The mysql image is hard coded with id 27. See my pod definition below

[screeley]

` apiVersion: v1 kind: Pod metadata: name: mysql-gluster labels: name: mysql-gluster spec: containers: - image: openshift3/mysql-55-rhel7 name: mysql-gluster volumeMounts: - name: glustervol mountPath: /var/lib/test/mysql env: - name: MYSQL_ROOT_PASSWORD value: foopass securityContext: runAsUser: 27 privileged: false volumes: - name: glustervol persistentVolumeClaim: claimName: glusterfs-claim`

[screeley]

@mick - formatting is terrible, sorry, but see answer below, if you have SELinux enforcing you may need to set those SELinux booleans, I have SELinux enforcing, so your pod does run but you are not able to access the gluster volume?

[mick]

I've try what you say. In a first time it works, and then I try to understant why it is not working with the openshift default template mysql-persistent. Adding runAsUser: 27 in the template didn't change anything. I've then see that in your pod definition, you mount the persistentVolume in the bad directory, /var/lib/test/mysql instead of /var/lib/mysql (I'm using the openshift/mysql-55-centos7 image). The database is starting because not using the persistent volume. In /var/lib/test/mysql I still get permission denied.

[mick]

With setsebool -P virt_use_fusefs 1 it's still in permission denied. I don't have the virt_sandbox_use_fusefs boolean.

[Balakrishnan]

I do have the same issue. Origin v1.3.0alpha2, Docker v1.9.1, Fedora 23