Kubernetes cgroup driver misconfiguration

Question

Default Docker installation in CentOS starts with systemd Cgroup. I installed Kubernetes from official YUM repo and systemd drop-in 10-kubeadm.conf has the following contents:

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS

I also tried to get the environmental variables to see if the systemd drop-in overrides properly (systemctl show --property=Environment kubelet | cat):

Environment=KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf\x20--kubeconfig=/etc/kubernetes/kubelet.conf KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests\x20--allow-privileged=true KUBELET_NETWORK_ARGS=--network-plugin=cni\x20--cni-conf-dir=/etc/cni/net.d\x20--cni-bin-dir=/opt/cni/bin KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10\x20--cluster-domain=cluster.local KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook\x20--client-ca-file=/etc/kubernetes/pki/ca.crt KUBELET_CADVISOR_ARGS=--cadvisor-port=0 KUBELET_CGROUP_ARGS=--cgroup-driver=systemd KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true\x20--cert-dir=/var/lib/kubelet/pki

When I initialize the cluster using kubeadm init --apiserver-advertise-address=X.X.X.X --pod-network-cidr=10.244.0.0/16, the process is successful. However, running kubelet version gives me a Cgroup misconfiguration error:

error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

Versions

• CentOS Linux release 7.4.1708 (Core)
• Kubernetes Version (kubectl version): 1.9 (GitVersion: 1.9.1; Go Version: go1.9.2)
• Docker version (docker version): 1.12.6

What is the right solution for this problem? I tried to change Cgroup of Kubernetes to cgroupfs and kubeadm kept giving me "kubelet unresponsive" error. How can I make Kubernetes to accept CGroup parameter given to it?

Solution 1

I had the same problems. You need to ensure that Docker and Kubelet run with same cgroup driver.

In Centos Docker runs in cgroupfs while kubelet runs in systemd cgroup. In order to change this you need to follow Kubelet documentation: https://kubernetes.io/docs/setup/independent/install-kubeadm/#configure-cgroup-driver-used-by-kubelet-on-master-node

To fix this you need:

sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Then restart kubelet

systemctl daemon-reload
systemctl restart kubelet

Solution 2

Modify the file /etc/sysconfig/kubelet with your cgroup-driver value, like so:

KUBELET_EXTRA_ARGS=--cgroup-driver=<value>

To get your Docker Cgroup driver:

docker info | grep -i 'cgroup driver'

Then reload systemd daemon and restart kubelet:

systemctl daemon-reload
systemctl restart kubelet