kubelet failed with kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"
Solution 1
On my environment it only worked the other way around. Setting systemd results always in an error. Here is my current setup
OS: CentOS 7.6.1810
Minikube Version v1.0.0
Docker Version 18.06.2-ce
The solution for me was:
Check /etc/docker/daemon.json
and change systemd to cgroupfs
{
"exec-opts": ["native.cgroupdriver=cgroupfs"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
Then reload systemctl systemctl daemon-reload
Kill the previous minikub config minikube delete
and start the minikube again minikube start --vm-driver=none
Now check the command line the output should find cgroupfs
in both outputs
docker info | grep -i cgroup
cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
In the end you should see
kubectl is now configured to use "minikube"
= Done! Thank you for using minikube!
Simple solution: Start your minikube with the Extra config parameter
--extra-config=kubelet.cgroup-driver=systemd
The complete command to start up minikube is the next line
minikube start --vm-driver=none --extra-config=kubelet.cgroup-driver=systemd
All the best and have fun
Solution 2
Possible cause
kubelet 1.7.3 not reading config file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf #50748
Solution
If you are using CentOS and encounter difficulty while setting up the master node, verify that your Docker cgroup driver matches the kubelet config:
docker info | grep -i cgroup
cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
If the Docker cgroup driver and the kubelet config don’t match, change the kubelet config to match the Docker cgroup driver. The flag you need to change is --cgroup-driver. If it’s already set, you can update like so:
sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
This can be replaced with:
CG=$(sudo docker info 2>/dev/null | sed -n 's/Cgroup Driver: \(.*\)/\1/p')
sed -i "s/cgroup-driver=systemd/cgroup-driver=$CG/g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Solution 3
This is caused by miscofiguration during the initial startup. For example forgeting to change docker cgroup driver before executing kubeadm init
command.
To remedy this under CentOS, open /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
or locate the file under your operating system. Locate the entry with EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
. Open this file and change the value of --cgroup-driver
to systemd
or to be the same as docker cgroup driver.
Old Content:
KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1
New Content:
KUBELET_KUBEADM_ARGS=--cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1
Solution 4
To get minicube running on a CentOS 7, I need to start it with a --extra-config=kubelet.cgroup-driver=systemd
as suggested in https://github.com/kubernetes/minikube/issues/2192.
Solution 5
It may be better to do the reverse and make kubelet to use systemd
In Kubernetes site, they recommend using systemd https://kubernetes.io/docs/setup/production-environment/container-runtimes/ More details here
And you can change kubelet to use systemd by following https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ adding this to /etc/sysconfig/kubelet
cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd
Comments
-
Yuwen Yan about 2 years
Configuration for
cgroup driver
is right in/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
I also checked the
Environment
with cli$ systemctl show --property=Environment kubelet | cat Environment=KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf\x20--require-kubeconfig=true KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests\x20--allow-privileged=true KUBELET_NETWORK_ARGS=--network-plugin=cni\x20--cni-conf-dir=/etc/cni/net.d\x20--cni-bin-dir=/opt/cni/bin KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10\x20--cluster-domain=cluster.local KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook\x20--client-ca-file=/etc/kubernetes/pki/ca.crt KUBELET_CADVISOR_ARGS=--cadvisor-port=0 KUBELET_CGROUP_ARGS=--cgroup-driver=systemd
KUBELET_CGROUP_ARGS=--cgroup-driver=systemd
How to reproduce it:
- yum install -y docker-1.12.6
- systemctl enable docker && systemctl start docker
- setenforce 0
- yum install -y kubelet kubeadm
- systemctl enable kubelet && systemctl start kubelet
- systemctl daemon-reload
- systemctl restart kubelet
- kubelet log
Environment:
- Kubernetes version (use
kubectl version
): 1.7.3 - Cloud provider or hardware configuration**: 4 core 16G RAM
- OS (e.g. from /etc/os-release): CentOS Linux 7 (Core)
- Kernel (e.g.
uname -a
): Linux 10-8-108-92 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 23 17:05:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux - Install tools: kubeadm