How fix this error "watch chan error: etcdserver: mvcc: required revision has been compacted"?

Question

I have a kubernetes cluster with two nodes, and two nodes for etcd,also I am using authentication for my kubernetes. When I run this command: kubectl get cs I get this output:

NAME                 STATUS      MESSAGE                                                                                        ERROR
scheduler            Unhealthy   Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection refused   
controller-manager   Healthy     ok                                                                                             
etcd-1               Healthy     {"health": "true"}                                                                             
etcd-0               Healthy     {"health": "true"}   

when I check logs of my kube-apiserver I get this error:

E0516 22:28:14.312855       1 watcher.go:188] watch chan error: etcdserver: mvcc: required revision has been compacted

what's this error for and how can I fix this?

also I get this warning for my kube-controller-manager:

W0516 22:28:14.314137       1 reflector.go:323] k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:192: watch of <nil> ended with: etcdserver: mvcc: required revision has been compacted

and in for my kube-scheduler:

W0516 10:43:38.097940       1 reflector.go:323] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: watch of *v1.Pod ended with: too old resource version: 2153673 (2154157)

my kubernetes version is 1.6.3

I really appreciate your help:)

Solution

update etcd version to 3.0.11 or later

https://github.com/kubernetes/kubernetes/issues/45506

Comments

[Ladan Nekuii]

Thanks Arika, This update fixed my error with "too old resource version" but I still get this error " 127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection refused" , I changed the livenessProbe host IP in my kube-scheduler file to 0.0.0.0 but I don't know why kube-scheduler still insist on using 127.0.0.1? do you have any idea?Thanks!

[Arika Chen]

Maybe hard code in pkg/registry/core/rest/storage_core.go. 10251 is already opened by scheduler or do you behind some proxy?

[Ladan Nekuii]

oh I found it! I added another switch for my port under the command section of my container section by mistakecontainers: - name: kube-scheduler image: gcr.io/google-containers/kube-scheduler:v1.6.3 command: - kube-scheduler - --address=0.0.0.0 **- --port=7251** - --master=http://srv2:8080 - --v=1 livenessProbe: httpGet: host: 0.0.0.0 path: /healthz port: 10251 initialDelaySeconds: 15 timeoutSeconds: 15 , I removed it and everything is working and my cluster is healthy. Thanks for your help:)