Kubernetes kube-dns pod is pending

Question

I tried this doc to install and setup Kubernetes in Ubuntu VM. I have finished upto 3/4 and now kube-dns pod is in pending status. How can i figure out this? here is the result for kubectl get pods --namespace=kube-system and kubectl describe pod <pod name>

# kubectl get pods --namespace=kube-system
NAME                              READY     STATUS    RESTARTS   AGE
dummy-2088944543-jk2t2            1/1       Running   0          3h
etcd-ubuntu                       1/1       Running   0          3h
kube-apiserver-ubuntu             1/1       Running   0          3h
kube-controller-manager-ubuntu    1/1       Running   0          3h
kube-discovery-1769846148-h88v4   1/1       Running   0          3h
kube-dns-2924299975-dfp17         0/4       Pending   0          3h
kube-proxy-zdcxw                  1/1       Running   0          3h
kube-scheduler-ubuntu             1/1       Running   0          3h
weave-net-xwfhj                   2/2       Running   0          2h

# kubectl describe pod kube-dns-2924299975-dfp17
Error from server (NotFound): pods "kube-dns-2924299975-dfp17" not found

Solution

Cause

Most likely a lack of available computing resources in your cluster.

If you're using the example in cluster/addons/dns you're certainly using a Deployment with resource requests, highlighted if you click the link. It could be that your other pods are already requesting all the available resources in the cluster, therefore your pod doesn't get scheduled.

You can confirm that theory with kubectl --namespace=kube-system describe pod kube-dns-2924299975-dfp17 and look for the following event:

Reason                Message
------                -------
FailedScheduling      pod (kube-dns-2924299975-dfp17) failed to fit in any node
fit failure summary on nodes : Insufficient cpu (3)

You can also describe your nodes with kubectl describe node <node-name> and look at the last information:

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.
  CPU Requests  CPU Limits      Memory Requests Memory Limits
  ------------  ----------      --------------- -------------
  320m (8%)     300m (7%)       150Mi (1%)      150Mi (1%)

In your case either the CPU or memory allocation should be close to 100%.

Solution

• Add more computing resources / nodes to your cluster (preferred)
• Remove the resource requests from your pod(s), at the risk of overcommitting your resources

Comments

[gsaslis]

this was also the issue in my case. Upgraded masters from t2.micro to t2.small and that fixed it.

[Antoine Cotten]

@gsaslis Masters should not be schedulable in general, for security reasons. Upgrading them so that you can schedule pods on them is quite an anti-pattern.

[gsaslis]

yes, good point. In my case it was the kube-scheduler that couldn't be deployed... ; )

[Stanislav Mekhonoshin]

Is it possible to decrease/remove kube-dns default requests?