K8s coredns and flannel nameserver limit exceeded

Question

i have been trying to setup k8s in a single node,everything was installed fine. but when i check the status of my kube-system pods,

CNI -> flannel pod has crashed, reason -> Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: x.x.x.x x.x.x.x x.x.x.x

CoreDNS pods status is ContainerCreating.

In My Office, the current server has been configured to have an static ip and when i checked /etc/resolv.conf

This is the output

# Generated by NetworkManager
search ORGDOMAIN.BIZ
nameserver 192.168.1.12
nameserver 192.168.2.137
nameserver 192.168.2.136
# NOTE: the libc resolver may not support more than 3 nameservers.
# The nameservers listed below may not be recognized.
nameserver 192.168.1.10
nameserver 192.168.1.11

i'm unable to find the root cause, what should i be looking at?

Solution

In short, you have too many entries in /etc/resolv.conf.

This is a known issue:

Some Linux distributions (e.g. Ubuntu), use a local DNS resolver by default (systemd-resolved). Systemd-resolved moves and replaces /etc/resolv.conf with a stub file that can cause a fatal forwarding loop when resolving names in upstream servers. This can be fixed manually by using kubelet’s --resolv-conf flag to point to the correct resolv.conf (With systemd-resolved, this is /run/systemd/resolve/resolv.conf). kubeadm (>= 1.11) automatically detects systemd-resolved, and adjusts the kubelet flags accordingly.

Also

Linux’s libc is impossibly stuck (see this bug from 2005) with limits of just 3 DNS nameserver records and 6 DNS search records. Kubernetes needs to consume 1 nameserver record and 3 search records. This means that if a local installation already uses 3 nameservers or uses more than 3 searches, some of those settings will be lost. As a partial workaround, the node can run dnsmasq which will provide more nameserver entries, but not more search entries. You can also use kubelet’s --resolv-conf flag.

If you are using Alpine version 3.3 or earlier as your base image, DNS may not work properly owing to a known issue with Alpine. Check here for more information.

You possibly could change that in the Kubernetes code, but I'm not sure about the functionality. As it's set to that value for purpose.

Code can be located here

const (
    // Limits on various DNS parameters. These are derived from
    // restrictions in Linux libc name resolution handling.
    // Max number of DNS name servers.
    MaxDNSNameservers = 3
    // Max number of domains in search path.
    MaxDNSSearchPaths = 6
    // Max number of characters in search path.
    MaxDNSSearchListChars = 256
)