Kubernetes on k3s can't resolve domains from custom dns server (fritz.box with dnsmasq)

Question

I have a dns server running at 192.168.0.19 for custom domains like .fritz.box. Having a single node cluster on k3s, Rancher was installed using a subdomain server2.fritz.box using this command:

    helm install rancher-latest/rancher \
      --name rancher \
      --namespace cattle-system \
      --set hostname=server2.fritz.box

Rancher itself shows that some services are not avaliable and the logs from cattle say server2.fritz.box is not avaliable. Since Kubernetes has its own dns system, I looked at the documentation and it seems that I need to set my .19 dns server so that Kubernetes knows how to resolve .fritz.box domains. Some questions also have similar problems like https://stackoverflow.com/questions/41448095/kube-dns-does-not-resolve-external-hosts-on-kubeadm-bare-metal-cluster

So I created the following yaml:

# https://github.com/kubernetes/kops/issues/4986
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
data:
  stubDomains: |
    {"fritz.box": ["192.168.0.19"]}
  upstreamNameservers: |
    ["192.168.0.19"]

Loaded with kubectl apply -f dns.yml. Now created a busybox test pod:

~$ kubectl exec -it busybox -- ping server2.fritz.box
ping: bad address 'server2.fritz.box'

Why is this not working? And what must be done to resolve a custom dns server in Kubernetes?

Solution 1

I believe this is an current bug in k3s that upstream DNS is hardcoded to 1.1.1.1. this should be resolved shortly https://github.com/rancher/k3s/issues/53

Solution 2

ibuildthecloud9 gave me the right hint. Since the github issue doesn't describe how to midify the dns, I figured it out and want to document it here in case someone need to change it, too. It's stored in the configmap coredns as Corefile:

proxy . 1.1.1.1

You need to replace this by your dns server (192.168.0.19 in my case). It could be done manually using kubectl edit cm -n kube-system coredns. In case you also want to automate this process:

kubectl get cm -n kube-system coredns -o yaml | sed "s/proxy . 1.1.1.1/proxy . 192.168.0.19/g" > coredns-fixed.yml

Now you'll have the fixed yaml file, which got loaded by

kubectl apply -f coredns-fixed.yml

Test it

Create busybox.yml

apiVersion: v1
kind: Pod
metadata:
    name: busybox
spec:
    containers:
    # for arm
    #- image: hypriot/armhf-busybox
    - image: busybox
      command:
          - sleep
          - "3600"
      imagePullPolicy: IfNotPresent
      name: busybox
    restartPolicy: Always

Create the pod: kubectl create -f busybox.yml And try to ping a host resolved by your dns:

~$ kubectl exec -it busybox -- ping -c1 server2.fritz.box
PING server2.fritz.box (192.168.0.37): 56 data bytes
64 bytes from 192.168.0.37: seq=0 ttl=61 time=0.386 ms

Before applying our dns (so 1.1.1.1 was used, which belongs to Cloudflare) this throws the following resolving error:

*~$ kubectl exec -it busybox -- ping -c1 server2.fritz.box
ping: bad address 'server2.fritz.box'*