Configuring multiple DNS name servers on 18.04 server

Question

I set up a static network during installation of Ubuntu 18.04, and it seems to have generated a file at /etc/netplan/50-cloud-init.yaml with the details I provided.

Things seemed to work, but Ubuntu periodically forgets about the DNS settings I entered. For example if I try to nslookup another-server.lan.example.com it will sometimes not find it.

After running sudo netplan apply it works again. Running this command seems to be required a couple times a day.

Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   another-server.lan.mydomain.com
Address: 10.10.1.1›

/etc/netplan/50-cloud-init.yaml

# This file is generated from information provided by
# the datasource.  Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        ens160:
            addresses:
            - 10.10.1.10/16
            gateway4: 10.10.0.1
            nameservers:
                addresses:
                - 10.10.1.1
                - 8.8.8.8
                search:
                - lan.mydomain.com
            optional: true
    version: 2

What I think is happening is at some point during the day a lookup fails with the primary (first listed) internal DNS server, and so systemd switches over to the secondary (second listed) google DNS server. But then for whatever reason when the secondary DNS server fails to resolve a local address, systemd doesn't retry the primary DNS server.

A similar issue is discussed here: https://github.com/systemd/systemd/issues/5755

Solution

I can confirm the issue with 18.04.1 LTS and netplan as I am experiencing it myself with a different workaround.

The Setup: I have two 18.04.1 LTS servers that are configured more or less identically. One server hosts applications and is internet accessible. The other is a data server behind the firewall that only the application server can access.

The Network: I have a static environment with two DNS servers. (Windows OS). For added redundancy I use 8.8.8.8 in the event of internal DNS failure.

The Situation: Some of my apps resolve via hostname and not IP address. A few times a day the apps will crash or throw errors because they cannot resolve the hostname. If I run netplan apply it works for a bit.

Even though I have 8.8.8.8 third in the list I believe that netplan is eventually ignoring my local DNS servers in favor of google and never checking internal DNS if it cannot resolve a host name against the primary DNS.

The Workaround: When I added my local hostname to my public DNS record (I know big no no) without reapplying netplan I was able to get my application server to resolve without issue.

Ultimately I believe their is a nameserver preference or "permanent" nameserver failure issue with netplan as implemented in Ubuntu 18.04.1 LTS