How to persist “systemd-resolve” configuration for a particular network interface?

Question

I have upgraded my Ubuntu Desktop to Bionic Beaver, which switched to systemd-resolve. With that change, the LXD DNS resolution stopped working.

In order to make LXD containers discoverable, I can run the below command. Note, the IP 10.78.38.1 is the IP of the lxdbr0 bridge.

$ sudo systemd-resolve --interface lxdbr0 --set-dns 10.78.38.1 --set-domain lxd

With this in place, I can discover LXD container by their name and the configuration looks as follows.

$ systemd-resolve --status
.
.
.
Link 10 (lxdbr0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 10.78.38.1
          DNS Domain: lxd
.
.
.

However, after a system reboot, this configuration is gone.

$ systemd-resolve --status
.
.
.
Link 10 (lxdbr0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
.
.
.

In order to make the above persistent, I have reviewed the systemd-resolve manpage http://manpages.ubuntu.com/manpages/bionic/man8/systemd-resolved.service.8.html. It suggests to create a /etc/systemd/resolved.conf.d/lxd.conf file but there are no parameters available that would allow me to make such configuration specific to one particular NIC.

Solution 1

Systemd-networkd store network configurations in (at least) three directories, maybe more. The one that seems most relevant to you is:

/etc/systemd/network/ 

(it overrides the other two, namely /lib/systemd/network and /run/systemd/network).

Create a text file in /etc/systemd/network similar to this:

[Match]
Name=enp0s4

[Network]
DHCP=yes
DNS=192.168.1.1

The file can have any name as long as it ends with .network. Mine is called enp0s4.network. Getting the content right is critical. If you get it wrong, bad things happen, like your networking stops working, at least on that interface, until you restore a working config. Which is not easy – I messed up mine and it took me like an hour to sort it out.

For examples how to set up a bridge, read from here on:

man systemd.network |grep -A 42 "Example 3"

The examples given there plus my snippet of code can set up:

Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 10.78.38.1

Not sure what the syntax is for DNS domain, it should be possible, the man page is like 30 pages long! If you can, skip it – trying Domain= screwed up my system. So find the right syntax or do not put it in unless you can recover.

NOTE: The correct syntax is Domains= not DOMAIN=!

Here are references I used:

• How to check if the network is running: https://superuser.com/questions/1187633/how-to-debug-systemd-networkd?rq=1#comment1807294_1187633
• How to enable if it is disabled: https://askubuntu.com/posts/681768/revisions

Restart systemd-resolved without reboot:

sudo systemctl restart systemd-resolved

FYI: there is a way to tell systemd to let Network Manager handle an interface, as well as leave it unmanaged and run with flat file configurations like /etc/resolv.conf. But I would make it work as is if I were in your shoes.

Solution 2

This sounds plausible but has does not seem to have any effect:

According to the linked manpage man 8 systemd-resolved.service, you have to use the per link configuration file in /etc/systemd/network:

The DNS servers contacted are determined from the global settings in /etc/systemd/resolved.conf, the per-link static settings in /etc/systemd/network/*.network files, the per-link dynamic settings received over DHCP and any DNS server information made available by other system services.

To make your configuration persistent, you have to create the file /etc/systemd/network/lxdbr0.conf:

[Match]
Name=lxdbr0

[Resolve]
DNS=10.78.38.1
Domains=lxd

Solution 3

For Ubuntu users (18.04, 18.10, 19.04, 19.10)

I followed hours worth of advice to work around, over, under, through, and behind systemd-resolve, including tapping several merry jigs on top of it as it grunted at me.

What finally worked for me was

1) leave systemd-resolve alone. Don't try to modify it, don't disable it, don't try to uninstall it, don't un-link it... Don't ... Otherwise endless misery will be your constant companion. "But I want to save space..." Trust me, just accept the extra kilobytes, and move forward...

2) Then, (re)install resolvconf sudo apt install resolvconf. It will dance with systemd-resolve like they were meant for each other.

3) after a reboot, make changes if needed like you used to through resolvconf's mechanisms, such as adding content to /etc/resolv.conf.d/resolv.conf/head, or otherwise.

I found that the music that these two systems make together, without my having to participate, was ... relaxing.

Secondarily... perhaps you came from ifupdown, where you were able to declare your DNS servers in /etc/network/interfaces along with you other ip settings, and have them actually work. But now, having been subverted by netplan and systemd-resolve, you're trying to fly through the new magic hoops, wasting your precious limited minutes, hours, searching for the new magic sequence to get network working again...

If that is the case, once again, accept a few extra kilobytes (ridiculous, I know!!); just (re)install ifupdown **sudo apt install ifupdown** and work through your /etc/network/interfaces again.

Both legacy systems fully and seamlessly integrate into the new systems which aren't working for you and many other people. All for the price of a few extra kilobytes (and it's worth it).

It would really be gracious if the Uber-Powers would post educational commentary on the nature of their new subsystems while (as well as just before) the new system is being installed, instead of making the user find out that things don't work, and begin a long search for solutions. (Why is Microsoft coming to mind...?)

Solution 4

This question still doesn't have an accepted answer and as I've also had tremendous problems resolving container names on the host, I thought I should provide my solution. I am also doing this, as this page is the first google result, when looking up persistent changes to sytemd-resolved.service. Lastly I want to mention that this problem seems to be specific to Ubuntu >= 18.04 and it's version of systemd-resolved.service.

So, the manpage systemd-resolved.service has been mentioned already. In case of LXD, misleading conclusions have been drawn out of it however. The LXD interface doesn't need to be defined in /etc/systemd/network/<iface>.{conf|network}. The interface is already working and therefore doesn't need to be defined for systemd's network manager. By default LXD already has a working DNS server listening on it's networks first host address. All we need to accomplish is getting systemd-resolved.service to recognize this server.

The manpage (linked above) mentions this for /etc/systemd/resolved.conf:

The default configuration is defined during compilation, so a configuration file is only needed when it is necessary to deviate from those defaults. By default, the configuration file in /etc/systemd/ contains commented out entries showing the defaults as a guide to the administrator. This file can be edited to create local overrides.

I want to highlight This file can be edited to create local overrides. @quat in fact you also refer to this in your actual question:

In order to make the above persistent, I have reviewed the systemd-resolve manpage http://manpages.ubuntu.com/manpages/bionic/man8/systemd-resolved.service.8.html. It suggests to create a /etc/systemd/resolved.conf.d/lxd.conf file but there are no parameters available that would allow me to make such configuration specific to one particular NIC.

1. Adding lxd's dnsmasq service to systemd-resolved.service global dns settings:
I am not understanding your last sentence however. You don't need to specify the NIC. You merely need to specify the LXD dns server address. Using your network information a working config for /etc/systemd/resolved.conf would look like this (uncomment to change defaults):

[Resolve]
DNS=10.78.38.1
#OBFallbackDNS=
Domains=LXD
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#Cache=yes
#DNSStubListener=yes

After saving the file either restart your server or just systemctl restart systemd-resolved.service For me this worked just fine on a fresh Ubuntu-18.04.02 installation. If it doesn't work for you, I presume you either did a non-default lxd install or you are having problems/conflicts due to your os upgrade.

2. UPDATE (NIC specific setup):
According to Simos' Blog the only way to persist $ sudo systemd-resolve --interface lxdbr0 --set-dns 10.78.38.1 --set-domain lxd is creating a custom systemd service initializing the NIC specific configuration at system start. This way is currently the "accepted" approach on "semi-official" linuxcontainers.org

Example mentioned on the blog:
Shell script enabling nic specific dns via systemd-resolve (lxdhostdns_start.sh)

$ cat /usr/local/bin/lxdhostdns_start.sh 
#!/bin/sh

LXDINTERFACE=lxdbr0
LXDDOMAIN=lxd
LXDDNSIP=`ip addr show lxdbr0 | grep -Po 'inet \K[\d.]+'`

/usr/bin/systemd-resolve --interface ${LXDINTERFACE} \
                         --set-dns ${LXDDNSIP} \
                         --set-domain ${LXDDOMAIN}

Shell script disabling nic specific dns via systemd-resolve (lxdhostdns_stop.sh)

$ cat /usr/local/bin/lxdhostdns_stop.sh 
#!/bin/sh

LXDINTERFACE=lxdbr0

/usr/bin/systemd-resolve --interface ${LXDINTERFACE} --revert

Creating a systemd service to manage the two scripts and haven them initialized at system startup:

$ sudo cat /lib/systemd/system/lxd-host-dns.service 
[Unit]
Description=LXD host DNS service
After=multi-user.target

[Service]
Type=simple
ExecStart=/usr/local/bin/lxdhostdns_start.sh
RemainAfterExit=true
ExecStop=/usr/local/bin/lxdhostdns_stop.sh
StandardOutput=journal

[Install]
WantedBy=multi-user.target

Last but not least run systemctl enable lxd-host-dns.service and reboot host to verify the changes being persistent by running systemd-resolve --status. The lxbr0 interface should now be in dns scope (Current Scopes: DNS). Considering your network it should look like this:

Link 3 (lxdbr0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 10.78.38.1
          DNS Domain: lxd

I am well aware of it not being the ideal solution and technically it still doesn't give an answer to your question about persisting this config within systemd-resolve. The problem with persisting it however, lays in the fact that the lxbr0 interface isn't managed by netplan but by systemd on Ubuntu18.04. As netplan doesn't know about the interface, you cannot use the aformentioned *.network interface configurations to add dns and domain settings to lxbr0.

This answer is a long long wall of text now but I hope it at least clarifies the problem a little bit. Both solutions resolve container names on an ubuntu host. The second one is more accurate, but if the only requirement is name resolution on the host, making lxd's dnsmasq service globally available via /etc/systemd/resolved.conf is a viable alternative, especially as it only requires two edits in a config file.

View more solutions

Comments

[Piotr Dobrogost]

Why not asking this at unix.stackexchange.com?

[Piotr Dobrogost]

but there are no parameters available that would allow me to make such configuration specific to one particular NIC – there is such an option; see Name option inside [Match] section.

[quat]

This sounds plausible but this setting does not seem to have any effect on my system with LXD. After posting the initial question, I had created a Systemd service that executes the aforementioned command in order to configure DNS for LXD, which runs on boot. That does not reliably work either, since the lxdbr0 interface sometimes does not exist yet at that point in time. I therefore have to start the custom Systemd service manually. I guess, this might also be the reason for your solution not to work with LXD although it appears to be the right way to go...

[ctx]

You are right. But somehow I managed to get the desired output from systemd-resolve --status without entering that command. The other problem is, that name resolving does not work inside containers. I will fix my answer, if I find a better solution.

[quat]

Thank you for your suggestion. Unfortunately this does not quite hit the nail on its head. With the suggested setting, the DNS settings are global and not network interface (NIC) specific. If you were to run systemd-resolve --status with your settings in place, you will notice that the Link 10 (lxdbr0) section does not have the DNS configuration in place but instead its global. As of now, I believe that @ctx answer is the closest call superuser.com/a/1365864/433376 but possibly a bug is the reason for it not to work. Just now I have tested this with Ubuntu 19.04.

[tobi]

Yes, that's correct. The question is, what did you want to accomplish? If I understood you correct, then you wanted to have container names be resolved on the host and my suggestion is doing that. I mean, why does it have to be nic specific in your case? Oh, and in regards of ctx. I think it doesn't work with .conf in network manager as there is no .network defined anywhere in network manager for lxcbr0. At least I couldn't find one on my system.

[quat]

The scope of this question is to persist the settings that can be made by the use of the 'systemd-resolve' command (see question for the full command). YMMV but there are use cases where NIC specific DNS settings are required. In regards to the network manager, you might be right about this. Methinks this problem could be due to a mixup between the old style DNS resolution and the new systemd-resolve way of doing it.

[tobi]

Ok, you are right. My focus lay in finding a workaround by lifting the lxd dnsmasq server into global scope. I honestly doubt that there is a way to persist the systemd-resolve command. Basically because lxd not using netplan but systemd to configure the network on startup. In return you'd also need systemd to persist the changes. On linuxcontainers.org my claim finds some [backup] (discuss.linuxcontainers.org/t/…). I am going to update my answer with this information.

[Piotr Dobrogost]

This should work. The reason it does not work might be due to the wrong filename as it should have been lxdbr0.network not lxdbr0.conf.

[Wang]

on ubuntu 20.04, the file in/etc/systemd/network/ has no effect at all

[Anton Eliasson]

I don't think systemd-resolved needs to be restarted afterwards. In my case I needed to restart systemd-network.service instead, in order to get systemd-resolve --status to show the new configuration. Then I also needed to restart NetworkManager.service in order to get a new IPv4 address. This is on Ubuntu 20.04.

[Lethargos]

The correct directive is Domains, in plural, not Domain (talking about critical :) ). In my case it ignores the file completely. I'm not sure why. I have to keep trying.

[Lethargos]

as far as I can see [Resolve] doesn't exist as a section for the *.network files. freedesktop.org/software/systemd/man/systemd.network.html DNS and Domains should be set under [Network]

[Lethargos]

I was happy with the systemd script that I could copy/paste directly :D Better than nothing until I understand how this shitty systemd works. It doesn't make any sense :)