How to ensure nfs-server is not started until it can resolve hostnames? (16.10)

Question

Summary: Is there a way to make sure that the NFS server isn't started by systemd until it can properly resolve client machine names specified in /etc/exports?

Problem description: I have found that NFS shares are not made properly available after the server (running 16.10) reboots. Clients get "access denied by server" errors, until exportfs -ra or service nfs-server restart is manually run on the server. After that, everything works as expected.

The server's /etc/exports contains only:

/mnt/raidarray clientmachine(rw)

where clientmachine is the hostname of the NFS client machine on the local network.

Problem identification: The output of systemctl status nfs-server (below) makes the problem clear: the name of the client can't be resolved at the time that the NFS server was started.

● nfs-server.service - NFS server and services
Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled
Active: active (exited) since Tue 2017-01-17 16:47:38 CST; 26min ago
Main PID: 1520 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 4915)
CGroup: /system.slice/nfs-server.service
Jan 17 16:47:38 servermachine exportfs[1511]: exportfs: Failed to resolve clientmachine
Jan 17 16:47:38 servermachine systemd[1]: Started NFS server and services.

NetworkManager-wait-online.service is enabled, which I had understood serves to make sure that network.target (on which nfs-server depends) is not satisfied until network-online.target is satisfied.

Things that did not help:

1. In case I misunderstood what NetworkManager-wait-online.service does, I tried adding an explicit After=network-online.target and Wants=network-online.target to nfs-server.service. This doesn't fix anything. I see the new dependency show up in systemctl list-dependencies, but name resolution still fails at boot. So it seems that network-online.target doesn't guarantee that hostname resolution can happen.

2. Some googling suggested that requiring nss-lookup.target would ensure that network resolution is available, but adding it as a Wants and/or After dependency to nfs-server.service doesn't fix things either!

3. Adding a Wants and/or After dependency on systemd-resolved.service instead of nss-lookup.target doesn't fix things either.

After adding all of these dependencies, nfs-server starts up very very late in the boot process (just before the desktop login things), but it still can't resolve hosts. Based on systemd-analyze plot, it appears that nmbd is wedged around this time, but I don't know if that's related.

Configuration information: This is on a desktop version of kubuntu 16.10 that is basically a fresh install.

NetworkManager.service is enabled and systemd-networkd.service is disabled -- I didn't change that from the default.

Here is the NetworkManager-wait-online service definition:

[Unit]
Description=Network Manager Wait Online
Documentation=man:nm-online(1)
Requisite=NetworkManager.service
After=NetworkManager.service
Before=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/bin/nm-online -s -q --timeout=30
RemainAfterExit=yes
[Install]
WantedBy=network-online.target

As a workaround I could hard-code IP addresses instead of hostnames into /etc/exports or /etc/hosts, but this seems somewhat brittle. Is there a better answer?

EDIT - Update: following @muru's advice below, I tried making a script to wait to resolve hostnames -- just to see how long it takes. My script has to wait tens of seconds after systemd-resolved starts before it can actually resolve a host. This is very bizarre. I wonder if it's actually a local networking issue? Sounds like maybe a hard-coded (and auto-updating, perhaps, as suggested by @mark-stosberg) /etc/exports or /etc/hosts file is warranted.

Solution

Based on @muru's comment, I made a python script to wait until DNS resolution:

import socket
import time
import itertools

TIMEOUT = 30
HOSTS = ['stanford.edu', 'google.com', 'example.com']

def main():
    hosts = itertools.cycle(HOSTS)
    t0 = time.time()
    for host in hosts:
        t = time.time()
        elapsed = t - t0
        if elapsed > TIMEOUT:
            break
        try:
            socket.getaddrinfo(host, None, proto=socket.IPPROTO_TCP)
            print('Resolved {} at t = {}'.format(host, elapsed))
            break
        except socket.gaierror:
            print('Could not resolve {} at t = {}'.format(host, elapsed))
        time.sleep(0.25)
        t = time.time()

if __name__ == '__main__':
    main()

I saved that script as /etc/systemd/system/nfs-server.service.d/wait_for_dns.py (one has to make the parent directory first), and then ran sudo systemctl edit --full nfs-server, adding the following before other ExecStartPre lines:

ExecStartPre=/usr/bin/python3 /etc/systemd/system/nfs-server.service.d/wait_for_dns.py

This worked fine, though it certainly feels a bit hackish and could be improved in many ways. (I eventually just gave up on NFS in this context; it was more of a pain than it was worth.)

Comments

[Admin]

Hack: Create a script which waits for successful DNS resolution, then use that as an ExecStartPre for the nfs-server service

[Admin]

Welcome to Ask Ubuntu! ;-) Could you do a systemd-analyze plot > /tmp/systemd.svg, post a link to systemd.svg and post the contents of /etc/systemd/system/network-online.target.wants/NetworkManag‌​er-wait-online.servi‌​ce as well? then drop me a note @fabby

[Admin]

Hrmpf! What you already did was what I was going to propose... BTW, you should edit your question and provide the above critical info there so the next user looking at your question doesn't have to dig through the entire comment section.

[Admin]

@muru IMNSHO, your suggestion is the best one. Could you provide an answer for that??? If you ping me, I'll come back and upvote! ;-)

[Admin]

Why hard-code and auto-update /etc/hosts instead of /etc/exports itself?

[Admin]

If /etc/exports is hard-coded, it will just contain the raw IP addresses -- so you need a template file (or special comment lines) to store the hostname to look up. That's a reasonable solution. Alternately, if /etc/hosts is used, that stores the hostname and IP both, and then the automatic updater would just need to revise the hostname to IP mapping. Both seem like reasonable approaches that might be useful in different situations.

[zachrahan]

Thanks! I'm not sure which seems like a more robust / less kludgey solution: the suggestion from @muru to make a script that waits for successful name resolution as an ExecStartPre component of nfs-server.service, or make a cron script to edit /etc/hosts with the latest DNS information for hosts of interest. Thoughts anyone?

[Mark Stosberg]

If you wait for successful name resolution in ExecStartPre, it's possibly it will take very long or maybe never happen. If you update /etc/hosts with valid DNS entries only when valid DNS queries can be completed, the file will be maximally up to date.

[Fabby]

Copy-paste of comments as an answer. -1

[Mark Stosberg]

@Fabby I'm not sure what you are referring to- I didn't copy/paste anything.

[Fabby]

A "virtual" copy-paste. You read through the comments and reworded them as an answer without adding anything of value is what I meant... ¯\_(ツ)_/¯

[Mark Stosberg]

@Fabby, No, I did not. And if someone had a great answer, they should have left it as an answer, not a comment!

[zachrahan]

@Fabby This answer was useful to me, at least, so it couldn't have been a total copy-paste of what I said. (And nobody other than I said anything about /etc/hosts...)The idea of automatically updating /etc/hosts was a good one, too, and hadn't occurred to me. Plus the context of this being a common systemd bugaboo is useful.

[Fabby]

If the post is edited I can upvote instead of downvote... Leave one more comment after edit

[Mark Stosberg]

I've (trivially) editted the post.