systemd: failed to mount NFS share: mount.nfs: Network is unreachable until later in boot process

networking boot systemd nfs usb-device

25,080

Solution 1

This works for me with systemd. The automount, requires delays the mount request until the network is online.

I am not sure about the nofail. I use it without issue but it doesn't seem quite right. It should be unnecessary with the requires... Perhaps someone has an idea.

10.0.0.110:/export/3T  /mnt/3T  nfs nofail,x-systemd.automount,x-systemd.requires=network-online.target,x-systemd.device-timeout=10 0 0

Solution 2

On CentOS 7 network filesystem mounts depend on network-online.target.
The network-online.target is reached once an interface is up and an ip has been set.
This presumes the first ip on the first interface to get up is enough for the hostname to be resoved. This is however doesn't have to be the case.

We can write systemd services to test if network is supposed to be flagged as "up" and make the network-online.target dependent on that service.

For a template look at: https://gitlab.com/ggeurts/extend-network-online.target

Solution 3

I had similar issues, in particular a dependency for NFS mounts on network-online.target.

This is particularly problematic as with my Ubuntu desktop network-online.target is controlled by NetworkManager and this will indicate that network-online.target has been reached even before any DNS or default route are installed via DHCP. The problem here is that any mount.nfs command issued, from either generator units created via fstab, or .mount units manually crafted will fail, with "Network unreachable" if the IP layer does not have a default route to the remote target.

I wanted a system that would reliably boot without delay, bringing me to the desktop regardless of if NFS had successfully mounted or not. Furthermore I wanted NFS to automatically establish mounts, if and when the network became available or the NFS server suddenly appeared on the network.

To achieve this goal was harder than I expected, but I shall share my experience as others may find it useful.

I have removed all mount definitions from fstab apart from '/'. I have a feeling that fstab is on the decline and eventually we will see all mount actions performed naively with systemd. Even though the generator does make the use of fstab 'easy'. Therefore I moved all my NFS mounts into mount units that I created myself. Also I found that Nautilus or Nemo, choose your poison, will not show NFS mounts in the left 'Devices' pane, unless they are either mounted in a user home directory or they are mounted in /media. When I finally have the system up and running naturally I wish to use the GUI tools to manage files. To this end I used a combination of a single NFS mount onto the server and then a cluster of mount units that --bind mount each of the exports from the server to a location I wanted on my desktop machine. When a bind mount is created onto an underlying NFS mount point, systemd actually creates an NFS mount behind the scenes. Therefore I only have a single NFS mount started by the system which attaches the root of the NFS server into /mnt.

To overcome the dependencies at boot time and to stop any NFS mount from timing out if network or server is not available I initiate the NFS mount operation from a mnt-NFS-mx.service unit.

[Unit]
Description=Attempting to mount server 'mx' using NFS

After=network.target

[Service]
Type=oneshot

# Add a high metric default to 'lo' interface into route table.
# This prevents the mount command from failing with "Network unreachable" is case no external phy link is available.
ExecStart=-/bin/ip route add default dev lo metric 4294967295
ExecStart=/bin/mount 10.0.0.10:/mnt/ /mnt/NFS/mx -t nfs -o nolock,nfsvers=4,fg,retry=10000,hard,timeo=20,retrans=1

# Remove the default once the mount command is successful
ExecStop=-/bin/ip route del default dev lo metric 4294967295
ExecStopPost=-/bin/ip route del default dev lo metric 4294967295

[Install]
WantedBy=network.target

This is pulled in by network.target at boot time and must start after network.target as it adds a default route, to prevent the mount command failing when no default is found. A suitably high metric is used, that will not interfere with a genuine default when acquired (FYI, this metric is max). It then calls NFS mount and the service will stay in 'activating' while the mount command runs. If it mounts the NFS server the service stops and removes the superfluous default route. If not, it hangs around ... forever (approx 1 week), waiting for the server. But it does not prevent any further targets being reached on the system. Note this is an NFSv4 only environment. If earlier versions are needed, you must ensure that all the RPC infrastructure is started already. I have all these RPC services masked, however NFS mount will still fail if it cant find parts of this legacy environment as it still looks to interface with RPC's file locking, even though NFSv4 does not use it. Therefore 'nolock' is needed in the mount command. It seems this is simply a hangover in the mount implementation of NFS. Also I have adjusted the 'timeo' and 'retrans' options to a more aggressive level, helping to make sure that the server is quickly found when a 'useable' default route becomes available. With the specified settings, the client waits 2sec for a response, and if none arrives it does not retransmit the same request. This ensures the client does not enter exponential back-off, which would otherwise result in the retransmission intervals growing between each attempt to find the server. However as the retry timer is set at 10000mins, it immediately tries again with a new attempt using the same maximum timeout of 2Sec.... and on and on until it gets a connection. On my my system I have even seen all NFS mounts in place and ready even before the network-online.target is reached.

Assuming that at some stage NFS mounts. I have the 'mx' server root @ /mnt/NFS/mx/. The activation of my mnt-NFS-mx.service unit triggers a series of bind mounts due to the 'WantedBy=mnt-NFS-mx.service' in the install section. An example of this see below: media-NAS_Public.mount

[Unit]
# To make remote NFS shares appear under the devices pane of Nemo
Description=--bind NFS NAS /Public onto /media/NAS_Public

DefaultDependencies=no

After=mnt-NFS-mx.service

[Mount]
What=/mnt/NFS/mx/shared_sun_NAS/Public
Where=/media/NAS_Public
Type=none
Options=bind,_netdev
TimeoutSec=0

[Install]
WantedBy=mnt-NFS-mx.service

Important to note in this mount unit is the use of 'DefaultDependencies=no' in combination with Option '_netdev'. The _netdev is normally used to ensure systemd understands that this mount unit is actually network related, and not a part of the local file system. However, this adds a series of default dependencies of After= on remote-fs-pre.target, network.target and network-online.target and importantly a Before= on remote-fs.target. This Before= is troublesome as this prevents the system reaching the remote-fs.target if these mount units do not either succeed or TIMEOUT!!! Hence the use of 'DefaultDependencies=no', this removes these dependencies and prevents the boot hanging if this unit is not started directly at boot. I left the _netdev option in place just as an aide memoir, that this bind mount is actually going to create an NFS mount directly to the mount point destination.

I created a bunch of these .mount units both into /media and also directly into my home directory for my personal Documents directory etc. As a final catch all I ensure that any mount point on my Desktop machine is created in advance and that the immutable flag is set (chattr +i /media/NAS_Public) on these directories to ensure that nothing can be written into these locations if the mount procedure fails.

Although not important in my case, it should be clear that using 'DefaultDependencies=no' for these bind mounts will prevent any dependency logic for applications that MUST have an NFS connection from being aware of failure. Having said that it could be possible to use an After=mnt-NFS-mx.service as a substitute. I have not tried this.

At the end of the day for my purposes, this gives me a stable and fast booting system, regardless of the state of my network or server, with the bonus that if a server fault is rectified, the mounts will suddenly appear.

Hope someone else finds this useful.

25,080

Jarek

You may be interested in the story of SE moderator Monica Cellio and how she was unfairly treated by the corporate management of this site. More info here. An update is available. Let's hope we can cultivate a more fair environment for content creators and moderators going forward.

Updated on September 18, 2022

Comments

Jarek over 1 year

My laptop has a USB Ethernet adapter. Apparently the network doesn't come up until later in the boot process than normal. My NFS shares are not mounting at boot. However, as soon as the laptop is booted up, I can mount all shares with this command:

mount -a -t nfs,nfs4

There are no errors and everything is fine at that point. However, I would like the shares to mount at boot time automatically. Therefore, this is not an NFS configuration question. I believe this issue is related to systemd, the boot process and maybe the timing of the USB Ethernet adapter availability.

Here is everything I know to check including the relevant parts of the journal showing that the network comes up after the NFS shares have given up trying to mount. I included the relevant config files, but they have not been changed from the defaults (as far as I know).

Jul 31 21:22:32 host systemd[1]: Mounted /var/log.
...
Jul 31 21:22:33 host mount[898]: mount.nfs: Network is unreachable
...
Jul 31 21:22:34 host mount[880]: mount.nfs: Resource temporarily unavailable
Jul 31 21:22:34 host systemd[1]: share1.mount: Mount process exited, code=exited status=32
Jul 31 21:22:34 host systemd[1]: share1.mount: Failed with result 'exit-code'.
Jul 31 21:22:34 host autossh[896]: starting ssh (count 9)
Jul 31 21:22:34 host autossh[896]: ssh child pid is 1033
Jul 31 21:22:34 host systemd[1]: Failed to mount /share1.
...
Jul 31 21:22:37 host autossh[899]: ssh: connect to host 10.10.0.201 port 22: Network is unreachable
Jul 31 21:22:37 host autossh[899]: ssh exited with error status 255; restarting ssh
...
Jul 31 21:22:38 host NetworkManager[792]: <info>  [1533086558.3237] manager: NetworkManager state is now CONNECTING
...
Jul 31 21:22:38 host NetworkManager[792]: <info>  [1533086558.3487] manager: NetworkManager state is now CONNECTED_LOCAL
Jul 31 21:22:38 host NetworkManager[792]: <info>  [1533086558.3492] manager: NetworkManager state is now CONNECTED_SITE
Jul 31 21:22:38 host NetworkManager[792]: <info>  [1533086558.3493] policy: set 'Wired connection 1' (eth1) as default for IPv4 routing and DNS
Jul 31 21:22:38 host NetworkManager[792]: <info>  [1533086558.3497] device (eth1): Activation: successful, device activated.
Jul 31 21:22:38 host nm-dispatcher[1017]: req:3 'up' [eth1]: new request (0 scripts)


# systemctl status  NetworkManager.service 
● NetworkManager.service - Network Manager
Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/NetworkManager.service.d
        └─NetworkManager-ovs.conf
Active: active (running) since Tue 2018-07-31 21:22:33 EDT; 42min ago
    Docs: man:NetworkManager(8)
Main PID: 792 (NetworkManager)
    Tasks: 3 (limit: 4915)
Memory: 33.4M
CGroup: /system.slice/NetworkManager.service
        └─792 /usr/bin/NetworkManager --no-daemon

Jul 31 22:00:59 host NetworkManager[792]: <info>  [1533088859.6259] dhcp4 (eth1):   gateway 192.168.0.1
Jul 31 22:00:59 host NetworkManager[792]: <info>  [1533088859.6334] dhcp4 (eth1): state changed bound -> bound
Jul 31 22:03:12 host NetworkManager[792]: <info>  [1533088992.7755] dhcp4 (eth1):   address 192.168.0.237
Jul 31 22:03:12 host NetworkManager[792]: <info>  [1533088992.7756] dhcp4 (eth1):   plen 24
Jul 31 22:03:12 host NetworkManager[792]: <info>  [1533088992.7756] dhcp4 (eth1):   expires in 300 seconds
Jul 31 22:03:12 host NetworkManager[792]: <info>  [1533088992.7757] dhcp4 (eth1):   nameserver '192.168.0.1'
Jul 31 22:03:12 host NetworkManager[792]: <info>  [1533088992.7758] dhcp4 (eth1):   domain name 'oaks'
Jul 31 22:03:12 host NetworkManager[792]: <info>  [1533088992.7758] dhcp4 (eth1):   hostname 'host'
Jul 31 22:03:12 host NetworkManager[792]: <info>  [1533088992.7759] dhcp4 (eth1):   gateway 192.168.0.1
Jul 31 22:03:12 host NetworkManager[792]: <info>  [1533088992.7769] dhcp4 (eth1): state changed bound -> bound


# systemctl status share1.mount
● share1.mount - /share1
Loaded: loaded (/etc/fstab; generated)
Active: failed (Result: exit-code) since Tue 2018-07-31 21:22:34 EDT; 25min ago
    Where: /share1
    What: server:/share1/
    Docs: man:fstab(5)
        man:systemd-fstab-generator(8)

Jul 31 21:22:33 host systemd[1]: Mounting /share1...
Jul 31 21:22:34 host mount[880]: mount.nfs: Resource temporarily unavailable
Jul 31 21:22:34 host systemd[1]: share1.mount: Mount process exited, code=exited status=32
Jul 31 21:22:34 host systemd[1]: share1.mount: Failed with result 'exit-code'.
Jul 31 21:22:34 host systemd[1]: Failed to mount /share1.


# cat /usr/lib/systemd/system/NetworkManager.service
[Unit]
Description=Network Manager
Documentation=man:NetworkManager(8)
Wants=network.target
After=network-pre.target dbus.service
Before=network.target 

[Service]
Type=dbus
BusName=org.freedesktop.NetworkManager
ExecReload=/usr/bin/dbus-send --print-reply --system --type=method_call --dest=org.freedesktop.NetworkManager /org/freedesktop/NetworkManager org.freedesktop.NetworkManager.Reload uint32:0
#ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/bin/NetworkManager --no-daemon
Restart=on-failure
# NM doesn't want systemd to kill its children for it
KillMode=process
CapabilityBoundingSet=CAP_NET_ADMIN CAP_DAC_OVERRIDE CAP_NET_RAW CAP_NET_BIND_SERVICE CAP_SETGID CAP_SETUID CAP_SYS_MODULE CAP_AUDIT_WRITE CAP_KILL CAP_SYS_CHROOT

# ibft settings plugin calls iscsiadm which needs CAP_SYS_ADMIN
#CapabilityBoundingSet=CAP_SYS_ADMIN

ProtectSystem=true
ProtectHome=read-only

[Install]
WantedBy=multi-user.target
Alias=dbus-org.freedesktop.NetworkManager.service
Also=NetworkManager-dispatcher.service

# We want to enable NetworkManager-wait-online.service whenever this service
# is enabled. NetworkManager-wait-online.service has
# WantedBy=network-online.target, so enabling it only has an effect if
# network-online.target itself is enabled or pulled in by some other unit.
Also=NetworkManager-wait-online.service

# cat /usr/lib/systemd/system/NetworkManager.service.d/NetworkManager-ovs.conf 
[Unit]
After=openvswitch.service

# cat /etc/NetworkManager/NetworkManager.conf
# Configuration file for NetworkManager.
# See "man 5 NetworkManager.conf" for details.

# tree /etc/NetworkManager/conf.d/
/etc/NetworkManager/conf.d/
0 directories, 0 files

# cat /usr/lib/systemd/system/NetworkManager-wait-online.service
[Unit]
Description=Network Manager Wait Online
Documentation=man:nm-online(1)
Requires=NetworkManager.service
After=NetworkManager.service
Before=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/nm-online -s -q --timeout=30
RemainAfterExit=yes

[Install]
WantedBy=network-online.target


Arch Linux 4.17.11-arch1 #1 SMP PREEMPT Sun Jul 29 10:11:16 UTC 2018 x86_64 GNU/Linux
KDE

UPDATE: here is the requested info regarding fstab:

All nfs mounts have these options:

nfs     _netdev,defaults,noatime,nodiratime,soft,retrans=6,timeo=20,retry=0,rsize=32768,wsize=32768,proto=tcp   0 0

I am indeed using _netdev along with systemd. After reading Filipe Brandenburger's comment, I'm not sure whether this option is meaningless under systemd or harmful.

systemctl cat share1.mount
# /run/systemd/generator/share1.mount
# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=remote-fs.target

[Mount]
Where=/share1
What=server:/share1/
Type=nfs
Options=_netdev,defaults,noatime,nodiratime,soft,retrans=6,timeo=20,retry=0,rsize=32768,wsize=32768,proto=tcp

# systemctl show -p Wants network-online.target
Wants=NetworkManager-wait-online.service

Here is an update with the information requested by @sourcejedi

sudo systemctl list-dependencies share1.mount
share1.mount
● ├─-.mount
● ├─home.mount
● ├─system.slice
● └─network-online.target

sudo systemctl list-dependencies --after share1.mount
share1.mount
● ├─-.mount
● ├─home.mount
● ├─system.slice
● ├─systemd-journald.socket
● ├─network-online.target
● │ └─network.target
● │   ├─dhcpcd.service
● │   ├─NetworkManager.service
● │   ├─wpa_supplicant.service
● │   └─network-pre.target
● ├─network.target
● │ ├─dhcpcd.service
● │ ├─NetworkManager.service
● │ ├─wpa_supplicant.service
● │ └─network-pre.target
● └─remote-fs-pre.target

# sudo journalctl -b -u share1.mount -u network-online.target -u NetworkManager-wait-online -u NetworkManager --no-pager
-- Logs begin at Sun 2018-06-10 17:09:08 EDT, end at Sat 2018-12-22 19:07:16 EST. --
Dec 22 17:58:38 laptop1 systemd[1]: Starting Network Manager...
Dec 22 17:58:38 laptop1 NetworkManager[463]: <info>  [1545519518.3825] NetworkManager (version 1.14.5dev+17+gba83251bb-1) is starting... (for the first time)
Dec 22 17:58:38 laptop1 NetworkManager[463]: <info>  [1545519518.3826] Read config: /etc/NetworkManager/NetworkManager.conf (lib: 20-connectivity.conf)
Dec 22 17:58:38 laptop1 NetworkManager[463]: <info>  [1545519518.3836] wifi-nl80211: (wlan0): using nl80211 for WiFi device control
Dec 22 17:58:38 laptop1 NetworkManager[463]: <info>  [1545519518.3895] bus-manager: acquired D-Bus service "org.freedesktop.NetworkManager"
Dec 22 17:58:38 laptop1 systemd[1]: Started Network Manager.
Dec 22 17:58:38 laptop1 NetworkManager[463]: <info>  [1545519518.3900] manager[0x55b8543f4060]: monitoring kernel firmware directory '/lib/firmware'.
Dec 22 17:58:38 laptop1 systemd[1]: Reached target Network is Online.
Dec 22 17:58:38 laptop1 systemd[1]: share1.mount: Directory /share1 to mount over is not empty, mounting anyway.
Dec 22 17:58:38 laptop1 systemd[1]: Mounting /share1...
Dec 22 17:58:39 laptop1 mount[546]: mount.nfs: Network is unreachable
Dec 22 17:58:39 laptop1 systemd[1]: share1.mount: Mount process exited, code=exited status=32
Dec 22 17:58:39 laptop1 systemd[1]: share1.mount: Failed with result 'exit-code'.
Dec 22 17:58:39 laptop1 systemd[1]: Failed to mount /share1.
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.3695] hostname: hostname: using hostnamed
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.3696] hostname: hostname changed from (none) to "laptop1"
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.3698] dns-mgr[0x98059644079A]: init: dns=default, rc-manager=symlink
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.3708] rfkill2: found WiFi radio killswitch (at /sys/devices/pci0000:00/0000:00:1c.1/0000:03:00.0/ieee80211/phy0/rfkill2) (driver iwlwifi)
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.3711] manager[0x55b8543f4060]: rfkill: WiFi hardware radio set disabled
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.3711] manager[0x55b8543f4060]: rfkill: WWAN hardware radio set enabled
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.3926] settings: Loaded settings plugin: NMSKeyfilePlugin (internal)
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.3933] settings: Loaded settings plugin: NMSIbftPlugin ("/usr/lib/NetworkManager/1.14.5dev+17+gba83251bb-1/libnm-settings-plugin-ibft.so")
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.3958] keyfile: new connection /etc/NetworkManager/system-connections/USB Ethernet Adapter 1 (fc9310,"USB Ethernet Adapter 1")
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.3975] keyfile: new connection /etc/NetworkManager/system-connections/walden2
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.3983] keyfile: new connection /etc/NetworkManager/system-connections/NETGEAR4
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.3991] keyfile: new connection /etc/NetworkManager/system-connections/Tls5
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.3999] keyfile: new connection /etc/NetworkManager/system-connections/JONES
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4062] manager: rfkill: WiFi disabled by radio killswitch; disabled by state file
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4063] manager: rfkill: WWAN enabled by radio killswitch; enabled by state file
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4064] manager: Networking is enabled by state file
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4066] dhcp-init: Using DHCP client 'internal'
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4129] Loaded device plugin: NMAtmManager (/usr/lib/NetworkManager/1.14.5dev+17+gba83251bb-1/libnm-device-plugin-adsl.so)
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4210] Loaded device plugin: NMBluezManager (/usr/lib/NetworkManager/1.14.5dev+17+gba83251bb-1/libnm-device-plugin-bluetooth.so)
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4232] Loaded device plugin: NMOvsFactory (/usr/lib/NetworkManager/1.14.5dev+17+gba83251bb-1/libnm-device-plugin-ovs.so)
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4404] Loaded device plugin: NMTeamFactory (/usr/lib/NetworkManager/1.14.5dev+17+gba83251bb-1/libnm-device-plugin-team.so)
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4420] Loaded device plugin: NMWifiFactory (/usr/lib/NetworkManager/1.14.5dev+17+gba83251bb-1/libnm-device-plugin-wifi.so)
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4427] Loaded device plugin: NMWwanFactory (/usr/lib/NetworkManager/1.14.5dev+17+gba83251bb-1/libnm-device-plugin-wwan.so)
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4435] device (lo): carrier: link connected
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4439] manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1)
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4449] manager: (eth0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2)
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4461] device (eth0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4481] manager: (eth1): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4491] device (eth1): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4504] device (wlan0): driver supports Access Point (AP) mode
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4508] manager: (wlan0): new 802.11 WiFi device (/org/freedesktop/NetworkManager/Devices/4)
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4513] device (wlan0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4516] device (wlan0): set-hw-addr: set MAC address to 28:ab:29:45:F9:2B (scanning)
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4586] bluez: use BlueZ version 5
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.4656] bluez5: NAP: added interface 77:ab:29:45:F9:2F
Dec 22 17:58:39 laptop1 NetworkManager[463]: <info>  [1545519519.5063] supplicant: wpa_supplicant running
Dec 22 17:58:42 laptop1 NetworkManager[463]: <info>  [1545519522.0759] device (eth1): carrier: link connected
Dec 22 17:58:42 laptop1 NetworkManager[463]: <info>  [1545519522.0792] device (eth1): state change: unavailable -> disconnected (reason 'carrier-changed', sys-iface-state: 'managed')
Dec 22 17:58:42 laptop1 NetworkManager[463]: <info>  [1545519522.0798] policy: auto-activating connection 'USB Ethernet Adapter 1' (fc9310)
Dec 22 17:58:42 laptop1 NetworkManager[463]: <info>  [1545519522.0804] device (eth1): Activation: starting connection 'USB Ethernet Adapter 1' (fc9310)
Dec 22 17:58:42 laptop1 NetworkManager[463]: <info>  [1545519522.0806] device (eth1): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Dec 22 17:58:42 laptop1 NetworkManager[463]: <info>  [1545519522.0810] manager: NetworkManager state is now CONNECTING
Dec 22 17:58:42 laptop1 NetworkManager[463]: <info>  [1545519522.0895] device (eth1): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Dec 22 17:58:42 laptop1 NetworkManager[463]: <info>  [1545519522.0899] device (eth1): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Dec 22 17:58:42 laptop1 NetworkManager[463]: <info>  [1545519522.0902] dhcp4 (eth1): activation: beginning transaction (timeout in 45 seconds)
Dec 22 17:58:45 laptop1 NetworkManager[463]: <info>  [1545519525.7908] device (eth1): carrier: link connected
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.1025] dhcp4 (eth1):   address 192.168.5.2
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.1026] dhcp4 (eth1):   plen 24
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.1026] dhcp4 (eth1):   expires in 108000 seconds
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.1027] dhcp4 (eth1):   nameserver '192.168.5.1'
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.1027] dhcp4 (eth1):   domain name 'wrkgrp'
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.1027] dhcp4 (eth1):   hostname 'usb_laptop1'
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.1028] dhcp4 (eth1):   gateway 192.168.5.1
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.1040] dhcp4 (eth1): state changed unknown -> bound
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.1075] device (eth1): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.1108] device (eth1): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.1117] device (eth1): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.1134] manager: NetworkManager state is now CONNECTED_LOCAL
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.1153] manager: NetworkManager state is now CONNECTED_SITE
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.1155] policy: set 'USB Ethernet Adapter 1' (eth1) as default for IPv4 routing and DNS
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.1161] device (eth1): Activation: successful, device activated.
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.1167] manager: startup complete
Dec 22 17:58:47 laptop1 NetworkManager[463]: <info>  [1545519527.3936] manager: NetworkManager state is now CONNECTED_GLOBAL

(In regard to my controversial use of the soft mount option, the decision to use it was made after years of problems with the hard option. With the hard option, any loss of connectivity freezes the system and forces a hard restart. I'm running BTRFS and I do not wish to risk that. I have decided that my soft is a better fit for my situation. However, along with soft I tweaked the retrans,timeo,and retry options. I'm not sure I have them optimized, but overall the experience has been much better than with the hard option.)

ErikF almost 6 years

What does your /etc/fstab look like? Are you using _netdev as a mount option?
filbranden almost 6 years

According to systemd.mount(5), systemd will detect network filesystems automatically, not requiring _netdev on those. It also explains what it does for network filesystems. Can you run systemctl cat share1.mount? Also, systemctl show -p Wants network-online.target (to confirm it's pulling NetworkManager and not systemd-networkd)?
Jarek almost 6 years

question updated with the info requested. Thanks for your interest in my question.
sergej over 5 years

This might help askubuntu.com/a/1069863/117559
Jarek over 5 years

systemd auto mount causes big problems on laptops which are sometimes not connected to the network where the shares are located. For example, in that situation, if you click on a mount point in the file browser, your whole system will hang. Users may not know it is a mount point, or you may click by accident, but the result is disastrous. So I don't use systemd auto mount.
user2948306 over 5 years

sudo journalctl -b -u share1.mount -u network-online.target -u NetworkManager-wait-online -u NetworkManager should show the relative timings, I don't think you've shown all of them. You can also check systemctl list-dependencies share1.mount, and systemctl list-dependendencies --after share1.mount (yes, --after, not --before, it's sort of the wrong way round). But, can you also check through sudo journalctl -b -p err, and check you don't have an error from systemd, maybe about a dependency "loop" / "cycle" ?
Jarek over 5 years

@sourcejedi - please see updated question with the outputs you suggested.
user2948306 over 5 years

@MountainX I don't see any mention of the error log I requested. If you mean that it didn't show anything, please say so. / I can't find any mention of NetworkManager's wait-online service in the log you did post, it does not even say "Starting ...". / The log showing network-online.target started immediately after NetworkManager, and also the absence of an ordering dependency between network-online.target and NetworkManager-wait-online.service in list-dependencies --after, seems very suspicious.
sagarcool89 about 4 years

I have the same issue under Debian Buster. Looks like we have a very similar setup: On my system NFS /home is successfully mounted while NFS /mnt/share is not. This is reproducable and doesn't seem to be a race condition. systemctl list-dependencies shows the same item including network-online.target. But only for the /mnt/share mount fails with Network is unreachable. The unit is created correctly from /etc/fstab but the condition is not awaited properly:-(

Nur almost 6 years

You could also check this link, which might help you to find a solution ervikrant06.wordpress.com/2014/09/30/…
Jarek almost 6 years

Sorry, I should have mentioned in my question that I am already using _netdev in /etc/fstab. I added that mount option to all NFS shares from the beginning. Any idea why it would not have expected effect?
Jarek almost 6 years

thanks for the link, but I'm running systemd and I don't see anything in there that applies to systemd.
Jarek about 4 years

Thank you for the template.
Florian Heigl about 4 years

this works and it is just hilarious how many options need to be added to this thing to solve a task from the early 90s.
Mikaelblomkvistsson about 4 years

I cannot find any spec about x-systemd.device-ut=10 what's this?
Stephen Boston about 4 years

@Mikaelblomkvistsson . I asked about that as well askubuntu.com/questions/1142887/… and received no response. I think it must be a typing or paste error for x-systemd.device-timeout= which I see is what my fstab has now. manpages.ubuntu.com/manpages/xenial/man5/systemd.mount.5.htm‌l Thanks. I'll update it.