Where are NFS v4 logs under systemd?

Question

My error is:

mount.nfs4: access denied by server while mounting fileserver:/export/path/one

My question is:

where would the detailed log information be on the server (under systemd)?

More information:

I asked a similar question from the Ubuntu client perspective on AskUbuntu. My focus in this question is on the Arch Linux server. In particular, I am looking for logs on the server that will help me understand the problem.

Here's the background:

Our small LAN is running an Arch Linux NFS v4 file server. We have several clients running Ubuntu 15.10 and 16.04. We have one client running Ubuntu 14.04. The 14.04 client will not connect to the file server. The others all connect fine. The settings are the same on all clients. And all clients are listed in /etc/exports on the server.

I need to find more detailed error information on the Arch linux server. However, journalctl does not show anything related to nfs and it does not contain any entries that are related to the nfs access denied errors.

The 14.04 client can ping the fileserver as well as log in via SSH. The user name / ID as well as group match. (I'm using the same user account / uid on both client and server. It is uid 1000.)

Even more info:

$ sudo mount -a (on client)
mount.nfs4: access denied by server while mounting fileserver:/export/path/one
mount.nfs4: access denied by server while mounting fileserver:/export/path/two

The client can ping the fileserver (and vice versa):

$ ping fileserver
PING fileserver (192.168.1.1) 56(84) bytes of data.
64 bytes from fileserver (192.168.1.1): icmp_seq=1 ttl=64 time=0.310 ms

The client successfully logs into the LAN-based fileserver:

$ ssh fileserver
Last login: Tue Aug 16 14:38:26 2016 from 192.168.1.2
[me@fileserver ~]$ 

The fileserver's mount export and rpcinfo are exposed to the client:

$ showmount -e fileserver    # on client
Export list for fileserver:
/export/path/one/ 192.168.1.2
/export/path/two/ 192.168.1.2,192.168.1.3

$ rpcinfo -p fileserver (on client)
    program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  58344  status
    100024    1   tcp  58561  status
    100005    1   udp  20048  mountd
    100005    1   tcp  20048  mountd
    100005    2   udp  20048  mountd
    100005    2   tcp  20048  mountd
    100005    3   udp  20048  mountd
    100005    3   tcp  20048  mountd
    100003    4   tcp   2049  nfs
    100003    4   udp   2049  nfs

This is the error when mounting the export directly:

$ sudo mount -vvv -t nfs4 fileserver:/export/path/one /path/one/

    mount: fstab path: "/etc/fstab"
    mount: mtab path:  "/etc/mtab"
    mount: lock path:  "/etc/mtab~"
    mount: temp path:  "/etc/mtab.tmp"
    mount: UID:        0
    mount: eUID:       0
    mount: spec:  "fileserver:/export/path/one"
    mount: node:  "/path/one/"
    mount: types: "nfs4"
    mount: opts:  "(null)"
    mount: external mount: argv[0] = "/sbin/mount.nfs4"
    mount: external mount: argv[1] = "fileserver:/export/path/one"
    mount: external mount: argv[2] = "/path/one/"
    mount: external mount: argv[3] = "-v"
    mount: external mount: argv[4] = "-o"
    mount: external mount: argv[5] = "rw"
    mount.nfs4: timeout set for Tue Aug 16 16:10:43 2016
    mount.nfs4: trying text-based options 'addr=192.168.1.1,clientaddr=192.168.1.2'
    mount.nfs4: mount(2): Permission denied
    mount.nfs4: access denied by server while mounting fileserver:/export/path/one

Solution 1

I was having exactly the same problem, with both client and server Arch linux. The solution was to use the hostname in /etc/exports instead of the IP address. I changed this:

/srv/nfs 192.168.10(rw,fsid=root,no_subtree_check)
/srv/nfs/media 192.168.10(rw,no_subtree_check)
/srv/nfs/share 192.168.10(rw,no_subtree_check)

To this:

/srv/nfs iguana(rw,fsid=root,no_subtree_check)
/srv/nfs/media iguana(rw,no_subtree_check)
/srv/nfs/share iguana(rw,no_subtree_check)

This resulted in a slightly different problem:

[root@iguana data]# mount -t nfs4 frog:/srv/nfs/media /data/media
mount.nfs4: Protocol not supported

I don't have a lot of experience with NFS4; apparently you are not supposed to include the NFS root path in the mount command. This finally worked and mounted the volume:

[root@iguana data]# mount -t nfs4 frog:/media /data/media

Solution 2

It looks like the underlying problem was solved by @pgoetz, but for posterity I'll address the original question about how capture NFS logs (I had similar issues but couldn't find an answer about logging either!).

---

Detailed logging for both the NFS server and its clients can be obtained using rpcdebug, which will generate kernel logs (so they'll show up in, e.g. /var/log/messages, /var/log/syslog, etc. depending on your distro).

The general form is rpcdebug -m [module] -s [flags]:

• -m [module]: this specifies the module to log. This can be nfs, nfsd, rpc, or nlm -- for general NFS logs on the server, use nfsd; for general NFS logs on the client, use nfs
• -s [flags]: this sets the debugging flag(s) that you want the kernel to log. The flags available depend on the module

(see man rpcdebug for more information)

So, example commands to start logging all flags can be:

• NFS server: rpcdebug -m nfsd -s all
• NFS clients: rpcdebug -m nfs -s all

Note: this will be very verbose. The flags can be tweaked to get the right logging level, but starting with all flags can help narrow down the issue.

For fewer log entries, clear the all flag and then set specific flags to track, e.g.:

$ rpcdebug -m nfsd -c all    # clear all flags to stop logging them
$ rpcdebug -m nfsd -s auth proc  # set the auth and proc flags instead of logging all flags

Comments

[Stephen Harris]

Your showmount -e is showing a / character at the end of each directory. That might be confusing things. On fileserver ensure your exports says /export/path/one and not /export/path/one/

[Jarek]

I made the change to remove trailing slash from /etc/exports. I ran exportfs -ra and systemctl restart nfs-server.service. I still get the same error on the client.

[Mark Plotnick]

If you try to mount fileserver:/path/one , does that work?

[Jarek]

@MarkPlotnick - no, mount fileserver:/path/one does not work. Trying different permutations of the trailing slash doesn't seem to have any effect.

[Mark Plotnick]

OK. With NFSv4, it's customary to omit the /exports portion of the path when asking to mount the filesystem.

[Jarek]

@MarkPlotnick - Thanks again for your help. The Arch Wiki says including the server's export root is required for Debian/RHEL/SLES. I found that to be true for Ubuntu clienits as well (and this client is Ubuntu 14.04). However, I tried it without the export root just to see. In that case the error is "no such file or directory". WIth the export root added back, the error is again "access denied by server..."

[Mark Plotnick]

Thanks, good to know. At this point, I'd try doing a tcpdump on the file server, looking at traffic sent by the client, and see if there's any difference between a mount request from a working client and from the 14.04 client.

[Jarek]

I don't know how to use tcpdump. Furthermore, I don't know how to read the traffic. Sounds very advanced.

[pgoetz]

I'm having exactly the same problem and am similarly frustrated by the apparent lack of logging. journalctl -f shows nothing, journalctl -xe | grep nfs just shows that the nfs-server process is running properly. WIll report back here if I can figure it out.