mount.nfs: access denied by server while mounting

networking 13.10 nfs debug

67,761

Solution 1

I had the same issue as the OP. This used to work for me but then after an update it didn't. The solution was to add the "nfsvers=3" option.

Solution 2

It could be worth mentioning (even if it's probably not the case here) that if you're suddenly getting this error after renaming a folder or attempting to access a new one, you might have forgotten that you actually have to specify server-side what folders are shared and what aren't in /etc/exports -- this was the issue in my case.

After you've modified that file, you also need to do sudo service nfs-kernel-server restart

Solution 3

My two colleagues at work just solved a similar issue within Vagrant. AppArmor was the culprit, solution can be found here, citing:

mount: cannot mount block device /some/path read-only

and dmesg shows:

[ 6944.194280] type=1400 audit(1385049795.420:32): apparmor="DENIED" operation="mount" info="failed type match" error=-13 parent=6631 profile="lxc-container-default" name="/some/other/path" pid=6632 comm="mount" srcname="/some/path" flags="rw, bind"

AppArmor is blocking mount -o bind inside the LXC container.

To enable id add in /etc/apparmor.d/lxc/lxc-default:

profile lxc-container-default flags=(attach_disconnected,mediate_deleted) {
  ...
    mount options=(rw, bind),
  ...

Reload apparmor:

# /etc/init.d/apparmor reload

67,761

puk

Updated on September 18, 2022

Comments

puk over 1 year

I have a problem I have been googling for at least ~~6 hours~~ 28 hours with no luck. Whenever I try to mount a directory I get the following error (verbose version at end):

mount.nfs: access denied by server while mounting :

Initally I was using computer1 as the nfs server, but I have moved it over to computer2 and I am trying to mount on computers3-8. As a further note, computer2 is now acting as the dhcp router for all other computers (even computer1), but I am almost certain that the problem has to do with computer2's nfs settings as I have tried the following:

Try mounting a folder from the original computer1 server to one of the computers3-8 clients -- SUCCESS
Try mounting a folder from the original computer1 server to the new computer2 server -- SUCCESS
Try mounting a folder from the new computer2 server to one of the computers3-8 clients -- FAILURE
Try mounting a folder from the new computer2 server to the old computer1 server -- FAILURE
Try exporting then mounting any other folder on computer2 -- FAILURE
ssh to and from computer2 -- SUCCESS

In addition to this, I have also checked to make sure that the relevant parts of the /etc/exports file do not have any mistakes. Therefore, I am almost absolutely certain this is not a permission issue.

Question: Given that this is not a network error, can anyone advise me as to how I can start debugging this problem.

Verbose output:

mount: proc already mounted on /proc
mount: /dev/mmcblk0p5 already mounted on /boot
mount: tmpfs already mounted on /var/tmp
mount: tmpfs already mounted on /var/log
mount.nfs: timeout set for Fri Nov 15 07:27:02 2013
mount.nfs: trying text-based options 'vers=3,rsize=8192,wsize=8192,nolock,addr=192.168.7.1'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.7.1 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.7.1 prog 100005 vers 3 prot UDP port 37300
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting 192.168.7.1:/home/raspi/Documents/raspi/raspi

Verbose Output2:

Here is a more direct approach which isolates the mount to only the directory in question here:

$ sudo mount -v -o "vers=3" 192.168.7.1:/home/raspi/Documents/raspi/raspi /home/raspi/test
mount: no type was given - I'll assume nfs because of the colon
mount.nfs: timeout set for Fri Nov 15 07:47:38 2013
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified

I had come across this rpc.statd is not running but is required for remote locking error before, but all the googling in the world did not help me at all...

EDIT:

Chaos asked me to execute the following commands and display the results:

First on the server:

SERVER$ sudo exportfs -rv
exporting raspi1:/home/raspi/Documents/raspi/raspi
exporting raspi2:/home/raspi/Documents/raspi/raspi
exporting raspi3:/home/raspi/Documents/raspi/raspi
exporting raspi4:/home/raspi/Documents/raspi/raspi
exporting raspi5:/home/raspi/Documents/raspi/raspi
exporting raspi6:/home/raspi/Documents/raspi/raspi
exporting raspi7:/home/raspi/Documents/raspi/raspi
exporting raspi8:/home/raspi/Documents/raspi/raspi
exporting raspi9:/home/raspi/Documents/raspi/raspi
exporting raspi10:/home/raspi/Documents/raspi/raspi
exporting raspi11:/home/raspi/Documents/raspi/raspi
exporting raspi12:/home/raspi/Documents/raspi/raspi
exporting raspi13:/home/raspi/Documents/raspi/raspi
exporting raspi14:/home/raspi/Documents/raspi/raspi
exporting raspi15:/home/raspi/Documents/raspi/raspi
exporting raspi16:/home/raspi/Documents/raspi/raspi
exporting raspi17:/home/raspi/Documents/raspi/raspi
exporting raspi18:/home/raspi/Documents/raspi/raspi
exporting raspi19:/home/raspi/Documents/raspi/raspi
exporting raspi20:/home/raspi/Documents/raspi/raspi
exporting raspi21:/home/raspi/Documents/raspi/raspi
exporting raspi22:/home/raspi/Documents/raspi/raspi
exporting raspi23:/home/raspi/Documents/raspi/raspi
exporting raspi24:/home/raspi/Documents/raspi/raspi
exporting raspi25:/home/raspi/Documents/raspi/raspi
exporting raspi26:/home/raspi/Documents/raspi/raspi
exporting raspi27:/home/raspi/Documents/raspi/raspi
exporting raspi28:/home/raspi/Documents/raspi/raspi
exporting raspi29:/home/raspi/Documents/raspi/raspi
exporting raspi30:/home/raspi/Documents/raspi/raspi
exporting raspi5:/home/raspi/Downloads
exporting 192.168.7.105:/home/raspi/Documents/raspi

And then on the client

CLIENT$ showmount -e 192.168.7.1
Export list for 192.168.7.1:
/home/raspi/Documents/raspi       192.168.7.105
/home/raspi/Documents/raspi/raspi raspi30,raspi29,raspi28,raspi27,raspi26,raspi25,raspi24,raspi23,raspi22,raspi21,raspi20,raspi19,raspi18,raspi17,raspi16,raspi15,raspi14,raspi13,raspi12,raspi11,raspi10,raspi9,raspi8,raspi7,raspi6,raspi5,raspi4,raspi3,raspi2,raspi1
/home/raspi/Downloads             raspi5

CLIENT$ sudo mount -a
mount.nfs: access denied by server while mounting 192.168.7.1:/home/raspi/Documents/raspi/raspi

CLIENT$ sudo tail -f /var/log/daemon.log 2>&1
Nov 15 21:35:55 raspi5 dhclient: bound to 192.168.7.105 -- renewal in 250 seconds.
Nov 15 21:40:05 raspi5 dhclient: DHCPREQUEST on eth0 to 192.168.7.1 port 67
Nov 15 21:40:05 raspi5 dhclient: DHCPACK from 192.168.7.1
Nov 15 21:40:05 raspi5 dhclient: bound to 192.168.7.105 -- renewal in 294 seconds.
Nov 15 21:44:59 raspi5 dhclient: DHCPREQUEST on eth0 to 192.168.7.1 port 67
Nov 15 21:44:59 raspi5 dhclient: DHCPACK from 192.168.7.1
Nov 15 21:44:59 raspi5 dhclient: bound to 192.168.7.105 -- renewal in 262 seconds.
Nov 15 21:49:21 raspi5 dhclient: DHCPREQUEST on eth0 to 192.168.7.1 port 67
Nov 15 21:49:21 raspi5 dhclient: DHCPACK from 192.168.7.1
Nov 15 21:49:21 raspi5 dhclient: bound to 192.168.7.105 -- renewal in 273 seconds.

EDIT 2:

Chaos asked me to further execute the following commands and display the results:

SERVER$ ping raspi5
PING raspi5 (192.168.7.105) 56(84) bytes of data.
64 bytes from raspi5 (192.168.7.105): icmp_seq=1 ttl=64 time=0.699 ms
64 bytes from raspi5 (192.168.7.105): icmp_seq=2 ttl=64 time=0.683 ms

SERVER$ sudo tail -f /var/log/daemon.log
tail: cannot open ‘/var/log/daemon.log’ for reading: No such file or directory

Here is my /etc/exports file on the server. As you can see I am trying to use its name as defined in /etc/hosts, its ip address, and I am also trying to different folders to export. All other lines I omitted as they are just duplicates of the last line, just redirected to different raspis.

# /etc/exports: the access control list for filesystems which may be exported
#       to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#

/home/raspi/Downloads raspi5(rw,sync,no_subtree_check,no_root_squash)
/home/raspi/Documents/raspi/ 192.168.7.105(rw,sync,no_subtree_check,no_root_squash,fsid=0)
/home/raspi/Documents/raspi/raspi raspi5(rw,sync,no_subtree_check,no_root_squash,fsid=0)

Here is my /etc/fstab file on my raspi5 client

proc            /proc           proc    defaults          0       0
/dev/mmcblk0p5  /boot           vfat    defaults          0       2
/dev/mmcblk0p6 /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, so no using swapon|off from here on, use  dphys-swapfile swap[on|off]  for that

#automatically mount the shared raspi folder(s)
# Remember that NFS4 is AIDS, so use NFS3
192.168.7.1:/home/raspi/Documents/raspi/raspi /home/raspi/ nfs nfsvers=3,rsize=8192,wsize=8192,rw,auto,nolock 0 0

# Move highly used directories to RAM
tmpfs /var/tmp tmpfs nodev,nosuid,size=50M 0 0
tmpfs /var/log tmpfs nodev,nosuid,size=50M 0 0

EDIT 3:

Following some of the advice here I also set the insecure flag in exports, but to no avail, and here is the output of rpcinfo -p on the client and the server:

CLIENT$ sudo rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  53553  status
    100024    1   tcp  60026  status

SERVER$ rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  42430  status
    100024    1   tcp  49377  status
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    2   tcp   2049
    100227    3   tcp   2049
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    2   udp   2049
    100227    3   udp   2049
    100021    1   udp  33106  nlockmgr
    100021    3   udp  33106  nlockmgr
    100021    4   udp  33106  nlockmgr
    100021    1   tcp  47922  nlockmgr
    100021    3   tcp  47922  nlockmgr
    100021    4   tcp  47922  nlockmgr
    100005    1   udp  50875  mountd
    100005    1   tcp  53329  mountd
    100005    2   udp  53583  mountd
    100005    2   tcp  42062  mountd
    100005    3   udp  36556  mountd
    100005    3   tcp  39984  mountd

EDIT 4:

Further following some of the advice here I have gotten the following debugging logs:

[ 2233.803852] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 2233.803885] NFSD: starting 90-second grace period (net ffffffff81cd29c0)
[ 3261.750455] nfsd: last server has exited, flushing export cache
[ 3262.912654] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 3262.912688] NFSD: starting 90-second grace period (net ffffffff81cd29c0)
[ 5231.798334] nfsd: last server has exited, flushing export cache
[ 5232.956633] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 5232.956668] NFSD: starting 90-second grace period (net ffffffff81cd29c0)
[ 5350.889640] nfsd: last server has exited, flushing export cache
[ 5352.023534] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 5352.023577] NFSD: starting 90-second grace period (net ffffffff81cd29c0)
[ 5517.837425] nfsd: last server has exited, flushing export cache
[ 5518.987793] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[ 5518.987826] NFSD: starting 90-second grace period (net ffffffff81cd29c0)
[56604.080162] Netfilter messages via NETLINK v0.30.
[56707.453535] nfsd: last server has exited, flushing export cache
[56708.605020] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[56708.605057] NFSD: starting 90-second grace period (net ffffffff81cd29c0)
[57338.926093] NFSD: laundromat service - starting
[57338.926105] NFSD: laundromat_main - sleeping for 90 seconds
[57363.216188] nfsd: freeing readahead buffers.
[57363.216244] nfsd: last server has exited, flushing export cache
[57364.351041] set_max_drc nfsd_drc_max_mem 6008832 
[57364.354764] nfsd: creating service
[57364.354771] nfsd: allocating 32 readahead buffers.
[57364.356023] nfsd4_umh_cltrack_upcall: cltrack_prog is disabled
[57364.356040] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
[57364.356072] NFSD: starting 90-second grace period (net ffffffff81cd29c0)
[57364.369688] nfsd_dispatch: vers 3 proc 0
[57372.738225] nfsd_dispatch: vers 4 proc 0
[57372.753507] nfsd_dispatch: vers 4 proc 1
[57372.753521] nfsv4 compound op #1/3: 24 (OP_PUTROOTFH)
[57372.755550] exp_export: export of invalid fs type.
[57372.755679] found domain 192.168.7.105,raspi5
[57372.755685] found fsidtype 1
[57372.755690] found fsid length 4
[57372.755695] Path seems to be <>
[57372.755722] nfsv4 compound op ffff8801a47ec080 opcnt 3 #1: 24: status 2
[57372.755731] nfsv4 compound returned 2
[57372.779429] nfsd_dispatch: vers 3 proc 0
[57372.792595] exp_export: export of invalid fs type.

EDIT 5:

I've made some progress and have deduced the following

It's not a port issue as client can telnet into host telnet 192.168.7.1 2049
It's probably not a network issue as the server can not even mount to itself
The old version which supported (and still supports) mounting was running nfs-kernel-server 1:1.2.4-1 whereas the new server which does not support mounting is running nfs-kernel-server 1:1.2.8-2
The command sudo /etc/init.d/portmap restart works on the old server, but not the new server, which tells me that I am missing portmap on the new server. However, when I try to install it via sudo apt-get install portmap, it tells me Note, selecting 'rpcbind' instead of 'portmap', and thatrpcbind is already the newest version`

Edit 6:

Here is the output of iptables -L on my server:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
LOG        all  --  anywhere             anywhere             LOG level warning

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

and the correct ports are open:

CLIENT$ nmap -P0 192.168.7.1

Starting Nmap 6.00 ( http://nmap.org ) at 2013-11-16 06:44 UTC
Nmap scan report for raspiserver (192.168.7.1)
Host is up (0.0095s latency).
Not shown: 996 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
111/tcp  open  rpcbind
2049/tcp open  nfs

Nmap done: 1 IP address (1 host up) scanned in 1.55 seconds

chaos over 10 years

At your information given, I think that your client is raspi5, right? Can you lookup this hostname on the server? ping raspi5... And I meant the daemon.log on the server while mounting =) Also post the exports file: cat /etc/exports and your fstab cat /etc/fstab. It must be a configuration issue, the exports file is very precise.
puk over 10 years

@chaos just updated the question
chaos over 10 years

In you /etc/fstab it looks unusual... The entries should have the following syntax: 1.2.3.4:/exported/folder /mount/point nfs rw 0 0. Can you try to mount it via terminal? mount -v -t nfs 192.168.7.1:/home/raspi/Documents/raspi/raspi /mnt/
puk over 10 years

@chaos I don't know what happened there, but I had the wrong lines for fstab. In any case, I tried the command line version and that did not work either. Also updated the fstab
puk over 10 years

@chaos I just realized that mounting does not even work locally, and now even my old server doesn't mount properly (locally or otherwise)
puk over 10 years

By the way, this is a HUGE problem. I have come across at least 15 cases, while googling this, of people having this exact same problem, and there is no solution yet.

puk over 10 years

wrt point 1, should I execute that on the server or client (I am guessing client).
chaos over 10 years

The service should be started on both systems (the server and the client).
puk over 10 years

It's already running on the Ubuntu 13.10 but it does not exist on the raspbian. However, I can still mount nfs directories on the same raspi in question from another Ubuntu on my network (11.10), hence why I don't think this is a raspberry pi issue.
chaos over 10 years

What's the error then, with nfsvers=3?
puk over 10 years

mount.nfs: access denied by server while mounting
chaos over 10 years

Can you please execute the following command on the server: exportfs -rv. Then showmount -e <ip-of-the-server> on the client, and while mounting the nfs share: tail -f /var/log/daemon.log on the server. And add the output of all the commands please.
puk over 10 years

sure, I will put the information in the question EDIT I will have to do it tomorrow as I just got home and the changes I made to the server make it inaccessible from home. Sorry
puk over 10 years

Just now added the information you requested to the question
BlueBird almost 9 years

I had the same issue. This helped me.
muru about 8 years

The gist is short enough to be included here. Please do. Link-only answers are useless if the link goes dead for some reason.
dhill about 8 years

@muru Good point.