NFS Client Issue on CentOS 6: rpcbind not listening on port 111

Question

I'm trying to setup a NFS client on a new CentOS 6.6 server to an existing NFS server and am having problems that I cannot figure out. When starting rpcbind/nfs I get:

# /etc/init.d/rpcbind start
# /etc/init.d/nfs start
    Starting NFS services:                                     [  OK  ]
    Starting NFS quotas: Cannot register service: 
         RPC: Unable to receive; errno = Connection refused
         rpc.rquotad: unable to register (RQUOTAPROG, RQUOTAVERS, udp).
                                                               [FAILED]
    Starting NFS mountd:                                       [  OK  ]
    Starting NFS daemon:                                       [  OK  ]

rpcinfo -p works fine

# rpcinfo -p
program vers proto   port  service
100000    4   tcp    111  portmapper
    .. snip
100021    4   tcp  47627  nlockmgr

but rpcinfo -p 127.0.0.1 (or localhost) returns an error:

# rpcinfo -p 127.0.0.1
rpcinfo: can't contact portmapper: RPC: Remote system error - Connection refused

Checking the ports with (netstat -an) it seems the portmap/rpcbind is not listening on port 111. Nothing is listening on port 111 and rpcbind is listening on port 3421 (edit: not random like I originally thought).

Things I've checked or tried include:

• Disabling selinux (was not enabled)
• Clearing /etc/hosts.deny
• Clearing iptables (it had no rules to start)
• Check logs (found nothing relevant)
• Restarted server (no effect)
• Checked on CentOS 5 servers (portmap is listening there on port 111)
• Checked Google: A lot of similar error messages but nothing specifically about rpcbind not listening on port 111 (and none the solutions worked anyways).
• 'telnet' to port 111 (timed out which confirms that nothing is listening there)
• Checked 127.0.0.1/localhost settings (they work fine)

At this point I don't really have any options or things left to check and need some ideas. I'll pretty much try anything at this point as the server is relatively useless without NFS.

Update 1:

Running rpcinfo -s localhost on the client works but rpcinfo -s server does not work. I'm wondering if there is an issue with having the client be CentOS 6 and the server be CentOS 5.

Update 2:

Output from showmount is identical on the new client and the NFS server:

# showmount -e 10.2.212.10
Export list for 10.2.212.10:
/shared/uesp/maps           67.205.112.104
/shared/uesp/skins          67.205.112.104
/shared/phpsessions         10.2.212.30,[...snip...],10.2.212.11
/shared/uesp/filecache      10.2.212.30,[...snip...],10.2.212.11
/shared/uesp/wikiimages     10.2.212.30,[...snip...],10.2.212.11
/shared/eqwiki/filecache    10.2.212.30,[...snip...],10.2.212.11
/shared/eqwiki/wikiimages   10.2.212.30,[...snip...],10.2.212.11
/shared/davewiki/filecache  10.2.212.30,[...snip...],10.2.212.11
/shared/davewiki/wikiimages 10.2.212.30,[...snip...],10.2.212.11

I've snipped the long list of IPs, but 10.2.212.30 is the new client. Trying to mount a NFS share on the new client results in this:

# mount -v 10.2.212.10:/shared/phpsessions /mnt/phpsessions
mount: no type was given - I'll assume nfs because of the colon
mount.nfs: timeout set for Thu Jan 15 18:53:43 2015
mount.nfs: trying text-based options 'vers=4,addr=10.2.212.10,clientaddr=10.2.212.30'
mount.nfs: mount(2): Operation not permitted
mount.nfs: trying text-based options 'addr=10.2.212.10'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query retrying: RPC: Unable to receive
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Unable to receive - Connection refused
mount.nfs: trying text-based options 'vers=4,addr=10.2.212.10,clientaddr=10.2.212.30'
mount.nfs: mount(2): Operation not permitted
mount.nfs: trying text-based options 'addr=10.2.212.10'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query retrying: RPC: Unable to receive
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Unable to receive - Connection refused
...

The last few lines repeat until it times out.

Update 3:

So if I use iptables to redirect port 111 to the port rpcbind is using on the client then I can get rpcinfo -p localhost to work on the client. However, this doesn't do anything with the mount issues. I've tried forcing the NFS to use v2 like:

# mount -v -t nfs -o nfsvers=3 10.2.212.10:/shared/phpsessions /mnt/phpsessions

but that results in the same error message as above. The fact that when I run on the client I get:

# rpcinfo -s server
rpcinfo: can't contact rpcbind: : RPC: Unable to receive; errno = Connection refused

which is the same error message from mount. I'm guessing that mount on the CentOS 6 client is trying to use the "new" rpcbind on the CentOS 5 server which is running the "old" portmap so it is failing. I would assume it should be possible to mix CentOS versions like this somehow.

Update 4:

I created a test NFS share on the client and was able to successfully mount it on the server. This seems to match the theory that it is something to do with a portmap/rpcbind mismatch. The client can act as a NFS server as rpcbind (rpc v3/4) is backwards compatible with portmap (rpc v2) but portmap is not forwards compatible with rpcbind.

Update 5 (Possible Solution)

Looking at the client more carefully I noticed rpcbind seems to always be on port 3421 (I thought it was random initially). On the server I used iptables to forward port 3421 to port 111. After this I was able to successfully mount the a NFS share from the server on the client. This may be a valid solution since I'll only have a mix of CentOS 5/6 servers for a few weeks as I upgrade all servers to CentOS 6.

Unless a better answer comes up I'll answer my own question in a few days.

Solution

The root issue turned out to be a port mismatch for portmap/rpcbind. On CentOS 6 my rpcbind was listening on port 3421 instead of the expected 111 like portmap does in CentOS 5. This caused both the issue with starting nfs and the mounting of shares between the two machines.

The solution was to use itpables to redirect port 111 on CentOS 6 to port 3421:

iptables -t nat -I PREROUTING -p tcp --dport 111 -j REDIRECT --to-ports 3421
iptables -t nat -I PREROUTING -p udp --dport 111 -j REDIRECT --to-ports 3421
iptables -t nat -I OUTPUT -p tcp -o lo --dport 111 -j REDIRECT --to-ports 3421
iptables -t nat -I OUTPUT -p udb -o lo --dport 111 -j REDIRECT --to-ports 3421

The same thing was done on the CentOS 5 server except in reverse, redirecting port 3421 to 111. After this I was able to successfully mount the NFS share from the CentOS 5 server on the CentOS 6 client.

In order to get nfs completely working on the CentOS 6 server I also had to uncomment all the port related lines in /etc/sysconfig/nfs to use fixed port addresses. Before doing this I had intermittent issues with the lockd service not responding.

Comments

[Admin]

It's not clear if the command output in the question is from the NFS client or server. My first though is that it appears you are trying to get NFS running on what is to be the NFS client.

[uesp]

All the above output is from the client...sorry for the confusion.

[Admin]

From your NFS client all you should need to do is mount an NFS export. Is there expected output from showmount -e nfs_server_name when run on the server and client?

[uesp]

showmount -e server show identical output on the new client and the server. Trying to mount from the new client gives me the similar "RPC: Unable to receive - Connection refused" errors. I'll confirm that the server has been correctly updated to permit the new client to connect.

[Admin]

Can you add output from the showmount commands run on the NFS server and client in addition to the mount command you are using on the client to your question?

[danone]

after 10 hours research try this is the solution only works thanks so much.