autofs mounts not disconnecting after inactive

Question

I have autofs installed on several linux servers that are connecting to central NFS server for the users /home directories. It works great when mounting the directories on login, but the mounts never seem to timeout. I've checked /etc/sysconfig/autofs and the default is indeed set to 300, so these should be timing out after 5 minutes.

Restarting autofs does umount all of the directories, so I know it's capable.

I've attempted using lsof randomly on the directories but no files appear open at any time.

I've also mounted a random directory that I know is not active, but these never umount themselves. Some of these boxes have 10+ users that have logged in once, and the mounts never drop.

I'm just trying to find out of there is a better method to finding out why. I don't see anything specific in any logs.

Any suggestions are appreciated. Thanks!

UPDATE

I turned on debugging for autofs but it doesn't seem to reveal anything out of the ordinary. These logs were generated 7 minutes after the /home/user1 was initially mounted and after 6 minutes of inactivity. According to the 5 minute default, this should have been unmounted. I never saw a log come through that indicated an attempt was even made to umount.

Jan 11 12:52:00 linux automount[26505]: st_expire: state 1 path /home
Jan 11 12:52:00 linux automount[26505]: expire_proc: exp_proc = 3055176592 path /home
Jan 11 12:52:00 linux automount[26505]: expire_proc_indirect: expire /home/user1
Jan 11 12:52:00 linux automount[26505]: expire_proc_indirect: expire /home/user2
Jan 11 12:52:00 linux automount[26505]: expire_proc_indirect: expire /home/user3
Jan 11 12:52:00 linux automount[26505]: 3 remaining in /home
Jan 11 12:52:00 linux automount[26505]: expire_cleanup: got thid 3055176592 path /home stat 7
Jan 11 12:52:00 linux automount[26505]: expire_cleanup: sigchld: exp 3055176592 finished, switching from 2 to 1
Jan 11 12:52:00 linux automount[26505]: st_ready: st_ready(): state = 2 path /home

Update 2 After talking to Red Hat support about this, the solution ended up being to just shorten the timeout value for home directories. I did that and looks well. Something apparently is traversing the mount point every 2 1/2 to 3 minutes and causing this to stay up.

The solution was to add the timeout value to the /etc/auto.master file for that mapping:

 /home     /etc/auto_home --timeout=120

Solution 1

Besides TIMEOUT variable autofs has a checking interval:

# cat /var/log/messages
Jan 11 21:45:35 client automount[24804]: mounted offset on /net/server/share with timeout 300, freq 75 seconds

It is equal to TIMEOUT/4. Every TIMEOUT/4 seconds autofs asks kernel when directory was accessed last time. So in your environment you have directory umnounted after 375 seconds of inactivity.

To get more detailed log you should add LOGGING="debug" to /etc/sysconfig/autofs

Solution 2

For any one else experiencing similar issues there are GUI processes on modern desktops that scan the drives continually. In particular Nautilus on Gnome and Dolphin on KDE along with file indexing applications like Baloo. These are all capable of causing the symptom.

For me (running KDE) the only clue from automount debug logging was "1 remaining" eg:

    Feb 13 00:00:44 fig automount[19026]: expire_proc: exp_proc = 139620739028736 path /mnt/vchanger
    Feb 13 00:00:44 fig automount[19026]: expire_proc_indirect: expire /mnt/vchanger/fb207cd6-6931-4af4-8293-c82ee0d2394c
    Feb 13 00:00:44 fig automount[19026]: 1 remaining in /mnt/vchanger

This didn't really identify the source. Also none of lsof, fuser and auditctl (auditd) gave any insight.

Eventually by process of elimination I determined there were 2 applications:

• KSysGuard (KDE System monitor)
• Dolphin (File Manager)

The issue with Dolphin could be fixed in this case by "hiding" the offending mounted disk in its tree view.

KSysGuard doesnt appear configurable but then its perhaps unusual to have it running long term unless you are debugging something. Hopefully other applications might be more configurable in permitting exclusions to prevent the automount mount point from being scanned.

Solution 3

I had a similar problem. I reinstalled our 10 year old RHEL 4.7 ProLiant server with CentOS 6 over the Christmas break. I had 2 newer ProLiants which I was able to install CentOS 7 on more recently (in April).

I configured automounting the home directories from the CentOS 6 server using a line in /etc/auto.master on the CentOS 7 servers like so:

/home   /etc/auto.home

Then I created a new /etc/auto.home file on the CentOS 7 servers initially with a line:

*      sam:/home/&

The home directories wouldn't unmount however. I also found that some of the file ownerships in the home directories would from time to time end up with a huge UID and GID number against them. It would change a few minutes later.

I set the logging level to 'debug' in /etc/autofs.conf and starting watching with journalctl -fu autofs.service. I saw almost identical messages as shown above, which seemed to contain no clues.

As I hadn't yet been able to understand NFS 4, and I knew our CentOS 6 server was exporting its shares as NFS 4 by default, I tried adding nfsvers=3 to the /etc/auto.home file like so:

training      -nfsvers=3,noac,soft,intr  sam:/home/training

I was also seeing the odd message about trying to mount directories like /home/lib, so added the individual home directories on separate lines. (Probably should have tried direct mounts at this point, or tried systemd automounts.)

Now I started to see messages like:

Apr 27 09:32:28 betty automount[13501]: expire_proc_indirect: expire /home/fred
Apr 27 09:32:28 betty automount[13501]: handle_packet: type = 4
Apr 27 09:32:28 betty automount[13501]: handle_packet_expire_indirect: token 21, name fred
Apr 27 09:32:28 betty automount[13501]: expiring path /home/fred
Apr 27 09:32:28 betty automount[13501]: umount_multi: path /home/fred incl 1
Apr 27 09:32:28 betty automount[13501]: umount_subtree_mounts: unmounting dir = /home/fred
Apr 27 09:32:28 betty automount[13501]: spawn_umount: mtab link detected, passing -n to mount
Apr 27 09:32:29 betty automount[13501]: rm_unwanted_fn: removing directory /home/fred
Apr 27 09:32:29 betty automount[13501]: expired /home/fred
Apr 27 09:32:29 betty automount[13501]: dev_ioctl_send_ready: token = 21
Apr 27 09:32:29 betty automount[13501]: handle_packet: type = 4
Apr 27 09:32:29 betty automount[13501]: handle_packet_expire_indirect: token 22, name barney
Apr 27 09:32:29 betty automount[13501]: expiring path /home/barney
Apr 27 09:32:29 betty automount[13501]: umount_multi: path /home/barney incl 1
Apr 27 09:32:29 betty automount[13501]: umount_subtree_mounts: unmounting dir = /home/barney
Apr 27 09:32:29 betty automount[13501]: spawn_umount: mtab link detected, passing -n to mount
Apr 27 09:32:29 betty automount[13501]: rm_unwanted_fn: removing directory /home/barney
Apr 27 09:32:29 betty automount[13501]: expired /home/barney
Apr 27 09:32:29 betty automount[13501]: dev_ioctl_send_ready: token = 22
Apr 27 09:32:29 betty automount[13501]: expire_proc_indirect: expire /home/barney
Apr 27 09:32:29 betty automount[13501]: expire_proc_indirect: expire /home/wilma
Apr 27 09:32:29 betty automount[13501]: 1 remaining in /home

The home directories now started unmounting after 10 minutes like they should - so it was a problem with misconfigured NFS 4 in my case.

Important: after reconfiguring the maps, simply doing systemctl daemon-reload or systemctl reload autofs doesn't have any effect. I had to do systemctl restart autofs

Solution 4

I spent hours today trying to debug and similar problem. Here is what I found and how I worked it around.]

setup: I wanted to auto mount dir containing users' home dirs from nfs server "srv1:/srv/homes" at /mnt/nfs/homes on clients. NFS servers exports NFS4. autofs version 5.1.3

I had configured every client like that:

/etc/auto.mount: file containing following:

... 
/mnt/nfs /etc/auto.home
...

/etc/auto.home:

homes  -rw,soft,intr,rsize=8192,wsize=8192 srv1:/srv/homes

Eventually this represents an indirect map. Auto mount works like a charm. I get the NFS volume properly mounted and working. But... it never gets auto unmounted. Even though autofs.conf file says:

and mount shows 600 seconds timeout:

#1# /etc/auto.home on /mnt/nfs type autofs (rw,relatime,fd=18,pgrp=5054,timeout=300,minproto=5,maxproto=5,indirect) 
srv1:/srv/homes on /mnt/nfs/homes type nfs4 (rw,relatime,vers=4.2,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=x.x.x.x,local_lock=none,addr=y.y.y.y)

I was seeing exactly the same in the (debug log level activate) autofs logs from journalctl as wanpelaman

automount[53593]: st_expire: state 1 path /mnt/nfs
automount[53593]: expire_proc: exp_proc = 139645987374848 path /mnt/nfs
automount[53593]: expire_proc_indirect: expire /mnt/nfs/homes
automount[53593]: 1 remaining in /mnt/nfs
automount[53593]: expire_cleanup: got thid 139645987374848 path /mnt/nfs stat 3
automount[53593]: expire_cleanup: sigchld: exp 139645987374848 finished, switching from 2 to 1
automount[53593]: st_ready: st_ready(): state = 2 path /mnt/nfs

At that time I gave up on autofs and decided to replicate automount config with systemd. Actually I ran it and at this time everything worked great - auto mount, auto unmount after predefined idle period. Just perfect. But systemd ... a little bit clumsy (don't shoot at me, I actually like it). Then I looked at how systemd handles auto mounting:

#2# systemd-1 on /mnt/nfs/homes type autofs (rw,relatime,fd=35,pgrp=1,timeout=20,minproto=5,maxproto=5,direct)
srv1:/srv/homes on /mnt/nfs/homes type nfs4 (rw,relatime,vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=x.x.x.x,local_lock=none,addr=y.y.y.y)

Difference between #1# and #2# is that latter is direct map whereas #1# is indirect. Hence I immediately decided to reconfigure autofs on another client and create direct map like that:

/etc/auto.master

/-   /etc/auto.home

/etc/auto.home

/mnt/nfs/homes  -rw,soft,intr,rsize=8192,wsize=8192 srv1:/srv/homes

And this has eventually solved the problem. Both auto mount and auto umount worked fine. umount was successfully run after the predefined idle time in /etc/autofs.conf

Absolutely no modification on NFS server was needed.

View more solutions

Comments

[devarni]

what command(s) are you using to determine that these mounts are present? I assume df, but just want to clarify.

[SteveHNH]

Yes, I'm using df to check for the mounted space. I just cd to the directories as root to get them to mount.

[Earl Ruby]

df will access the home directories in order to display the fact that they're mounted. So if you're doing something like watch -n 1 df -h to see if the directories are mounted they will never unmount because all mounted directories are being accessed every second.

[SteveHNH]

I see. Thanks for the clarification. The logs above continued well after the 6 minutes of inactivity, and exceeded 375 seconds. I keep thinking that something has to be accessing these directories, or the umount would be attempted. I guess my real objective is to find out what is accessing this directory, if anything. That can be the only reason I can think of that it wouldn't umount.

[G-Man Says 'Reinstate Monica']

FYI, if you would login before editing your post, you won’t need to approve it later (or wait hours for other people to approve it).