NFS mount: Device or resource busy

Question

I referred the following link, the solution works.

How to get over "device or resource busy"?

The above solution works when you are manually deleting the file. But I have a python script that deletes the files (automated process). Sometimes I get "Device or resource busy error" when the script tries to delete the files. Consequently, my script fails. I don't know how to resolve this issue using my python script.

EDIT: The script downloads the logs files from a log server. These files are then processed by my script. After the processing is done, the script deletes these log files. I don't think that there is anything wrong with the design.

Exact Error:

OSError: [Errno 16] Device or resource busy: '/home/johndoe/qwerty/.nfs000000000471494300000944'

Solution

These files are NFS placeholders:

/home/johndoe/qwerty/.nfs000000000471494300000944

Some background

In a typical UNIX filesystem, a file that is currently in use and open can be deleted but its contents will not actually disappear until the last filehandle to it is closed. You can see this in action with code like this:

$ ps -ef >/tmp/temporaryfile
$ ls -l /tmp/temporaryfile
-rw-r--r-- 1 roaima roaima 6758 Mar  2 14:02 /tmp/temporaryfile

$ ( sleep 60 ; cat ) </tmp/temporaryfile &
[1] 4864

$ rm /tmp/temporaryfile
$ ls -l /tmp/temporaryfile
ls: cannot access /tmp/temporaryfile: No such file or directory

$ fg    # Wait for the rest of the minute
( sleep 60; cat ) < /tmp/temporaryfile
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 09:44 ?        00:00:02 init [2]
root         2     0  0 09:44 ?        00:00:00 [kthreadd]
root         3     2  0 09:44 ?        00:00:00 [ksoftirqd/0]
root         5     2  0 09:44 ?        00:00:00 [kworker/0:0H]
...
roaima    4857  4786  0 14:02 pts/1    00:00:00 -bash
roaima    4858  4857  0 14:02 pts/1    00:00:00 ps -ef

(Note that this is opposite to Microsoft Windows, where files cannot be deleted while they are still open.)

Explanation

A file on an NFS server may have one or more clients accessing it. NFS itself is (mostly) stateless and so needs to emulate the functionality that allows an open file to be accessed even after it's been deleted.

The emulation is handled by removing the file from its place in the filesystem but leaving it in place as a file whose name starts with .nfs. When the last reader/writer closes their filehandle to this file it will be properly removed from the filesystem.

Here's an example of this in action:

$ ps -ef > /var/autofs/net/nfsserver/tmp/temporaryfile
$ ls -l /var/autofs/net/nfsserver/tmp/temporaryfile
-rw-r--r-- 1 roaima roaima 6766 Mar  2 14:14 /var/autofs/net/nfsserver/tmp/temporaryfile

$ ( sleep 60 ; cat ) </var/autofs/net/nfsserver/tmp/temporaryfile &
[1] 4987

$ rm /var/autofs/net/nfsserver/tmp/temporaryfile
$ ls -l /var/autofs/net/nfsserver/tmp/temporaryfile
ls: cannot access /var/autofs/net/nfsserver/tmp/temporaryfile: No such file or directory

$ ls -lA /var/autofs/net/nfsserver/tmp/
total 8
-rw-r--r-- 1 roaima roaima 6766 Mar  2 14:14 .nfs000000000100000300000001

$ rm /var/autofs/net/nfsserver/tmp/.nfs000000000100000300000001
rm: cannot remove ‘/var/autofs/net/nfsserver/tmp/.nfs000000000100000300000001’: Device or resource busy

$ fg    # Wait for the rest of the minute
( sleep 60; cat ) < /var/autofs/net/nfsserver/tmp/temporaryfile
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 09:44 ?        00:00:02 init [2]
root         2     0  0 09:44 ?        00:00:00 [kthreadd]
root         3     2  0 09:44 ?        00:00:01 [ksoftirqd/0]
...
roaima    4983  4712  0 14:14 pts/0    00:00:00 ps -ef

Corollary

You should ignore files on an NFS mount whose names begin with .nfs. Furthermore, your code needs to cope with the possibility that a remote directory cannot be deleted until all these files have actually disappeared.

NFS isn't quite as transparent to applications as one might hope.

Comments

It may be that the reason the log files are still open is that they are still being used by the logger process on your remote system. Generally the approach to this would be to cycle the log files and only download and delete the previous log files, leaving the current ones in the filesystem for use by the logger process.

Utilities such as logrotate handle this with specific configuration elements such as delaycompress that (attempt to) ensure a log file is not compressed while it's still in use. (See /etc/logrotate.d/apache2 on at least Debian systems for an example.)

Comments

[Touchstone]

The script creates a bunch of files and later on it deletes them. As the entire process is automated, I want to delete the file automatically. Moreover, this script is a scheduled job run by a cron.

[Wildcard]

I'll take a bet that this problem could/should be solved at a higher level of abstraction. :) Why make the files if you're just going to delete them? Could you perhaps use memory for that (i.e. data structures)? In a shell script (not Python), the answer would be "pipelines." Also relevant: unix.stackexchange.com/q/254296/135943

[dirkt]

So have the script wait until all the processes that use the files are done.

[Touchstone]

@dirkt yes, it waits

[Touchstone]

@roaima it is a Red Hat Enterprise Linux Server release 6.8 and I am deleting log files.

[Touchstone]

@Wildcard the thing is I download the logs from a log server and then process them. Due to space constraint, I cannot keep on downloading the logs. That is why I delete them after the processing is done.

[terdon]

@roaima go for it.