df -l is showing 100% full but the file was removed

Question

I got an alert that a local disk was full;

dm@fooserv:/local/data/plog $ df -l
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/rootvg-datavol
                     121790564 115659468         0 100% /local/data
tmpfs                   102400      1028    101372   2% /var/asagent/lib/asagen

I checked the directory and saw the file.

user@fooserv:/local/data/plog $ ls -ltr
total 84926904
lrwxrwxrwx 1 user ers_gsd          37 Aug 15 03:00 bomb.log -> /local/data/plog/bomb.31655.log
-rw-rw-rw- 1 user ers_gsd           0 Aug 15 03:00 recovery.log
drwxrwxrwt 2 user ers_gsd        4096 Aug 15 03:00 log/
-rw-rw-rw- 1 user ers_gsd           0 Aug 15 03:00 dropping.log
-rw-rw-rw- 1 user ers_gsd       10109 Aug 15 09:20 proc_fooserv.log
-rw-rw-rw- 1 user ers_gsd      381083 Aug 15 10:25 trip_bomb.rip.1.log
-rw-rw-rw- 1 user ers_gsd    60563456 Aug 15 13:35 bomb.31655.log
-rw-rw-rw- 1 user ers_gsd           0 Aug 15 13:37 bomb.stats
-rw-rw-rw- 1 user ers_gsd 86819237888 Aug 15 13:37 process-one.log

I found out what process was creating the files and killed them :

user@fooserv:/local/data/plog $ ps -ef | grep 12077
user    12077     1  0 09:20 ?        00:00:00 /bin/bash /home/user/bin/process_big.sh /local/data/plog/process-one.log
user    12085 12077  0 09:20 ?        00:00:35 tail -f /local/data/plog/process-one.log
user    12088 12077  0 09:20 ?        00:01:31 grep ERR
user    12095 12077  0 09:20 ?        00:02:06 grep -v FIXME
user    12098 12077 61 09:20 ?        02:38:56 /bin/bash /home/user/bin/process_big.sh /local/data/plog/process-one.log
user    22836 32756  0 13:36 pts/0    00:00:00 grep 12077
user@fooserv:/local/data/plog $ kill 12098
user@fooserv:/local/data/plog $ kill 12100

I removed the file:

user@fooserv:/local/data/plog $ rm process-one.log

the df still says that the directory is full:

dm@fooserv:/local/data/plog $ df -l
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/rootvg-datavol
                    121790564 115659468         0 100% /local/data
   tmpfs                   102400      1028    101372   2% /var/asagent/lib/asagent
user@fooserv:/local/data/plog $

~

Solution

Trying checking if the process still running which is causing the file resources to be held.

lsof -nP | grep '(deleted)'

Should give you a starting point.

Comments

[Nate]

I'm not sure how to help (sorry), but +1 for the well documented question.

[capser]

if i removed a file without killing a process - does that lock the file? - i don't see and inodes for the file

[hymie]

"Removing a file" only takes away the link to access that file in the future. As long as the file is "in use" by one or more active running programs, then the underlying file system will continue to mark the space as "used". You can't get to it with a new process, because the space has no accessible link in the file system. But the program that is already using that file space still has it, and it's still active, and it's still in use.