df in linux not showing correct free space after file removal
Solution 1
Deleting the filename doesn't actually delete the file. Some other process is holding the file open, causing it to not be deleted; restart or kill that process to release the file.
Use
lsof +L1
to find out which process is using a deleted (unlinked) file.
Solution 2
as Ignacio mentions, deleting the file won't free the space until you delete the processes that have open handles against that file.
Nevertheless, you can reclaim the space without killing the processes. All you need to do is to remove the file descriptors.
First execute lsof | grep deleted to identify the process holding the file
[hudson@opsynxvm0055 log]$ /usr/sbin/lsof |grep deleted
java 8859 hudson 1w REG 253,0 3662503356 7578206 /crucible/data/current/var/log/fisheye.out (deleted)
Then execute:
cd /proc/PID/fd
then
[hudson@opsynxvm0055 fd]$ ls -l |grep deleted
total 0
l-wx------ 1 hudson devel 64 Feb 7 11:48 1 -> /crucible/data/current/var/log/fisheye.out (deleted)
The "1" will be the file descriptor. Now type "> FD" to reclaim that space
> 1
You might need to repeat the operation if there are other processes holding the file.
Solution 3
If partition has been configured to reserve certain portion of disk space only for root usage, df
will not include this space as available.
[root@server]# df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/optvol 625G 607G 0 100% /opt
...
Even after space will be reclaimed by deleting files/directories, non-root user won't be able to write to particular partition.
You can easily check if that's your case by trying to create a file on a device as root and non-root user.
Additionally you can check filesystem configuration by running
tune2fs -l <device> | egrep "Block count|Reserved block count
and calculating actual % on your own.
To change disk % reserved for root-only usage, execute
tune2fs -m <percentage> <device>
Solution 4
The file is still locked by the process opening it. To free up space, do these steps:
Run
sudo lsof | grep deleted
and see which process is holding the file. Example result:$ sudo lsof | grep deleted COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME cron 1623 root 5u REG 0,21 0 395919638 /tmp/tmpfPagTZ4 (deleted)
Kill the process using
sudo kill -9 {PID}
. In above sample, the PID is 1623.$ sudo kill -9 1623
Run
df
to check if space is already freed up. If it's still full, maybe you need to wait a few seconds and check again.
Solution 5
One possibility is that the file(s) you deleted have more references in the filesystem. If you've created hardlinks, several filenames will point to the same data, and the data (the actual contents) won't be marked as free/usable until all references to it has been removed. Before you delete files, either stat them (Entry named Links) or do ls -l on them (should be the second column).
If it does turn out that the files are referenced elsewhere, I guess you'll have to ls -i the file(s) to find the inode-number, and then do a find with -inum <inode-number> to find the other references to that file (you probably also want to use -mount to stay within the same filesystem as well).
Related videos on Youtube
Aminah Nuraini
Do you want to contact me? Please click GREEN CONTACT BUTTON in my Upwork profile link below. https://www.upwork.com/fl/aminahnuraini My passion is to build an efficient and best-practice solution in web scraping and automatization, especially using a steady and best-practice framework like Scrapy. Demonstrated problem-solving skills in various challenging websites to scrape, including dynamic websites. Focusing on Scrapy for over 4 years. Familiar with Scrapy's cutting-edge tools like Scrapy Splash, Scrapy Spidermon, Scrapy Crawlera, Scrapy Middlewares, Scrapy Pipelines, and Scrapy Shell. Experienced in managing both small-scale and large-scale projects, even handling over 250 million data using ElasticSearch. As an engineer and architect, I have a proven ability in improving the system's stability and maintainability. I have managed and worked with a geographically dispersed team. Additional significant experience includes database (MySQL, MongoDB, MS SQL), Python 2.x and Python 3.x, Django, XPATH, Regex, Javascript, PHP, Linux, and Networking. If you want to get a VIP consultation with me, you can call me here https://clarity.fm/aminahnuraini Notable Scrapy answer: https://stackoverflow.com/questions/21009027/split-scrapys-large-csv-file/40654790#40654790
Updated on September 17, 2022Comments
-
Aminah Nuraini over 1 year
I have file servers which are used to store files. Files might reside there for a week, or for a year. Unfortunately, when I remove files from the server,
df
command doesn't reflect the freed up space. So eventually, the server gets filled up (df
shows 99%), and my script doesn't send any more files there, except there might be a few dozen GB of free space on there.I got
noatime
flag on the mounted partitions if that makes any difference.-
Khaled over 13 yearsIs this happening on a single partition or on all partitions?
-
Admin over 13 yearsWell, its happening on my main data partition, which is the only one I care about, since I only write/remove files onto it.
-
Muhammad Danish over 13 yearsWhat filesystem(s)? DF does a stat of the superblock, it may be that your filesystem is not updating the sb inode. Have you tried flushing cache?
-
Admin over 13 yearsUsing ext4. How do you flush caches?
-
shoover about 7 yearsJoined this Stack today just to upvote this question and its answers because I was stuck and this got me unstuck.
-
AncientSwordRage over 2 yearsDoes this answer your question? df says disk is full, but it is not
-
-
Admin over 13 yearsFiles that are removed were not accessed in over a month, and the only process that accesses them is nginx, so its doubtful.
-
pehrs over 13 years+1. Also, "lsof +L1" will tell you which program is holding the files open.
-
ScottZ over 13 yearsas root run "lsof -n | grep file," you'd be surprised at how long files can stick around due to processes keeping them open for whatever reason. If all else fails, reboot, I feel bad suggesting it but it will definitely make sure nothing is holding on to the file. Per pehrs, lsof +L1 is probably the better way to go.
-
Admin over 13 yearsThat returns a single file, which is being encoded with ffmpeg.
-
Deer Hunter over 11 yearsDear Marcellus, your solution is encompassed by the accepted answer; and sometimes you don't want to do a reboot if you are not forced to...
-
Andrew B about 11 yearsThis doesn't really add much; the accepted answer covered the logic behind that in 2001. When you have 50 rep, use comments if you want to add qualifiers to the existing answers.
-
Luke Cousins about 10 yearsYou just saved me! Deleted a 93G log file and didn't get the space back and couldn't work out why. Thanks.
-
Nick almost 10 yearsAlong the same lines and in case this helps others, I erased a large nginx access.log file but was only able to reclaim the space after restarting nginx: service nginx restart
-
FilBot3 almost 9 yearswhat does the
> FD
do? -
Adrián Deccico almost 9 yearsit removes the file descriptor
-
Ashish Karpe over 8 yearsI have same issue with apache2 access logs, ihad even stopped n started appache still no space returned i am using ext4
-
ariera over 8 yearsdoes this
>
command have a name? i had to switch from zsh to bash in order to be able to use it. Is it possible to run it on zsh? -
kontinuity over 7 yearsNever knew about this. Learning something every day.
-
aross over 7 yearsIt helps to know that you need to run this as root/sudo. I could guess and try, but I generally don't try stuff with sudo if I don't know the consequences.
-
Aaron Hudon about 7 yearsit is a output redirect and therefore truncates the file. The long from would be "echo -n > 1" or "true > 1". It does not really remove the FD, it just points to a empty file afterwards.
-
Chaminda Bandara over 5 yearsI got a error
-bash: lsof: command not found
-
dr0i over 4 yearsOne shouldn't say "thank you" in comments. Here I do it nonetheless.
-
chiappa about 4 yearsI got an easier to read format in MB with this command; lsof +L1 | numfmt --field=7 --to=iec --invalid=ignore
-
J.Ko almost 4 years... so how do you close those processes?
-
Michael Hampton over 3 yearsThis will just kill a bunch of programs. You won't be able to know which ones in advance. It's also not clear why you would want to do this.
-
peterh over 3 yearsScript copy-paste is not a very well going thing, I think. I suggest to explain, what is your command doing and how.
-
enharmonic over 2 yearsRun
sudo lsof +L1
to see non-user processes, as well, and thensudo kill [pid]
the relevant PID. -
Admin almost 2 years+1. Note to ppl like me: Before doing
> 1
it's important that we are in the /proc/<PID>/fd dir, as it's mentioned in the answer. I usually avoid going deep in dirs and use appropriate paths in commands, which doesn't work in this case.