Tell fs to free space from deleted files NOW

Question

Is there a way to tell the kernel to give back the free disk space now? Like a write to something in /proc/ ? Using Ubuntu 11.10 with ext4.

This is probably an old and very repeated theme. After hitting 0 space only noticed when my editor couldn't save source code files I have open, which to my horror now have 0 byte size in the folder listing, I went on a deleting spree.

I deleted 100's of MB of large files both from user and from root, and did some hardlinking too.

Just before I did apt-get clean there was over 900MB in /var/cache/apt/archives, now there is only 108KB:

# du
108 /var/cache/apt/archives

An hour later still no free space and cannot save my precious files opened in the editor, but notice the disparity below:

# sync; df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda4             13915072  13304004         0 100% /

Any suggestions? I shut off some services/processes but not sure how to check who might be actively eating disk space.

More info

# dumpe2fs  /dev/sda4
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              884736
Block count:              3534300
Reserved block count:     176715
Free blocks:              422679
Free inodes:              520239
First block:              0
Block size:               4096
Fragment size:            4096

Solution 1

Check with lsof to see if there are files held open. Space will not be freed until they are closed.

sudo /usr/sbin/lsof | grep deleted

will tell you which deleted files are still held open.

Solution 2

Use lsof to find the deleted, but open, file still consuming space:

lsof | grep deleted | grep etilqs_1IlrBRwsveCCxId
chrome     3446       user  128u      REG              253,2              16400       2364626 /var/tmp/etilqs_1IlrBRwsveCCxId (deleted)  

Find the entry in /proc/<pid>/fd/ that cooresponds to the filehandle:

ls -l /proc/3446/fd/etilqs_1IlrBRwsveCCxId
lrwx------. 1 user unix 64 Feb 11 15:31 128 -> /var/tmp/etilqs_1IlrBRwsveCCxId (deleted)

Now, just cat /dev/null into the fd:

cat /dev/null > /proc/3446/fd/128

Note that the inode is still open, but now it's 0 length

chrome     3446       user  128u      REG              253,2         0    2364626 /var/tmp/etilqs_1IlrBRwsveCCxId (deleted)

Solution 3

df will not show space reserved for root (even when run as root):

# df -h
Filesystem            Size  Used Avail Use% Mounted on
...
/dev/optvol           625G  607G     0 100% /opt
...

How to change "reserved block percentage"

1. Reduce reserved space to 4%

   # tune2fs -m4 /dev/sda4

df -h now showed 45M free.

2. Saved my files quickly

3. Put it back to 5%

   # tune2fs -m5 /dev/sda4

Solution 4

In Ubuntu, if you deleted files using your trash bin, your files were more than likely not be completely removed.

Even after emptying your trash your files will remain in ~/.local/share/Trash/expunged until after a reboot and maybe even longer.

I haven't found a good reason for this, but if I run out of space, I always manually rm the expunged trash files.

Solution 5

sudo lsof | grep "(deleted)$" | sed -re 's/^\S+\s+(\S+)\s+\S+\s+([0-9]+).*/\1\/fd\/\2/' | while read file; do sudo bash -c ": > /proc/$file"; done

Explanation:
Grep lsof output to extract only deleted files. Sed extract the process id and filedescriptor id from each line, and create a string in format {pid}/fd/{fid}. While loop and output nothing to each file, setting them to empty.

View more solutions

Comments

[hhaamu]

The file system frees the space immediately. However, the root-reserved blocks feature of ext[234] and how the kernel keeps open files reserved may give the appearance of lost space.

[vonbrand]

If you have several filesystems (patitions), freeing up space in one won't do any good in the other one.

[Psddp]

Why was I able to fill the partition before the "reserved" 5Go blocks reclaim themselves?

[YoloTats.com]

The for root reserved space is nowadays nearly always too big. You can reduce it to some few percent. df displays the for normal user usable space. As apt runs as root, the reserved space is only useful to protect against fill-ups caused by non-root users (=normal users and services that have their own user).

[RIQ]

Good. Showed me some mysqld locks in /tmp, but many apport-gt uses of extinct files in /var/lib/apt/lists/partial/ that apparently have been accumulating. So I might killall apport-gt but will investigate it first.

[RIQ]

sync never helped. As for logs, it's an Ubuntu system so it's pretty buggy, so yeah they're typically noisy. apport has be deploying often because every nightly apt-get update crashes, though /var/crash has only 77MB. Also noticed atd has flooded /var/log/syslog with repeating lines like atd[8892]: File a0015c0152ab76 is in wrong format - aborting probably since the few files in /var/spool/cron/atspool were all 0 sized, making the problem circular of course

[RIQ]

I agree; an mkfs these days should reserve eg. 5% or 300MB, whichever is less. Just re-tuned some of my servers to 2% and freed GBs back!

[psusi]

@jofel, no, it isn't. Any time you go over 90% utilization, you start getting lots of fragmentation. You need to free up some more space, not run closer to 100% utilization.

[YoloTats.com]

@psusi You are true, thanks for your comment. But the opportunity to use (temporary) nearly all available space as normal user could be really practical and with ext4, the things are not that bad anymore, see unix.stackexchange.com/a/7965/15241

[RIQ]

Good point. Even though I'm one of those who live and die by the command line, and rarely use graphical file manager. Haven't yet noticed that expunged folder as a hiding place though--an Empty Trash click had always been final and returned my disk space when needed.

[Martin Fido]

You can also use lsof +L1 (select open files that have been unlinked).

[200_success]

Superfluous use of cat to truncate. In Bourne shell, just > /proc/3446/fd/128 will do.

[Michael Galaxy]

Do NOT do this if your program is actually expected to re-read any portion of the file in the future which may or may not be available in the page cache.

[marcelm]

The information in this answer is correct, but the problem the OP was having was likely not caused by this but by root reserved space (which an other answer addresses).

[Majid Golshadi]

I got an Error " syntax error near unexpected token `(' "

[shivams]

This happened for me when some of the files inside the Trash were owned by root instead of the normal user. As I deleted them from Trash, they ended up inside the expunged folder and I had to manually sudo rm them.

[hpaknia]

I got the syntax error, but the script worked!