df reports an ext4 partition is full, but there is no data

Question

I recently received a warning, that my home-partition is full. It is an Ext4-Partition mounted on /home, /dev/sda is a 240 GB SSD:

hannes@XFLR6 ~> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              19G  5.2G   13G  30% /
none                  3.9G  792K  3.9G   1% /dev
none                  3.9G  2.4M  3.9G   1% /dev/shm
none                  3.9G  712K  3.9G   1% /var/run
none                  3.9G     0  3.9G   0% /var/lock
/dev/sda5             193G  175G  7.9G  96% /home
/dev/sdb5             357G   92G  264G  26% /mnt/schacht

as you can see, df -h (and gparted) reports that /dev/sda5 is 96% full. However, the Ubuntu Disk Usage Analyzer and du -h only find about 89 GB of data. ~/.gvfs is empty and there are no other filesystems mounted below /home. How can that be? I have already tried running du as root, but it doesn't change anything.

root@XFLR6 ~# sudo fdisk -l /dev/sda

Disk /dev/sda: 240.1 GB, 240057409536 bytes
255 heads, 63 sectors/track, 29185 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0003e4c5

Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        2432    19530752   83  Linux
/dev/sda2            2432       28450   208984065    5  Extended
/dev/sda5            2432       27963   205077504   83  Linux
/dev/sda6           27963       28450     3905536   82  Linux swap / Solaris

Edit: whoops - I only ran du on ~ not on /home – there was a lot of data unintentionally copied to /home. My bad, sorry.

Solution 1

It could be that some process still has deleted files open. If this is the case then they will not appear in the du output but would still be counted in the df output.

One quick way to check for these is to list /proc as user root (hint sudo su should get you a root shell). Any open, but deleted files will have (deleted) at the end of the symbolic link target name.

ls -l /proc/*/fd/* | grep deleted | grep /home

should give you a list of any files open. Once you have that then an ls -lL of the specific file should give you the size of the file.

As an example (using /tmp on my system because there are no examples on /home here) I see a few files owned by user mysql.

richm@viking:/$ sudo su
root@viking:/# ls -l /proc/*/fd/* | grep deleted | grep /tmp
lrwx------ 1 root     root     64 Oct 13 06:30 /proc/1489/fd/11 -> /tmp/ibwmCqpg (deleted)
lrwx------ 1 root     root     64 Oct 13 06:30 /proc/1491/fd/12 -> /tmp/ib9MTMQi (deleted)
root@viking:/# ls -lL /proc/1489/fd/11
-rw------- 0 mysql2 mysql2 0 Aug 24 14:09 /proc/1489/fd/11
root@viking:/# ls -lL /proc/1491/fd/12
-rw------- 0 mysql mysql 1320 Oct 15 13:40 /proc/1491/fd/12

If you have any processes with large deleted files open then stoping the process should be enough to reclaim the disk space. Alternatively a reboot should do the same thing.

Solution 2

Each file system only has a certain amount of inodes and blocks that can be stored on it. Even in case there is enough space, you can't go further.

Check your settings with

dumpe2fs /dev/sda5

(only the top 50 lines are important here).

If you have a lot of small files that are smaller than the blocksize, a lot of space is wasted.

Comments

[Admin]

Can you please show the output of the mount command?

[Admin]

Did you check Trash! If there are files moved to trash whilst being root you do not own these files and emptying trash does not remove them...

[psusi]

He is out of space, not inodes

[ddeimeke]

Did you read the last sentence?

[psusi]

what does that have to do with the original question? If you are out of space, then you are out of space, no matter how many or what size files you have. df reports the correct used size no matter if the files are large or small, unless you use the -b flag.

[Mechanical snail]

I think lsof +L1 is a more reliable way to find open deleted files.

[tvn]

I used the following command to get an nice overview: ls -l /proc/*/fd/* | grep deleted | awk '{print $9}' | xargs -i ls -lhL {}

[kbulgrien]

+1 for lsof +L1. Recently I had a full /var/log but du showed only 50% of the filesystem in use. ls -l /proc/*/fd/* was kind of complicated relative to identifying the problem (piping/sub-processes needed to get detail) whereas lsof +L1 just did the right thing and showed a deleted file holding 1.6GB of space.