linux - disk space missing

Question

# df -h /
Filesystem            Size  Used Avail Use% Mounted on
rootfs                9.9G  7.2G  2.2G  77% /

# du -hx --max-depth=0 /
3.2G    /

As you can see, df says 7.2GB is used, but du can only find 3.2GB of it. The server has been rebooted since I noticed this, so it's not a deleted file. Additionally, lsof doesn't show me anything interesting. What else could it be?

Solution 1

I was having the exact same issue on my ext4 system and just wanted to post my solution for future reference. When my drive initially filled out, I deleted a bunch of logs out of /var/log. That cleared up a couple GB, but within a few days I ran out of space again, and du -h and "mount --bind / /mnt" did not point to the culprit. What finally got it was when I ran lsof.

lsof
...
rsyslogd   1766      root    2w      REG                9,1   12672375940     264014 /var/log/messages (deleted)
...

When I had deleted the messages log file, the rsyslog service still kept it open, but hidden. Running a "touch /var/log/messages;service rsyslog restart" cleared up the problem and my disk space was reclaimed.

The lsof output can be a bit overwhelming, especially if you have a busy system (it was over 1000 lines long on mine). If you grep for "deleted" in the lsof output, it should help to pinpoint the problem process.

Solution 2

Since you're using the -x option, I assume you have other filesystems mounted? It could be that you have another partition mounted on top of a directory that wasn't empty.

Solution 3

Make sure all folders "behind" your mountpoints are empty. Out of experience i'd say most probably you hide some data behind a mountpoint.

you can check what is in the folder behind the mountpoint without having to unmount the disk/partitions (which can be nice if for example you would have to unmount /usr). Just do a

# mount --bind / /mnt
# du -shx /mnt

also as a side note:

# du -shx /

does the same as your du... but is quite a bit shorter (-s stands for summarize)

Solution 4

There could be a couple of problems: (1) a process has a large file or number of files open that have been deleted, or (2) you might have some type of filesystem problem remedied by an fsck of the drive. However, this can only be done when the drive is not mounted. Several Linux provide a method of FULL fsck of the root file at boot time by:

 touch /forcefsck

and then rebooting. I know this exists in SuSE and RedHat varients.

View more solutions

Comments

[Zoredache]

Have you tried a fsck?

[MastaJeet]

Is your filesystem pathological in some way? E.g., many symlinks, many small files, many bad blocks? Assuming the fs type is ext[23] the output of e2freefrag and dumpe2fs may be helpful.

[Zoredache]

He did mention he has rebooted recently, which should rule out the open file issue.

[wes]

this was exactly it, thanks. the --bind flag was also very helpful - I would have had a much harder time finding it without it.

[xwild]

You save me a lot of time, I have a backup script which performs its operations by anacron from ssd to hdd, and when I temporary removed hdd it had pushed a backup to hdd mount point. And of course I didn't catch this.

[dgould]

Same here, except in my case it was both /var/log/messages and var/log/secure (both held by rsyslogd). Thanks!