df command not showing correct values

Question

This is a RHEL server, I'm running a MySQL server on it, the database and log files (however, logging is disabled) are located on the /srv directory where plenty of space is available.

Recently I had a crashed table, so I tried fixing it but on the next day I found that MySQL can't respond to many of queries with an error indicating that there is no disk space:

ERROR 1030 (HY000): Got error 28 from storage engine

So I ran the following command to see what's taking space

[root@tms /]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/server-slash
                      9.9G  9.5G     0 100% /
tmpfs                 7.8G     0  7.8G   0% /dev/shm
/dev/sda1             485M   58M  402M  13% /boot
/dev/mapper/server-var
                      739G  252G  450G  36% /srv

Surprisingly it's the / directory. But more surprisingly is that the directories under / do not indicate used space more than 2 GB, while df shows total space 9.9 GB for /.

[root@tms /]# du -sh /*
7.5M    /bin
48M     /boot
200K    /dev
24M     /etc
4.0K    /home
223M    /lib
21M     /lib64
16K     /lost+found
4.0K    /media
4.0K    /mnt
183M    /opt
...deleted some file-not-found errors for files under /proc
0       /proc
144K    /root
14M     /sbin
4.0K    /selinux
252G    /srv
0       /sys
44K     /tmp
917M    /usr
259M    /var

So why does df show wrong values ? And how can I findout what's actually taking space ?

Solution 1

A common programming technique is to create a temporary file and immediately unlink() it. This leaves the file (and its space) available for the duration of the program but automatically causes its removal when the program using it terminates. One advantage is that no epilog (cleanup) code is necessary to write.

To determine if you have a process holding an unlinked file open, do:

lsof -a +L1 /dev/server_slash

(or)

lsof +D /dev/server_slash +L1

Look for any files with an NLINK value of zero (0). These would be files with a zero link count that will vanish when the last process terminates. The SIZE/OFFSET column will offer the character size of the file in question.

Solution 2

1) df doesn't report the space reserved for root (5% by default) on unix-style filesystems. So dfwill always report less than you ought to have.

2) Here though, I will guess you've run your database without your srv-partition mounted. Without the srv-partition mounted, things will have been written to the mount-point ie. to under the srv-directory in the /-partition. (root-partition), thus using up very much space on the /-partition.

However, when you do mount the srv-partition on the srv-directory, all the files under the srv-directory of the /-partition becomes "hidden" by the partition you've "mounted over" it - but the space is still used-up, although you can't see it or access it to delete it.

Try unmounting the srv-partition (go to singleuser-moder/runlevel 1/maintenance mode so everything is stopped), check that it really is unmounted (with mount), and see if there are files hiding under the directories you usually use as mountpoints (usr, srv, home, var, tmp, ...) in the /-partition. I'll bet you'll find something big in some of them. When you've done and reboots, make sure the srv-partition actually gets mounted the way it should.

This - that you can have stuff in directories that usually are hidden by mounted partitions - can actually sometimes be useful.

Comments

[Muhammad Gelbana]

Pardon me but I'm not an admin and I'm commanding the server remotely, so hopefully there is simple/remote way to solve this. I ran the mount command though and it returned the following: /dev/mapper/server-slash on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sda1 on /boot type ext4 (rw) /dev/mapper/server-var on /srv type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) Which indicates that the /srv directory is mounted already.

[Muhammad Gelbana]

If I switch to runlevel 1, could this disconnect me ?

[terdon]

@MuhammadGelbana yes, it is mounted (otherwise it would not appear in df), that is the problem. What Baard Kopperud is suggesting is that at some point you ran he database when srv was not mounted and later mounted srv thus hiding files that may have been in the /srv directory. And yes, switching to runlevel 1 will disconnect you.

[BillThor]

When I want to know what is on a partition that contains mounts (/ and others), I mount it on an second mount point. This exposes files hidden behind mount points. /mnt is usually available and unused. Remember to unmount it when you are done.

[Muhammad Gelbana]

Unfortunately I ended the process before trying your suggesting but actually ending the process released A LOT of space so my best guess is that you are correct. Thank you for your insightful answer :)