mysqld is not starting up anymore due filled disk

Question

How to clean logs or find largest files and reduce the space? also i do not understand 1.7T is not even used in this critical moment.

Mysqld is not running because 99% is used.

# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs           20G   19G  279M  99% /
/dev/root        20G   19G  279M  99% /
devtmpfs         32G  268K   32G   1% /dev
/dev/md3        1.8T  196M  1.7T   1% /home
tmpfs            32G     0   32G   0% /dev/shm
/dev/root        20G   19G  279M  99% /var/named/chroot/etc/named
/dev/root        20G   19G  279M  99% /var/named/chroot/var/named
/dev/root        20G   19G  279M  99% /var/named/chroot/etc/named.conf
/dev/root        20G   19G  279M  99% /var/named/chroot/etc/named.rfc1912.zones
/dev/root        20G   19G  279M  99% /var/named/chroot/etc/rndc.key
/dev/root        20G   19G  279M  99% /var/named/chroot/usr/lib64/bind
/dev/root        20G   19G  279M  99% /var/named/chroot/etc/named.iscdlv.key
/dev/root        20G   19G  279M  99% /var/named/chroot/etc/named.root.key

Solution 1

MySQL data is stored in /var/lib. There is no more space in /var. MySQL does not start. Pretty simple actually. Have a look at /var/log and clean it up. I would recommend something like (delete all gz files in /var/log) :

$ find /var/log -iname "*.gz" -delete

Of course, you might want to check what's being deleted first:

$ find /var/log -iname "*.gz"

1.7TB is given to /home. However /var is not /home. So /home is not being used. Simple as well.

You clearly should have separated /var on a much larger partition when you set up your system. Now, the best solution I can come across is to move the MySQL datadir to a larger partition (/home) :

$ mv /var/lib/mysql /home
$ ln -s /home/mysql /var/lib/mysql

If you don't want to use a link, you should reconfigure MySQL to use the new datadir stored in /home. Edit /etc/mysql/my.cnf and change this:

datadir = /var/lib/mysql

to this:

datadir = /home/mysql

Now that MySQL has space available, it should start without trouble. You should really reconsider your partition scheme if you're hosting such an important database though.

You might also want to investigate the contents of your logs, since applications shouldn't be writing so much to them if everything is going well. Something may be struggling to work properly somewhere.

Solution 2

This is a useful command to find the largest files:

du -ak /var | sort -nr | less

Usage:

du -ak summarizes the disk usage of all (-a) files in the /var partition and prints the size in kilobytes (-k).

sort -nr Concatenates the list of files and sorts them into a reversed (-r) numerical (-n) order.

less Will paginate the output so that you can see the largest offenders at the top and move back and forward through the list.