/var file system gets full quickly

Question

/var file system gets full quickly. Even the inodes' usage reaches 100%. We have to clear the logs regularly. We are not able to ssh also because of this. We have do a hard reboot eventually. We are not able to identify the exact root cause for the same. Kindly help us out in pin pointing the exact cause and resolving the issue.

What is the proper way to prevent /var from filling the disk space on a server with limited storage space?

Solution 1

First of all, the pipe on head -n10 is only going to print out the first 10 directory entries in /var. If you have more than 10 directories, they are not being reported on. Secondly, du -h is not quite the best tool to use here as it is reporting duplicated allocation because child directories are being reported on.

Try this instead to get a better idea on allocation of all first level directories under /var:

# find /var -maxdepth 1 -type d -exec du -smh '{}' \;

21G     /var
18G     /var/lib
4.0K    /var/local
384K    /var/www
86M     /var/cache
3.3G    /var/log
12K     /var/mail
16K     /var/lost+found
7.8M    /var/backups
840K    /var/spool
4.0K    /var/tmp
4.0K    /var/opt

If you need to drill down into one of those top levels, to report allocation under the parent, simply add the sub directory to the original command. I can easily see elasticsearch is the culprit behind 18G of allocation within /var on my system:

# find /var/lib -maxdepth 1 -type d -exec du -smh '{}' \;
18G     /var/lib
...
8.0K    /var/lib/vim
18G     /var/lib/elasticsearch

@fpmurphy suggest the yum directory may be involved with your problem. To extrapolate on that, whenever you work with yum all the packages which are downloaded to update the system are retained under /var/cache/yum by default. You may want to check the contents of the file /etc/yum.conf and check for the keepcache setting. What you post suggests that keepcache has a value of 1.

If /var/cache/yum is indeed filling up, you may be able to runyum clean all to fix things for now. There is evidence this command may not work as advertised however so take heed. I would suggest allocating more space to /var if possible.

Solution 2

/var is the location where most of the data is stored by applications. This includes log files, libraries or even binary data for applications like MySQL or MongoDB.

Coming to your problem, as you have mentioned clearing logs frequently, you need to monitor what writes to /var/log the most and set appropriate logrotate policy for the same.

To find the top 10 directories (inside /var) by disk usage:

sudo du -h /var | sort -hr | head -n10

To find the top 10 files (inside /var/log) by disk usage

sudo find /var/log -maxdepth 2 ! -path . -printf "%s %p\n" | sort -rn -k1  | head

---

What is the disk size available to the machine?

How frequently does it get full?

What is the output of the above two commands?

Comments

[David Schwartz]

Your question includes almost no useful information. How big the partition? How quickly does it fill? Do you see files appear? If so, what are there names, how big are they, what's in them? What tools have you used to look at the usage? What did they tell you?

[alchemy]

Also, see here: unix.stackexchange.com/questions/85184/… var-partition-gets-full-whats-the-solution

[yaxe]

my problem was with yum.pid not getting created because of cache being full. thanks for predicting the scenario. Last I cleaned the /var and then ssh wasn't working (and it shouldn't have right.)