No free space left (inode shortage)

Question

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        75G   52G   20G  73% /
udev             10M     0   10M   0% /dev
tmpfs           793M  8.9M  784M   2% /run
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs           397M     0  397M   0% /run/user/0

This is the status of Debian 3.16.43-2+deb8u2 VM running on Oracle VM. I see the free space available but it still throws the error of space not available. Please suggest what will be the fix. If I search, most of links redirect and show that at least one of disks is full. I don't see such case in mine.

df -i output :

Filesystem      Inodes   IUsed  IFree IUse% Mounted on
/dev/sda1      4980736 4935456  45280  100% /
udev            505228     327 504901    1% /dev
tmpfs           507332     545 506787    1% /run
tmpfs           507332       1 507331    1% /dev/shm
tmpfs           507332       7 507325    1% /run/lock
tmpfs           507332      13 507319    1% /sys/fs/cgroup
tmpfs           507332       4 507328    1% /run/user/0

Example Error :

ls -al /tm-bash: cannot create temp file for here-document: No space left on device

Basically if I press tab after any command, I get this error. Rebooting the system fixes it for few minutes. Any other Log file I need to see to find the reason?

Solution

You have run out of inodes on the root filesystem.

It seems you have more files on that filesystem that a typical installation (4,935,456). You have a little growth available to the root user but all non-root accounts are now unable to create new files.

You haven't said what type of filesystem you're using, but if it's ext4 then this related question, How can I increase the number of inodes in an ext4 filesystem?, may be of interest.

Unfortunately it is not possible to increase the number of inodes on an ext4 filesystem without increasing the allocated disk space proportionately. If that isn't an option you will need to backup all of the data, reformat the filesystem with a larger number of inodes, and copy back all your data. (Since you would be wiping the root filesystem this would need to be performed via a rescue disk boot.)

Comments

[Raman Sailopal]

Looks like your inode usage is the issue. The root file system "reserves" a certain amount of space to keep the system operational in such scenarios. Use du / --inodes --max-depth 1. This will give you details of the inode usage in each directory. Change directory and max usage to delve "deeper and deeper" into where the inode usage issue is.

[WangJie]

I agree with @RamanSailopal, inodes seem to be the issue. You can also run df -ih and would take less time than du.

[Raman Sailopal]

df -i has already been performed with the output posted