df command shows 100% full, even after deleting files it shows same usage (100%)

Question

Nand Flash storage partition shows as 100% full with df command. When manually calculated the usage it is about 7-80%(max) only. Deleted few files (say around 50~60MB), still df command output did not change. I can not create a new file, error: "there is no space left on the storage device". Tried sync command, no difference.

Filesystem           1K-blocks      Used Available Use% Mounted on
tmpfs                   262144     20780    241364   8% /tmp
tmpfs                      512         0       512   0% /dev
/dev/ubi0_0            1275540   1275540         0  100% /storage

When the actual storage space is not full (as per the manual calculation), why doe df command shows it as 100% full?

Solution

A reboot and re-mount would solve the problem I believe.

Reason: The reason behind this is that df utilize statfs(2) system call to get the File system stat. What it means that it checks open kernel file descriptors to count free space as it named (df = disk free). You will get a different result if you use du. df showing 100% used because the files were deleted are not yet released from the kernel file descriptor.

May be the following scenario will help you to understand:

1. A running process named xyz.service are using a file named something.dump which resides at /storage partition.

2. The file discriptor of something.dump is listed in the process file descriptor table of xyz.service.

3. Then you deleted something.dump but xyz.service is still running.

4. The xyz.service is not aware of the updated status of something.dump file. So file descriptor of something.dump is still there.

5. Then you ran df command and kernel checked all the process table to see used filed descriptor to calculate free space.

6. Kernel saw that the inodes of that something.dump file is still listed at the xyz.service process's file descriptor table. So it counts something.dump is still at the file system hence it doesn't consider something.dump's inodes as free. Which is the reason you are seeing more usage.

7. So when you SIGKILL xyz.service those inodes will be released and you probably can see the free spaces.

8. Restarting the system will do the same thing as step 7.