iotop showing 1.5 MB/s of disk write, but all programs have 0.00 B/s

Question

I don't understand iotop output: it shows ~1.5 MB/s of disk write (top right), but all programs have 0.00 B/s. Why?

The video was taken as I was deleting the content of a folder with a few millions of files using perl -e 'for(<*>){((stat)[9]<(unlink))}', on Kubuntu 14.04.3 LTS x64.

iotop was launched using sudo iotop.

Solution

The information shown by iotop isn't gathered in the same way for individual processes and for the system as a whole. The “actual” global figures are not the sum of the per-process figures (that's what “total” is).

All information is gathered from the proc filesystem.

• For each process, iotop reads data from /proc/PID/io, specifically the rchar and wchar values. These are the number of bytes passed in read and write system calls (including variants such as readv, writev, recv, send, etc.).
• The global “actual” values are read from /proc/vmstat, specifically the pgpgin and pgpgout values. These measure the data exchanged between the kernel and the hardware (more precisely, this is the data shuffled around by the block device layer in the kernel).

There are many reasons why the per-process data and the block device layer data differ. In particular:

• Caching and buffering mean that I/O happening at one layer may not be happening at the same time, or the same number of times, at the other layer. For example, data read from the cache is accounted as a read from the process that accesses it, but there's no corresponding read from the hardware (that already happened earlier, possibly on behalf of another process).
• The process-level data includes data exchanged on pipes, sockets, and other input/output that doesn't involve an underlying disk or other block device.
• The process-level data only accounts for file contents, not metadata.

That last difference explains what you're seeing here. Removing files only affects metadata, not data, so the process isn't writing anything. It may be reading directory contents to list the files to delete, but that's small enough that it may scroll by unnoticed.

I don't think Linux offers any way to monitor file metadata updates. You can monitor per-filesystem I/O via entries under /sys/fs for some filesystems. I don't think you can account metadata I/O against specific processes, it would be very complicated to do in the general case since multiple processes could be causing the same metadata to be read or changed.

Comments

[Gilles 'SO- stop being evil']

@RuiFRibeiro You can watch which file rm -r is currently processing by straceing it, but that won't give you a very useful estimate of the completion percentage since the order of traversal in each directory is somewhat unpredictable. If there's only one massive operation going on in that filesystem, and there aren't too many hard links involved, watching df -i tells you how many files have been processed.