How to monitor which files lots of data is being written to on Linux?

Question

I can monitor I/O with tools such as iotop. It allows me to identify which processes do how much I/O.

Now I'd like to know which files those processes write all that data to. How would I do that?

E.g. something like "ok I know Tomcat is doing a lot of I/O. Which files is it mostly reading/writing from/to"?

Solution

I know two ways to get that kind of information.

1. Manually using lsof.
   The good old lsof can show you what files are being accessed by a process or thread, along with a few other pieces of information. In iotop -o, eyeball and take note of the TID (Thread ID) value of the process or thread you need to inspect. Then close iotop and run lsof -p [pid/tid]. If you need to sort the output, pipe it to sort. For example, lsof -p [pid or tid] | sort -n -k 7,7 -r will sort the output from lsof by the seventh column (SIZE/OFF) in reverse order (biggest to smallest).

2. Using fatrace.
   This new addition to Linux is similar to inotify, except it doesn't target specific files/directories. It shows you aggregate disk I/O based on files being accessed. Depending on your distribution, you might or might not have access to this nifty little program in precompiled executable form. The oldest distribution that provides fatrace in its official repositories that I know of is Ubuntu 12.04. Debian 7, which I'm using, doesn't have it.

Comments

[Johannes Ernst]

These seem to tell me which files my process accesses, but doesn't tell me which of those files are the ones that all the data is being written to. The "size" in lsof seems to be the file size, NOT the rate at which the process writes into the file. And I don't see anything related to size/rate in manpages.ubuntu.com/manpages/wily/man1/fatrace.1.html at all.

[mpr]

fatrace is perfect. Put it in the toolbox.