I have high io stat. High writes. But what files are being written?

Question

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11.50    0.02    5.38    0.07    0.00   83.04

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdc               0.01    89.92    0.26   41.59     3.36   457.19    22.01     0.23    5.60   0.09   0.38
sdb               0.10    15.59    0.40   14.55     8.96   120.57    17.33     0.04    2.91   0.07   0.11
sda               0.13    45.37    0.96    8.09    20.06   213.56    51.63     0.02    2.64   0.16   0.14
sde               0.01    31.83    0.09   11.34     0.94   103.56    18.29     0.04    3.52   0.14   0.16
sdd               0.01    48.01    0.13   19.81     1.58   202.16    20.44     0.11    5.62   0.13   0.25

Is there a way to know what files are being written? 457 kB/s

Also this other linux machine have this same problem.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          20.50    0.00   46.48   20.74    0.00   12.28

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.17    11.61    0.99    3.51    36.65    59.43    42.70     0.10   23.20   3.84   1.73
sdb               0.55   224.18   24.30   97.45   246.48  1287.12    25.19     3.96   32.53   7.88  95.91
sdd               0.53   226.75   25.56   90.96   283.50  1271.69    26.69     3.43   29.44   8.22  95.75
sdc               0.00     1.76    0.28    0.06     4.73     7.26    70.41     0.00   12.00   2.12   0.07
dm-0              0.00     0.00    1.11   14.77    36.41    58.92    12.01     1.00   62.86   1.09   1.74
dm-1              0.00     0.00    0.04    0.12     0.17     0.49     8.00     0.00   21.79   2.47   0.04
dm-2              0.00     0.00    0.01    0.00     0.05     0.01     8.50     0.00    7.90   2.08   0.00

1200 write request per second for a server that host nothing

Solution 1

Well, you could try the following commands which worked for me in RHEL6:

1) Whatever device you see in "iostat" output performing more I/O, use it with fuser command as follows:

fuser -uvm device

2) You will get a list of processes with the user name causing more I/O. Select those PIDS and use it in the lsof command as follows:

lsof -p PID | more

3) You will get a list of files/directories along with the user performing maximum I/O.

Solution 2

It seems there is no tools to find out I/O throughput per file other than within process using the file. But there are ways to find out process I/O throughput.

iotop - It is a top/iftop like utility that show process I/O throughput.

After pin pointing which process is having heavy I/O, use following to find out what file is being used

lsof -c <process name>

That should narrow down the scope and help pin point the source.

Solution 3

You can use inotifywait from inotify-tools to find out exactly which file is being written to. This won't tell you how much data is being written, but it will at least tell you what files are being written to as it happens.

As an example, this command will print the file name as soon as any are created, modified, or deleted in /tmp:

$ sudo inotifywait -e modify -e attrib -e move -e create -e delete -m -r /tmp
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
/tmp/ CREATE test
/tmp/ MODIFY test

Unfortunately you will need to guess which directory contains the files being written to. This fails if you try to use it on the root directory, though apparently that can be overridden:

$ sudo inotifywait -e modify -e attrib -e move -e create -e delete -m -r /
Setting up watches.  Beware: since -r was given, this may take a while!
Failed to watch /; upper limit on inotify watches reached!
Please increase the amount of inotify watches allowed per user via `/proc/sys/fs/inotify/max_user_watches'.

Solution 4

csysdig. https://github.com/draios/sysdig/wiki/Csysdig-Overview

They even have a link to a video demo-ing how to do it. https://www.youtube.com/watch?v=UJ4wVrbP-Q8

Solution 5

Use systemtap to monitor each write/pwrite syscall. You can do some accounting and you will see which file/fd gets the most bytes.

View more solutions

Comments

[Valerio Bozz]

It would be interesting to explain how the first output was obtained, I think it's iostat but not sure 100% now

[user2935706]

how to get the process? lsof also produce nothing. Something is writing so many things and I don't know what

[John Siu]

Sorry, I put the wrong io utility. It should be iotop, not iostat. I corrected my answer.

[John Siu]

(1) Can you also update what distro(redhat/centos/ubuntu ...) you are running? (2) Is this a fresh install? (3) Any file sharing like samba running?

[user2935706]

root@host [~]# fuser -uv /home4 root@host [~]# fuser -uvm /home4 USER PID ACCESS COMMAND /home4: newgames 18297 ..c.. (newgames)php cuntschi 18407 ..c.. (cuntschi)php newgames 18457 ..c.. (newgames)php