iotop and iostat disagree

Question

I'm trying to understand what's going on with my server's disks. It's running Ubuntu 12.04LTS and I'm used to using iostat to know when the disk is the bottleneck. But iostat -xd looks to me like the disk is pretty idle:

Linux 3.2.0-41-generic (loki)   07/05/2013  _x86_64_    (8 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.16     1.23    0.69    1.49     9.94    37.03    43.05     0.52  239.02   14.16  343.06   3.65   0.80
sdb               0.00     0.00    0.00    0.00     0.00     0.00     8.10     0.00    0.25    0.25    0.00   0.25   0.00
dm-0              0.00     0.00    0.82    2.70     9.92    37.03    26.67     0.72  203.84   16.80  260.64   2.27   0.80
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     7.99     0.00  198.63    9.33 1232.65  13.97   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     7.99     0.00  497.11   14.27 1858.12  20.96   0.00

Whereas iotop tells a very different picture:

Total DISK READ:      31.84 M/s | Total DISK WRITE:      19.55 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                     
27472 be/4 mongodb    31.35 M/s    3.92 K/s  0.00 % 46.56 % mongod --config /etc/mongodb.conf
  318 be/3 root        0.00 B/s   94.06 K/s  0.00 % 44.52 % [jbd2/dm-0-8]
 1493 be/4 mongodb     0.00 B/s   48.88 M/s  0.00 %  9.13 % mongod --config /etc/mongodb.conf
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
    6 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
    7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]

with mongod writing tens of megabytes per second.

Where is the discrepancy coming from and which should I believe?

Solution

iostat -xd will give you the averages since the last counter reset (typically the last system reboot) whereas iotop produces averages of the last second. If you want comparable results, you should append a refresh time interval to your iostat call like

iostat -xdmy 5

to average over a time period of 5 seconds.

Comments

[Petter H]

Hi! With an await of 343.06 I would not say that the disk is idle. probably iotop IO% is measured differently than iostat %util...

[I. Cantrell]

Thanks! I thought the -m switch just caused it to automatically refresh.

[the-wabbit]

@Leopd man iotop: -m Display statistics in megabytes per second instead of blocks or kilobytes per second.