How to test disk latency on Linux?

Question

Made an upgrade of RAID firmware and latency as it seems have increased.

Could you please advise on the most accurate way to examine disk read and write latencies?

Solution 1

IOzone great benchmark.

Run Random Read, Random Write, Random Mix test to check latencies.

Solution 2

you can use dd to mesure the latency of a disk

eg.

dd if=/dev/zero of=/root/testfile bs=512 count=1000 oflag=dsync

Solution 3

The common tool in Linux hard disk tuning and basic performance monitoring is hdparm.

Solution 4

I would suggest using fio as I described in answer https://askubuntu.com/a/991311/50254

The fio allows you to get more info about the distribution of latencies for different work loads (anything between single threaded huge sequential writes QD32 and mixed random read and write 4k QD1 with multiple threads).

I think that if you assume that hardware has higher latency than previously, you should test single threaded random 4k read QD1 (this is the most latency sensitive operation that I can think):

fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=randread --size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 --iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting

That will create test file fio-tempfile.dat in the current working directory so make sure that you first change to device you want to test. (If you want to test raw device performance instead of performance with a file system, you can use raw device as the filename. You're going to lose current contents of said device, though.)

You can also test the same thing with writing instead of reading data:

fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=randwrite --size=500m --io_size=10g --blocksize=4k --ioengine=libaio --fsync=1 --iodepth=1 --direct=1 --numjobs=1 --runtime=60 --group_reporting

If you see huge increase in latency with write you can be pretty sure that old firmware allowed write back cache and the new firmware does not allow that. Unless you have battery backed RAID controller or correctly configured UPS, you do not want write back cache if you consider your data important.

View more solutions

Comments

[Admin]

time command can be used. copy a big file and calculate how much time it taking

[Admin]

Look at this question: Linux Tool command to check file system performance

[goetzc]

I you are gonna give half of an answer, better include the description of the article you copied this from: thomas-krenn.com/en/wiki/Linux_I/O_Performance_Tests_using_d‌​d e.g. In this test, 512 bytes were written one thousand times. Thereby, the 0.084 seconds that were measured for one thousand accesses corresponded to precisely 0.084 ms for each access.

[Mithrandir]

@goetzc feel free to edit my answer directly.