linux: how to simulate hard disk latency? I want to increase iowait value without using a cpu power

Question

Is it possible to set an arbitrary delay for I/O operations?

I would like to learn how do iowait and disk %util impact the performance. There are a lot of tools to stress the disk (bonnie++ etc.) but I'm looking for a way to introduce a delay like the tc (traffic control) tool does for network. It is possible to disable some optimizations with hdparm tool, like set AAM to quiet or disable DMA and switch to PIO mode - which are only a indirect way to decrease I/O.

Solution 1

device-mapper "delay" devices

Look at the "delay" target for device-mapper devices. This is exactly why it exists.

Example

Here's an example of how to get that going:

Create a place to read/write from

[root@centos6 ~]# dd if=/dev/zero of=/tmp/100M-of-zeroes bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.141834 s, 739 MB/s

Make it into a block device

Dev-mapper only maps from one block device to another, not between files and block devices. That's the job of the loopback device.

[root@centos6 ~]# losetup --show --find /tmp/100M-of-zeroes
/dev/loop0

Get the device size in blocks

Since this is what dev-mapper will need here in a moment...

[root@centos6 ~]# blockdev --getsize /dev/loop0
204800

Set up a "slow" device

# echo "0 204800 delay /dev/loop0 0 200" | dmsetup create dm-slow
(about a 30 second pause here with no output)

The fields in the device mapper setup table in the "echo" command above are:

1. starting sector of this section of the device mapper device (0)
2. number of sectors of this section of the device mapper device (204800)
3. the type of device mapper device for this section (delay)
4. the first argument to "delay", which is the device to use for real reads/writes after the delay (/dev/loop/0)
5. the second argument to "delay" which is the offset in the source device to use (0)
6. the third argument to "delay" which is the ms of time to delay reads (or reads and writes if no further parameters are specified.) (200)

We only have one line since we're treating the entire device mapper device the same, but this lets you have different sectors with different backing devices, only have some of them slow, only having some of them give errors, etc.

See https://linux.die.net/man/8/dmsetup for more info, including the possibly-also-useful "flakey" mapper type. Authoritative documentation on device-mapper's delay feature is at https://www.kernel.org/doc/Documentation/device-mapper/delay.txt

Is it slow?

[root@centos6 ~]# dd if=/dev/mapper/dm-slow of=/dev/null count=25000
25000+0 records in
25000+0 records out
12800000 bytes (13 MB) copied, 10.2028 s, 1.3 MB/s

Yeah, that's pretty slow, especially compared to the original:

[root@centos6 ~]# dd if=/dev/loop0 of=/dev/null count=25000
25000+0 records in
25000+0 records out
12800000 bytes (13 MB) copied, 0.0361308 s, 354 MB/s

So the mapped device is definitely introducing a delay.

Combine the above

I intentionally broke things apart so the process was easy to follow. However, you could easily combine steps above into fewer commands.

Solution 2

With fio and blktrace, you can replay an I/O trace. What type of disruption are you trying to simulate?

If the delays you wish to induce are from an existing production system, you can use that as the basis for your trace.

Comments

[Michael Hampton]

Run your actual workload.

[user156233]

sounds interesting! I'll try them out and came back with a feedback.

[ewwhite]

Good luck. Does this seem like it could do what you're asking for?

[user156233]

I want to learn more about I/O, for example which performance impact will be added during high I/O caused by other VM guests on the same VM. How await, iowait and %util correlate etc.

[slm]

Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference.

[Matt Faus]

To undo this setup, I ran dmsetup remove dm-slow ; losetup -d /dev/loop0 and confirmed with dmsetup ls ; losetup -l

[U. Windl]

This answer is just adding I/O load to an undetermined set of devices, and it does not set a specific I/O delay to a device.

[U. Windl]

I wonder: I have a similar requirement, but reads should be delayed only now and then; is there also a solution for this? The cited flakey seems unable to do that.

[Steve Bonds]

In a word, no. The source code for the dm-delay driver curently uses a simple countdown timer: github.com/torvalds/linux/blob/master/drivers/md/dm-delay.c One possible (incomplete) workaround would be to set up a device with certain sections mapped to different delays.