Why does high disk I/O reduce system responsiveness/performance?

Question

I never quite understood why high disk I/O slowed the system so much. It's strange to me because I would expect the slow-down to affect only those processes dependent on the hard/optical drive data, but the slow-down affects even stuff loaded onto RAM. I'm here referring to iowait.

Why does the processor wait, instead of doing other work? Can anyone explain this limitation and why it hasn't been solved in Linux kernel? Is there a kernel out there that doesn't have this problem?

[note] There has been some progress in this performance area. For one, the later kernels (2.6.37 in my case) are much more responsive.

Solution 1

Operating systems make use of virtual memory so that more memory can be used than there is physical RAM available. When the kernel decides that it has a better use for a physical memory page, its content may be "paged out" for storage on disk. When such a virtual memory page is accessed while paged out, it generates a page fault and is moved back from the disk to RAM.

Page faults are a disaster for performance because disk latency is measured in milliseconds, while RAM latency is measured in nanoseconds. (1 millisecond = a million nanoseconds!)

Memory is not only used by user processes, but also by the kernel for things like file system caching. During file system activity, the kernel will cache recently used data. The assumption is that there is a good chance that the same data will be used again shortly, so caching should improve I/O performance.

Physical memory being used for the file system cache cannot be used for processes, so during file system activity more process memory will be paged out and page faults will increase. Also, less disk I/O bandwidth is available for moving memory pages from and to the disk. As a result processes may stall.

Solution 2

As far as I understand it, IOwait means that a process, not the processor, is waiting for IO to become available. Processors have gained much more speed than hard drives, meaning more code will finish faster and then the disk will need to be read. When multiple more needs to be read than the drive can do fast enough, you end up with processor's waiting. The way it's decided who gets to read/write to the disk is determined by the block scheduler, in most cases now CFQ. If you're using CFQ, and you need a process to use less of the overall IO time to increase system responsiveness, you can use ionice -c3 <processid>. This tells the system to only give this process IO only when nothing else needs it.

This is still interesting and explains the iowait problem better.

Comments

[Michael Mrozek]

Didn't xeno explain exactly how this has been solved in the Linux kernel the last time you asked?

[Steven D]

Given the edits, I think the intent is to have the previous question be about the progress being made towards fixing the problem while this question is about why the problem exists.

[tshepang]

@mic Steven is right. We had a long discussion about what I meant with the previous question. xeno's answer was so good that I edited the question to fit it, and I re-asked the original question here.

[Michael Mrozek]

I understand, but your question seems to contradict the other one; here you say "Can anyone explain this limitation and why it hasn't been solved in Linux kernel? Is there a kernel out there that doesn't have this problem?", but xeno's answer starts with "I think for the most part it has been solved."

[tshepang]

@mic Not really. The kernel still does iowait, meaning it still waits. I view xeno's answer as more of system responsiveness has been improved. And that I agree, as I've noted on the question.

[Bratchley]

I know this is as old as dirt but, depending on how the it's coming in, high amounts of I/O could result in a lot of interrupts being generated and the resulting context switches wasting CPU time.