Understanding Ruby and OS I/O buffering

Question

How does IO buffering work in Ruby? How often is data flushed to the underlying stream when using the IO and File classes? How does this compare to OS buffering? What needs to be done to guarantee that given data has been written to disk, before confidently reading it back for processing?

Solution 1

The Ruby IO documentation is not 100% clear on how this buffering works, but this is what you can extract from the documentation:

• Ruby IO has its own internal buffer
• In addition to that the underlying operating system may or may not further buffer data.

The relevant methods to look at:

• IO.flush: Flushes IO. I also looked at the Ruby source and a call to IO.flush also calls the underlying OS fflush(). This should be enough to get the file cached, but does not guarantee physical data to disk.
• IO.sync=: If set to true, no Ruby internal buffering is done. Everything is immidiately sent to the OS, and fflush() is called for each write.
• IO.sync: Returns the current sync setting (true or false).
• IO.fsync: Flushes both the Ruby buffers + calls fsync() on the OS (if it supports it). This will guarantee a full flush all the way to the physical disk file.
• IO.close: Closes the Ruby IO and writes pending data to the OS. Note that this does not imply fsync(). The POSIX documentation on close() says that it does NOT guarantee data is physically written to the file. So you need to use an explicit fsync() call for that.

Conclusion: flush and/or close should be enough to get the file cached so that it can be read fully by another process or operation. To get the file all the way to the physical media with certainty, you need to call IO.fsync.

Other related methods:

• IO.syswrite: Bypass Ruby internal buffers and do a straight OS write. If you use this then do not mix it with IO.read/write.
• IO.sysread: Same as above, but for reading.

Solution 2

Ruby does its internal buffering on top of the OS. When you do file.flush Ruby flushes its internal buffer. To ensure the file is written to disk you need to do file.fsync. But in the end you can not be certain the file is written to disk anyway, it depends on the OS, the hdd controller and the hdd.