Does C++ ofstream file writing use a buffer?

Question

Below are two programs that write 50,000,000 bytes to a file.

The first program, written in C, utilizes a buffer, that once filled to a arbitrary value, writes to disk, and then repeats that process until all 50,000,000 bytes are written. I noticed that as I increased the size of the buffer, the program took less time to run. For instance, at BUFFER_SIZE = 1, the program took ~88.0463 seconds, whereas at BUFFER_SIZE = 1024, the program only took ~1.7773 seconds. The best time I recorded was when BUFFER_SIZE = 131072. As the BUFFER_SIZE increased higher than that, I noticed that it began to actually take a little longer.

The second program, written in C++, utilizes ofstream to write one byte at a time. To my surprise, the program only took ~1.87 seconds to run. I expected it to take a minute or so, like the C program with BUFFER_SIZE = 1. Obviously, the C++ ofstream handles file writing differently than I thought. According to my data, it is performing pretty similarly to the C file with BUFFER_SIZE = 512. Does it use some sort of behind-the-scenes buffer?

Here is the C program:

const int NVALUES = 50000000; //#values written to the file
const char FILENAME[] = "/tmp/myfile";
const int BUFFER_SIZE = 8192; //# bytes to fill in buffer before writing

main()
{
    int fd;  //File descriptor associated with output file
    int i;
    char writeval = '\0';
    char buffer[BUFFER_SIZE];

    //Open file for writing and associate it with the file descriptor
    //Create file if it does not exist; if it does exist truncate its size to 0
    fd = open(FILENAME, O_WRONLY|O_CREAT|O_TRUNC, S_IRUSR|S_IWUSR);

    for(i=0;i<NVALUES;i++)
    {
        //Package bytes into BUFFER_SIZE chunks 
                //and write a chunk once it is filled
        buffer[i%BUFFER_SIZE] = writeval;
        if((i%BUFFER_SIZE == BUFFER_SIZE-1 || i == NVALUES-1))
            write(fd, buffer, i%BUFFER_SIZE+1);

    }

    fsync(fd);

    close(fd);
}

Here is the C++ program:

int main()
{
    ofstream ofs("/tmp/iofile2");
    int i;

    for(i=0; i<50000000; i++)
        ofs << '\0';

    ofs.flush();
    ofs.close();

    return 0;
}

Thank you for your time.

Solution 1

Yes, ostreams use a stream buffer, some subclass of an instantiation of the template basic_streambuf. The interface of basic_streambuf is designed so that an implementation can do buffering if there's an advantage in that.

However this is a quality of implementation issue. Implementations are not required to do this but any competent implementation will.

You can read all about it in chapter 27 of the ISO standard, though maybe a more readable source is The C++ Standard Library: A Tutorial and Reference (google search).

Solution 2

Yes, all stream operations are buffered, though by default the standard input, output and error output are not so that interactions with the C IO is less surprising.

As already alluded, there is a base class streambuf that is used behind the scenes. It is provided with its own buffer, which size is an implementation detail.

You can check (experimentally) how much this buffer is by using streambuf::in_avail, assuming that input filestream and output filestream are setup with the same buffer size...

There are two other operations that you can do here that might be of interest:

• you can change the streambuf object used by a stream, to switch to a custom version
• you can change the buffer used by the streambuf object

both should be done either right after creating the stream or after a flush, lest some data is lost...

To illustrate the buffer change, check out streambuf::putsetbuf:

#include <fstream>
#include <vector>

int main () {
  std::vector<char> vec(512);

  std::fstream fs;
  fs.rdbuf()->pubsetbuf(&vec.front(), vec.size());

  // operations with file stream here.
  fs << "Hello, World!\n";

  // the stream is automatically closed when the scope ends, so fs.close() is optional
  // the stream is automatically flushed when it is closed, so fs.flush() is optional

  return 0;
}

Now you can repeat the experiments you did in C to find the sweet spot :)

Solution 3

Per this, ofstream has an internal filebuf pointer, can be read through the rdbuf function, which points to a streambuf object, which is this:

streambuf objects are usually associated with one specific character sequence, from which they read and write data through an internal memory buffer. The buffer is an array in memory which is expected to be synchronized when needed with the physical content of the associated character sequence.

I bolded the important bits, it seems that it does make use of a buffer, but I don't know or haven't found out what kind of buffer it is.