How to read a binary file into a vector of unsigned chars

Question

Lately I've been asked to write a function that reads the binary file into the std::vector<BYTE> where BYTE is an unsigned char. Quite quickly I came with something like this:

#include <fstream>
#include <vector>
typedef unsigned char BYTE;

std::vector<BYTE> readFile(const char* filename)
{
    // open the file:
    std::streampos fileSize;
    std::ifstream file(filename, std::ios::binary);

    // get its size:
    file.seekg(0, std::ios::end);
    fileSize = file.tellg();
    file.seekg(0, std::ios::beg);

    // read the data:
    std::vector<BYTE> fileData(fileSize);
    file.read((char*) &fileData[0], fileSize);
    return fileData;
}

which seems to be unnecessarily complicated and the explicit cast to char* that I was forced to use while calling file.read doesn't make me feel any better about it.

---

Another option is to use std::istreambuf_iterator:

std::vector<BYTE> readFile(const char* filename)
{
    // open the file:
    std::ifstream file(filename, std::ios::binary);

    // read the data:
    return std::vector<BYTE>((std::istreambuf_iterator<char>(file)),
                              std::istreambuf_iterator<char>());
}

which is pretty simple and short, but still I have to use the std::istreambuf_iterator<char> even when I'm reading into std::vector<unsigned char>.

---

The last option that seems to be perfectly straightforward is to use std::basic_ifstream<BYTE>, which kinda expresses it explicitly that "I want an input file stream and I want to use it to read BYTEs":

std::vector<BYTE> readFile(const char* filename)
{
    // open the file:
    std::basic_ifstream<BYTE> file(filename, std::ios::binary);

    // read the data:
    return std::vector<BYTE>((std::istreambuf_iterator<BYTE>(file)),
                              std::istreambuf_iterator<BYTE>());
}

but I'm not sure whether basic_ifstream is an appropriate choice in this case.

What is the best way of reading a binary file into the vector? I'd also like to know what's happening "behind the scene" and what are the possible problems I might encounter (apart from stream not being opened properly which might be avoided by simple is_open check).

Is there any good reason why one would prefer to use std::istreambuf_iterator here?
(the only advantage that I can see is simplicity)

Solution 1

When testing for performance, I would include a test case for:

std::vector<BYTE> readFile(const char* filename)
{
    // open the file:
    std::ifstream file(filename, std::ios::binary);

    // Stop eating new lines in binary mode!!!
    file.unsetf(std::ios::skipws);

    // get its size:
    std::streampos fileSize;

    file.seekg(0, std::ios::end);
    fileSize = file.tellg();
    file.seekg(0, std::ios::beg);

    // reserve capacity
    std::vector<BYTE> vec;
    vec.reserve(fileSize);

    // read the data:
    vec.insert(vec.begin(),
               std::istream_iterator<BYTE>(file),
               std::istream_iterator<BYTE>());

    return vec;
}

My thinking is that the constructor of Method 1 touches the elements in the vector, and then the read touches each element again.

Method 2 and Method 3 look most promising, but could suffer one or more resize's. Hence the reason to reserve before reading or inserting.

I would also test with std::copy:

...
std::vector<byte> vec;
vec.reserve(fileSize);

std::copy(std::istream_iterator<BYTE>(file),
          std::istream_iterator<BYTE>(),
          std::back_inserter(vec));

In the end, I think the best solution will avoid operator >> from istream_iterator (and all the overhead and goodness from operator >> trying to interpret binary data). But I don't know what to use that allows you to directly copy the data into the vector.

Finally, my testing with binary data is showing ios::binary is not being honored. Hence the reason for noskipws from <iomanip>.

Solution 2

std::ifstream stream("mona-lisa.raw", std::ios::in | std::ios::binary);
std::vector<uint8_t> contents((std::istreambuf_iterator<char>(stream)), std::istreambuf_iterator<char>());

for(auto i: contents) {
    int value = i;
    std::cout << "data: " << value << std::endl;
}

std::cout << "file size: " << contents.size() << std::endl;

Solution 3

Since you are loading the entire file into memory the most optimal version is to map the file into memory. This is because the kernel loads the file into kernel page cache anyway and by mapping the file you just expose those pages in the cache into your process. Also known as zero-copy.

When you use std::vector<> it copies the data from the kernel page cache into std::vector<> which is unnecessary when you just want to read the file.

Also, when passing two input iterators to std::vector<> it grows its buffer while reading because it does not know the file size. When resizing std::vector<> to the file size first it needlessly zeroes out its contents because it is going to be overwritten with file data anyway. Both of the methods are sub-optimal in terms of space and time.

Solution 4

I would have thought that the first method, using the size and using stream::read() would be the most efficient. The "cost" of casting to char * is most likely zero - casts of this kind simply tell the compiler that "Hey, I know you think this is a different type, but I really want this type here...", and does not add any extra instrucitons - if you wish to confirm this, try reading the file into a char array, and compare the actual assembler code. Aside from a little bit of extra work to figure out the address of the buffer inside the vector, there shouldn't be any difference.

As always, the only way to tell for sure IN YOUR CASE what is the most efficient is to measure it. "Asking on the internet" is not proof.

View more solutions

Comments

[Mats Petersson]

Yes, if the content doesn't need to be in a vector, that is definitely the best method.

[superhero]

Is there a way to read a specific size into the array instead of the whole file as described here?

[jiggunjer]

I thought you only need file.unsetf(std::ios::skipws); if using the operator>>

[jiggunjer]

rather than resize, reserve doesnt initialize.

[jiggunjer]

meaning you can pass the iterators to a reserved vector to avoid redundant resizing. Referring to your last paragraph.

[Maxim Egorushkin]

@jiggunjer Well, that would not work because you cannot access the reserved capacity without resizing the vector first.

[jiggunjer]

I don't understand. If you use reserve(x) no reallocation will happen if you add less than x elements to the vector.

[Maxim Egorushkin]

@jiggunjer Ah, I understand now that what you are saying is already in the accepted answer...

[underscore_d]

For someone reading without reference to the standard, this is unclear. It doesn't explain how to map to memory - I assume streambuf and basic do this?. Also, the terminology assumes Linux/UNIX is the OS used, which feels like it might not be applicable for all platforms - do the same concepts and best practices exist in all OSes targetable by C++?

[Maxim Egorushkin]

@underscore_d C++ standard library does not provide portable facilities for mapping files into memory. Boost does, see Boost Interprecess library, Memory Mapped Files for more details.

[Maxim Egorushkin]

@underscore_d I assume Linux OS because it is free to use, learn and modify and plenty of free documentation is available.

[phoenix]

I needed file.unsetf(std::ios::skipws); even when using std::copy to copy to a vector, otherwise I would lost data. This was with Boost 1.53.0.

[tomi.lee.jones]

@jiggunjer std::istream_iterator uses >> operator internally to extract data from the stream.

[Admin]

Tried more than 8 snippet, none of them worked but this , Thanks a lot! +1

[leiyc]

It seems we don't need add file.unsetf(std::ios::skipws) under gcc 8.2.0

[Anton Breusov]

Use of vector::insert() and iterators is awfully slow. Probably because of calling a lot of virtual functions that read each byte. I cannot even wait until it finishes reading a huge file (3 GB in my case), and this is in Release mode. By changing the last part to this I got a multitude of speedup. std::vector<uint8_t> vec; vec.resize(fileSize); file.read(reinterpret_cast<std::ifstream::char_type*>(&vec.f‌​ront()), fileSize);