ifstream:: What is the maximum file size that a ifstream can read

Question

I tried to read a 3GB data file using ifstream and it gives me wrong file size, whereas when I read a 600MB file, it gives me the correct result. In addition to wrong file size, I am also unable to read the entire file using ifstream.

Here is the code that I used

        std::wstring name;
        name.assign(fileName.begin(), fileName.end());
        __stat64 buf;
        if (_wstat64(name.c_str(), &buf) != 0)
            std::cout << -1; // error, could use errno to find out more

        std::cout << " Windows file size : " << buf.st_size << std::endl;;


        std::ifstream fs(fileName.c_str(), std::ifstream::in | std::ifstream::binary);
        fs.seekg(0, std::ios_base::end);

        std::cout << " ifstream  file size: " << fs.tellg() << std::endl;

The output for 3GB file was

 Windows file size : 3147046042
 ifstream  file size: -1147921254

Whereas the output for 600 MB file was

 Windows file size : 678761111
 ifstream  file size: 678761111

Just in case, I also tested for 5GB file and 300 MB file,

The output for 5GB file was

Windows file size : 5430386900
 ifstream  file size: 1135419604

The output for 300MB file was

Windows file size : 318763632
 ifstream  file size: 318763632

It looks to me like it is reaching some limit.

I am testing the code using Visual Studio 2010 on a Windows Machine which has plenty of memory and disk space.

I am trying to read some large files. Which is a good stream reader to use if ifstream can't read large files?

Solution 1

I think you want to say:

std::cout << " ifstream  file size: " << fs.tellg().seekpos() << std::endl;

At least that works correctly for a 6GB file I have laying around. But I'm compiling with Visual Studio 2012. And even your original code works fine too on that environment.

So I suspect that this is an bug in the std library on VS 2010 that got fixed in VS 2012. Whether it's a bug in the operator overloading for the pos_type or if that class isn't 64-bit aware is unknown. I'd have to install VS 2010 to validate, but that is likely the problem.

Solution 2

I modified your code slightly, to something that would compile:

#include <fstream>
#include <iostream>
#include <string>
#include <windows.h>

int main() { 

    std::wstring name(L"whatever.txt");

    __stat64 buf;
    if (_wstat64(name.c_str(), &buf) != 0)
        std::cout << -1; // error, could use errno to find out more

    std::cout << " Windows file size : " << buf.st_size << std::endl;;


    std::ifstream fs(name.c_str(), std::ifstream::in | std::ifstream::binary);
    fs.seekg(0, std::ios_base::end);

    std::cout << " ifstream  file size: " << fs.tellg() << std::endl;

    return 0;
}

I tried this on a ~3 Gigabyte file. With VS 2012 (either 32- or 64-bit) it produced:

 Windows file size : 3581853696
 ifstream  file size: 3581853696

With 32-bit VS 2008 (sorry, don't have a copy of VS 2010 installed right now) I got:

 Windows file size : 3581853696
 ifstream  file size: -713113600

So, it would appear that older versions of VS/VC++ used a 32-bit signed number for file sizes, so their practical limit for iostreams was probably 2 gigabytes. With VS 2012, that has apparently been corrected.

Comments

[Ben Voigt]

I think you mean the Standard library, not the compiler... but the Standard library can only be the limiting factor if you're not using a library appropriate for the OS.

[Ben Voigt]

According to the Standard, it doesn't look like fpos<char_traits<char>::state_type> (the type returned by tellg() is supposed to have a public seekpos() member. Is this an implementation-specific extension?

[veda]

Yes, it did solve the problem. Now I am getting the correct result.

[Thomas Matthews]

Does the Standard Library determine the size of the filepos variable or the compiler? I know that length of size_t is set by the compiler.

[Ben Voigt]

The Standard library does. The types used for file position are not specified to be size_t or ssize_t, and generally should not be, since filesystem limits greatly exceed pointer limits (even on 64-bit systems, on which filesystems are starting to use 128-bit lengths!).

[Brandlingo]

Note that MS deprecated seekpos() with Visual Studio 15.8, they also changed the implementation to always return 0.