ifstream:: What is the maximum file size that a ifstream can read
Solution 1
I think you want to say:
std::cout << " ifstream file size: " << fs.tellg().seekpos() << std::endl;
At least that works correctly for a 6GB file I have laying around. But I'm compiling with Visual Studio 2012. And even your original code works fine too on that environment.
So I suspect that this is an bug in the std library on VS 2010 that got fixed in VS 2012. Whether it's a bug in the operator overloading for the pos_type or if that class isn't 64-bit aware is unknown. I'd have to install VS 2010 to validate, but that is likely the problem.
Solution 2
I modified your code slightly, to something that would compile:
#include <fstream>
#include <iostream>
#include <string>
#include <windows.h>
int main() {
std::wstring name(L"whatever.txt");
__stat64 buf;
if (_wstat64(name.c_str(), &buf) != 0)
std::cout << -1; // error, could use errno to find out more
std::cout << " Windows file size : " << buf.st_size << std::endl;;
std::ifstream fs(name.c_str(), std::ifstream::in | std::ifstream::binary);
fs.seekg(0, std::ios_base::end);
std::cout << " ifstream file size: " << fs.tellg() << std::endl;
return 0;
}
I tried this on a ~3 Gigabyte file. With VS 2012 (either 32- or 64-bit) it produced:
Windows file size : 3581853696
ifstream file size: 3581853696
With 32-bit VS 2008 (sorry, don't have a copy of VS 2010 installed right now) I got:
Windows file size : 3581853696
ifstream file size: -713113600
So, it would appear that older versions of VS/VC++ used a 32-bit signed number for file sizes, so their practical limit for iostreams was probably 2 gigabytes. With VS 2012, that has apparently been corrected.
veda
Currently, I am a software Engineer at Schlumberger WesternGeco
Updated on June 08, 2022Comments
-
veda almost 2 years
I tried to read a 3GB data file using ifstream and it gives me wrong file size, whereas when I read a 600MB file, it gives me the correct result. In addition to wrong file size, I am also unable to read the entire file using ifstream.
Here is the code that I used
std::wstring name; name.assign(fileName.begin(), fileName.end()); __stat64 buf; if (_wstat64(name.c_str(), &buf) != 0) std::cout << -1; // error, could use errno to find out more std::cout << " Windows file size : " << buf.st_size << std::endl;; std::ifstream fs(fileName.c_str(), std::ifstream::in | std::ifstream::binary); fs.seekg(0, std::ios_base::end); std::cout << " ifstream file size: " << fs.tellg() << std::endl;
The output for 3GB file was
Windows file size : 3147046042 ifstream file size: -1147921254
Whereas the output for 600 MB file was
Windows file size : 678761111 ifstream file size: 678761111
Just in case, I also tested for 5GB file and 300 MB file,
The output for 5GB file was
Windows file size : 5430386900 ifstream file size: 1135419604
The output for 300MB file was
Windows file size : 318763632 ifstream file size: 318763632
It looks to me like it is reaching some limit.
I am testing the code using Visual Studio 2010 on a Windows Machine which has plenty of memory and disk space.
I am trying to read some large files. Which is a good stream reader to use if ifstream can't read large files?
-
Ben Voigt almost 11 yearsI think you mean the Standard library, not the compiler... but the Standard library can only be the limiting factor if you're not using a library appropriate for the OS.
-
Ben Voigt almost 11 yearsAccording to the Standard, it doesn't look like
fpos<char_traits<char>::state_type>
(the type returned bytellg()
is supposed to have a publicseekpos()
member. Is this an implementation-specific extension? -
veda almost 11 yearsYes, it did solve the problem. Now I am getting the correct result.
-
Thomas Matthews almost 11 yearsDoes the Standard Library determine the size of the
filepos
variable or the compiler? I know that length ofsize_t
is set by the compiler. -
Ben Voigt almost 11 yearsThe Standard library does. The types used for file position are not specified to be
size_t
orssize_t
, and generally should not be, since filesystem limits greatly exceed pointer limits (even on 64-bit systems, on which filesystems are starting to use 128-bit lengths!). -
Brandlingo over 5 yearsNote that MS deprecated
seekpos()
with Visual Studio 15.8, they also changed the implementation to always return 0.