C++ std::bad_alloc error
There's nothing wrong with your code, you're simply running on a platform likely with small memory limits, likely an old compiler and likely an old C++ library. It all conspires against you. You'll have to micro-optimize :(
Here's what you can do, starting with lowest hanging fruit first:
Do a dry run through the file, just counting the lines. Then
values.resize(numberOfLines)
, seek to the beginning and only then read the values. Of course you won't be usingvalues.push_back
anymore, merelyvalues[lineNumber] = tmp
. Resizing thevalues
vector as you add to it may more than double the amount of memory your process needs on a temporary basis.At the end of the line, do
tmp.resize(tmp.size()
- it'll shrink the vector to just fit the data.You can reduce overheads in the existing code by storing all the values in one vector.
If each line has a different number of elements, but you access them sequentially later on, you can store an empty string as an internal delimiter, it may have lower overhead than the vector.
If each line has same number of values, then splitting them by lines adds unnecessary overhead - you know the index of the first value in each line, it's simply
lineNumber * valuesPerLine
, where first line has number0
.
Memory-map the file. Store the beginning and end of each word in a structure element of a vector, perhaps with a line number as well if you need it split up in lines.
Achintha Gunasekara
I am a senior Devops Engineer with experience in Banking, Engineering, Telecommunication and Transport. Proficient with Java, PHP, C#, C++, Shell scripting, PowerShell and JavaScript. Extensive knowledge of database systems both in operational and development roles. Full stack infrastructure management and agile monitoring.
Updated on October 08, 2020Comments
-
Achintha Gunasekara over 3 years
I'm working on a C++ program (C++ 98). It reads a text file with lots of lines (10000 lines). These are tab separated values and then I parse it into Vector of Vector objects. However It seems to work for some files (Smaller) but One of my files gives me the following error (this file has 10000 lines and it's 90MB). I'm guessing this is a memory related issue? Can you please help me?
Error
terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Abort
Code
void AppManager::go(string customerFile) { vector<vector<string> > vals = fileReader(customerFile); for (unsigned int i = 0; i < vals.size();i++){ cout << "New One\n\n"; for (unsigned int j = 0; j < vals[i].size(); j++){ cout << vals[i][j] << endl; } cout << "End New One\n\n"; } } vector<vector<string> > AppManager::fileReader(string fileName) { string line; vector<vector<string> > values; ifstream inputFile(fileName.c_str()); if (inputFile.is_open()) { while (getline(inputFile,line)) { std::istringstream iss(line); std::string val; vector<string> tmp; while(std::getline(iss, val, '\t')) { tmp.push_back(val); } values.push_back(tmp); } inputFile.close(); } else { throw string("Error reading the file '" + fileName + "'"); } return values; }