Memory Error in Python 3 and Windows 64

Question

I know Memory Error related questions have been asked before, for example here, here, here, here, or here. And the suggested solutions always are to switch to Python 3 and/or to Window 64bit, or in case of faulty code, to fix the code. However, I am already on Python 3 and Win 64. I also can see from windows task manager that I have several GB of my 64GB of RAM still available when Python throw the Memory Error.

I have about 15 date-indexed, pandas data frames each with 14000 rows and on average 5000 columns of float data, and about 40-50% NaN values, that I read in from the hard drive. I can not simply drop NaNs because different columns have NaNs at different dates. Memory Error happens when I try to concatenate them with pd.concat(). So it's not a matter of some faulty code or while loop. If I leave some of the data frames out of concatenation, Memory Error does not happen in concatenation, but then when I try to do a Scikit learn decision tree analysis on the concatenated data, it happens.

My question is how can I get Python to use all the available memory and not throw Memory Error?

Edit: screenshots added IPython interpreter screenshot (I don't have Python 2 even installed):

System information screenshot:

Solution 1

Here are some links with snippets of information from them, I hope it helps.

How to give programs more RAM

Increase your pagefile. The pagefile is a section of the hard drive that is used as RAM by the processor and is also called Virtual Memory. Although it is not as fast as RAM because it is physically located on a hard drive, increasing it can sometimes increase program performance. To access it, open your Control Panel. Click "System," then "Advanced system settings," and then "Settings" in the "Performance" tab.

How do you set the memory usage for python programs

if u wanna limit the python vm memory usage,you can try this: 1、Linux， ulimit command to limit the memory usage on python 2、you can use resource module to limit the program memory usage; if u wanna speed up ur program though giving more memory to ur application, you could try this: 1\threading, multiprocessing 2\pypy 3\pysco on only python 2.5

Assign memory application

!.Right-click MY COMPUTER 2.Properties>Advanced>Performance>Settings>Advanced 3.Under, Virtual Memory check Paging File size,(if memory serves, the rule of thumb is 1/2 of total V. Memory may be used. If you use more, the paging file fights between the app. and the OS. Hope this help's

Allocate memory process server

Because some programs need more memory than others to work, you can use the Task Manager to allocate extra memory to specific processes in order to increase performance.

Others

http://stackoverflow.com/questions/1760025/limit-python-vm-memory

http://stackoverflow.com/questions/2308091/how-to-limit-python-heap-size

Solution 2

My PC's configuration is 8GB Memory and Windows 10 x64 OS. Also, Python 3 is installed there. I was getting this kind of exception while CSV files were being read by the python script. Fortunately, once pagefile's initial and max capacity was increased, the issue has been resolved.

For instructions of how to increase pagefiles values you can take a look this SO answer :here