python del not freeing all the memory

Question

In my python program, I use pandas to read a csv file and store in memory:

data = pandas.read_csv('data.csv')

Before running the above command I check the free memory with free -m and the output is 1704. After running the above command the output is 729. I run

del(data)

to free the memory used by data. Now when I check the free memory the output is 1093 which is much less than the original 1704. Where did the rest go? How can I free it? I'm running all these in ipython and even exiting ipython doesn't free up that memory.

Thanks.

Solution 1

"Exiting ipython doesn't free up that memory" means that you're seeing the effect at the OS level. You're probably seeing the effect of memory caching. That data is saved in memory until the space is needed or the file is read again - instead of reading the file from disk it will be read from the 'inactive' section of memory.

Assuming you're on some unix/linux variant, for details about ram usage

cat /proc/meminfo

Your available memory is Free + Inactive. The Free is available now, and the OS will reclaim space from Inactive as necessary. I keep the following alias (in csh) around to check

alias 'freemem' cat /proc/meminfo | grep -i 'inactive:\|memfree:'

Python's del will remove your variable from the namespace. So if that memory isn't referenced anywhere else in your code, python's garbage collection will kick in and clear data from memory. Or if you want to force it:

import gc
foo = range(10000000)
del foo
gc.collect()

Regardless of all this, the OS will keep the file contents cached so it doesn't have to do redundant disk reads when you try to read that same file again.

Solution 2

del is not C's free or C++'s delete

6.5. The del statement

del_stmt ::= "del" target_list

Deletion is recursively defined very similar to the way assignment is defined. Rather than spelling it out in full details, here are some hints.

Deletion of a target list recursively deletes each target, from left to right.

Deletion of a name removes the binding of that name from the local or global namespace, depending on whether the name occurs in a global statement in the same code block. If the name is unbound, a NameError exception will be raised.

It is illegal to delete a name from the local namespace if it occurs as a free variable in a nested block.

Deletion of attribute references, subscriptions and slicings is passed to the primary object involved; deletion of a slicing is in general equivalent to assignment of an empty slice of the right type (but even this is determined by the sliced object).

source Python spec

You're not freeing the memory. You're telling python you're done with that variable.