No module named mem_profile

Question

I'm using this program to measure the time and memory used by two functions and compare which is better for processing a large amount of data. My understanding is that to measure the memory usage we need the mem_profile module, but during the pip install mem_profile it gave me the error No module named mem_profile.

import mem_profile
import random
import time

names = ['Kiran','King','John','Corey']
majors = ['Math','Comps','Science']

print 'Memory (Before): {}Mb'.format(mem_profile.memory_usage_resource())

def people_list(num_people):
    results = []
    for i in num_people:
        person = {
                    'id':i,
                    'name': random.choice(names),
                    'major':random.choice(majors)
                  }
        results.append(person)
    return results

def people_generator(num_people):
    for i in xrange(num_people):
        person = {
                    'id':i,
                    'name': random.choice(names),
                    'major':random.choice(majors)
                  }
        yield person

t1 = time.clock()
people = people_list(10000000)
t2 = time.clock()


# t1 = time.clock()
# people = people_generator(10000000)
# t2 = time.clock()

print 'Memory (After): {}Mb'.format(mem_profile.memory_usage_resource())
print 'Took {} Seconds'.format(t2-t1)

What has caused this error? And are there any alternative packages I could use instead?

Solution 1

Use this for calculating time:

import time

time_start = time.time()
#run your code
time_elapsed = (time.time() - time_start)

As referenced by the Python documentation:

time.time() → float Return the time in seconds since the epoch as a floating point number. The specific date of the epoch and the handling of leap seconds is platform dependent. On Windows and most Unix systems, the epoch is January 1, 1970, 00:00:00 (UTC) and leap seconds are not counted towards the time in seconds since the epoch. This is commonly referred to as Unix time. To find out what the epoch is on a given platform, look at gmtime(0).

Note that even though the time is always returned as a floating point number, not all systems provide time with a better precision than 1 second. While this function normally returns non-decreasing values, it can return a lower value than a previous call if the system clock has been set back between the two calls.

The number returned by time() may be converted into a more common time format (i.e. year, month, day, hour, etc…) in UTC by passing it to gmtime() function or in local time by passing it to the localtime() function. In both cases a struct_time object is returned, from which the components of the calendar date may be accessed as attributes.

Reference: https://docs.python.org/3/library/time.html#time.time

---

Use this for calculating memory:

import resource

resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

Reference: http://docs.python.org/library/resource.html

Use this if you using python 3.x:

Reference: https://docs.python.org/3/library/timeit.html

Solution 2

1)First import module

pip install memory_profiler

2)include it in your code like this

import memory_profiler as mem_profile

3)change code

mem_profile.memory_usage_psutil() to memory_usage()

4)convert you print statements like this

print('Memory (Before): ' + str(mem_profile.memory_usage()) + 'MB' )
print('Memory (After) : ' + str(mem_profile.memory_usage()) + 'MB')
print ('Took ' + str(t2-t1) + ' Seconds')

5)you will have something like this code:

import memory_profiler as mem_profile
import random
import time

names = ['John', 'Corey', 'Adam', 'Steve', 'Rick', 'Thomas']
majors = ['Math', 'Engineering', 'CompSci', 'Arts', 'Business']

# print('Memory (Before): {}Mb '.format(mem_profile.memory_usage_psutil()))
print('Memory (Before): ' + str(mem_profile.memory_usage()) + 'MB' )

def people_list(num_people):
    result = []
    for i in range(num_people):
        person = {
                    'id': i,
                    'name': random.choice(names),
                    'major': random.choice(majors)
                }
        result.append(person)
    return result

def people_generator(num_people):
    for i in range(num_people):
        person = {
                    'id': i,
                    'name': random.choice(names),
                    'major': random.choice(majors)
                }
        yield person

# t1 = time.clock()
# people = people_list(1000000)
# t2 = time.clock()

t1 = time.clock()
people = people_generator(1000000)
t2 = time.clock()

# print 'Memory (After) : {}Mb'.format(mem_profile.memory_usage_psutil())
print('Memory (After) : ' + str(mem_profile.memory_usage()) + 'MB')

# print 'Took {} Seconds'.format(t2-t1)
print ('Took ' + str(t2-t1) + ' Seconds')

Now it work fine i m using python 3.6 and its working without any error.

Solution 3

Was going through the same tutorial and encountered the same problem. But upon further research, I discovered the author of the tutorial used a package called memory_profiler, whose main file he changed to mem_profile, which he imported in the code tutorial.

Just go ahead and do pip install memory_profiler. Copy and rename the file to mem_profile.py in your working directory and you should be fine. If you are on Windows, make sure you install the dependent psutil package as well.

Hope this helps somebody.

Solution 4

Adding to Adebayo Ibro's answer above. Do the following :

• In terminal, run $ pip install memory_profiler
• In your script, replace import mem_profile with import memory_profiler as mem_profile
• In your script, replace all mem_profile.memory_usage_resource() with mem_profile.memory_usage().

Hope this helps!

Solution 5

That module is hand written (not in python packages). I got this from Corey Schafer's comment in his youtube video. Just save this code as the module's name:

from pympler import summary, muppy
import psutil
import resource
import os
import sys

def memory_usage_psutil():
    # return the memory usage in MB
    process = psutil.Process(os.getpid())
    mem = process.get_memory_info()[0] / float(2 ** 20)
    return mem

def memory_usage_resource():
    rusage_denom = 1024.
    if sys.platform == 'darwin':
        # ... it seems that in OSX the output is different units ...
        rusage_denom = rusage_denom * rusage_denom
    mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / rusage_denom
    return mem

View more solutions

Comments

[Corey Goldberg]

don't use time.clock(). it is misleading and also deprecated. The reference you quoted is outdated and has been updated in newer versions of python3 documentation

[Devansh]

Are you talking about this. @CoreyGoldberg

[Devansh]

I don't have much experience in python as of now i'm working on python 2.7 so i found solution on the basis of that. i agree with you @CoreyGoldberg when using python 3.x.

[Corey Goldberg]

yes that. time.clock() also has different behavior on Windows vs other platforms. I suggest removing this answer

[Corey Goldberg]

this answer provides bad advice for timing code, no matter which version of python you are using

[user3785966]

Hey, the profiling for generator is misleading, as nothing was "generated" in people_generator.

[Pyzard]

You can use time.time(), it returns the exact number of seconds (a float) since the Epoch. So import time; t1 = time.time(); t2 = time.time(); print(t2) would output the elapsed time since the creation of t1.