AWS lambda - Release /tmp storage after each execution

Question

I have 4 lambda functions which will be invoked at same time (by SNS), the frequence of SNS's event is 5 minutes. Each function process the large mount of data and images(~300MB) so I store them on /tmp folder (500MB limit).

At the beginning of function, I wrote some code to the clean up /tmp folder, to make sure it's not out of memory (Because I've known that AWS lambda sometimes reuses previous container to improve performance).

I check it manually (create message and publish by SNS to 4 lambda functions), it worked fine.

But when it runs automatically (invoked each 5 minutes) the result is not as my expectation. The first execution is fine, but the next times after, 1 of 4 or even 4 lambda functions throw out the error related to "out of memory": "No space left on device", cannot load lib, ...

Previous, I use nodejs(4.3) it worked fine both case.

But I have to change to python for some reason, the main flow and the mount of created data is the same. But it's failed when run automatically.

I think that the issue came from the cache of previous container (reused container), I checked the /tmp after clean (ls -alh /tmp) there's no files but when check the storage (df /tmp) it show that used is 77%.

Any suggestion to make clean /tmp folder or work around solution is very appreciate. Thank!

Edited: Code I use to clean /tmp folder:

from subprocess import call
...
call('rm -rf /tmp/*', shell=True)

Solution 1

Yes, lambda being a managed service; they do reuse the same underlying resource if the lambda is getting invoked repeatedly. This was a production problem we faced and fixed by deleting the /tmp. On a separate note AWS should mention this in their FAQs.

if os.path.exists(tmp_file_path):
        os.remove(tmp_file_path)
        print("Removed the file %s" % tmp_file_path)     
else:
    print("Sorry, file %s does not exist." % tmp_file_path)

Solution 2

I tried to replicate this issue using lambdash, a great function to test out commands in your "dev" account. It allows you to run arbitrary UNIX commands in the Lambda environment.

I ran this command repeatedly, and did not see the issue appear. Note: The commands are not actually in the deployed code so this test does not fully replicate the potential issue.

lambdash "echo Checking:;file /tmp/nullfile;rm -f /tmp/nullfile;df -h /tmp;dd if=/dev/zero bs=1024 count=88888 >> /tmp/nullfile; echo ==========;df -h /tmp"

Solution 3

Containers are often reused, but not concurrently. Clean up your temp directory when the function finishes and see if issue resolves.