Pandas in AWS lambda gives numpy error

Question

I've been trying to run my code in AWS Lambda which imports pandas. So here is what I've done. I have a python file which contains a simple code as follows(This file has the lambda handler)

import json
print('Loading function')
import pandas as pd
def lambda_handler(event, context):
    return "Welcome to Pandas usage in AWS Lambda"

1. I have zipped this python file along with numpy, pandas and pytz libraries as a deployment package (Did all these in Amazon EC2 linux machine)
2. Then uploaded the package into S3
3. Created a lambda function(runtime=python3.6) and uploaded the deployment package from S3

But when I test the lambda function in AWS Lambda, I get the below error:

Unable to import module 'lambda_function': Missing required dependencies ['numpy']

I already have numpy in the zipped package but still I get this error. I tried to follow the hints given at Pandas & AWS Lambda but no luck.

Did anyone ran into the same issue. Would appreciate any hint or suggestions to solve this problem.

Thanks

Solution 1

After doing a lot of research I was able to make it work with Lambda layers.

Create or open a clean directory and follow the steps below:

Prerequisites: Make sure you have Docker up and running

1. Create a requirements.txt file with the following:

pandas==0.23.4
pytz==2018.7

2. Create a get_layer_packages.sh file with the following:

#!/bin/bash

export PKG_DIR="python"

rm -rf ${PKG_DIR} && mkdir -p ${PKG_DIR}

docker run --rm -v $(pwd):/foo -w /foo lambci/lambda:build-python3.6 \
    pip install -r requirements.txt --no-deps -t ${PKG_DIR}

3. Run the following commands in the same directory:

chmod +x get_layer_packages.sh

./get_layer_packages.sh

zip -r pandas.zip .

4. Upload the layer to a S3 bucket.

5. Upload the layer to AWS by running the command below:

aws lambda publish-layer-version --layer-name pandas-layer --description "Description of your layer"
--content S3Bucket=<bucket name>,S3Key=<layer-name>.zip
--compatible-runtimes python3.6 python3.7

6. Go to Lambda console and upload your code as a zip file or use the inline editor.

7. Click on Layers > Add a layer> Search for the layer (pandas-layer) from the Compatible layers and select the version.

8. Also add the AWSLambda-Python36-SciPy1x layer which is available by default for importing numpy.

Selecting the layer from the console

9. Test the code. It should work now!!!!

Thanks to this medium article https://medium.com/@qtangs/creating-new-aws-lambda-layer-for-python-pandas-library-348b126e9f3e

Solution 2

EDIT: I figured out finally how to run pandas & numpy in a AWS Lambda python 3.6 runtime environment.

I have uploaded my deployment package to the following repo:

git clone https://github.com/pbegle/aws-lambda-py3.6-pandas-numpy.git

Simply add your lambda_function.py to the zip file by running:

zip -ur lambda.zip lambda_function.py

Upload to S3 and source to lambda.

ORIGINAL:

The only way I have gotten Pandas to work in a lambda function is by compiling the pandas (and numpy) libraries in an AWS Linux EC2 instance following the steps from this blog post and then using the python 2.7 runtime for my lambda function.

Solution 3

To include numpy in your lambda zip follow the instructions on this page in the AWS docs...

How do I add Python packages with compiled binaries to my deployment package and make the package compatible with AWS Lambda?

To paraphrase the instructions using numpy as an example:

1. Open the module pages at pypi.org. https://pypi.org/project/numpy/

2. Choose Download files.

3. Download:

For Python 2.7, module-name-version-cp27-cp27mu-manylinux1_x86_64.whl

e.g. numpy-1.15.2-cp27-cp27m-manylinux1_x86_64.whl

For Python 3.6, module-name-version-cp36-cp36m-manylinux1_x86_64.whl

e.g. numpy-1.15.2-cp36-cp36m-manylinux1_x86_64.whl

4. Uncompress the wheel file on the /path/to/project-dir folder. You can use the unzip command on the command line to do this. There are other ways obviously.

unzip numpy-1.15.2-cp36-cp36m-manylinux1_x86_64.whl

When the wheel file is uncompressed, your deployment package will be compatible with Lambda.

Hope that all makes sense ;)

The end result might look something like this. Note: you should not include the whl file in the deployment package.

Solution 4

AWS Lambda use Amazon Linux operating system. Idea is download Pandas and NumPy compatible with Amazon Linux. What you download using pip is specific to Windows or Mac. You need to download the compatible version for Linux, so that your Lambda function can understand it. These files are called wheel files.

Create new local directory with lambda_function.py file. Install Pandas to local directory with pip:

$ pip install -t . pandas

Navigate to https://pypi.org/project/pandas/#files. Search for and download newest *manylinux1_x86_64.whl package. In my case, I'm using Python 3.6 on my Lambda function, so I downloaded the following:

• Pandas - pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl

• NumPy - numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl

Download whl files to directory with lambda_function.py. Remove pandas, numpy, and *.dist-info directories. Unzip whl files.

$ rm -r pandas numpy *.dist-info
$ unzip numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl
$ unzip pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl

Remove whl files, *.dist-info, and __pycache__. Prepare zip.zip archive:

$ rm -r *.whl *.dist-info __pycache__
$ zip -r zip.zip .

Upload the zip.zip file in your Lambda function.

Source: https://medium.com/@korniichuk/lambda-with-pandas-fd81aa2ff25e

Solution 5

To get additional libraries in Lambda we need to compile them on Amazon Linux (this is important if the underlying library is based on C or C++ like for Numpy) and package them in a ZIP file together with the python script you want to run in Lambda.

To get the Amazon Linux compiled version of the libraries. You can either find a version that someone already compiled, like the one by @pbegle, or compile it yourself. To compile it ourself there are two options: - compile the libraries on an EC2 instance https://streetdatascience.com/2016/11/24/using-numpy-and-pandas-on-aws-lambda/ - compile the libraries on a docker version of Lambda environment https://serverlesscode.com/post/scikitlearn-with-amazon-linux-container/

Following the last option with Docker, it is possible to make it work using the instructions in the blog post above and by adding:

pip install --use-wheel pandas

in the script to compile the libraries:

https://github.com/ryansb/sklearn-build-lambda/blob/master/build.sh#L21

View more solutions