Missing required dependencies ['numpy'] in AWS Lambda after installing numpy into directory, how to fix?

Question

I am trying upload my python code into AWS Lambda. I have been following this guide to create the deployment package (https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html).

I have created a folder 'project-dir' on my desktop and moved my python file 'Twilio_Alerts_AWS.py' into the folder. I have used the command:

pip install module-name -t /path/to/project-dir

to install all my libraries into the folder. Next I highlighted everything and hit 'compress' by right clicking on the highlighted files in the folder. This produces one zipped file called 'archive'

I put the 'archive.zip' in a S3 bucket on AWS and called it into AWS Lambda. I keep getting error Unable to import module 'Twilio_Alerts_AWS': Missing required dependencies ['numpy'] even though I have installed numpy into the folder.

Not sure what I am doing wrong.

Code I am trying to upload:

from twilio.rest import Client
import time
import datetime
import requests
import pandas as pd
from pandas.io.json import json_normalize




def lambda_handler(event, context):
    # Your Account SID from twilio.com/console
    account_sid = "xxx"
    # Your Auth Token from twilio.com/console
    auth_token  = "xxx"

    client = Client(account_sid, auth_token)


    current_datetime = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')


    headers = {
        'Accept': 'application/json',
        'Content-Type': 'application/json',
        'x-api-key': 'xxx',
        'x-organization-id': 'xxx',
        'x-facility-id': 'xxx',
        'x-user-id': 'xxx',
    }



    orders_staging_api_call = requests.get('URL', headers=headers, verify=False)
    consumers_staging_api_call = requests.get('URL', headers=headers, verify=False)
    inventory_staging_api_call = requests.get('URL', headers=headers, verify=False)

    lst = ["[phone]"]

    ##Consumers API Alert
    if consumers_staging_api_call.status_code !=200:
        for i in lst:
            message = client.messages.create(
                    to=i, 
                    from_="[phone]",
                    body="API connection between A and B has failed for: Consumers.Datetime of check:{}".format(current_datetime))
            time.sleep(5)
        print(message.sid)
    else:
        print('done')

edit: using osx machine.

Solution 1

Answer here helped me: Pandas in AWS lambda gives numpy error.

TLDR: libs compiled on mac don't work on linux, so you need to make sure to get the linux versions one way or another (e.g. Docker).

Solution 2

The advice from AWS docs is to use the .whl file for certain dependencies.

You can unpackage the numpy .whl file from the python project download files, there's a fuller answer here

Pandas in AWS lambda gives numpy error - Answer

Solution 3

Here is a short way only to provide Numpy in AWS lambda: Simply add the scipy-numpy layer, provided publicy from Amazon, to your lambda function (In AWS Lambda: layers -> add layer -> the numpy scipy layer should get suggested already).

If you have problems with several packages which are needed to compile, I want to add a hint to an useful docker container which can be used to get the packages compiled for Linux: https://hub.docker.com/r/lambci/lambda/

There are other solutions to solve the compiling problem as well, one is the serverless-python-requirements npm-package if you're using serverless in your project. But I experienced, that this is not working if you run the serverless deploy command in a custom gitlab-runner with serverless and serverless-python-requirements (for ci/cd purpose). In this case I'm currently using AWS Lambda Layers, to provide the needed dependencies. Here is a good explanation to create a layer for pandas: https://medium.com/@qtangs/creating-new-aws-lambda-layer-for-python-pandas-library-348b126e9f3e

My answer has maybe gone too far, but the above mentioned options could be useful for other readers coming here.

Comments

[thepurpleowl]

Does the same go for Windows too??