How do I add python libraries to an AWS lambda function for Alexa?

Question

I was following the tutorial to create an Alexa app using Python:

Python Alexa Tutorial

I was able to successfully follow all the steps and get the app to work.I now want to modify the python code and use external libraries such as import requests or any other libraries that I install using pip. How would I setup my lambda function to include any pip packages that I install locally on my machine?

Solution 1

As it is described in the Amazon official documentation link here It is as simple as just creating a zip of all the folder contents after installing the required packages in your folder where you have your python lambda code.

As Vineeth pointed above in his comment, The very first step in moving from an inline code editor to a zip file upload approach is to change your lambda function handler name under configuration settings to include the python script file name that holds the lambda handler.

lambda_handler => {your-python-script-file-name}.lambda_handler.

Other solutions like python-lambda and lambda-uploader help with simplifying the process of uploading and the most importantly LOCAL TESTING. These will save a lot of time in development.

Solution 2

The official documentation is pretty good. In a nutshell, you need to create a zip file of a directory containing both the code of your lambda function and all external libraries you use at the top level.

You can simulate that by deactivating your virtualenv, copying all your required libraries into the working directory (which is always in sys.path if you invoke a script on the command line), and checking whether your script still works.

Solution 3

You may want to look into using frameworks such as zappa which will handle packaging up and deploying the lambda function for you.

You can use that in conjunction with flask-ask to have an easier time making Alexa skills. There's even a video tutorial of this (from the zappa readme) here

Solution 4

To solve this particular problem we're using a library called juniper. In a nutshell, all you need to do is create a very simple manifest file that looks like:

functions:
  # Name the zip file you want juni to create
  router:
    # Where are your dependencies located?
    requirements: ./src/requirements.txt.
    # Your source code.
    include:
    - ./src/lambda_function.py

From this manifest file, calling juni build will create the zip file artifact for you. The file will include all the dependencies you specify in the requirements.txt.

The command will create this file ./dist/router.zip. We're using that file in conjunction with a sam template. However, you can then use that zip and upload it to the console, or through the awscli.

Solution 5

Echoing @d3ming's answer, a framework is a good way to go at this point. Creating the deployment package manually isn't impossible, but you'll need to be uploading your packages' compiled code, and if you're compiling that code on a non-linux system, the chance of running into issues with differences between your system and the Lambda function's deployed environment are high.

You can then work around that by compiling your code on a linux machine or Docker container.. but between all that complexity you can get much more from adopting a framework.

Serverless is well adopted and has support for custom python packages. It even integrates with Docker to compile your python dependencies and build the deployment package for you.

If you're looking for a full tutorial on this, I wrote one for Python Lambda functions here.

View more solutions

Comments

[Vineet Shah]

I tried following the documentation. I pip installed the requests into the library. I took the color_game.py and added import requests at the top, requests directory, and requests-2.11.0.dist-info directory and sent those three to a zip file. I uploaded the zip file to lambda but when I try to run the Alexa function; it does not work. I get The remote endpoint could not be called, or the response it returned was invalid.

[Vineet Shah]

I figured it out! I realized I wasn't changing the name of my handler. It should be the filename.lambda_handler if you go from inline code to a zip file. Thanks!

[Bela Vizy]

We just started a project bstpy to expose a Python lambda as an http service. You may find it useful for testing. You can throw json payloads at it with curl or postman. If you try it with other Bespoken Tools you can a have very nice development environment.

[Jeremy Leipzig]

not blaming the messenger but this is a kluge on AWS's part. There should really be a more structured way to from pip/conda requirements to lambda deployment.

[Hrushikesh Patel]

This way is not working for python 3.6 script with psycopg2. Getting error "Unable to import module 'ProcessRawRetailSalesUsingCOPY': No module named 'psycopg2._psycopg'" i am using windows7 prof with pycharm. Need some help on this if you can.

[user124384]

The official documentation is pretty good. Gonna have to disagree with you there. The title and the first sentence of the page you linked to alone are entirely unclear to me as to whether they even have anything to do with importing packages to my AWS instance. (Is it about packaging something I've created? Who knows!)

[Pedro]

We started juniper as a way to automates the creation of python based artifacts as described in the official documentation. Building the zip manually is not a scalable solution, juniper was born out of our frustration building these artifacts with custom shell scripts.

[Sarah]

Can we do that and still be able to modify the code inline? It seems that after adding a number of libraries, the zip file is large and it doesn't allow me to update the code inline anymore (th code itself is not large). Any workaround for this?

[Santhosh Gandhe]

@Sarah package all your libraries in to a lambda layer and just keep your source code in your lambda zip and layer dependency to your lambda. This will reduce the size and you should be able to edit your code in line. Also checkout aws sam templates here => docs.aws.amazon.com/serverless-application-model/latest/…

[Admin]

Can you take a look here? I am already doing what you suggested i guess stackoverflow.com/questions/72295967/…

[Admin]

Can you take a look here? I am already doing what you suggested i guess stackoverflow.com/questions/72295967/…