AWS python lambda function:No module named requests

Question

I am fairly new to AWS and I am having some issues. Here is my code:

from __future__ import print_function
from urllib2 import Request, urlopen, URLError
import requests
import boto3
import json

def lambda_handler(event, context):
    url = "https://globalcurrencies.xignite.com/xGlobalCurrencies.json/GetHistoricalRatesRange?Symbol=BTCUSD&PriceType=Mid&StartDate=01/01/2017&EndDate=10/27/2017&PeriodType=Daily&FixingTime=22:00&_token=some_token_xyz"
    response = requests.get(url).json()
    # print json.dumps(response, indent=4) # gives a syntax error
    return response

Name of the file is lambda_function.py; I have checked similar problems on stackoverflow and some mentioned that I have to change the file naming. But it didn't help. Here is how python method was named:
Here is the error I am getting: START RequestId: cf24e9be-bbef-11e7-97b4-d9b586307f3e Version: $LATEST Unable to import module 'lambda_function': No module named requests And when try to print it gives me a syntax error. Sorry for the formatting. Any suggestions?

Solution 1

requests is not a standard library in AWS lambda.

So two ways to solve this:

1- Import it from the Botocore libraries stack as: (Option to be deprecated after the answer was given)

from botocore.vendored import requests

Here there is a list of all the available libraries to import in lambda

2- Create a deployment package with virtualenv.

Solution 2

If you want to include libraries which are not part of Python's standard library, such as requests, you can use lambda's layers.

---

1. Create the zip

This is a zip which contains all the libraries you want the lambda function to use. First, create a folder called python:

$ mkdir python
$ cd python

then, install the Python libraries you need in there. You can do this either with a single library:

$ pip install --target . requests

or with a list of libraries (requirements.txt)

$ pip install --target . -r requirements.txt

finally, zip up everything.

... in Bash:

$ zip -r dependencies.zip ../python

... in Powershell:

$ Compress-Archive -Path . -DestinationPath dependencies.zip

---

2. Create a layer

You can do this either in the AWS console or in the CLI. Follow these instructions.

---

3. Add the layer to the lambda function

You can do this either with the Add a layer option in the lambda page, or in the CLI. Follow these instructions.

Solution 3

This is because it is missing the requests library when running in the lambda - its likely that its installed globally on your local machine. If you run:
pip install requests -t .
in the same directory as your source code it will install the requests package in that directory then you can upload it to lambda along with your lambda_function.py. You may need to do the same for boto3 and json:
pip install boto3 -t .
pip install json -t .

Solution 4

You want to use Layers. You download a python module e.g pip install requests -t . then simply put all the module folders into a folder called python and zip it and upload it in the lambda layers section of the aws console. Add it to your lambda function and it will work. The dir structure is very important so for requests when you unzip the folder it should be RequestLayer/python/requests/"requests and all the other folders that are part of that package". This guy gives a good step by step.

Solution 5

For future readers using Python: If you created your Lambda app directly in Cloud9, you would notice that there is a virtual environment for your app. To install requests (or any other package for that matter), do the following:

1. Right-click on the folder for your app
2. Select "Open terminal here"
3. In the terminal type source venv/bin/activate to activate your virtual environment
4. Type pip3 install requests to install requests

That is it. You can now use requests with import requests as per normal.

View more solutions

Comments

[Zafar]

No, there's another error: Unable to import module 'main': No module named requests

[Admin]

do you have the .zip folder with the request library inside? before upload to lambda? thats the problem

[Zafar]

I am using virtual environment in my machine. How can I upload and retrieve the dependancy from requirements.txt?

[ilvidel]

Option number 1 shouts this message: /var/runtime/botocore/vendored/requests/api.py:67: DeprecationWarning: You are using the get() function from 'botocore.vendored.requests'. This is not a public API in botocore and will be removed in the future. Additionally, this version of requests is out of date. We recommend you install the requests package, 'import requests' directly, and use the requests.get() function instead.

[Nilo_DS]

I just added the heads up. Thanks @ilvidel for the heads up!

[LoMaPh]

What should be the folder structure? What do you mean by "RequestLayer"? I tries your version and a bunch of others (with or without "RequesLayer", with or without "requests") but no luck.

[McGlothlin]

According to an updated version of the warning: "This dependency was removed from Botocore and will be removed from Lambda after 2020/03/31." Stick with option 2.

[nnsense]

boto3 and json are both already included in AWS lambda, see gist.github.com/gene1wood/4a052f39490fae00e0c3

[Anum Sheraz]

I don't see any such option "Open terminal here"

[ninsonTan]

@LoMaPh I tried without RequestLayer and it works. So after installing the package using pip, place all content within a folder called python, zip that and add that as a Layer.

[Riccardo]

Great answer! In my case, step number 2 threw the error ERROR: Can not combine '--user' and '--target'. As it turns out, this happens if Python is installed as an app from the store and in some other cases. The issue is easily solved by adding a --no-user at the end of the command: pip install --target . requests --no-user. Source