AWS lambda Unable to import module 'lambda_function': No module named PIL

Question

I am using a lambda function of SearchFacesbyimage And I am using this doc https://aws.amazon.com/blogs/machine-learning/build-your-own-face-recognition-service-using-amazon-rekognition/

where for comparison I am using this

from PIL import Image

And I am getting this error Unable to import module 'lambda_function': No module named PIL

Solution 1

You are getting this error as PIL for Python 2.x or PILLOW for 3.x are not standard libraries available in python lambda environment.

To use such a library , you have to make a custom deployment package of all libraries you need as well as the python code you want to deploy. This package can be made easily either in docker or by using EC2 instance .

here is the procedure how you will make that deployment package on EC2 :

1. Suppose you have your file named CreateThumbnail.py

2. If your source code is on a local host, copy it over to EC2.

   scp -i key.pem /path/to/my_code.py ec2-user@public-ip-address:~/CreateThumbnail.py

3. Connect to a 64-bit Amazon Linux instance via SSH.

   ssh -i key.pem ec2-user@public-ip-address

4. Install Python 3.6 and virtualenv using the following steps:

   a) sudo yum install -y gcc zlib zlib-devel openssl openssl-devel

   b) wget https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tgz

   c) tar -xzvf Python-3.6.1.tgz

   d) cd Python-3.6.1 && ./configure && make

   e) sudo make installfsudo /usr/local/bin/pip3 install virtualenv

5. Choose the virtual environment that was installed via pip3

   /usr/local/bin/virtualenv ~/shrink_venv

   source ~/shrink_venv/bin/activate

6. Install libraries in the virtual environment

   pip install Pillow

   pip install boto3

7. Add the contents of lib and lib64 site-packages to your .zip file. Note that the following steps assume you used Python runtime version 3.6. If you used version 2.7 you will need to update accordingly.

   cd $VIRTUAL_ENV/lib/python3.6/site-packages

   zip -r9 ~/CreateThumbnail.zip

   note- To include all hidden files, use the following option:

   zip -r9 ~/CreateThumbnail.zip

8. Add your python code to the .zip file

   cd ~

   zip -g CreateThumbnail.zip CreateThumbnail.py

Now CreateThumbnail.zip is your custom deployment package , just copy it to s3 and upload it to your lambda.

This example is taken from official AWS documentation at https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-deployment-pkg.html

Solution 2

Even though the documentation clearly outlines the steps used to manually create the zip artifact for your lambda function. This solution is not very scalable. I've been using a very small package called juniper to seamlessly package python lambda functions.

In your particular case this are the steps you need to take:

Assuming this is your folder structure:

.
├── manifest.yml
├── src
│   ├── requirements.txt
│   ├── lambda_function.py

In the requirements.txt you would include only the dependencies of your lambda function, in this case, the PIL library.

Pillow==6.0.0

Now, you just have to create a small file to tell juniper what to include in the zip file. The manifest.yml would look like:

functions:
  reko:
    requirements: ./src/requirements.txt.
    include:
    - ./src/lambda_function.py

Now you need to pip install juniper in your local environment. Execute the cli command:

juni build

Juniper will create: ./dist/reko.zip. That file will have your source code as well as any dependency you include in your requirements.txt file.

By default juniper uses docker containers and the build command will use python3.6. You can override that default.

Solution 3

I also ran into this exact same problem. There are two steps that you can take here: manual versus automated packaging and deploying.

The manual step would involve creating the correct virtualenv and install dependencies in that virtual environment. Then zip everything and upload to AWS.

To automate stuff, I always prefer to use the Serverless framework to package and deploy Lambda functions. Specifically the python-requirements-plugin helps with packaging. But I do have to specify the following things to tell the framework to build within a docker container and don't strip any libraries:

custom:
  pythonRequirements:
    dockerizePip: true
    strip: false

Comments

[ephemeral]

CreateThumbnail.py is lambda function ?

[Mausam Sharma]

you can give it any name , but always configure your handler in this way filename.handler-function , e.g disco.lambda_handler where disco.py is filename and lambda_handler is code handler module for lambda

[ephemeral]

All these installations from step 4 are on local machine or EC2 ?

[Mausam Sharma]

Installations need to be done on EC2, from local machine the only thing you need to do is copying your python code to Ec2

[ephemeral]

still Unable to import module 'lambda_function': cannot import name '_imaging' :|

[Songtham T.]

This is a direct copy and paste from docs.aws.amazon.com/lambda/latest/dg/…

[getup8]

This took a little bit to get set up but is working great. Using python3.7+ you just have to add an additional argument, see: github.com/eabglobal/juniper#python37-and-beyond