How can I install GraphicsMagick or ImageMagick on AWS Lambda?

Question

I am using the gm package for Node.js along with the default ImageMagick installation that is available on AWS Lambda.

const gm = require('gm').subClass({ imageMagick: true });

For some reason, the resize functionality fails for certain images.

I created an EC2 instance with Amazon Linux AMI (ami-hvm-2016.03.3.x86_64-gp2). I installed the (old) 6.x version of ImageMagick that is available from yum. When I run my script with that install on the EC2 instance, it reproduces the failure I see when the code runs on Lambda, confirming it is something with this version of IM that is causing the failure.

If I install GraphicsMagick with sudo yum install GraphicsMagick. This allows my script to perform the resizes without error.

const gm = require('gm').subClass({ imageMagick: false });

However, I'm not sure how to bundle this in my deploy with serverless. If I install GraphicsMagick to the same folder as the script with sudo yum --installroot=/var/task install GraphicsMagick, and run my script using this require statement instead:

const gm = require('gm').subClass({ imageMagick: false, appPath: './usr/bin/' });

The resizes work when I run my script on the EC2 instance. But when I deploy with serverless, and the script runs in Lambda, the executable appears to be broken. gm fails with the following error on a call to gm(buffer).size(/*...*/).

could not get the image size: ERR: {"code":"EPIPE","errno":"EPIPE","syscall":"write"}

How can I build a version of ImageMagick or GraphicsMagick that can be deployed with serverless?

Solution 1

I spun up the latest aws linux and ran the commands below.

yum -y install gcc-c++ libpng-devel libjpeg-devel libtiff-devel wget
wget https://downloads.sourceforge.net/project/graphicsmagick/graphicsmagick/1.3.26/GraphicsMagick-1.3.26.tar.gz
tar zxvf GraphicsMagick-1.3.26.tar.gz
cd GraphicsMagick-1.3.26
./configure --prefix=/var/task/graphicsmagick --enable-shared=no --enable-static=yes
make
sudo make install
tar zcvf ~/graphicsmagick.tgz /var/task/graphicsmagick/

I scp the dir down into my local and threw it in the package to be zipped and deployed. My layout is similar to the aws repo code linked, but modified for serverless.

Lambda code:

// graphicsmagick dir is at the root of my project
const BIN_PATH = process.env['LAMBDA_TASK_ROOT'] + "/graphicsmagick/bin/";
const Gm = require('gm').subClass({ appPath: BIN_PATH });

// below is inside the handler
process.env['PATH'] = process.env['PATH'] + ':' + BIN_PATH;

serverless.yml

package:
  artifact: /path/to/function.zip

I use the artifact and build my own zip. If you run into the issue below I suggest you do that. https://github.com/serverless/serverless/issues/3215

# -y to keep the symlinks and thus reduce the size from 266M to 73M
cd lambda && zip -FS -q -r -y ../dist/function.zip *

Ideas grabbed from:

https://gist.github.com/bensie/56f51bc33d4a55e2fc9a

https://github.com/awslabs/serverless-image-resizing

Edit: Might want to also check out lambda layers. May only need to do this kind of thing once.

Solution 2

I was struggling on this for a couple of days, ended up going through the process myself and it does indeed work.

ImageMagick is no longer bundled with the Node.js 10.x runtime. There are 3 options to get ImageMagick working with your Node.js 10.x function:

1) Package the dependency and include it in your uploaded ZIP file (like this one)

https://image-magick-example.s3-us-west-2.amazonaws.com/image-magick-example.zip

https://github.com/hmagdy/imagemagick-aws-lambda-Node.js10.x/tree/master/option1_image-magick-example-zip

But with option: The deployment package of your Lambda function "image-magick-example-zip-demo" is too large to enable inline code editing. However, you can still invoke your function.

or

2) Create or use a Lambda Layer that includes ImageMagick, to do that:

clone [email protected]:hmagdy/imagemagick-aws-lambda-Node.js10.x.git
cd imagemagick-aws-lambda-2
start Docker services
make all

That would create a layer.zip inside build folder. But to save you some time here’s a zip file you can use to create a Lambda Layer.

https://image-magick-layer.s3-us-west-2.amazonaws.com/layer.zip

When you create the layer make sure you add Node.js 10.x as a supported runtime. You can then set your function to use the latest Node.js 10.x and add the layer you created. The image conversion should then work again!

Then you can create your aws lambda function like this

https://github.com/hmagdy/imagemagick-aws-lambda-Node.js10.x/tree/master/option2_image-magick-example-c_lib_layer/index.js

3) NodeJS Runtime Environment (npm) with AWS Lambda Layers, to do that:

Also, if you want to use

const imageThumbnail = require('image-thumbnail');

and got

Runtime.ImportModuleError: Error: Cannot find module 'image-thumbnail'

you should follow option 3:

https://github.com/hmagdy/imagemagick-aws-lambda-Node.js10.x/tree/master/option3_image-magick-example-npm_layer

Inspired by:

https://medium.com/@anjanava.biswas/nodejs-runtime-environment-with-aws-lambda-layers-f3914613e20e

Solution 3

If you want to tackle image resizing, you may also take a look at the serverless sharp image library which uses Sharp, a high performance Node.js library for image resizing which is about 3x - 5x faster compared to GM/IM. You didn't provide enough information to say that it fits your use case requirements but I just wanted to mention it since this library already saved me a lot of AWS Lambda costs so far.

By the way: I am not related to this project (but licences are MIT/Apache License 2.0 anyway).

Solution 4

For node.js, you can use node-lambda, it simplifies packaging using a docker image :

node-lambda package -I lambci/lambda:build-nodejs6.10 -A . -x '*.lock *.zip'

The -I argument will launch a docker image and launch npm i in your project so it will compile the binary node_modules against the right architecture.

Solution 5

All dependencies can be packed and uploaded as a part of your AWS Lambda function

You can mostly use any package you want from AWS Lambda, if you can fit it within the allowed size limits and upload the zip file. Take a look at the AWS Lambda Deployment Limits section

Also, here's an example of how to package dependencies (for python code) https://stackoverflow.com/a/36093281/358013

View more solutions