Call aws-cli from AWS Lambda

Question

is there ANY way to execute aws-cli inside AWS Lambda? It doesn't seem to be pre-installed. (I've checked with "which aws" via Node.js child-process, and it didn't exist.）

Solution 1

Now we can use Layers inside Lambda. Bash layer with aws-cli is available at https://github.com/gkrizek/bash-lambda-layer

handler () {
    set -e

    # Event Data is sent as the first parameter
    EVENT_DATA=$1

    # This is the Event Data
    echo $EVENT_DATA

    # Example of command usage
    EVENT_JSON=$(echo $EVENT_DATA | jq .)

    # Example of AWS command that's output will show up in CloudWatch Logs
    aws s3 ls

    # This is the return value because it's being sent to stderr (>&2)
    echo "{\"success\": true}" >&2
}

Solution 2

Not unless you include it (and all of its dependencies) as part of your deployment package. Even then you would have to call it from within python since Lambda doesn't allow you to execute shell commands. Even if you get there, I would not recommend trying to do a sync in a Lambda function since you're limited to a maximum of 5 minutes of execution time. On top of that, the additional spin-up time just isn't worth it in many cases since you're paying for every 100ms chunk.

So you can, but you probably shouldn't.

EDIT: Lambda does allow you to execute shell commands

Solution 3

aws-cli is a python package. To make it available on a AWS Lambda function you need to pack it with your function zip file.

1) Start an EC2 instance with 64-bit Amazon Linux;

2) Create a python virtualenv:

mkdir ~/awscli_virtualenv
virtualenv ~/awscli_virtualenv

3) Activate virtualenv:

cd ~/awscli_virtualenv/bin
source activate

4) Install aws-cli and pyyaml:

pip install awscli
python -m easy_install pyyaml

5) Change the first line of the aws python script:

sed -i '1 s/^.*$/\#\!\/usr\/bin\/python/' aws

6) Deactivate virtualenv:

deactivate

7) Make a dir with all the files you need to run aws-cli on lambda:

cd ~
mkdir awscli_lambda
cd awscli_lambda
cp ~/awscli_virtualenv/bin/aws .
cp -r ~/awscli_virtualenv/lib/python2.7/dist-packages .
cp -r ~/awscli_virtualenv/lib64/python2.7/dist-packages .

8) Create a function (python or nodejs) that will call aws-cli:

For example (nodejs):

var Q = require('q');
var path = require('path');
var spawn = require('child-process-promise').spawn;    

exports.handler = function(event, context) {

    var folderpath = '/folder/to/sync';
    var s3uel = 's3://name-of-your-bucket/path/to/folder';

    var libpath = path.join(__dirname, 'lib');
    var env = Object.create(process.env);
    env.LD_LIBRARY_PATH = libpath;

    var command = path.join(__dirname, 'aws');
    var params = ['s3', 'sync', '.', s3url];
    var options = { cwd: folderpath };

    var spawnp = spawn(command, params, options);

    spawnp.childProcess.stdout.on('data', function (data) {
        console.log('[spawn] stdout: ', data.toString());
    });

    spawnp.childProcess.stderr.on('data', function (data) {
        console.log('[spawn] stderr: ', data.toString());
    });

    return spawnp
    .then(function(result) {

        if (result['code'] != 0) throw new Error(["aws s3 sync exited with code", result['code']].join(''));

        return result;

    });

}

Create the index.js file (with the code above or your code) on ~/awscli_lambda/index.js

9) Zip everything (aws-cli files and dependencies and your function):

cd ~
zip -r awscli_lambda.zip awscli_lambda

Solution 4

Now you can simply run it as Docker container within lambda along with AWS CLI.

Solution 5

You can use the AWS node.js SDK which should be available in Lambda without installing it.

var AWS = require('aws-sdk');
var lambda = new AWS.Lambda();
lambda.invoke({
    FunctionName: 'arn:aws:lambda:us-west-2:xxxx:function:FN_NAME',
    Payload: {}, 
  },
  function(err, result) {
    ...
});

As far as I can tell you get most, if not all the cli functionality. See the full documentation here: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html

View more solutions

Comments

[Hirofumi Okino]

Thanks Ryan, but I want some functions which do NOT exit in node.js SDK and do exist in aws-cli, because it seems easier to use aws-cli in some occasions, such as s3 sync across buckets.

[Ryan]

Ah, I see now. In that case it seems you would need to install it on every run which sounds kind of painful and inefficient.

[halil]

you cant install it, yum and pip install is not available in nodejs environment

[Noah]

See alestic.com/2016/11/aws-lambda-awscli for a good overview on how to accomplish bundling the aws-cli python package for use in lambda

[Yves M.]

Actually using aws cli IS a good idea when using Lambda for a Cloudformation Custom resource for example

[Yves M.]

The question is about aws cli, not aws sdk

[domdambrogia]

This means that you need to have an ec2 instance sitting around at all times rather than implementing the usefulness of lambdas - small scope & run only when needed.

[Jose A]

Omg!!! Thank you! I was literally fighting to get something like this working for 4 hours! Thanks :D :D

[Christian Dechery]

I'm trying to use the aws cli from this layer with my python code. But I'm getting nothing back.

[Jai]

This is bash layer. I am not sure why you need it in python. You may just use the python SDK.

[Sashi]

To add some context - AWS Lambda now supports container images - aws.amazon.com/blogs/aws/…