AWS SES S3 process inbound email

Question

I'm working on a publish by email system based on AWS SES. For all incoming emails I've set routing to save messages in an S3 bucket so I can asynchronously process them. The problem I have is that the messages are saved in the S3 bucket in a raw format: headers, email body, etc + the encrypted attachment (a huge string) - all in a single file.

Is there a way to break the email message apart form the attachment and save both in separate files at AWS SES level? I'm trying to get the data in the format I need straight from AWS and avoid adding another processing step to the process.

If AWS SES doesn't provide such a feature, what would be the proper way to process these messages to obtain the result described above?

Solution 1

It doesn't look possible to have SES automatically split up the email for you. As per the documentation here:

Amazon SES provides you the raw, unmodified email, which is typically in Multipurpose Internet Mail Extensions (MIME) format.

I would use S3 or SNS to trigger a Lambda function whenever SES puts a new email file to S3. The Lambda function could split the file however you wish, then write those new files to another S3 bucket.

Solution 2

For anyone coming back later on to this question, this is the link to the JSON structure that you get when you invoke a Lambda function from SES.

http://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiving-email-notifications-examples.html

It took some searching to arrive at that page ;-)

From the link, a Lambda notification would look like this,

{
"notificationType": "Received",
"receipt": {
    "timestamp": "2015-09-11T20:32:33.936Z",
    "processingTimeMillis": 406,
    "recipients": [
        "[email protected]"
    ],
    "spamVerdict": {
        "status": "PASS"
    },
    "virusVerdict": {
        "status": "PASS"
    },
    "spfVerdict": {
        "status": "PASS"
    },
    "dkimVerdict": {
        "status": "PASS"
    },
    "action": {
        "type": "S3",
        "topicArn": "arn:aws:sns:us-east-1:012345678912:example-topic",
        "bucketName": "my-S3-bucket",
        "objectKey": "\email"
    }
},
"mail": {
    "timestamp": "2015-09-11T20:32:33.936Z",
    "source": "[e-mail]",
    "messageId": "d6iitobk75ur44p8kdnnp7g2n800",
    "destination": [
        "[email protected]"
    ],
    "headersTruncated": false,
    "headers": [
        {
            "name": "Return-Path",
            "value": "<[e-mail]>"
        },
        {
            "name": "Received",
            "value": "from a9-183.smtp-out.amazonses.com (a9-183.smtp-out.amazonses.com [54.240.9.183]) by inbound-smtp.us-east-1.amazonaws.com with SMTP id d6iitobk75ur44p8kdnnp7g2n800 for [email protected]; Fri, 11 Sep 2015 20:32:33 +0000 (UTC)"
        },
        {
            "name": "DKIM-Signature",
            "value": "v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1442003552; h=From:To:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Date:Message-ID:Feedback-ID; bh=DWr3IOmYWoXCA9ARqGC/UaODfghffiwFNRIb2Mckyt4=; b=p4ukUDSFqhqiub+zPR0DW1kp7oJZakrzupr6LBe6sUuvqpBkig56UzUwc29rFbJF hlX3Ov7DeYVNoN38stqwsF8ivcajXpQsXRC1cW9z8x875J041rClAjV7EGbLmudVpPX 4hHst1XPyX5wmgdHIhmUuh8oZKpVqGi6bHGzzf7g="
        },
        {
            "name": "From",
            "value": "[email protected]"
        },
        {
            "name": "To",
            "value": "[email protected]"
        },
        {
            "name": "Subject",
            "value": "Example subject"
        },
        {
            "name": "MIME-Version",
            "value": "1.0"
        },
        {
            "name": "Content-Type",
            "value": "text/plain; charset=UTF-8"
        },
        {
            "name": "Content-Transfer-Encoding",
            "value": "7bit"
        },
        {
            "name": "Date",
            "value": "Fri, 11 Sep 2015 20:32:32 +0000"
        },
        {
            "name": "Message-ID",
            "value": "<[email protected]>"
        },
        {
            "name": "X-SES-Outgoing",
            "value": "2015.09.11-54.240.9.183"
        },
        {
            "name": "Feedback-ID",
            "value": "1.us-east-1.Krv2FKpFdWV+KUYw3Qd6wcpPJ4Sv/pOPpEPSHn2u2o4=:AmazonSES"
        }
    ],
    "commonHeaders": {
        "returnPath": "[e-mail]",
        "from": [
            "[email protected]"
        ],
        "date": "Fri, 11 Sep 2015 20:32:32 +0000",
        "to": [
            "[email protected]"
        ],
        "messageId": "<[email protected]>",
        "subject": "Example subject"
    }
}
}

Solution 3

Regarding the question on how to write a Lambda. Here is a portion of our Lambda. The main thing to take out of it is the parseEvent function. and data.event.Records[0] which will give you details

exports.handler = function(event, context, callback) {

    var AWS = require('aws-sdk');

    // Validate characteristics of a SES event record.
    if (!event ||
      !event.hasOwnProperty('Records') ||
      event.Records.length !== 1 ||
      event.Records[0].hasOwnProperty('eventSource') ||
      event.Records[0].eventSource !== 'aws:ses' ||
      event.Records[0].eventVersion !== '1.0') {
      callback(null, {'disposition':'STOP_RULE_SET'});      
    }

    email = data.event.Records[0].ses.mail;
    subjectLine = event.Records[0].ses.mail.commonHeaders.subject;
}

The key is the event.Record[0].ses.mail. Unfortunately, I can't find the structure of it via a Google search, I am sure I had seen it before.

Comments

[johnborges]

What's your workflow for downloading and viewing the raw emails? I'm just getting started with AWS SES.

[shiuu]

You need to write code to parse the MIME data. If you write Java, this might help: stackoverflow.com/questions/3444660/java-email-message-parse‌​r.

[greener]

Is there any documentation that you know of that might describe how to create the Lambda function? I have incoming emails to SES where I simply need to copy the attachments to S3.

[Birdy]

Mark would it make more sense to parse the headers and content one needs from the ses object stored in s3 and then store them locally either by file or db record so no further pull requests are needed to S3? A use case is that in the backend, We can view /admin/emails for example and each email is looped over to show the information (Like a webmail client) and each page load would need to make a s3 request as a pose to a local directory or local db request, What would you advise on that? Stay with S3 or use local environment?

[k00k]

Please note that this is the event that gets passed into Lambda. This is not the actual email itself, hence there is no body. If you don't need it and you can make do with what is provided, great, but if you need the body, you will likely need to save the SES email to S3, get the filename (key) from this event, read the email from S3 and then parse the file contents (MIME email format) to pull out the body.

[eugenevd]

Shouldn't the line "event.Records[0].hasOwnProperty('eventSource')" be negated?

[bytor99999]

Good news is our code hasn't broken. It works perfectly for us. And it has been a long time since I wrote that code, almost 2 years ago that I don't even remember. It could be needing the "!" or not. Thanks for pointing it out though.

[user1819575]

Anyone looking to use the above suggestion to save to the s3 and read it in a lambda know that AWS creates a buffer e.g. <Buffer 54 68 69 73 20 69 73 20 61 etc... when it saves the ses s3 file... so when you read it you will need to decode it. e.g. data["Body"].toString('utf8'); see: hackernoon.com/…

[freethejazz]

For a few additional details on a specific implementation of this (in python), see this blog post by Casper: medium.com/caspertechteam/…. TL;DR: they use a python library to parse out the attachment and save them to an S3 bucket.

[freethejazz]

For a full reference implementation, see: github.com/martysweet/aws-lambda-attachment-extractor