Elegant way to create thumbnails of images stored on s3 with ec2 and communicate with rails on finish?

Question

OK, so a quick summary of my setup and what i want to accomplish:

1. I have a rails 2.3.5 server that runs my website. I have a flash application on my site where users can upload images directly to s3.

2. When an upload is completed, rails is notified.

3. At the point where the image is finished uploading to s3 and rails is notified, i want rails to send some post to something located on ec2 to create two thumbnails (110x110 and 600x600).

4. When the thumbnails are created and transferred to s3, i want whatever process on ec2 to send a post back to rails to notify that the thumbnail creation is finished and are on s3.

What is the best way to accomplish this? I have looked at tools such as knife very briefly but I'm not familiar at all with using ec2 or similar services.

Thanks

Solution 1

The process that I would use is the following:

1. Once the image is uploaded to S3, rails gets notified and it adds a message to an Amazon SQS Queue (see http://aws.amazon.com/sqs/)

2. A background process running on EC2 checks the queue and processes any messages, generating the thumbnails

3. Once a thumbnail is generated, a notification is sent using Amazon SNS (see http://aws.amazon.com/sns/) and your rails app respons to this notification

Solution 2

for those like me who looked this up, AWS now offers Lambda

AWS Lambda is a compute service that makes it easy for you to build applications that respond quickly to new information. AWS Lambda runs your code in response to events such as image uploads, in-app activity, website clicks, or outputs from connected devices. You can use AWS Lambda to extend other AWS services with custom logic, or create your own back-end that operates at AWS scale, performance, and security. With AWS Lambda, you can easily create discrete, event-driven applications that execute only when needed and scale automatically from a few requests per day to thousands per second.

Here is a great walkthrough that answers this question perfectly, Handling Amazon S3 Events. The idea is to have a node.js package -the Labmda- get notified about S3 Bucket events (object-created in our case), get the uploaded object, resize it then finally save it in some other bucket for thumbnails. Since you will have a node.js app, you can basically make any kind of requests to any service you want after the thumbnail got saved.