Creating an Amazon EC2 Backup solution to Amazon S3

Question

I currently use Amazon S3 as a backup location for my local machines - Primarily using JungleDisk to backup nightly all my local files to my S3 account.

I have been looking at creating a more intelligent backup solution for remote files - Websites on Windows and Linux boxes, along with SQL Server and MySQL databases.

I have been pondering the idea of creating an Amazon EC2 instance which can operate in the cloud as my remote backup machine - Ideally, some sort of script or schedule would trigger the machine to start, and when it does, it would execute a variety of tools or scripts to connect to and back-up my web servers, backing everything up to my Amazon S3 account. Hopefully, when complete the instance can be programmed to shut-down saving execution time.

Am I dreaming? Is this a possibility? Can anyone point me in the right direction?

Thanks,

GW

Solution 1

What sort of pointing are you looking for? This sounds like a perfectly workable idea. There are a variety of libraries to access both EC2 and S3 from a scripting language such as Python or Ruby. You would create a machine instance that starts up, reads from a configuration file to find out which machine(s) to connect to, logs in remotely, fetches new versions of files or database dumps that might be present, uploads that to S3, and finally shuts itself down.

Solution 2

I had the exact same thought about getting an EC2 instance to be my dynamic rsync machine. Just wrote an entry about it on our blog, but basically the solution is indeed to create an EC2 volume, not S3 and dynamically startup and shutdown an EC2 instance, mount the volume and rsync to it whenever you want to backup.

See my entry here: Using Amazon EC2/EBS/S3 for automated backups

Solution 3

If you like to do the backup process by your self and the way you want,

then S3CMD is your best solution I think. You can access the amazon S3 like a folder and do everything you want from Cron Job. To know more about it, have a look.

http://s3tools.org/s3cmd

Comments

[Geniuswood]

Thanks Greg - Nice to know I'm on the right track! By pointing, I'm just trying to get some initial info to get started with... For example: Do I need a Windows or Linux instance? What kinds of services might I need to configure on the instance? Any links on where I might get decent scripts to do what I'm after? I'm primarily a Windows man, but am hoping to use this as a way to get into a bit of Linux...

[Greg Hewgill]

One way to get started with programmatic access to EC2 and S3 is to use Boto: code.google.com/p/boto This is a Python library that lets you do AWS things in an automated way. Because it's Python, it is cross-platform and you wouldn't need to decide between Windows or Linux just yet. In fact, with care (and testing) you could make it work on either OS.

[whatnick]

There are fuse-fs systems allowing S3 based disks.

[Veer Abheek Singh Manhas]

If you are scripting, you could look at Lambdas to execute the code so you don't have to pay for EC2. It would be very cost effective. You can schedule it using CloudWatch Alarms.