I need a simple way to backup files to Amazon glacier from the command line?

Question

I have a CentOS server online with a directory of cPmove files I need to back up to Amazon Glacier. I only have command line access, no GUI.

I need a command line client that is relatively easy to use to upload the files. For example I have used dropbox_uploader.sh before ( https://github.com/andreafabrizi/Dropbox-Uploader ) with this simple bash script:

#!/bin/bash
find /backup_folder -maxdepth 1 -name "cpmove-*.tar.gz" | while read CPMOVE
do
   ./dropbox_uploader.sh upload ${CPMOVE}
done

I saw this one: https://github.com/carlossg/glacier-cli but I don't have java installed on the server and am a bit hesistant to install it, due to all the security issues lately.

I do have python and ruby:

me@server [/home]# python -V
Python 2.4.3
root@server [/home]# ruby --version
ruby 1.8.7 (2012-06-29 patchlevel 370) [i686-linux]
me@server [/home]#

Is there a ruby (preferable) or python or other language (less preferable) amazon glacier command line client?)

Solution 1

The canonical library for interfacing with AWS via Python is Boto. Though it is intended to be used as a library in a Python script, it is simple enough to use independently. You can skim the fine documentation, and this page has an overview of how to use Boto, but here are the important parts:

Put your AWS credentials in ~/.boto:

[Credentials]
aws_access_key_id = FOO
aws_secret_access_key = BAR

List your vaults:

$ glacier vaults

Upload a file:

$ glacier upload foo-vault foo.zip

List pending jobs on a particular vault:

$ glacier jobs foo-vault

Though it really isn't easier to use than Boto itself, Amazon Glacier CLI Interface is based on Boto and is in fact designed for end users. Here is a tutorial for it.

Lastly the official AWS-CLI is rather easy to use. Put the credentials in ~/.aws/credentials and then simply use these commands:

Create a vault:

$ aws glacier create-vault --account-id - --vault-name foo-vault

Upload a file:

$ aws glacier upload-archive --account-id - --vault-name foo-vault --body foo.zip

Solution 2

github.com/numblr/glaciertools provides bash scripts that orchestrate the mulitpart upload of a large file with the official AWS command line client (AWS CLI). This is useful in the case when your data exceeds the 4GB limit for uploading an archive in a single operation with the AWS CLI.

To use the scripts you need to have the AWS CLI installed and setup, which requires python. The link contains additional information about the setup.

Then you can create an archive of your backup and upload it to a vault on glacier with

> ./glacierupload -v myvault /path/to/my/archive

More detailed information is contained in the documentation there.

Solution 3

Try vsespb/mt-aws-glacier – Perl multithreaded multipart sync to Amazon Glacier with easy deploy instructions for CentOS.

Solution 4

There is another software also named glacier-cli (https://github.com/basak/glacier-cli) but in python that seems to work pretty well.

View more solutions

Comments

[Admin]

Sorry, I had not looked into Glacier in depth, I thought you had normal ssh access. Answer deleted.

[Admin]

I have since found out that I can upload to s3 with ruby's fog gem. Then, from s3's aws management console, I can set an expiration policy and have the files archived to glacier. Works. Also, fog supports Glacier directly but I haven't looked into that yet. For anyone that's curious, here's my ruby script for uploading to s3: gist.github.com/ivanoats/4712421

[cryanbhu]

funny thing is your step upload-archive is not even covered in the shoddy documentation at docs.aws.amazon.com/cli/latest/userguide/…

[foobarbecue]

These instructions will only work if your vault is in the default region for boto (us-east-1). Otherwise, you need specify the region. E.g. if your vault is in CA, the command should be aws glacier vaults --region us-west-1.