In Amazon EC2, what's the best way to clone a private github repository on boot?

Question

I need to configure an autoscale group of EC2 instances that on boot will clone a particular branch and commit from a private github repository, then execute code from inside that repository.

I'm using an Ubuntu-derived AMI that has the cloud-init launch system on it, so it seems to me the simplest way to implement this would be a cloud-init user script that installs a GitHub deploy key, uses that to clone the repository, and then starts the appropriate scripts within the cloned repository.

Here's my first attempt (the deploy key, branch and commit hash are inserted by a management process that manages the autoscale cluster before initializing the LaunchConfiguration ):

#!/usr/bin/env python

# ** IMPORTS OMITTED **

DEPLOY_KEY = '''
--- RSA PRIVATE KEY GOES HERE ---
'''
REPO_URL = '[email protected]:github_user/MyRepository.git'
BRANCH = 'master'
COMMIT = '6dba9ae2cb77dc30c525ce14aeb82b072c88042b'
USER_HOMEDIR = os.environ['HOME']
WORKING_PATH = USER_HOMEDIR

try:
    os.makedirs(os.path.join(USER_HOMEDIR, '.ssh'))
except:
    pass

# Install the deploy key as the user's default identity file
keypath = os.path.join(USER_HOMEDIR, '.ssh', 'id_rsa')
try:
    with open(keypath, 'w') as keyfile:
        keyfile.write(DEPLOY_KEY)
except:
    with open(keypath, 'r') as keyfile:
        assert keyfile.read() == DEPLOY_KEY

# openSSH requires identitiy file mode to be 600
os.chmod(keypath, stat.S_IREAD)

# Disable strict host checking for github.com so we don't get the prompt
with open(os.path.join(USER_HOMEDIR, '.ssh', 'config'), 'a') as sshconfig:
    sshconfig.write("\nHost github.com\n\tStrictHostKeyChecking no\n")

# clone the get repo
os.chdir(WORKING_PATH)
subprocess.call('git clone %s --branch %s' % (REPO_URL, BRANCH), shell=True)
if COMMIT != 'HEAD':
    os.chdir(os.path.join(WORKING_PATH, 'LFAnalyze'))
    subprocess.call('git checkout ' + COMMIT, shell=True)

This script works from a login shell, but it fails to to run within cloud-init's environment, because the $HOME environment variable isn't set yet (cloud-init runs within an Upstart job at runlevel 2, with access to these environment variables).

git-clone doesn't seem to have an option to pass in SSH options such as would let me use a particular key file. It just looks in ~/.ssh/ for the default identity key. If $HOME isn't defined when this script is executing, how can I tell git-clone where to look for the Deploy Key?

Alternatively, is there some way to make cloud-init execute a script as a particular user, emulating a login shell environment?

Or am I just approaching this entire problem the wrong way? Is there an established best practice or standard way of doing this?

Solution

Normally, I would recommend that you create a working EC2 instance with the SSH key and git repo clone (a working instance with your ideal configuration).

Then create a AMI of that instance and in the launch configuration enter your scripts in the userdata so that the scripts would run each time (basically update git and launch command) when the instance is launched.

Sample script would be (assuming /opt/src is where your repo is):

#!/bin/sh
cd /opt/src
git pull
source run_cmd.sh

This way your script is contained and well maintained. Hope that helps.

Comments

[Sirex]

try the ssh-agent answer from: stackoverflow.com/questions/4565700/… perhaps ?

[Sirex]

also, look into the $GIT_SSH env var. You can have a script call ssh in the way you wish and have git use that for its ssh connection.

[leted]

The $GIT_SSH solution worked! I set it to point to a script like this: #!/bin/bash ssh -o StrictHostKeyChecking=no -i /path/to/deploy_key "$@"