Controlling rsync with Python?

Question

I've been wanting to write a python script that would run several instances of rsync in sequence for backing up data to a different computer.

At the moment I just have this text file with the commands I use and I've just been copy-pasting them into the terminal, and it seems kinda silly.

I want to be able to use python to do this for me. I know very vaguely how to use subprocess.popen, but I have no clue how to get python to interact with rsync directly, like for entering my password for me. Can python do that?

Something like:

if theProccess.proccessResponse == "Password:" :
    theProccess.respond(string)

Or is the best that I can do is just have it, or even a bash script, just run the rsyncs in sequence and have to type my password in over and over again?

Thanks in advance.

Solution 1

If you'd like to interact with a subprocess in general, you can use pexpect as mentioned elsewhere. But for your particular case, assuming your rsync is running over ssh (the default), then you may want to consider setting up a passwordless ssh connection between the two hosts, which will eliminate the need to enter the password. This is a key-based solution and will be much more secure than storing your password in your source code.

Here's a blogger who discusses your exact problem and decides to go with passwordless ssh.

Solution 2

There's a great Python module written by Colin Stewart called RSyncBackup that's little known and little documented, but very useful.

By default, it doesn't contain any methods for including a password in your rsync commands, so I modified the module and talked about it in this blog post: http://technofart.blogspot.com/2012/02/rsync-controlled-by-python.html

A link to my modified module can be found in the Download section of my post.

A key-based solution is also a great idea. Also, many rsync implementation will look for the RSYNC_PASSWORD environment variable, which you can set. Just be careful if your environment variables are visible to other users.

Solution 3

I use rsync to back up all of my clients' web sites. A script is triggered by cron and it uses Makefiles for each client because of their different needs.

Rather than do something that enters the password, use ssh-keygen to create a public/private key pair and put your public key on the remote machine. This gives you secure, no-password connections. This also means you don't have to expose the rsync port to the world. After you get past the learning curve on this (and it's not very steep) ssh is most definitely your friend.

Solution 4

I feel bad, for answering this late, but I feel like everyone else's answer was wrong. They did KINDA answer your question, but not directly as they could have.

More to the point, you had asked how to grab the password interactively. To do so I would suggest the built-in getpass. In short, you are not interacting with rsync. You are grabbing the password from the user RIGHT before you execute rsync, and passing that into rsync. Another option is to allow the user to pass it in as an option, most of my command line scripts use optparse

import getpass
password = getpass.getpass('Password for %s: ' % opts.user)
try:
    #code that uses password
except Exception, e:
    # test to see if str(e) is really an invalid password error, if so tell the user and return or loop, up to you
    # else 
    raise Exception(e) # the error that was raised in the first place

To continue, I stumbled upon your question because I was looking for something similar. Just an FYI to anyone else out there, I ended up referencing the two of these stack overflow links: calling rsync from python subprocess.call and Python Subprocess.Popen from a thread

Solution 5

If you need to programatically control a sub-process, you should look into using pexpect.

View more solutions