Read remote file using python subprocess and ssh?

Question

How can I read data from a big remote file using subprocess and ssh?

Solution 1

import subprocess
ssh = subprocess.Popen(['ssh', 'user@host', 'cat', 'path/to/file'],
                       stdout=subprocess.PIPE)
for line in ssh.stdout:
    line  # do stuff

Solution 2

The answer above will work, but you'll have to setup your ssh login to use no password between your boxes. There are other ways to transfer files between computers using Python. A simple way, without authentication is to setup an apache server and use an http request.

Comments

[user2921139]

Ross, I have the exact same command that you suggest but that only reads the first line and does not read the other lines.. How do I make it read all the lines in the file?

[user2921139]

I should say that by using the above you have, the file contents are read character by character.. how can i read a whole line completely? "split('\n')" does not seem to work..

[Elle]

Remove .readline() after ssh.stdout, you want to loop on the output not the first line from it.

[Ross Patterson]

@JordanTrudgett Hmm, I submitted the edit when I said, but I can't find anything about it being rejected. Submitted it again.

[Haifeng Zhang]

@RossPatterson will this affect the servers bandwidth?

[Ross Patterson]

@haifzhan No more or less than any other way to transfer the file.

[CutePoison]

I am not getting any feedback - dont know if it can connect but not find the file or..