Why is Paramiko raising EOFError() when the SFTP object is stored in a dictionary?

Question

I'm having trouble with an application I'm writing that downloads and uploads files to and from other boxes via SSH. The issue I'm experiencing is that I can get (download) files just fine but when I try to put (upload) them onto another server I get an EOFError() exception. When I looked at _write_all() in paramiko\sftp.py it seemed like the error was caused when it couldn't write any data to the stream? I have no network programming experience so if someone knows what it's trying to do and could communicate that to me I'd appreciate it.

I wrote a simplified version of the function that handles my connections as ssh(). runCommand() shows how the upload is failing in my application while simpleTest() shows how sftp put does work, but I can't see any difference between runCommand() and simpleTest() other than how my SFTP objects are being stored. One is stored in a dictionary and the other by itself. It seems like if the dictionary was the problem that downloading files wouldn't work but that is not the case.

Does anyone know what could cause this behavior or could recommend another way to manage my connections if this way is causing problems?

I'm using Python 2.7 with Paramiko 1.7.6. I've tested this code on both Linux and Windows and got the same results.

EDIT: code included now.

import os
import paramiko

class ManageSSH:
     """Manages ssh connections."""
    def __init__(self):
        self.hosts = {"testbox": ['testbox', 'test', 'test']}
        self.sshConnections = {}
        self.sftpConnections = {}
        self.localfile = "C:\\testfile"
        self.remotefile = "/tmp/tempfile"
        self.fetchedfile = "C:\\tempdl"

    def ssh(self):
        """Manages ssh connections."""
        for host in self.hosts.keys():
            try:
                self.sshConnections[host]
                print "ssh connection is already open for %s" % host
            except KeyError, e:         # if no ssh connection for the host exists then open one
                # open ssh connection
                ssh = paramiko.SSHClient()
                ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
                ssh.connect(self.hosts[host][0], 22, self.hosts[host][1], self.hosts[host][2])
                self.sshConnections[host] = ssh
                print "ssh connection to %s opened" % host
            try:
                self.sftpConnections[host]
                print "sftp connection is already open for %s" % host
            except KeyError, e:
                # open sftp connection
                ssh = paramiko.SSHClient()
                ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
                ssh.connect(self.hosts[host][0], 22, self.hosts[host][1], self.hosts[host][2])
                self.sftpConnections[host] = ssh.open_sftp()
                print "sftp connection to %s opened" % host

    def runCommand(self):
        """run commands and return output"""
        for host in self.hosts:
            command = "if [ -d /tmp ]; then echo -n 1; else echo -n 0; fi"
            stdin, stdout, stderr = self.sshConnections[host].exec_command(command)
            print "%s executed on %s" % (command, host)
            print "returned %s" % stdout.read()
            self.sftpConnections.get(self.remotefile, self.fetchedfile)
            print "downloaded %s from %s" % (self.remotefile, host)
            self.sftpConnections[host].put(self.localfile, self.remotefile)
            print "uploaded %s to %s" % (self.localfile, host)
            self.sftpConnections[host].close()
            self.sshConnections[host].close()

    def simpleTest(self):
        host = "testbox"
        ssh = paramiko.SSHClient()
        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        ssh.connect(host, 22, 'test', 'test')
        sftp = ssh.open_sftp()
        print "sftp connection to %s opened" % host
        sftp.get(self.remotefile, self.fetchedfile)
        print "downloaded %s from %s" % (self.localfile, host)
        sftp.put(self.localfile, self.remotefile)
        print "uploaded %s to %s" % (self.localfile, host)
        sftp.close()

if __name__ == "__main__":
    test = ManageSSH()
    print "running test that works"
    test.simpleTest()
    print "running test that fails"
    test.ssh()
    test.runCommand()

output:

running test that works
sftp connection to testbox opened
downloaded C:\testfile from testbox
uploaded C:\testfile to testbox
running test that fails
ssh connection to testbox opened
sftp connection to testbox opened
if [ -d /tmp ]; then echo -n 1; else echo -n 0; fi executed on testbox
returned 1
downloaded /tmp/tempfile from testbox
Traceback (most recent call last):
  File "paramikotest.py", line 71, in <module>
    test.runCommand()
  File "paramikotest.py", line 47, in runCommand
    self.sftpConnections[host].put(self.localfile, self.remotefile)
  File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 561, in put

    fr = self.file(remotepath, 'wb')
  File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 245, in open
    t, msg = self._request(CMD_OPEN, filename, imode, attrblock)
  File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 627, in _request
    num = self._async_request(type(None), t, *arg)
  File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 649, in _async_request
    self._send_packet(t, str(msg))
  File "C:\Python27\lib\site-packages\paramiko\sftp.py", line 172, in _send_packet
    self._write_all(out)
  File "C:\Python27\lib\site-packages\paramiko\sftp.py", line 138, in _write_all

    raise EOFError()
EOFError

Solution 1

I was able to resolve my issue. I was supposed to be using Paramiko.Transport and then creating the SFTPClient with paramiko.SFTPClient.from_transport(t) instead of using open_sftp() from SSHClient().

The following code works:

t = paramiko.Transport((host, 22))  
t.connect(username=username, password=password)  
sftp = paramiko.SFTPClient.from_transport(t)

Solution 2

as i see it, with ssh=SSHClient() you create an SSHClient-Object, and then with sftp=ssh.open_sftp() you create an sftp-object. while you only want to use the sftp, you store the ssh in a local variable, which then gets gc'd, but, if the ssh is gc'd, the sftp magically stops working. don't know why, but try to store the ssh for the time your sftp lives.

Comments

[Vye]

Comment moved to answer below.

[Shaw]

however, use the same code, this exception is raised randomly