Prevent SFTP/SSH session timeout with paramiko

Question

I'm using paramiko to connect to an SFTP server on which I have to download and process some files.

The server has a timeout set to 5 minutes, but some days it happens that the processing of the files can take longer than the timeout. So, when I want to change the working directory on the server to process some other files sftp.chdir(target_dir)) I get an exception that the connection has timed out:

File buildbdist.win32eggparamikosftp://ftp.py, line 138, in _write_all raise EOFError()

To counter this I thought that activating the keep alive would be the best option so I used the "set_keepalive" on the transport to set it to 30 seconds:

ssh = paramiko.SSHClient()
ssh.set_missing_hostkey_policy(paramiko.AutoAddPolicy())
ssh.connect(ssh_server, port=ssh_port, username=ssh_user, password=password)
transport = ssh.get_transport()
transport.set_keepalive(30)
sftp = transport.open_sftp_client()

But nothing changes at all. The change has absolutely no effect. I don't know if I'm misunderstanding the concept of set_keepalive here or maybe the server (on which I have no access) ignores the keep alive packets.

Isn't this the right way to counter this problem or should I try a different approach? I don't like the idea of "manually" sending some ls command to the server to keep the session alive.

Solution

If the server is timing you out for inactivity, there's not much you can do from the client-side (other than perhaps send a simple command every now and again to keep your session from timing out).

Have you considered breaking apart your download and processing steps, so that you can download everything you need to start with, then process it either asynchronously, or after all downloads have completed?

Comments

[JimB]

Keepalive packets are ignored my the server. They are only for keeping network connections from timing out.

[Patric]

ok, I see. I was also thinking about a redesign of the process so that the processing/download is async but that would be a quite bit of work for which I currently I have no time. Seems I'm unlucky that the solution with "keep alive" will not work. Another attempt I could try might by to make a reconnect to the server when I notice that the session has had a timeout.

[Steve Mayne]

Yes, reconnection would be a good solution. You could reconnect upon socket.error and EOFError, and that should suffice.

[Gulzar]

Also how to catch a disconnect?

[Gulzar]

Why is "sending a simple command now and then" doing nothing? How can I do that cleanly with paramiko? Why is Keepalive not just that?