How to list all the folders and files in the directory after connecting through SFTP in Python

Question

I am using Python and trying to connect to SFTP and want to retrieve an XML file from there and need to place it in my local system. Below is the code:

import paramiko

sftpURL   =  'sftp.somewebsite.com'
sftpUser  =  'user_name'
sftpPass  =  'password'

ssh = paramiko.SSHClient()
# automatically add keys without requiring human intervention
ssh.set_missing_host_key_policy( paramiko.AutoAddPolicy() )

ssh.connect(sftpURL, username=sftpUser, password=sftpPass)

ftp = ssh.open_sftp()
files = ftp.listdir()
print files

Here connection is success full. And now I want to see all the folders and all the files and need to enter in to required folder for retrieving the XML file from there.

Finally my intention is to view all the folders and files after connecting to SFTP server.

In the above code I had used ftp.listdir() through which I got output as some thing like below

['.bash_logout', '.bash_profile', '.bashrc', '.mozilla', 'testfile_248.xml']

I want to know whether these are the only files present?

And the command I used above is right to view the folders too?

What is the command to view all the folders and files?

Solution 1

One quick solution is to examine the output of lstat of each object in ftp.listdir().

Here is how you can list all the directories.

>>> for i in ftp.listdir():
...     lstatout=str(ftp.lstat(i)).split()[0]
...     if 'd' in lstatout: print i, 'is a directory'
... 

Files are the opposite search:

>>> for i in ftp.listdir():
...     lstatout=str(ftp.lstat(i)).split()[0]
...     if 'd' not in lstatout: print i, 'is a file'
... 

Solution 2

The SFTPClient.listdir returns everything, files and folders.

Were there folders, to tell them from the files, use SFTPClient.listdir_attr instead. It returns a collection of SFTPAttributes.

from stat import S_ISDIR, S_ISREG

for entry in sftp.listdir_attr(remotedir):
    mode = entry.st_mode
    if S_ISDIR(mode):
        print(entry.filename + " is folder")
    elif S_ISREG(mode):
        print(entry.filename + " is file")

---

The accepted answer by @Oz123 is inefficient. SFTPClient.listdir internally calls SFTPClient.listdir_attr and throws most information away returning file and folder names only. The answer then uselessly and laboriously re-retrieves all that data by calling SFTPClient.lstat for each file.

See also How to fetch sizes of all SFTP files in a directory through Paramiko.

---

Obligatory warning: Do not use AutoAddPolicy – You are losing a protection against MITM attacks by doing so. For a correct solution, see Paramiko "Unknown Server".

Solution 3

Here is a solution I have come up with. Based on https://stackoverflow.com/a/59109706 . My solution gives a pretty output.

Update I have modified it slightly to incorporate Martin's suggestions. Now my code is considerably fast compared to my initial version using isdir and listdir

# prefix components:
space =  '    '
branch = '│   '
# pointers:
tee =    '├── '
last =   '└── '

def stringpath(path):
    # just a helper to get string of PosixPath
    return str(path)

from pathlib import Path
from stat import S_ISDIR
def tree_sftp(sftp, path='.', parent='/', prefix=''):
    """
    Loop through files to print it out
    for file in tree_sftp(sftp):
        print(file)
    """
    fullpath = Path(parent, path)
    strpath = stringpath(fullpath)

    dirs = sftp.listdir_attr(strpath)
    pointers = [tee] * (len(dirs) - 1) + [last]
    pdirs = [Path(fullpath, d.filename) for d in dirs]
    sdirs = [stringpath(path) for path in pdirs]

    for pointer, sd, d in zip(pointers, sdirs, dirs):
        yield prefix + pointer + d.filename
        if S_ISDIR(d.st_mode):
            extension = branch if pointer == tee else space
            yield from tree_sftp(sftp, sd, prefix=prefix + extension)

You can try it out like this using pysftp

import pysftp
with pysftp.Connection(HOSTNAME, USERNAME, PASSWORD) as sftp:
    for file in tree_sftp(sftp):
        print(file)

Let me know if if works for you.