SSH.NET SFTP Get a list of directories and files recursively

Question

I am using Renci.SshNet library to get a list of files and directories recursively by using SFTP. I can able to connect SFTP site but I am not sure how to get a list of directories and files recursively in C#. I haven't found any useful examples.

Has anybody tried this? If so, can you post some sample code about how to get these files and folders recursively.

Thanks,
Prav

Solution 1

This library has some quirks that make this recursive listing tricky because the interaction between the ChangeDirectory and ListDirectory do not work as you may expect.

The following does not list the files in the /home directory instead it lists the files in the / (root) directory:

sftp.ChangeDirectory("home");
sftp.ListDirectory("").Select (s => s.FullName);

The following does not work and returns a SftpPathNotFoundException:

sftp.ChangeDirectory("home");
sftp.ListDirectory("home").Select (s => s.FullName);

The following is the correct way to list the files in the /home directory

sftp.ChangeDirectory("/");
sftp.ListDirectory("home").Select (s => s.FullName);

This is pretty crazy if you ask me. Setting the default directory with the ChangeDirectory method has no effect on the ListDirectory method unless you specify a folder in the parameter of this method. Seems like a bug should be written for this.

So when you write your recursive function you'll have to set the default directory once and then change the directory in the ListDirectory call as you iterate over the folders. The listing returns an enumerable of SftpFiles. These can then be checked individually for IsDirectory == true. Just be aware that the listing also returns the . and .. entries (which are directories). You'll want to skip these if you want to avoid an infinite loop. :-)

EDIT 2/23/2018

I was reviewing some of my old answers and would like to apologize for the answer above and supply the following working code. Note that this example does not require ChangeDirectory, since it's using the Fullname for the ListDirectory:

void Main()
{
    using (var client = new Renci.SshNet.SftpClient("sftp.host.com", "user", "password"))
    {
        var files = new List<String>();
        client.Connect();
        ListDirectory(client, ".", ref files);
        client.Disconnect();
        files.Dump();
    }
}

void ListDirectory(SftpClient client, String dirName, ref List<String> files)
{
    foreach (var entry in client.ListDirectory(dirName))
    {

        if (entry.IsDirectory)
        {
            ListDirectory(client, entry.FullName, ref files);
        }
        else
        {
            files.Add(entry.FullName);
        }
    }
}

Solution 2

Try this:

var filePaths = client.ListDirectory(client.WorkingDirectory);

Solution 3

I have achieved this using recursion. Created a class TransportResponse like this

 public class TransportResponse
{
    public string directoryName { get; set; }
    public string fileName { get; set; }
    public DateTime fileTimeStamp { get; set; }
    public MemoryStream fileStream { get; set; }
    public List<TransportResponse> lstTransportResponse { get; set; }
}

I create a list of TransportResponse class. If the directoryName is not null, it will contain a list of the same class which will have the the files inside that directory as a MemoryStream ( this can be changed as per your use case)

List<TransportResponse> lstResponse = new List<TransportResponse>();
using (var client = new SftpClient(connectionInfo))
  {
          try
          {
                    Console.WriteLine("Connecting to " + connectionInfo.Host + " ...");
                    client.Connect();
                    Console.WriteLine("Connected to " + connectionInfo.Host + " ...");
           }
           catch (Exception ex)
           {
                    Console.WriteLine("Could not connect to "+ connectionInfo.Host +" server. Exception Details: " + ex.Message);
           }
           if (client.IsConnected)
           {
                    var files = client.ListDirectory(transport.SourceFolder);
                    lstResponse = downloadFilesInDirectory(files, client);
                    client.Disconnect();
            }
            else
            {
                    Console.WriteLine("Could not download files from "+ transport.TransportIdentifier +" because client was not connected.");
             }
   }



private static List<TransportResponse> downloadFilesInDirectory(IEnumerable<SftpFile> files, SftpClient client)
    {
        List<TransportResponse> lstResponse = new List<TransportResponse>();
        foreach (var file in files)
        {
            if (!file.IsDirectory)
            {
                if (file.Name != "." && file.Name != "..")
                {
                    if (!TransportDAL.checkFileExists(file.Name, file.LastWriteTime))
                    {
                        using (MemoryStream fs = new MemoryStream())
                        {
                            try
                            {
                                Console.WriteLine("Reading " + file.Name + "...");
                                client.DownloadFile(file.FullName, fs);
                                fs.Seek(0, SeekOrigin.Begin);
                                lstResponse.Add(new TransportResponse { fileName = file.Name, fileTimeStamp = file.LastWriteTime, fileStream = new MemoryStream(fs.GetBuffer()) });
                            }
                            catch(Exception ex)
                            {
                                Console.WriteLine("Error reading File. Exception Details: " + ex.Message);
                            }
                        }
                    }
                    else
                    {
                        Console.WriteLine("File was downloaded previously");
                    }
                }
            }
            else
            {
                if (file.Name != "." && file.Name != "..")
                {
                    lstResponse.Add(new TransportResponse { directoryName = file.Name,lstTransportResponse = downloadFilesInDirectory(client.ListDirectory(file.Name), client) });
                }                
            }
        }

        return lstResponse;
    }

Hope this helps. Thanks

Solution 4

Here is a full class. It's .NET Core 2.1 Http trigger function app (v2)

I wanted to get rid of any directories that start with '.', cause my sftp server has .cache folders and .ssh folders with keys. Also didn't want to have to deal with folder names like '.' or '..'

What I will end up doing is projecting the SftpFile into a type that I work with and return that to the caller (in this case it will be a logic app). I'll then pass that object into a stored procedure and use OPENJSON to build up my monitoring table. This is basically the first step in creating my SFTP processing queue that will move files off my SFTP folder and into my Data Lake (blob for now until I come up with something better I guess).

The reason I used .WorkingDirectory is because I created a user with home directory as '/home'. This lets me traverse all of my user folders. My app doesn't need to have a specific folder as a starting point, just the user 'root' so to speak.

using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
using Renci.SshNet;
using Renci.SshNet.Sftp;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

namespace SFTPFileMonitor
{
    public class GetListOfFiles
    {
        [FunctionName("GetListOfFiles")]
        public async Task<IActionResult> RunAsync([HttpTrigger(AuthorizationLevel.Anonymous, "get")] HttpRequest req, ILogger log)
        {
            log.LogInformation("C# HTTP trigger function processed a request.");

            List<SftpFile> zFiles;
            int fileCount;
            decimal totalSizeGB;
            long totalSizeBytes;

            using (SftpClient sftpClient = new SftpClient("hostname", "username", "password"))
            {
                sftpClient.Connect();
                zFiles = await GetFiles(sftpClient, sftpClient.WorkingDirectory, new List<SftpFile>());
                fileCount = zFiles.Count;
                totalSizeBytes = zFiles.Sum(l => l.Length);
                totalSizeGB = BytesToGB(totalSizeBytes);
            }

            return new OkObjectResult(new { fileCount, totalSizeBytes, totalSizeGB, zFiles });
        }
        private async Task<List<SftpFile>> GetFiles(SftpClient sftpClient, string directory, List<SftpFile> files)
        {
            foreach (SftpFile sftpFile in sftpClient.ListDirectory(directory))
            {
                if (sftpFile.Name.StartsWith('.')) { continue; }

                if (sftpFile.IsDirectory)
                {
                    await GetFiles(sftpClient, sftpFile.FullName, files);
                }
                else
                {
                    files.Add(sftpFile);
                }
            }
            return files;
        }
        private decimal BytesToGB(long bytes)
        {
            return Convert.ToDecimal(bytes) / 1024 / 1024 / 1024;
        }
    }
}

View more solutions

Comments

[Gargravarr]

sftp.ListDirectory("") doesn't work, but sftp.ListDirectory(".") does - remember, '.' means 'current directory'. However, it does not seem to support the '~' shortcut for home folder.

[Erik]

Thanks for the explanation, is there a complete example how to actually get a list or a tree of all directories and subdirectories, starting from a given root folder? I wonder why ssh.net does not offer a bool recursive in the ListDirectory method itself. Is there any better solution available for .net these days? I.e. a better lib that is also free, which implements these basic tasks already? However, a working example would be very helpful, thanks!

[Adrian Hum]

Just me, but since the BytesToGB is only used in the RunAsync(...) function, it could be made into a local function...

[jbusciglio acuity]

@AdrianHum Or an extension method in a helper class.

[Adrian Hum]

I actually have it as an extension method that calls the win32 long to string function that returns file sizes like windows does... to three digits... 1.23GB 12.3GB 0.81kb etc... And it provides consistence with windows.

[Kasper Olesen]

Works great for me. But I changed it from a List<string> to the sftp File type, so instead of using files.Add(entry.FullName) it only needs files.Add(entry) and you will have all the information for each file. I used this because I have a VERY slow FTP, and this can optimize a bit I think so it might not have to get the file information again when downloading it.