Download new and modified files from an FTP server

Question

I'm trying to get a list of the files on an FTP server, then one by one check if that file exists on the local system and if it does compare the dates modified and if the ftp file is newer download it.

private void btnGo_Click(object sender, EventArgs e)
{
    string[] files = GetFileList();
    foreach (string file in files)
    {
        if (file.Length >= 5)
        {
            string uri = "ftp://" + ftpServerIP + "/" + remoteDirectory + "/" + file;
            Uri serverUri = new Uri(uri);

            CheckFile(file);
        }
    }
    this.Close();
}

public string[] GetFileList()
{
    string[] downloadFiles;
    StringBuilder result = new StringBuilder();
    WebResponse response = null;
    StreamReader reader = null;

    try
    {
        FtpWebRequest reqFTP = (FtpWebRequest)FtpWebRequest.Create(new Uri("ftp://" + ftpServerIP + "/" + remoteDirectory));
        reqFTP.UseBinary = true;
        reqFTP.Credentials = new NetworkCredential(ftpUserID, ftpPassword);
        reqFTP.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
        reqFTP.Proxy = null;
        reqFTP.KeepAlive = false;
        reqFTP.UsePassive = false;
        response = reqFTP.GetResponse();
        reader = new StreamReader(response.GetResponseStream());

        string line = reader.ReadLine();
        while (line != null)
        {
            result.Append(line);
            result.Append("\n");
            line = reader.ReadLine();
        }
        result.Remove(result.ToString().LastIndexOf('\n'), 1);
        return result.ToString().Split('\n');
    }
    catch
    {
        if (reader != null)
        {
            reader.Close();
        }
        if (response != null)
        {
            response.Close();
        }
        downloadFiles = null;
        return downloadFiles;
    }
}

private void CheckFile(string file)
{
    string dFile = file;
    string[] splitDownloadFile = Regex.Split(dFile, " ");
    string fSize = splitDownloadFile[13];
    string fMonth = splitDownloadFile[14];
    string fDate = splitDownloadFile[15];
    string fTime = splitDownloadFile[16];
    string fName = splitDownloadFile[17];


    string dateModified = fDate + "/" + fMonth+ "/" + fYear;

    DateTime lastModifiedDF = Convert.ToDateTime(dateModified);

    string[] filePaths = Directory.GetFiles(localDirectory);

    // if there is a file in filePaths that is the same as on the server compare them and then download if file on server is newer
    foreach (string ff in filePaths)
    {

        string[] splitFile = Regex.Split(ff, @"\\");

        string fileName = splitFile[2];
        FileInfo fouFile = new FileInfo(ff);
        DateTime lastChangedFF = fouFile.LastAccessTime;
        if (lastModifiedDF > lastChangedFF) Download(fileName);
    }
}

In the check file method, for each file (they are .exe files) I keep getting different results when I split the string i.e. for one file the file name was at column 18 another it was at 16 etc. I also can't always get the year portion of the file.

Solution 1

Option A: I'd recommend that you use a higher-level FTP client library that handles some of these details for you, a few likely options are:

• http://ftplib.codeplex.com/
• http://ftps.codeplex.com/
• http://netftp.codeplex.com/

Option B: To answer your question more directly, I think the issue is with this line:

string[] splitDownloadFile = Regex.Split(dFile, " ");

It seems like the FTP server is using spaces to right-align the filenames. To deal with that, we want to adjust the regex to consume all whitespace between the fields:

string[] splitDownloadFile = Regex.Split(dFile, "\s+");

...where \s stands for any whitespace character (usually tabs or spaces), and + means one or more of the thing to the left of it. This will not handle edge cases, such as file names with spaces in them.

Solution 2

First of all, there are some components for which you can get info and download data from ftp can be found here: http://www.limilabs.com/ftp

I wrote some methods for get filename and last modified date from ftp.

This is how I get the filename from line:

private string GetFtpName(string line)
{
    for (int i = 0; i < 8; i++)
        line = line.Substring(line.IndexOf(" ")).Trim();
    return line;
}

And this is how I get the last modified date from ftp:

private DateTime GetFtpFileDate(string url, ICredentials credential)
{
    FtpWebRequest rd = (FtpWebRequest)WebRequest.Create(url);
    rd.Method = WebRequestMethods.Ftp.GetDateTimestamp;
    rd.Credentials = credential;

    FtpWebResponse response = (FtpWebResponse)rd.GetResponse();
    DateTime lmd = response.LastModified;
    response.Close();

    return lmd;
}

Solution 3

Try

ListDirectory + GetDateTimestamp 

instead of

ListDirectoryDetails

Solution 4

For this, you need to retrieve a remote directory listing, including timestamps.

Unfortunately, there's no really reliable and efficient way to retrieve timestamps using features offered by the .NET framework as it does not support the FTP MLSD command. The MLSD command provides listing of remote directory in a standardized machine-readable format. The command and the format is standardized by the RFC 3659.

Alternatives you can use, that are supported by the .NET framework (as the other answers show):

• the ListDirectoryDetails method (the FTP LIST command) to retrieve details of all files in a directory and then you deal with FTP server specific format of the details (*nix format similar to ls *nix command is the most common, drawback is that the format may change over time, as for newer files "May 8 17:48" format is used and for older files "Oct 18 2009" format is used)

  DOS/Windows format: C# class to parse WebRequestMethods.Ftp.ListDirectoryDetails FTP response
  *nix format: Parsing FtpWebRequest ListDirectoryDetails line

• the GetDateTimestamp method (an FTP MDTM command) to individually retrieve timestamps for each file. An advantage is that the response is standardized by the RFC 3659 to YYYYMMDDHHMMSS[.sss]. A disadvantage is that you have to send a separate request for each file, what can be quite inefficient.

  const string uri = "ftp://ftp.example.com/remote/path/file.txt";
  FtpWebRequest request = (FtpWebRequest)WebRequest.Create(uri);
  request.Method = WebRequestMethods.Ftp.GetDateTimestamp;
  FtpWebResponse response = (FtpWebResponse)request.GetResponse();
  Console.WriteLine("{0} {1}", uri, response.LastModified);
  

---

Alternatively you can use a 3rd party FTP client implementation that supports the modern MLSD command.

For example the WinSCP .NET assembly supports that.

You can use the Session.ListDirectory or the Session.EnumerateRemoteFiles methods and read the RemoteFileInfo.LastWriteTime of the files in returned collection.

Or even easier, you can use the Session.SynchronizeDirectories to have the library automatically download (synchronize) the modified files:

// Setup session options
SessionOptions sessionOptions = new SessionOptions
{
    Protocol = Protocol.Ftp,
    HostName = "ftp.example.com",
    UserName = "user",
    Password = "mypassword",
};

using (Session session = new Session())
{
    // Connect
    session.Open(sessionOptions);

    // Synchronize files
    session.SynchronizeDirectories(
        SynchronizationMode.Local, @"d:\www", "/remote/path", false).Check();
}

(I'm the author of WinSCP)

View more solutions

Comments

[Noelle]

That only gets the file name. I need to be able to see the date modified of the file

[abatishchev]

@Wellie: Use GetDateTimestamp against each file file returned by ListDirectory. That will rid you from string status representation parsing issues

[NightOwl888]

This was a bad idea. I tried ftplib and ftps, and both were unstable in that they threw exceptions intermittently. Even though I put my calls in a retry loop up to 5 times, I still couldn't get them to work reliably. ftplib threw exceptions that couldn't be caught on the main thread making my Windows service crash every hour or so. I cannot speak for netftp because I didn't try it. I ended up using the standard .NET FtpWebRequest class and a non-standard IIS configuration (ftps over port 21) to get it working. I did end up using the parser from ftplib to get directory listing details.