Using FTP to download each file *WHILE* getting the file list

Question

We need to get about 100 very small files from a remote FTP server using vb.net. Our company won't let us buy (or install) any 3rd party ftp libraries... so we are forced to use something like FtpWebRequest. (Or is there a better free, choice that is already a part of Visual Studio?)

This method works, but it is VERY slow. (I assume because of the constant logging in/out.)

Log in with user name and password.
Get a file-list from the remote server.
Log out
Use that file-list to get each file separtely:
Log in, get the file, log out.
Log in 99 more times, get each file, log out each time.

Instead, we probably should be doing this, but it never works:

Log in with user name and password.  ONCE.
Get a list of filenames.
Download each file.
Log out ONCE.

We found COUNTLESS examples online of "getting an FTP file list" and later "how to download 1 file with FTP"... but we never see "get EACH file name, and download it NOW".

Dim fwr As Net.FtpWebRequest = Net.FtpWebRequest.Create(ftpSite)
fwr.Credentials = New NetworkCredential(userName, password)
fwr.KeepAlive = True
fwr.Method = WebRequestMethods.Ftp.ListDirectory

   Dim sr As IO.StreamReader = Nothing
   Try
      sr = New IO.StreamReader(fwr.GetResponse().GetResponseStream())
      Do Until (sr.EndOfStream())
         fileName = sr.ReadLine()

         fwr.Method = WebRequestMethods.Ftp.DownloadFile

         output = ""
         Dim sr2 As IO.StreamReader = Nothing
         Try
            sr2 = New IO.StreamReader(fwr.GetResponse().GetResponseStream())
            output = sr2.ReadToEnd()

         Catch ex As Exception

         End Try

         If (Not sr2 Is Nothing) Then sr2.Close() : sr2 = Nothing

         Call MsgBox("Got " & fileName & LF & output)
        Loop

   Catch ex As Exception

   End Try

   If (Not sr Is Nothing) Then sr.Close() : sr = Nothing
   If (Not fwr Is Nothing) Then fwr = Nothing

Solution 1

Take a look at my FTP class, it might be exactly what you need.

Public Class FTP
        '-------------------------[BroCode]--------------------------
        '----------------------------FTP-----------------------------
        Private _credentials As System.Net.NetworkCredential
        Sub New(ByVal _FTPUser As String, ByVal _FTPPass As String)
            setCredentials(_FTPUser, _FTPPass)
        End Sub
        Public Sub UploadFile(ByVal _FileName As String, ByVal _UploadPath As String)
            Dim _FileInfo As New System.IO.FileInfo(_FileName)
            Dim _FtpWebRequest As System.Net.FtpWebRequest = CType(System.Net.FtpWebRequest.Create(New Uri(_UploadPath)), System.Net.FtpWebRequest)
            _FtpWebRequest.Credentials = _credentials
            _FtpWebRequest.KeepAlive = False
            _FtpWebRequest.Timeout = 20000
            _FtpWebRequest.Method = System.Net.WebRequestMethods.Ftp.UploadFile
            _FtpWebRequest.UseBinary = True
            _FtpWebRequest.ContentLength = _FileInfo.Length
            Dim buffLength As Integer = 2048
            Dim buff(buffLength - 1) As Byte
            Dim _FileStream As System.IO.FileStream = _FileInfo.OpenRead()
            Try
                Dim _Stream As System.IO.Stream = _FtpWebRequest.GetRequestStream()
                Dim contentLen As Integer = _FileStream.Read(buff, 0, buffLength)
                Do While contentLen <> 0
                    _Stream.Write(buff, 0, contentLen)
                    contentLen = _FileStream.Read(buff, 0, buffLength)
                Loop
                _Stream.Close()
                _Stream.Dispose()
                _FileStream.Close()
                _FileStream.Dispose()
            Catch ex As Exception
                MessageBox.Show(ex.Message, "Upload Error: ", MessageBoxButtons.OK, MessageBoxIcon.Error)
            End Try
        End Sub
        Public Sub DownloadFile(ByVal _FileName As String, ByVal _ftpDownloadPath As String)
            Try
                Dim _request As System.Net.FtpWebRequest = System.Net.WebRequest.Create(_ftpDownloadPath)
                _request.KeepAlive = False
                _request.Method = System.Net.WebRequestMethods.Ftp.DownloadFile
                _request.Credentials = _credentials
                Dim _response As System.Net.FtpWebResponse = _request.GetResponse()
                Dim responseStream As System.IO.Stream = _response.GetResponseStream()
                Dim fs As New System.IO.FileStream(_FileName, System.IO.FileMode.Create)
                responseStream.CopyTo(fs)
                responseStream.Close()
                _response.Close()
            Catch ex As Exception
                MessageBox.Show(ex.Message, "Download Error: ", MessageBoxButtons.OK, MessageBoxIcon.Error)
            End Try
        End Sub
        Public Function GetDirectory(ByVal _ftpPath As String) As List(Of String)
            Dim ret As New List(Of String)
            Try
                Dim _request As System.Net.FtpWebRequest = System.Net.WebRequest.Create(_ftpPath)
                _request.KeepAlive = False
                _request.Method = System.Net.WebRequestMethods.Ftp.ListDirectoryDetails
                _request.Credentials = _credentials
                Dim _response As System.Net.FtpWebResponse = _request.GetResponse()
                Dim responseStream As System.IO.Stream = _response.GetResponseStream()
                Dim _reader As System.IO.StreamReader = New System.IO.StreamReader(responseStream)
                Dim FileData As String = _reader.ReadToEnd
                Dim Lines() As String = FileData.Split(New String() {Environment.NewLine}, StringSplitOptions.RemoveEmptyEntries)
                For Each l As String In Lines
                    ret.Add(l)
                Next
                _reader.Close()
                _response.Close()
            Catch ex As Exception
                MessageBox.Show(ex.Message, "Directory Fetch Error: ", MessageBoxButtons.OK, MessageBoxIcon.Error)
            End Try
            Return ret
        End Function

        Private Sub setCredentials(ByVal _FTPUser As String, ByVal _FTPPass As String)
            _credentials = New System.Net.NetworkCredential(_FTPUser, _FTPPass)
        End Sub
    End Class

To initialize:

Dim ftp As New FORM.FTP("username", "password")

ftp.UploadFile("c:\file.jpeg", "ftp://domain/file.jpeg")

ftp.DownloadFile("c:\file.jpeg", "ftp://ftp://domain/file.jpeg")

Dim directory As List(Of String) = ftp.GetDirectory("ftp://ftp.domain.net/")
        ListBox1.Items.Clear()
        For Each item As String In directory
            ListBox1.Items.Add(item)
        Next

Solution 2

Something I just put together, the important part is the fwr.Proxy = Nothing, otherwise it tries to auto get the proxy settings which causes huge delays so setting it to nothing forces it to not use one.

If you are using a proxy obviously you need to set this to an actual proxy.

Dim fwr As Net.FtpWebRequest = Net.FtpWebRequest.Create(ftpAddress)
fwr.Credentials = New NetworkCredential(userName, password)
fwr.KeepAlive = True
fwr.Method = WebRequestMethods.Ftp.ListDirectory
fwr.Proxy = Nothing

Try
    Dim sr As New IO.StreamReader(fwr.GetResponse().GetResponseStream())
    Dim lst = sr.ReadToEnd().Split(vbNewLine)
    For Each file As String In lst
        file = file.Trim() 'remove any whitespace
        If file = ".." OrElse file = "." Then Continue For
        Dim fwr2 As Net.FtpWebRequest = Net.FtpWebRequest.Create(ftpAddress & file)
        fwr2.Credentials = fwr.Credentials
        fwr2.KeepAlive = True
        fwr2.Method = WebRequestMethods.Ftp.DownloadFile
        fwr2.Proxy = Nothing

        Dim fileSR As New IO.StreamReader(fwr2.GetResponse().GetResponseStream())
        Dim fileData = fileSR.ReadToEnd()
        fileSR.Close()
    Next

    sr.Close()
Catch ex As Exception
End Try

I know its a bit late but hopefully helps

Comments

[Jane T]

If you want ALL the files in the folder why not just use windows standard FTP with a batch file and use MGET?

[THE AMAZING]

Perhaps they are doing mass file manipulation as well.

[THE AMAZING]

You dont need to set proxy to nothing, by default it is nothing.

[THE AMAZING]

Also i would not suggest this method. You creating a multidimensional connection for something so simple.