How can I improve ftp upload speed for multiple files [Java]

Question

I implemented java code to upload files to server with org.apache.commons.net.ftp.FTPClient For multiple files the ftp upload speed is very slow. How can I improve the speed.

-Change library? What is the powerful FTP client class library for uploading multiple files?

-Use multiple threads? How can I implement ftp upload function with multiple thread? Could someone show me an example? I am a new for multiple threading programming.

---

After I read all answer, I try to change my code and test it.

Following is a sample FTPClient code:

// create instance of FTPClient
FTPClient ftp = new FTPClient();

ftp.setControlEncoding("UTF-8");
ftp.setDefaultTimeout(30000);


// connect to server
try
{
    ftp.connect("10.1.1.1", 990);
}
catch(Exception e)
{
    System.out.println("Cannot connect to server");
    return;
}

// login to server
if (!ftp.login("username", "password"))
    {
        ftp.logout();
        System.out.println("Cannot login to server");
        return;
    }

try
{
    ftp.setFileTransferMode(FTP.BINARY_FILE_TYPE);
    ftp.enterLocalPassiveMode();
    // ftp.setBufferSize(0); <-- someone suggest me to set buffer size to 0, but it throw error sometime.
}
catch(Exception e)
{
}
// create directory on server
// dirs is list of required directories on server
for (String dir : dirs)
    {
        try
        {
            ftp.makeDirectory(dir);
        }
        catch(IOException e)
        {
        }
    }


// files is a map of local file and string of remote file
// such as 
// file on client is "C://test/a.txt"
// location on server is "/test/a.txt"
for (Map.Entry<File, String> entry : files.entrySet())
    {
        File localFile = entry.getKey();
        String remoteFile = entry.getValue();

        FileInputStream input = null;
        try
        {
            input= new FileInputStream(localFile);
            ftp.storeFile(remoteFile, input);
        }
        catch (Exception e)
        {
            try
            {
                ftp.deleteFile(remoteFile);
            }
            catch (IOException e1)
            {
            }
        }
        finally
        {
            if (input != null)
            {
                try
                {
                    input.close();
                }
                catch (IOException e)
                {
                }
            }
        }
    }

// disconnect
if (ftp != null && ftp.isConnected())
    {
        try
        {
            ftp.disconnect();
        }
        catch (IOException f)
        {
            // do nothing
        }
    }

When I uploaded 1050 files (each file is about 1-20 KB), it took about 49406 - 51000 millisec (This is upload time only). I would like to improve the speed.

Some people suggest me to use ftp4j, but when I test the library with 1050 files, the upload speed of ftp4j is slowly than FTPClient about 10000 millisec. it took about 60000 millisec.

Following is sample ftp4j code:

// create instance of FTPClient 
FTPClient ftp = new FTPClient();

ftp.setCharset("UTF-8");

// connect to server
try
{
    ftp.connect("10.1.1.1", 990);
}
catch(Exception e)
{
    System.out.println("Cannot connect to server")
    return;
}

// login to server  
try
{
    ftp.login("username", "password");
}
catch (Exception e)
{
    try
    {
      ftp.logout();
    }
    catch (Exception e1)
    {
    }
    System.out.println("Cannot login to server")
    return;
}

try
{
    ftp.setType(FTPClient.TYPE_BINARY);
    ftp.setPassive(true);
}
catch(Exception e)
{   
}

// create directory on server  
// dirs is list of required directories on server  
for (String dir : dirs)
{
    try
    {
      ftp.createDirectory(dir);
    }
    catch (Exception e)
    {
    }
}

// files is a map of local file and string of remote file  
// such as   
// file on client is "C://test/a.txt"  
// location on server is "/test/a.txt" 
for (Map.Entry<File, String> entry : files.entrySet())
{
    final File localFile  = entry.getKey();
    final String remoteFile  = entry.getValue();

    BufferedInputStream input = null;
    boolean success = false;
    try
    {
      input = new BufferedInputStream(new FileInputStream(localFile));

       // ftp.upload(localFile); <-- if I use ftp.upload(File), it will took more time. 
       ftp.upload(remoteFile, input, 0, 2048, new MyTransferListener());
       success = true;
    }
    catch (Exception e)
    {
    }
    finally
    {
        if (input != null)
        {
            try
            {
                input.close();
            }
            catch (IOException e)
            {
            }
        }
        if (!success)
        {
            try
            {
              ftp.deleteFile(remoteFile);
            }
            catch (Exception e)
            {
            }
        }
    }
}

// disconnect
if (ftp != null && ftp.isConnected())  
{  
    try  
    {  
      ftp.disconnect();  
    }  
    catch (IOException f)  
    {  
      // do nothing  
    }  
} 

I try to use multiple threads.

Following is multiple threading code:

final CountDownLatch latch = new CountDownLatch(files.size());
ExecutorService pool = Executors.newFixedThreadPool(10);
for (Map.Entry<File, String> entry : files.entrySet())
{
    final File localFile = entry.getKey();                  
    final String remoteFile = entry.getValue();

    pool.execute(new Runnable() {
        public void run()
        {
            FileInputStream input = null;
            try
            {
                input= new FileInputStream(localFile);
                ftp.storeFile(remoteFile, input);
            }
            catch (Exception e)
            {
                try
                {
                    ftp.deleteFile(remoteFile);
                }
                catch (IOException e1)
                {
                }
            }
            finally
            {
                if (input != null)
                {
                    try
                    {
                        input.close();
                    }
                    catch (IOException e)
                    {
                    }
                }
               latch.countDown();
            }
        }
    });
}
try
{
  // waiting for all threads finish
  // see: http://stackoverflow.com/questions/1250643/how-to-wait-for-all-threads-to-finish-using-executorservice
  latch.await();
}
catch(Exception e)
{
}

Is it correct? It work correctly but it cannot improve speed. it took about 49000 - 51000 millisec same as the code without thread.

I test the speed with intranet. It will take more time for internet.

How should I do for improve upload speed?

Solution

I don't know why, but Apache Commons FTP is pretty slow in uploading, i had the same problem and i couldn't solve it.

Now i use FTP4j, it's pretty similar to apache commons ftp but uploads are really fast.

This is an example:

FTPClient client = new FTPClient();
client.connect("www.yoursite.com");
client.login("login", "password");
client.setPassive(true);
client.setType(FTPClient.TYPE_BINARY);
client.changeDirectory("a");
File f = new File("path/to/your/file");
client.upload(f);
client.disconnect(true);

With this library i uplodaded a 340KB file in less than one second, while with Apache Commons FTP it took about 1 minute.

If you want to transfer different files with threads, try to put every client.upload(f) into a different thread, but i'm not sure it will boost up the transfer.

---

Quoting @fge previous answer:

Basically, chances are, you can't.

Don't forget that FTP has two types of channels: the command channel and the data channels. One upload is initiated by sending the instructions over the command channel to open a data channel for the upload proper.

Now:

• most FTP servers are configured so that one command channel can only open one data channel at anyone time;
• there are bandwidth limits: your upstream bandwidth, and the server's downstream bandwidth.

Were it possible to upload several files in parallel, ie, opening more than one data channel, you'd have the problem that the overhead of TCP itself would in fact slow down the upload process in general.

Basically: keep one data channel open at any time. Trying and opening more than one is just not worth it. It may work in ~1% of cases in general. This is just not worth the hassle.

Comments

[fge]

Good recommendation as to an alternative library! However, as I explain in my answer, uploading multiple files at once is more often than not counterproductive... Merge my answer into yours and I'll delete mine?

[BackSlash]

@fge well, yes the last line was to say that in fact uploading more files at the same time would slow down the upload speed. Merge my answer into yours and I'll delete mine are you talking about copying entirely your answer to mine?

[BackSlash]

@fge FTPClient, by default, uses a 1KB buffer to upload files, and this makes the upload very slow. If you change the buffer size to 0, it should use the maximum buffer size it can use. But, I tried with setting the buffer size to 0 and it didn't increase upload speed

[Stephen C]

It is also possible that some of the answers to that question are wrong ... or apply in different scenarios. I'm merely reporting what I found in the hope that they will be useful. (I suggest you read the Answers and make up your own mind if they are likely to apply ... and do some experiments.)

[matbrgz]

"Took one minute" sounds like something else is broken. Where exactly was the time spent?

[BackSlash]

@ThorbjørnRavnAndersen I measured the time, when i call client.storeFile() from Apache Commons it takes over a minute to upload very small files. When i call client.upload() from FTP4j it takes 1-2 seconds to transfer the same file

[BackSlash]

@ThorbjørnRavnAndersen Isn't the difference between System.currentTimeMillis() (first taken before calling the upload funcion, second called after it) enaugh?

[matbrgz]

Depends om what you need to know. A very experienced programmer needing to find out why in order to fix it, would use a profiler to find out where the time is actually spent. A factor 100 difference may be an external factor - like DNS lookups - and profilers usually pinpoints the culprit quickly.

[BackSlash]

I would say that if a library is faster than another, the problem is obviously not in the internet connection.

[matbrgz]

You missed the point. DNS lookup was an example of something where the difference can be a minute without your code being wrong. If you do not know where the time goes, you cannot find a fix.