Java NIO FileChannel versus FileOutputstream performance / usefulness

Question

I am trying to figure out if there is any difference in performance (or advantages) when we use nio FileChannel versus normal FileInputStream/FileOuputStream to read and write files to filesystem. I observed that on my machine both perform at the same level, also many times the FileChannel way is slower. Can I please know more details comparing these two methods. Here is the code I used, the file that I am testing with is around 350MB. Is it a good option to use NIO based classes for File I/O, if I am not looking at random access or other such advanced features?

package trialjavaprograms;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;

public class JavaNIOTest {
    public static void main(String[] args) throws Exception {
        useNormalIO();
        useFileChannel();
    }

    private static void useNormalIO() throws Exception {
        File file = new File("/home/developer/test.iso");
        File oFile = new File("/home/developer/test2");

        long time1 = System.currentTimeMillis();
        InputStream is = new FileInputStream(file);
        FileOutputStream fos = new FileOutputStream(oFile);
        byte[] buf = new byte[64 * 1024];
        int len = 0;
        while((len = is.read(buf)) != -1) {
            fos.write(buf, 0, len);
        }
        fos.flush();
        fos.close();
        is.close();
        long time2 = System.currentTimeMillis();
        System.out.println("Time taken: "+(time2-time1)+" ms");
    }

    private static void useFileChannel() throws Exception {
        File file = new File("/home/developer/test.iso");
        File oFile = new File("/home/developer/test2");

        long time1 = System.currentTimeMillis();
        FileInputStream is = new FileInputStream(file);
        FileOutputStream fos = new FileOutputStream(oFile);
        FileChannel f = is.getChannel();
        FileChannel f2 = fos.getChannel();

        ByteBuffer buf = ByteBuffer.allocateDirect(64 * 1024);
        long len = 0;
        while((len = f.read(buf)) != -1) {
            buf.flip();
            f2.write(buf);
            buf.clear();
        }

        f2.close();
        f.close();

        long time2 = System.currentTimeMillis();
        System.out.println("Time taken: "+(time2-time1)+" ms");
    }
}

Solution 1

My experience with larger files sizes has been that java.nio is faster than java.io. Solidly faster. Like in the >250% range. That said, I am eliminating obvious bottlenecks, which I suggest your micro-benchmark might suffer from. Potential areas for investigating:

The buffer size. The algorithm you basically have is

• copy from disk to buffer
• copy from buffer to disk

My own experience has been that this buffer size is ripe for tuning. I've settled on 4KB for one part of my application, 256KB for another. I suspect your code is suffering with such a large buffer. Run some benchmarks with buffers of 1KB, 2KB, 4KB, 8KB, 16KB, 32KB and 64KB to prove it to yourself.

Don't perform java benchmarks that read and write to the same disk.

If you do, then you are really benchmarking the disk, and not Java. I would also suggest that if your CPU is not busy, then you are probably experiencing some other bottleneck.

Don't use a buffer if you don't need to.

Why copy to memory if your target is another disk or a NIC? With larger files, the latency incured is non-trivial.

Like other have said, use FileChannel.transferTo() or FileChannel.transferFrom(). The key advantage here is that the JVM uses the OS's access to DMA (Direct Memory Access), if present. (This is implementation dependent, but modern Sun and IBM versions on general purpose CPUs are good to go.) What happens is the data goes straight to/from disc, to the bus, and then to the destination... bypassing any circuit through RAM or the CPU.

The web app I spent my days and night working on is very IO heavy. I've done micro benchmarks and real-world benchmarks too. And the results are up on my blog, have a look-see:

• Real world performance metrics: java.io vs. java.nio
• Real world performance metrics: java.io vs. java.nio (The Sequel)

Use production data and environments

Micro-benchmarks are prone to distortion. If you can, make the effort to gather data from exactly what you plan to do, with the load you expect, on the hardware you expect.

My benchmarks are solid and reliable because they took place on a production system, a beefy system, a system under load, gathered in logs. Not my notebook's 7200 RPM 2.5" SATA drive while I watched intensely as the JVM work my hard disc.

What are you running on? It matters.

Solution 2

If the thing you want to compare is performance of file copying, then for the channel test you should do this instead:

final FileInputStream inputStream = new FileInputStream(src);
final FileOutputStream outputStream = new FileOutputStream(dest);
final FileChannel inChannel = inputStream.getChannel();
final FileChannel outChannel = outputStream.getChannel();
inChannel.transferTo(0, inChannel.size(), outChannel);
inChannel.close();
outChannel.close();
inputStream.close();
outputStream.close();

This won't be slower than buffering yourself from one channel to the other, and will potentially be massively faster. According to the Javadocs:

Many operating systems can transfer bytes directly from the filesystem cache to the target channel without actually copying them.

Solution 3

Based on my tests (Win7 64bit, 6GB RAM, Java6), NIO transferFrom is fast only with small files and becomes very slow on larger files. NIO databuffer flip always outperforms standard IO.

• Copying 1000x2MB

  1. NIO (transferFrom) ~2300ms
  2. NIO (direct datababuffer 5000b flip) ~3500ms
  3. Standard IO (buffer 5000b) ~6000ms

• Copying 100x20mb

  1. NIO (direct datababuffer 5000b flip) ~4000ms
  2. NIO (transferFrom) ~5000ms
  3. Standard IO (buffer 5000b) ~6500ms

• Copying 1x1000mb

  1. NIO (direct datababuffer 5000b flip) ~4500s
  2. Standard IO (buffer 5000b) ~7000ms
  3. NIO (transferFrom) ~8000ms

The transferTo() method works on chunks of a file; wasn't intended as a high-level file copy method: How to copy a large file in Windows XP?

Solution 4

Answering the "usefulness" part of the question:

One rather subtle gotcha of using FileChannel over FileOutputStream is that performing any of its blocking operations (e.g. read() or write()) from a thread that's in interrupted state will cause the channel to close abruptly with java.nio.channels.ClosedByInterruptException.

Now, this could be a good thing if whatever the FileChannel was used for is part of the thread's main function, and design took this into account.

But it could also be pesky if used by some auxiliary feature such as a logging function. For example, you can find your logging output suddenly closed if the logging function happens to be called by a thread that's also interrupted.

It's unfortunate this is so subtle because not accounting for this can lead to bugs that affect write integrity.^[1][2]

• ^[1] https://bugs.openjdk.java.net/browse/JDK-4469683
• ^[2] http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6608965

Solution 5

I tested the performance of FileInputStream vs. FileChannel for decoding base64 encoded files. In my experients I tested rather large file and traditional io was alway a bit faster than nio.

FileChannel might have had an advantage in prior versions of the jvm because of synchonization overhead in several io related classes, but modern jvm are pretty good at removing unneeded locks.

View more solutions

Comments

[Brett]

transferTo/transferFrom would be more conventional for copying files. Whichever technique shouldn't make your hard drive any faster or slower, although I guess there could be a problem if it read small chunks at a time and causes the head to spend an inordinate amount of time seeking.

[Brett]

(You don't mention which OS you are using, or which JRE vendor and version.)

[Keshav]

Oops sorry, I am using FC10 with Sun JDK6.

[Stu Thompson]

Did you get that mixed up? My own experience is that java.nio is faster with larger files than java.io, not smaller.

[tangens]

No, my experience is the other way round. java.nio is fast as long the file is small enough to be mapped to memory. If it gets bigger (200 MB and more) java.io is faster.

[Stu Thompson]

Wow. The total opposite of me. Note that you don't necessarily need to map a file to read it--one can read from the FileChannel.read(). There is not just one single approach to read files using java.nio.

[Andy Dufresne]

@Stu Thompson - Thanks for your post. I came across your answer since I am researching on the same topic. I am trying to understand the OS improvements that nio exposes to the java programmers. A couple of them being - DMA and memory mapped files. Have you come across more of such improvements? P.S - Your blog links are broken.

[Stu Thompson]

@AndyDufresne my blog is down at the moment, will be up later this week--in the process of moving it.

[Arthur Edelstein]

Here are the blog links on archive.org: web.archive.org/web/20120815094827/http://geekomatic.ch/2008‌​/… web.archive.org/web/20120821114802/http://geekomatic.ch/2009‌​/…

[Deckard]

How about just copying files from one directory to another? (Each different disk drive)

[Jeryl Cook]

Interesting: mailinator.blogspot.in/2008/02/…

[sinedsem]

@tangens did you check this?

[Smithfield]

"NIO is faster for larger files". May I ask what 'larger' means in terms of size here? Larger file is >1MB, >100MB, >350MB as in initial question, more?