How to set the buffer size on a BufferedWriter over a FileWriter
Solution 1
I solve the problem by using OutputStream, not writer, here is the code:
bw = new BufferedOutputStream(
new FileOutputStream(new File("/Users/liaoliuqing/Downloads/1.txt"),true),165537);
Solution 2
Is there a problem with my code?
A few. Mainly: potential IO and concurrency errors. File buffer size might be a lesser concern (and one you can't effectively deal with).
-
Trying to open already opened file. All your threads are trying to write into the same file (
1.txt
). That might be an issue. FileWriter documentation says:Some platforms, in particular, allow a file to be opened for writing by only one FileWriter (or other file-writing object) at a time. In such situations the constructors in this class will fail if the file involved is already open.
-
Lines might be cut and mixed. If you have several threads with their respective buffers flushing at some point into the same output, you might not even need weird race-conditions or threads stopped right of the middle or a write operation to see your output corrupted.
As I solution (If your threads must share the same output) you can use a shared object with synchronized access to take care of actual writing. I implemented
SafeAppender
in my example, but probably there are better alternatives out there. No flushing and closing buffers will mean (the tail of) your data will be lost (like tears in the rain). A finally block is usually good to take care of that.
Also, as stated by other users,
BufferedWriter
buffer size does not affect the buffer size inFileOutputStream
(and soFileWriter
). And it looks thejava.io
andjava.nio
APIs dont offer any way to mess with that. If you look at the Java library sources you might noticeBufferedWriter
buffer size just means the amount of chars you store before actually writing into the delegate output. The default size (8192) is optimal for most cases, and increasing it might mean more trouble (potentially losing more data) than benefits.
This is my code, if it serves you:
// http://stackoverflow.com/questions/32451526/how-to-set-the-buffer-size-on-a-bufferedwriter-over-a-filewriter
public class TestWriter {
public static class SafeAppender {
private BufferedWriter bw;
private int users = 0;
public SafeAppender(File f) throws IOException {
bw = new BufferedWriter(new FileWriter(f));
}
public synchronized void append(String s) throws IOException {
bw.write(s);
}
public synchronized void incrUsers() {
users ++;
}
public synchronized void decrUsers() {
if (--users <= 0) {
try {
bw.flush();
System.err.println("INFO-appender-flush()");
} catch (Throwable whatever) { /* log-if-you-care*/}
}
}
// Might be called by GC, or not
@Override protected void finalize() throws Throwable {
try {
bw.close();
System.err.println("INFO-appender-close()");
} catch (Throwable whatever) { /* log-if-you-care */}
super.finalize();
}
}
private static class MyRunnable implements Runnable {
final static String S = "{addffffffkkkljlkj2015dd}";
SafeAppender appender;
String threadId;
public MyRunnable (SafeAppender a, String tid) {
appender = a; threadId = tid;
}
public void run() {
appender.incrUsers();
try {
for(int i =0 ; i<1000; i++){
// NOTE: Not a good idea to printStackTrace if each line fails. Let thread fail
String line = String.format("%s-%03d-%s\n", threadId, i, S);
appender.append(line);
}
} catch (IOException e) {
System.err.printf("ERROR-%s-%s\n", threadId, e.toString());
} finally {
appender.decrUsers();
}
}
}
public static void main(String[] args) {
try {
File f = File.createTempFile("TestWriter", ".txt");
System.err.printf("INFO-main-Writing into %s\n", f.getCanonicalPath());
SafeAppender appender = new SafeAppender (f);
for(int i =0;i<10;i++){
MyRunnable r = new MyRunnable(appender, ""+i);
Thread t = new Thread(r);
t.start();
}
} catch (Throwable e) {
e.printStackTrace(System.err);
}
}
}
Solution 3
FileWriter
actually uses its own fixed-size 1024 byte buffer. The BufferedWriter
on the other hand, show that it uses and 8192 byte buffer size (default), which can be configured by the user to any other desired size.
And to further muddy the waters, the Java 6 implementation of OutputStreamWriter
actually delegates to a StreamEncoder
, which uses its own buffer with a default size of 8192 bytes. And the StreamEncoder
buffer is user-configurable, although there is no way to access it directly through the enclosing OutputStreamWriter.
jinhong_lu
Updated on July 21, 2022Comments
-
jinhong_lu almost 2 years
I met a problem with
BufferedWriter
when I write data to a single file with some threads.I set the buffer size of the
BufferedWriter
, but no matter what number I set, it flushes the data to disk when the buffer is 8192 (the default buffer size), not the size I set (here is 16384). Is there a problem with my code?This is how I'm constructing the
BufferedWriter
:new BufferedWriter(new FileWriter(fileName, true), 16384);
This is the full code:
import java.io.BufferedWriter; import java.io.File; import java.io.FileWriter; import java.io.IOException; public class Test1 { public static void main(String[] args) throws IOException { for(int i =0;i<10;i++){ MyThread r = new MyThread(); Thread t = new Thread(r); t.start(); } } } class MyThread implements Runnable { public void run() { String s = "{addffffffkkkljlkj2015dd}\n"; BufferedWriter bw = null; try { bw = new BufferedWriter(new FileWriter( "/Users/liaoliuqing/Downloads/1.txt", true),16384); } catch (IOException e) { e.printStackTrace(); } for(int i =0 ; i<1000; i++){ try { bw.write(String.format("%03d", i)+s); //bw.flush(); } catch (IOException e) { e.printStackTrace(); } } } }
-
Chris Martin over 8 yearsHow does that help you control the buffer size?
-
Chris Martin over 8 yearsThis answer appears to have been copied from stackoverflow.com/questions/6976893/…, which contains some additional useful links.
-
Raman Shrivastava over 8 yearsIndeed it is. But copy would be a wrong word. I could have rephrased it to suggest it i my answer, but the point was to pass the information and no point rewriting it if it's already there. i skipped those links to keep answer minimal and relevant and kept doors open for OP to do some research himself. :)
-
jinhong_lu over 8 yearsI set to this: bw = new BufferedWriter(new OutputStreamWriter( new FileOutputStream(new File("/Users/liaoliuqing/Downloads/1.txt"),true)),65536); bw.write(String.format("%03d", i)+s); bw.flush(); my string is 27340 bytes, so the buffer size is big enough to contain the string. but still it write to disk every 8192 bytes.
-
Javier over 8 yearsGood for you, but that's weird, since
Writer
delegates overOutputStream
and all the behavior (appending flag, buffer size, exact operations if writing ASCII) is identical. Any other changes? -
Marco Sandrini over 8 yearsMy answer was based on the documentation: as it states 'to specify those values' (plural), I assumed that in that case the FileOutputStream would not have a buffer of its own....