How to append to a file in Node?

Question

I am trying to append a string to a log file. However writeFile will erase the content each time before writing the string.

fs.writeFile('log.txt', 'Hello Node', function (err) {
  if (err) throw err;
  console.log('It\'s saved!');
}); // => message.txt erased, contains only 'Hello Node'

Any idea how to do this the easy way?

Solution 1

For occasional appends, you can use appendFile, which creates a new file handle each time it's called:

Asynchronously:

const fs = require('fs');

fs.appendFile('message.txt', 'data to append', function (err) {
  if (err) throw err;
  console.log('Saved!');
});

Synchronously:

const fs = require('fs');

fs.appendFileSync('message.txt', 'data to append');

But if you append repeatedly to the same file, it's much better to reuse the file handle.

Solution 2

When you want to write in a log file, i.e. appending data to the end of a file, never use appendFile. appendFile opens a file handle for each piece of data you add to your file, after a while you get a beautiful EMFILE error.

I can add that appendFile is not easier to use than a WriteStream.

Example with appendFile:

console.log(new Date().toISOString());
[...Array(10000)].forEach( function (item,index) {
    fs.appendFile("append.txt", index+ "\n", function (err) {
        if (err) console.log(err);
    });
});
console.log(new Date().toISOString());

Up to 8000 on my computer, you can append data to the file, then you obtain this:

{ Error: EMFILE: too many open files, open 'C:\mypath\append.txt'
    at Error (native)
  errno: -4066,
  code: 'EMFILE',
  syscall: 'open',
  path: 'C:\\mypath\\append.txt' }

Moreover, appendFile will write when it is enabled, so your logs will not be written by timestamp. You can test with example, set 1000 in place of 100000, order will be random, depends on access to file.

If you want to append to a file, you must use a writable stream like this:

var stream = fs.createWriteStream("append.txt", {flags:'a'});
console.log(new Date().toISOString());
[...Array(10000)].forEach( function (item,index) {
    stream.write(index + "\n");
});
console.log(new Date().toISOString());
stream.end();

You end it when you want. You are not even required to use stream.end(), default option is AutoClose:true, so your file will end when your process ends and you avoid opening too many files.

Solution 3

Your code using createWriteStream creates a file descriptor for every write. log.end is better because it asks node to close immediately after the write.

var fs = require('fs');
var logStream = fs.createWriteStream('log.txt', {flags: 'a'});
// use {flags: 'a'} to append and {flags: 'w'} to erase and write a new file
logStream.write('Initial line...');
logStream.end('this is the end line');

Solution 4

Besides appendFile, you can also pass a flag in writeFile to append data to an existing file.

fs.writeFile('log.txt', 'Hello Node',  {'flag':'a'},  function(err) {
    if (err) {
        return console.error(err);
    }
});

By passing flag 'a', data will be appended at the end of the file.

Solution 5

You need to open it, then write to it.

var fs = require('fs'), str = 'string to append to file';
fs.open('filepath', 'a', 666, function( e, id ) {
  fs.write( id, 'string to append to file', null, 'utf8', function(){
    fs.close(id, function(){
      console.log('file closed');
    });
  });
});

Here's a few links that will help explain the parameters

open
write
close

---

EDIT: This answer is no longer valid, look into the new fs.appendFile method for appending.

View more solutions

Comments

[angry kiwi]

look like supercobra constantly writes log to the log file, usage of fs.write is not recommended in this case, use fs.createWriteStream instead. Read nodejs.org/docs/v0.4.8/api/all.html#fs.write

[Ruben Tan]

The answer is no longer accurate as of nodejs v0.4.10.

[Zane Claes]

anything sync() is almost always a bad idea unless you're 100% sure you absolutely NEED it. Even then, you're probably doing it wrong.

[nelsonic]

Does anyone know if fs.appendFile keeps a link to the file open so appends are faster? (rather than open/close each write) nodejs.org/api/…

[Ya Zhuang]

it should be '0666' rather than 666.

[Luis R.]

Doesn't mean it's wrong. It just does it Synchronously. Might not be best practice for Node.js, but it's supported.

[Zane Claes]

I was using "ur doin it wrong" in the colloquial internet-meme sense of the phrase. Obviously it's supported =P

[Patrick Roberts]

@LuisR. The whole point in using node is to do things asynchronously, allowing the processes to happen in the background while freeing the internal event loop to handle other requests (if it's a server) or other tasks.

[bryanmac]

Agreed on async, but sometimes if you're just writing an interactive script, sync is fine.

[Jan Święcki]

Writing synchronously is absolutely ok if you are doing single user command line app (e.g. script to do some stuff). That way it is faster to do stuff. Why would node have sync methods if not for this purpose?

[Maël Nison]

@nelsonic According to the source code, there is no particular treatment.

[SilentSteel]

In case it's handy: Note that this is async. This can result in weird timing and other things. Ex: if you have process.exit() just after fs.appendFile, you may exit before the output is sent. (Using return is fine.)

[gumenimeda]

@SilentSteel I am seeing some wired behavior where not everything is getting appended to the file. This is at high volume where I am trying to append every 1ms. Do you know what I should be using instead?

[SilentSteel]

Worse case, you can use the synchronous version, appendFileSync. nodejs.org/api/… But you may lose one of the great benefits of Node, which is async operations. Make sure you catch errors. Perhaps on some OSes, you can get access denied if requesting the file handle at the same time. Not sure about that.

[Ben]

Also, this creates a file if it doesn't exist. Docs say: Asynchronously append data to a file, creating the file if it not yet exists. data can be a string or a buffer. (nodejs.org/api/…)

[Stormbytes]

missing first line! should be 'var fs = require('fs');'

[Marko Bonaci]

Or perhaps even better var fs = require('graceful-fs'), which hashed out some known problems. See the docs for more info.

[Gank]

fdW = fsW.openSync("", './log.txt'), 'a') ^ SyntaxError: Unexpected token )

[fadedbee]

Do not use fs.appendFile. It is a broken API which uses a new file descriptor for each call. On a busy server (hundreds of appendFile per second) this can result in all 1024 file descriptors being used up and the process not being able to open other files.

[zipzit]

@chrisdew Thanks for the update.. but... if we are not to use the accepted answer here, what are we supposed to do? How did you solve this dilema?

[binki]

Both the initial and end line are on the same line though :-p

[questionasker]

hi How to add each data with new line ?

[laurent]

Note that it won't automatically append a newline character, so if it's for something like a log file, a "\n" needs to be manually added at the end.

[Anish Nair]

Please note: If you are using fs.createWriteStream, then use flags. If you are using fs.writeFile then it's flag. Please refer Node JS Docs - File System for more information.

[Anish Nair]

Please note: If you are using fs.createWriteStream, then use flags. If you are using fs.writeFile then it's flag. Please refer Node JS Docs - File System for more information.

[Marcus]

@zipzit you asked chrisdew (which I assume is now fadedbee) for an alternative to fs.appendFile. How about the answer from Plaute? It says fs.createWriteStream("append.txt", {flags:'a'}).write(myText);

[Benny Neugebauer]

Be careful! The parameter is not "flags" but "flag" (singular): nodejs.org/api/…

[Aashish Kumar]

Thanks for the great answer, but my doubt is that due to asynchronous nature of Javascript, it will execute stream.end() before the stream.write(), so we shouldn't use stream.end(), also as you mentioned that AutoClose:True is a default option then why bother writing a line which is of no use.

[Sharcoux]

due to asynchronous nature of Javascript... What? Array.forEach is a synchronous operation. JS is synchronous. It just happens to provide some ways to manage asynchronous operations, like Promises and async/await.

[Qwiso]

@BennyNeugebauer the use of flags is correct, you are not. It was already posted many months before you You've linked docs to fs.writeFile which does use 'flag'. But this solution uses fs.createWriteStream and the parameter 'flags' is correct - nodejs.org/api/fs.html#fs_fs_createwritestream_path_options

[apple apple]

I'd guess fs.appendFile result in too many open files because you execute it in asynchronous manner (you're just asynchronously creating 10000 file handle), I believe appendFileSync would not have similar problem, also not fs.appendFile with proper interval (1s is probably more than enough) or queueing.

[Radvylf Programs]

@appleapple But you're still opening the file each time. For a log, it makes much more sense to keep it open.

[apple apple]

@RedwolfPrograms For busy server log, maybe true. For one-time per execution log, maybe not. Anyway, I just state that the point (at least the reason) in this answer is not correct.

[Dan]

For logs I would just use fs.appendFileSync (it can run a million times without issues). If the script crashes or exits right after you log something with stream.write or fs.appendFile, it is likely that you won't get the log written (which most likely will be needed to debug the crash/exit)

[noderman]

@fadedbee and @zipzit - old comment, but invaluable. I had a large loop with appends and noticed the thing would get exponentially slower. Best way is to createWriteStream with flags : 'a' and use subsequent <stream>.write for appending. I believe you can use await <stream>.write to make sure the appends are sequential.

[Petr Bugyík]

What is the reason why you deleted your linked project apickfs from GitHub?

[Uche Ozoemena]

Isn't this by far the easiest of all the answers?