Using async/await with a forEach loop

Question

Are there any issues with using async/await in a forEach loop? I'm trying to loop through an array of files and await on the contents of each file.

import fs from 'fs-promise'

async function printFiles () {
  const files = await getFilePaths() // Assume this works fine

  files.forEach(async (file) => {
    const contents = await fs.readFile(file, 'utf8')
    console.log(contents)
  })
}

printFiles()

This code does work, but could something go wrong with this? I had someone tell me that you're not supposed to use async/await in a higher-order function like this, so I just wanted to ask if there was any issue with this.

Solution 1

Sure the code does work, but I'm pretty sure it doesn't do what you expect it to do. It just fires off multiple asynchronous calls, but the printFiles function does immediately return after that.

Reading in sequence

If you want to read the files in sequence, you cannot use forEach indeed. Just use a modern for … of loop instead, in which await will work as expected:

async function printFiles () {
  const files = await getFilePaths();

  for (const file of files) {
    const contents = await fs.readFile(file, 'utf8');
    console.log(contents);
  }
}

Reading in parallel

If you want to read the files in parallel, you cannot use forEach indeed. Each of the async callback function calls does return a promise, but you're throwing them away instead of awaiting them. Just use map instead, and you can await the array of promises that you'll get with Promise.all:

async function printFiles () {
  const files = await getFilePaths();

  await Promise.all(files.map(async (file) => {
    const contents = await fs.readFile(file, 'utf8')
    console.log(contents)
  }));
}

Solution 2

With ES2018, you are able to greatly simplify all of the above answers to:

async function printFiles () {
  const files = await getFilePaths()

  for await (const contents of files.map(file => fs.readFile(file, 'utf8'))) {
    console.log(contents)
  }
}

See spec: proposal-async-iteration

---

2018-09-10: This answer has been getting a lot of attention recently, please see Axel Rauschmayer's blog post for further information about asynchronous iteration.

Solution 3

Instead of Promise.all in conjunction with Array.prototype.map (which does not guarantee the order in which the Promises are resolved), I use Array.prototype.reduce, starting with a resolved Promise:

async function printFiles () {
  const files = await getFilePaths();

  await files.reduce(async (promise, file) => {
    // This line will wait for the last async function to finish.
    // The first iteration uses an already resolved Promise
    // so, it will immediately continue.
    await promise;
    const contents = await fs.readFile(file, 'utf8');
    console.log(contents);
  }, Promise.resolve());
}

Solution 4

The p-iteration module on npm implements the Array iteration methods so they can be used in a very straightforward way with async/await.

An example with your case:

const { forEach } = require('p-iteration');
const fs = require('fs-promise');

(async function printFiles () {
  const files = await getFilePaths();

  await forEach(files, async (file) => {
    const contents = await fs.readFile(file, 'utf8');
    console.log(contents);
  });
})();

Solution 5

Here are some forEachAsync prototypes. Note you'll need to await them:

Array.prototype.forEachAsync = async function (fn) {
    for (let t of this) { await fn(t) }
}

Array.prototype.forEachAsyncParallel = async function (fn) {
    await Promise.all(this.map(fn));
}

Note while you may include this in your own code, you should not include this in libraries you distribute to others (to avoid polluting their globals).

View more solutions

Comments

[Demonbane]

Could you please explain why does for ... of ... work?

[Demonbane]

ok i know why... Using Babel will transform async/await to generator function and using forEach means that each iteration has an individual generator function, which has nothing to do with the others. so they will be executed independently and has no context of next() with others. Actually, a simple for() loop also works because the iterations are also in one single generator function.

[Bergi]

@Demonbane: In short, because it was designed to work :-) await suspends the current function evaluation, including all control structures. Yes, it is quite similar to generators in that regard (which is why they are used to polyfill async/await).

[arve0]

So files.map(async (file) => ... is equivalent to files.map((file) => new Promise((rej, res) => { ...?

[Bergi]

@arve0 Not really, an async function is quite different from a Promise executor callback, but yes the map callback returns a promise in both cases.

[Jay Edwards]

Minor addendum, don't forget to wrap your await/asyncs in try/catch blocks!!

[Bergi]

Have a look at How to define method in javascript on Array.prototype and Object.prototype so that it doesn't appear in for in loop. Also you probably should use the same iteration as native forEach - accessing indices instead of relying on iterability - and pass the index to the callback.

[jbustamovej]

This is a step in the wrong direction. Here's a mapping guide I created to help get folks stuck in callback hell into the modern JS era: github.com/jmjpro/async-package-to-async-await/blob/master/…‌​.

[Zachary Ryan Smith]

as you can see here, I am interested in and open to using async/await instead of the async lib. Right now, I think that each has a time and place. I'm not convinced that the async lib == "callback hell" and async/await == "the modern JS era". imo, when async lib > async/await: 1. complex flow (eg, queue, cargo, even auto when things get complicated) 2. concurrency 3. supporting arrays/objects/iterables 4. err handling

[Adi Sivasankaran]

This answer is the best one: await Promise.all(_.map(arr, async (val) => {...}); solved my issue. Of course, each async callback returns a promise that I was not awaiting on.

[Doug]

For those who don't know, Promise.all returns a single promise that resolves when all the promises in the array are resolved. (basically waits for all promises to conclude)

[doubleOrt]

I think the second paragraph should be "If you want to read the files in parallel, you cannot use for..of" ? Because you can read the files in parallel with the forEach, it's just that it will look very ugly because you cannot await it.

[Bergi]

@Taurus If you don't intend to await them, then for…of would work equally to forEach. No, I really mean that paragraph to emphasise that there is no place for .forEach in modern JS code.

[doubleOrt]

@Bergi Isn't that bad for performance ? If you need to make 3 requests, you can't make them all in parallel, so if each takes one second, your program will take 3 seconds to run. But if you use map or forEach, the requests will run in parallel.

[Bergi]

@Taurus I don't understand. What is bad for performance in comparison to what?

[doubleOrt]

for..of is bad for performance in comparison to forEach/map, because for..of will stop in each iteration, so if I make 3 requests, each request will have to wait for the preceding requests to complete. But with forEach/map, all the requests will be made in parallel.

[Bergi]

@Taurus No. queries.forEach(q => asyncRequest(q)); does exactly the same as for (const q of queries) asyncRequest(q);. There is no difference in performance, and both will run the requests in parallel. Of course, in neither you can wait for anything - for how to do that, see my answer.

[doubleOrt]

@Bergi I am talking about this: gist.github.com/doubleOrt/598659adfefdd941f2a98512f0f7f078

[Bergi]

@Taurus Yes, that's basically the two approaches from my answer. Neither of them uses forEach. Where's the problem?

[doubleOrt]

@Bergi I could have used forEach (would work the same as map regarding execution time). But as you can see from my previous comments, I was talking about either forEach or map versus for..of. I said: for..of is bad for performance in comparison to forEach/map.

[doubleOrt]

However, how is this true (from your answer): If you want to read the files in parallel, you cannot use forEach indeed. I think you can do that, same thing as your map example, except with forEach it will look ugly and unfit for an async/await.

[Bergi]

@Taurus But that's wrong: for…of is not bad for performance in comparison to forEach if used in the same unfit way without awaiting anything. And yes, "cannot be used" means "is unfit".

[doubleOrt]

@Bergi Are you suggesting a case where you don't await whatever request you make inside of a for..of ? If so, alright but you do agree that your second example is way more performant than your first example ?

[Bergi]

@Taurus Yes, that's what I wrote in my comment above. Of course it doesn't make any sense and is unfit to solve the OPs problem, regardless whether with forEach or for…of. And no, I would not compare the two (for…of+await vs Promise.all+map) in terms of "performance" at all - they are just different, in many other more important regards. Choosing sequential vs parallel executing is a decision that involves many other factors, and of course parallel typically finishes faster.

[doubleOrt]

@Bergi Of course there are cases where the for..of might be the only way to go (e.g if each request has a dependency on the previous request), but a beginner might always opt for the for..of solution because it is simpler.

[doubleOrt]

@Bergi However, I still don't understand how this is correct: If you want to read the files in parallel, you cannot use forEach indeed.

[Bergi]

@Taurus You cannot use it because you cannot await the result. Of course if you used it - like the OP did - then they would run in parallel, but forEach is absolutely unfit as you said yourself.

[Timothy Zorn]

You can use Array.prototype.reduce in a way that uses an async function. I've shown an example in my answer: stackoverflow.com/a/49499491/2537258

[parrker9]

This works perfectly, thank you so much. Could you explain what is happening here with Promise.resolve() and await promise;?

[Ryan Bayne]

This is pretty cool. Am I right in thinking the files will be read in order and not all at once?

[Timothy Zorn]

@parrker9 Promise.resolve() returns an already resolved Promise object, so that reduce has a Promise to start with. await promise; will wait for the last Promise in the chain to resolve. @GollyJer The files will be processed sequentially, one at a time.

[Antonio Val]

I don't think this answer address the initial question. for-await-of with a synchronous iterable (an array in our case) doesn’t cover the case of iterating concurrently an array using asynchronous operations in each iteration. If I'm not mistaken, using for-await-of with a synchronous iterable over non-promise values is the same as using a plain for-of.

[Vadim Shvetsov]

How we delegates files array to the fs.readFile here? It tooks from iterable?

[Shay Yzhakov]

Very cool use of reduce, thanks for the comment! I'll just denote that, in contrast to some of the other methods mentioned in the comments, this one is synchronous, meaning that the files are read one after another and not in parallel (since the next iteration of reduce function relies on the previous iteration, it must be synchronous).

[Timothy Zorn]

@Shay, You mean sequential, not synchronous. This is still asynchronous - if other things are scheduled, they will run in between the iterations here.

[Rafi Henig]

Using this solution each iteration would await for the previous, and in case of operation is making some long calculations or reading a long file it would block the executions of the next, as opposed to mapping all the functions to promises and waiting for them to complete.

[Bergi]

Your methods 1 and 2 are simply incorrect implementations where Promise.all should have been used - they do not take any of the many edge cases into account.

[PranavKAndro]

@Bergi: Thanks for the valid comments, Would you please explain me why method 1 and 2 are incorrect. It also serves the purpose. This works very well. This is to say that all these methods are possible, based on the situation one can decide on choosing one. I have the running example for the same.

[Bergi]

It fails on empty arrays, it doesn't have any error handling, and probably more problems. Don't reinvent the wheel. Just use Promise.all.

[PranavKAndro]

In certain conditions where its not possible it will be helpful. Also error handling is done by forEach api by default so no issues. Its taken care !

[Bergi]

No, there are no conditions where Promise.all is not possible but async/await is. And no, forEach absolutely doesn't handle any promise errors.

[PranavKAndro]

Cone on Bergi, its not the matter of which is best, but the various available solutions. You mentioned error handling as empty array, empty array error handling is taken care by forEach internally. Lets drop this conversation here. Again its all the possible ways you can achieve

[Bergi]

It's not about "good" vs "better" from a pool of valid solutions, but that your code is completely broken. Have you tried what happens when you pass in an empty array? It hangs forever. Have you tried what happens when someAPICall() rejects? You get an unhandled promise rejection and it hangs forever.

[PranavKAndro]

@Bergi: It is usual that if any developer use the async await to wait for the promise, then it should be surrounded by the try catch. someAPICall() is not the wrapper that I am providing, It is the developer function that developer write inside the callback that I am providing as part of wrapper, developer should surround that with try catch. If you want i can update the answer, Empty array is handled, you run and let me know. It works perfect.

[Bergi]

"developer should surround that with try catch" - and yet you didn't. If you think one always would need to do this, yes please at least do it yourself. But no, the usual expectation is that the promise returned by forEachAsync(…) should reject. And no, your code does not handle empty arrays, did you try it? await [].forEachAsync(() => {}); console.log("never happens");

[Admin]

Is there a way to let this apparently fast when you have an Express page that needs a list of resources to render...?

[Timothy Zorn]

If you need the async processes to finish as quickly as possible and you don't care about them being completed sequentially, try one of the provided solutions with a good amount of upvotes which uses Promise.all. Example: Promise.all(files.map(async (file) => { /* code */ }));

[Mark Odey]

The first senario is ideal for loops that needs to be ran in serie and you cant use for of

[Bergi]

Your option a involves the Promise constructor antipattern.

[jib]

This answer has the same issue as the OP: It accesses all files in parallel. The serialized printing of results merely hides it.

[010011100101]

You're a life saver. For the life of me I couldn't figure out why Object.entries({}).forEach wasn't returning the desired result. Switched it to: for (let [k, v] of Object.entries({})) {...} and it works great now. Thanks!

[Remi Mélisson]

my favorite option, I'd love to see a native Promise.chain() like all()

[Bergi]

It is also not good to open thousands of files at once to read them concurrently. One always has to do an assessment whether a sequential, parallel, or mixed approach is better. Sequential loops are not fundamentally bad, await actually makes them possible in the first place. Also they do not "aver the benefits" of asynchronous execution, as you can still run multiple such loops at once (e.g. two concurrent calls to printFiles).

[Ray Foss]

The main issue with this answer is that in for await, the if using a sync iterator like an Array, the map should return a promise. In node fs is not async by default... using fsPromises would make this run in parallel properly. I think its good to bring attention to this superior, well supported syntax.

[krupesh Anadkat]

If anyone wondering what vscode theme is that - its is github's official light theme. & If anyone hurt their eyes with so bright snapshot, my apologies 😅

[Damien Romito]

usage : await myArray. forEachAsyncParallel( async (item) => { await myAsyncFunction(item) })

[Bigyan Devkota]

This works perfectly -- thanks, here the important step is await promise; that's what does the heavy load

[Admin]

Less characters does not mean it is simpler. This is mostly convoluted and unreadable.

[leonheess]

Void function return value is used

[Bergi]

@leonheess What's the problem? Where (what code position) are you getting that warning? What tool do you use?

[leonheess]

@Bergi my bad I used forEach instead of mal

[Griffi]

the best answer ;) Useful is to just think what the transpiler is doing with async / await and how the final code will look like

[Bergi]

@Griffi Why are you still using a transpiler for async/await? And what will it transpile to?

[srmark]

Not quite the same. Promise.all will run all the promises concurrently. A for loop is meant to be sequential.

[Ido Bleicher]

I think it will be helpful if you can complete this example :) in the how to use section. For my case: await asyncForEach(configuration.groupNames, async (groupName) => { await AddUsersToGroup(configuration, groupName); })

[Bergi]

Better check urls.length before calling .shift() the first time, and better use urls[0] and urls.slice(1) instead of emptying the array that is being passed to the function.

[Bergi]

Why use finally instead of then? This will ignore errors, unlike async/await

[Matt Janssen]

This would be if you want to do every fetch, regardless of the success of preceding calls. Good idea on the empty check and not mutating the array! ✔

[Jesus Loves You]

best answer.....

[Augustin Riedinger]

There should be a forEachAwait operator that does just that!

[Bergi]

@AugustinRiedinger It's called for … of!

[Bergi]

This answer is wrong. files.map() returns an array of promises, not an asynchronous iterator, for which for await was made! It will cause unhandled-rejection crashes!

[Augustin Riedinger]

No, for ... of doesn't behave the same way: in one case, it takes a function (which can be returned, curried, composed etc.) while in the other it is simply a for loop that must be continued ...

[Bergi]

@AugustinRiedinger You cannot return anything useful from a forEach callback anyway

[Augustin Riedinger]

Well, those don't behave the same hence it could make sense that both versions exist. Just like for ... of and forEach also coexist.

[Bergi]

OP never requested not to use async/await. They state "I'm trying to loop through an array of files and await on the contents of each file."

[Bergi]

Also, why do you say require("async").forEach only works in nodejs?

[João Pimentel Ferreira]

@Bergi I explicitly said the OP didn't request exactly that and it just works with NodeJS. Although it still may be helpful for some people, because the example given by OP is to read file contents, and normally you do file reading in the backend.

[Bergi]

Oh, I misinterpreted that phrase as "does (not use async/await) as the OP requested" instead of "does not (use async/await as the OP requested)"

[Adrian Bartholomew]

You would need to return contents.

[Bergi]

@AdrianBartholomew OP only wants to log each file contents, and return nothing from the function. Of course if you want to create an array of file contents, you're free to return them from the map callback.

[Andrew Smith]

This is extremely clever, I love it.

[close]

I suggest using the phrase 'Before/After Loop' would make it less confusing when it's not a 'For Each Loop'.

[JulienRioux]

Thanks, nice solution!!

[User_coder]

The brother is out here just writing code using Githubs official like an absolute heathen. I'm not even mad. To each their own. Nonetheless, I would cache the length to speed that for loop up and prevent recalculations between every iteration.

[ado387]

So in short: foreach doesn't handle callbacks in asynchronous way, therefore no waiting.

[Normal]

@Matt, isn't it a problem to await fn in case it wasn't asynchronous? what if the given input was a synchronous function? stackoverflow.com/a/53113299/18387350