Scala's "for comprehension" with futures

Question

I am reading through the Scala Cookbook (http://shop.oreilly.com/product/0636920026914.do)

There is an example related to Future use that involves for comprehension.

So far my understanding about for comprehension is when use with a collection it will produce another collection with the same type. For example, if each futureX is of type Future[Int], the following should also be of type Future[Int]:

for {
   r1 <- future1
   r2 <- future2
   r3 <- future3
} yield (r1+r2+r3)

Could someone explain me what exactly happening when use <- in this code? I know if it was a generator it will fetch each element by looping.

Solution

First about for comprehension. It was answered on SO many many times, that it's an abstraction over a couple of monadic operations: map, flatMap, withFilter. When you use <-, scalac desugars this lines into monadic flatMap:

r <- monad into monad.flatMap(r => ... )

it looks like an imperative computation (what a monad is all about), you bind a computation result to the r. And yield part is desugared into map call. Result type depends on the type of monad's.

Future trait has a flatMap and map functions, so we can use for comprehension with it. In your example can be desugared into the following code:

future1.flatMap(r1 => future2.flatMap(r2 => future3.map(r3 => r1 + r2 + r3) ) )

Parallelism aside

It goes without saying that if execution of future2 depends on r1 then you can't escape sequential execution, but if the future computations are independent, you have two choices. You can enforce sequential execution, or allow for parallel execution. You can't enforce the latter, as the execution context will handle this.

val res = for {
   r1 <- computationReturningFuture1(...)
   r2 <- computationReturningFuture2(...)
   r3 <- computationReturningFuture3(...)
} yield (r1+r2+r3)

will always run sequentially. It can be easily explained by the desugaring, after which the subsequent computationReturningFutureX calls are only invoked inside of the flatMaps, i.e.

computationReturningFuture1(...).flatMap(r1 => 
    computationReturningFuture2(...).flatMap(r2 => 
        computationReturningFuture3(...).map(r3 => r1 + r2 + r3) ) )

However this is able to run in parallel and the for comprehension aggregates the results:

val future1 = computationReturningFuture1(...)
val future2 = computationReturningFuture2(...)
val future3 = computationReturningFuture3(...)

val res = for {
   r1 <- future1
   r2 <- future2
   r3 <- future3
} yield (r1+r2+r3)

Comments

[Impredicative]

The Scala Language Spec (scala-lang.org/files/archive/nightly/pdfs/ScalaReference.pd‌​f, now a little out of date) gives the precise rules for desugaring of for comprehensions on pages 89-90.

[nish1013]

@Alexlv Sorry ,what is monodic operations ?? seems like its something to do with collections by looking at your explanation.

[4lex1v]

@nish1013 A concept of a monad is more general, so no, it has nothing to do with collections in particular. A monadic function is a function of type A => M[A], so it's a function which gives us a monadic value, which is a value in some context which describes computation, i.e Future which describes a context in which computations are deferred for future.

[nish1013]

@AlexIv sorry to ask this stupid question , what exactly "monadic" means ? I got to know form you two variations i.e monodic functions and monodic value . But still "monodic" greek term is there. Could you please tell me what is "monodic" in plain English or direct me to a resource to understand the concept in programming context ? before I asked this from you I tried gooling but finally got more confused :(

[4lex1v]

@nish1013 That's a really too big idea and it was discussed too many times. Monadic is just about a concept of a Monad and computation flow. To understand monads i would highly recomend this book and this tutorial or just look for similar questions on StackOverflow

[nish1013]

@AlexIv by the way I purchased the early edition of manning.com/bjarnason, but it does not explain much what is Monad in detail than bunch of examples that hard to understand for a beginner. And in the tutorial it refereed to swiss.csail.mit.edu/classes/6.001/abelson-sussman-lectures this. In that book I searched for monad keyword, but non there :(

[4lex1v]

@nish1013 if you have bought a MEAP version, the manning sends you a fresh version after each update, check your mailbox or download the latest version. The latest is 11. It contains all chapters, 11 and 12 chapters is just what you need

[howard branch]

The comment, "BUT you should remember that this gonna be evaluated sequentially, not in parallel, because the result of result2 and result3 depends on r1" is somewhat misleading. Whether or not the underlying operation of future2 depends on the outcome of future1 it will be begun in the onComplete of future1 and thus be sequential. Future(1).map(_ => Future(2)) is sequential although the underlying functions are independent.

[redzedi]

Is the Future thus created is complete ? I tried something similar , it created a future ,tat was not complete

[Jonathan Stray]

This seems a very complex answer to what is really a very simple question. Yes, this describes the implementation well, but most of the time someone looking for the answer won't care about that.

[prashant]

Could you add the desugared version of the sequential example for clarity?

[4lex1v]

@DavidMoles please check the answer, there's already an example of desugared code

[prashant]

Hi @4lex1v, I see the desugared code for one case, but I don't understand how the other case desugars differently.

[Jason Wheeler]

The for comprehension doesn't allow them to run in parallel. They are running in parallel because the Futures were defined outside of the for comprehension, but if you were creating Futures inside of the for comprehension, they would run sequentially (ex: r1 <- Future(value1)). Sometimes you want them to run sequentially, for instance, you may have a future that gives you back a an id, which you need to make another call with, which is also a Future.

[Nathan Brown]

I've made an edit that explicitly shows the desugared code for the sequential case, hope that helps.

[Jus12]

Too bad though, because the code looks more compact when the future is created inside for.

[Saurabh Gangamwar]

How to make it parallel