How to wrap async function calls into a sync function in Node.js or Javascript?

Question

Suppose you maintain a library that exposes a function getData. Your users call it to get actual data:
var output = getData();
Under the hood data is saved in a file so you implemented getData using Node.js built-in fs.readFileSync. It's obvious both getData and fs.readFileSync are sync functions. One day you were told to switch the underlying data source to a repo such as MongoDB which can only be accessed asynchronously. You were also told to avoid pissing off your users, getData API cannot be changed to return merely a promise or demand a callback parameter. How do you meet both requirements?

Asynchronous function using callback/promise is the DNA of JavasSript and Node.js. Any non-trivial JS app is probably permeated with this coding style. But this practice can easily lead to so called callback pyramid of doom. Even worse, if any code in any caller in the call chain depends on the result of the async function, those code has to be wrapped in callback function as well, imposing a coding style constraint on caller. From time to time I find the need to encapsulate an async function (often provided in a 3rd party library) into a sync function in order to avoid massive global re-factoring. Searching for a solution on this subject usually ended up with Node Fibers or npm packages derived from it. But Fibers just cannot solve the problem I am facing. Even the example provided by Fibers' author illustrated the deficiency:

...
Fiber(function() {
    console.log('wait... ' + new Date);
    sleep(1000);
    console.log('ok... ' + new Date);
}).run();
console.log('back in main');

Actual output:

wait... Fri Jan 21 2011 22:42:04 GMT+0900 (JST)
back in main
ok... Fri Jan 21 2011 22:42:05 GMT+0900 (JST)

If function Fiber really turns async function sleep into sync, the output should be:

wait... Fri Jan 21 2011 22:42:04 GMT+0900 (JST)
ok... Fri Jan 21 2011 22:42:05 GMT+0900 (JST)
back in main

I have created another simple example in JSFiddle and looking for code to yield expected output. I'll accept a solution that only works in Node.js so you are free to require any npm package despite not working in JSFiddle.

Solution 1

deasync turns async function into sync, implemented with a blocking mechanism by calling Node.js event loop at JavaScript layer. As a result, deasync only blocks subsequent code from running without blocking entire thread, nor incuring busy wait. With this module, here is the answer to the jsFiddle challenge:

function AnticipatedSyncFunction(){
  var ret;
  setTimeout(function(){
      ret = "hello";
  },3000);
  while(ret === undefined) {
    require('deasync').runLoopOnce();
  }
  return ret;    
}


var output = AnticipatedSyncFunction();
//expected: output=hello (after waiting for 3 sec)
console.log("output="+output);
//actual: output=hello (after waiting for 3 sec)

(disclaimer: I am the co-author of deasync. The module was created after posting this question and found no workable proposal.)

Solution 2

There is a npm sync module also. which is used for synchronize the process of executing the query.

When you want to run parallel queries in synchronous way then node restrict to do that because it never wait for response. and sync module is much perfect for that kind of solution.

Sample code

/*require sync module*/
var Sync = require('sync');
    app.get('/',function(req,res,next){
      story.find().exec(function(err,data){
        var sync_function_data = find_user.sync(null, {name: "sanjeev"});
          res.send({story:data,user:sync_function_data});
        });
    });


    /*****sync function defined here *******/
    function find_user(req_json, callback) {
        process.nextTick(function () {

            users.find(req_json,function (err,data)
            {
                if (!err) {
                    callback(null, data);
                } else {
                    callback(null, err);
                }
            });
        });
    }

reference link: https://www.npmjs.com/package/sync

Solution 3

You've got to use promises:

const asyncOperation = () => {
    return new Promise((resolve, reject) => {
        setTimeout(()=>{resolve("hi")}, 3000)
    })
}

const asyncFunction = async () => {
    return await asyncOperation();
}

const topDog = () => {
    asyncFunction().then((res) => {
        console.log(res);
    });
}

I like arrow function definitions more. But any string of the form "() => {...}" could also be written as "function () {...}"

So topDog is not async despite calling an async function.

EDIT: I realize a lot of the times you need to wrap an async function inside a sync function is inside a controller. For those situations, here's a party trick:

const getDemSweetDataz = (req, res) => {
    (async () => {
        try{
            res.status(200).json(
                await asyncOperation()
            );
        }
        catch(e){
            res.status(500).json(serviceResponse); //or whatever
        }
    })() //So we defined and immediately called this async function.
}

Utilizing this with callbacks, you can do a wrap that doesn't use promises:

const asyncOperation = () => {
    return new Promise((resolve, reject) => {
        setTimeout(()=>{resolve("hi")}, 3000)
    })
}

const asyncFunction = async (callback) => {
    let res = await asyncOperation();
    callback(res);
}

const topDog = () => {
    let callback = (res) => {
        console.log(res);
    };

    (async () => {
        await asyncFunction(callback)
    })()
}

By applying this trick to an EventEmitter, you can get the same results. Define the EventEmitter's listener where I've defined the callback, and emit the event where I called the callback.

Solution 4

If function Fiber really turns async function sleep into sync

Yes. Inside the fiber, the function waits before logging ok. Fibers do not make async functions synchronous, but allow to write synchronous-looking code that uses async functions and then will run asynchronously inside a Fiber.

From time to time I find the need to encapsulate an async function into a sync function in order to avoid massive global re-factoring.

You cannot. It is impossible to make asynchronous code synchronous. You will need to anticipate that in your global code, and write it in async style from the beginning. Whether you wrap the global code in a fiber, use promises, promise generators, or simple callbacks depends on your preferences.

My objective is to minimize impact on the caller when data acquisition method is changed from sync to async

Both promises and fibers can do that.

Solution 5

Nowadays this generator pattern can be a solution in many situations.

Here an example of sequential console prompts in nodejs using async readline.question function:

var main = (function* () {

  // just import and initialize 'readline' in nodejs
  var r = require('readline')
  var rl = r.createInterface({input: process.stdin, output: process.stdout })

  // magic here, the callback is the iterator.next
  var answerA = yield rl.question('do you want this? ', r=>main.next(r))    

  // and again, in a sync fashion
  var answerB = yield rl.question('are you sure? ', r=>main.next(r))        

  // readline boilerplate
  rl.close()

  console.log(answerA, answerB)

})()  // <-- executed: iterator created from generator
main.next()     // kick off the iterator, 
                // runs until the first 'yield', including rightmost code
                // and waits until another main.next() happens

View more solutions

Comments

[abbr]

Promise only rephrases a callback parameter rather than turning the function into sync.

[actual_kangaroo]

yuu don't want it to be sync or your whole server will block! stackoverflow.com/questions/17959663/…

[abbr]

What's desirable is a sync call without blocking other events such as another request being handled by Node.js. A Sync function by definition only means it won't return to the caller until the result is produced (not merely a promise). It doesn't pre-exclude the server from handling other events while the call is blocked.

[abbr]

Thanks Bruno. But what if I need sync style in bootstrap code that needs to get executed before server binds to tcp port - such as config or data that have to be read from db which is opened async? I may ended up with wrapping entire server.js in Fiber, and I suspect that will kill concurrency at the entire process level. Nonetheless it's a suggestion that worth to verify. To me the ideal solution should be able to wrap an async function to provide a sync call syntax and only blocks next lines of code in the caller chain without sacrificing concurrency at process level.

[Bergi]

@fred: I think you're missing the point of promises. They're not simply an observer pattern abstraction, but they do provide a way to chain and compose asynchronous actions.

[Bruno Jouhier]

You can wrap your entire bootstrap code inside one big Fiber call. Concurrency should not be a problem because the bootstrap code usually needs to run to completion before you start serving requests. Also, a fiber does not prevent other fibers to run: every time you hit a yield call you give other fibers (and the main thread) a chance to run.

[abbr]

@Bergi, I use promise a lot and know exactly what it does. Effectively all it achieved is breaking down a single async function invocation into multiple invocations/statements. But it doesn't change the outcome - when the caller returns, it cannot return the result of async function. Check out the example I posted in JSFiddle. The caller in that case is function AnticipatedSyncFunction and async function is setTimeout. If you can answer my challenge using promise, please show me.

[abbr]

I have wrapped Express bootstrap file server.js with fiber. The execution sequence is what I am looking for, but that wrap doesn't have any effect on request handler. So I guess have to apply the same wrapper to EACH dispatcher. I gave up at this point because it doesn't seem to do any better to help avoid global re-factoring. My objective is to minimize impact on the caller when data acquisition method is changed from sync to async in DAO layer and Fiber still falls a little short to the challenge.

[Bergi]

@fred: Nothing can return the result of an async function.

[Bergi]

@fred: It doesn't make much sense to "synchronize" event streams like the request handler - you'd need to have a while(true) handleNextRequest() loop. Wrapping each request handler in a fiber would.

[abbr]

Then don't promise :). But thanks for the effort anyway.

[Bruno Jouhier]

@fred: fibers won't help you much with Express because Express' callback is not a continuation callback (a callback that is always called exactly once, either with an error or with a result). But fibers will solve the pyramid of doom when you have lots of code written on top of async APIs with continuation callbacks (like fs, mongodb, and lots of others).

[Bruno Jouhier]

@Bergi There are at least 3 technical approaches that make it possible to return from an async function: #1 fibers, #2 generators, #3 CPS transforms.

[abbr]

But can any approach address the challenge I gave in the first paragraph of my question? If so, it shouldn't be hard to give an answer by code to my JSFiddle example. So far I got none.

[Bruno Jouhier]

@fred There is nothing that can solve your challenge as stated, with all its constraints (although you did not completely rule out overriding setTimeout with a silly blocking call that pegs the CPU). But this does not mean that the solutions (CPS, fibers, generators) are worth nothing. They solve the problem with a little less constraints. Promises alone, on the other hand, don't relieve you of having a callback everwhere a value is returned.

[abbr]

@Bruno, I tend to believe there is no practical solution now. But I don't see it's theoretically impossible - sync and concurrency are not mutually exclusive, even in a single thread. A good C/C++ programmer should be able to come up with a npm package similar to Fibers to answer my challenge. Until then, the best mitigation I can think of is to adopt a framework that incorporates Fibers deeply at very early stage of the call stack. I believe Meteor is one of such framework and my problem can be addressed by Meteor._wrapAsync() function. Or better yet, include Fibers into Node.js core.

[abbr]

async works in your example b/c it's main, which doesn't care about caller. Imagine all your code is wrapped in a function which is supposed to return the result of one of your async function calls. It can be easily proofed not working by adding console.log('return'); at the end of your code. In such case the output of return will happen after in main but before step 1.

[Inanc Gumus]

@EruPenkman fibers will not block the server. they are async operations which are written as sync.

[Mark Amery]

@EruPenkman who said anything about a server? Node is usable for lots of things besides serverside code - in particular, lots of testing tools are written in it. The OP doesn't specify that he is writing code for a webserver.

[newman]

Anybody else had luck with this? I can't make it work.

[Alexander Mills]

I can't make it work properly. you should improve your documentation for this module, if you wish that it gets used more. I doubt the authors know exactly what the ramifications are for using the module, and if they do, they certainly don't document them.

[Alexander Mills]

this is the ABSOLUTE worst thing you can with Node.js: "synchronous-looking code that uses async functions and then will run asynchronously." if your API does that, you will ruin lives. if it is asynchronous, it should require a callback, and throw an error if no callback is provided. that is the best way to create an API, unless your goal is to trick people.

[Bergi]

@AlexMills: Yes, that would be horrible indeed. However, luckily this is nothing that an API can do. An asynchronous API always needs to accept a callback / return a promise / expect to be run inside a fiber - it doesn't work without. Afaik, fibers were mostly used in quick'n'dirty scripts that were blocking and don't have any concurrency, but want to use async APIs; just like in node there sometimes are cases where you'd use the synchronous fs methods.

[abbr]

So far there is one confirmed problem documented in github issue tracker. The problem has been fixed in Node v0.12. The rest I know of are merely groundless speculations that are not worth to document. If you believe your problem is caused by deasync, post a self-contained, duplicatable scenario and I will look into.

[tishma]

You don't want to answer to a question 'how?' by saying 'no!'.

[Alex]

I tried to use it and I get some improvements in my script but still I had no luck with the date. I modified the code as follow: function AnticipatedSyncFunction(){ var ret; setTimeout(function(){ var startdate = new Date() //console.log(startdate) ret = "hello" + startdate; },3000); while(ret === undefined) { require('deasync').runLoopOnce(); } return ret; } var output = AnticipatedSyncFunction(); var startdate = new Date() console.log(startdate) console.log("output="+output); and I expect to see 3 secs of different in the date output!

[Gandhi]

@abbr can this be browserified and used without node dependency>

[abbr]

@Gandhi, no. It depends on nodejs.

[Gandhi]

@abbr thanks for the response. Is there any way to achieve synchronous function call using vanilla JavaScript?

[Kris]

I generally like node. Especially if I can use typescript instead of pure js. But this whole async nonsense that permeates everything you do and literally infects every function in the call chain as soon as you decide to make a single async call is something I really... really hate. Async api is like an infectious disease, one call infects your entire code base forcing you to rewrite all the code you have. I really don't understand how anyone can possibly argue this is a good thing.

[Bergi]

@Kris Node uses an asynchronous model for IO tasks because it's fast and simple. You can do many things synchronously as well, but blocking is slow as you can't do anything concurrently - unless you go for threads, which make everything complicated.

[Kris]

@Bergi I read the manifesto so I know the arguments. But changing your existing code into async the moment you hit that first api call that has no sync equivalent is not simple. Everything breaks and every single line of code has to be scrutinised. Unless your code is trivial I guarantee... it will take a while to convert and get it working again after converting the whole thing to async idiom.

[user1742529]

@Alex read carefully developer.mozilla.org/en-US/docs/Web/JavaScript/EventLoop before expects waiting of exactly 3000 ms from setTimeout combined with js engine hacks. :-)

[Andrew]

This still ends up running a callback. I'm in a huge sync framework. I need to make a single call to aws sts get-caller-identity. I'd like to use the aws-sdk framework to do that... how do I write a function which encapsulates all the async stuff? For the life of me, I can't figure it out. :)