Using asyncio for Non-async Functions in Python?

Question

Suppose there is a library that makes various database queries:

import time

def queryFoo():
    time.sleep(4)
    return "foo"

def queryBar():
    time.sleep(4)
    return "bar"

I want to execute those 2 queries concurrently without having to add async to the method signature or adding a decorator. These functions should not depend on asyncio at all.

What is the best way to utilize those non-async functions within asyncio?

I am looking for something of the form:

#I need an 'asyncWrapper'

results = asyncio.gather(asyncWrapper(queryFoo()), asyncWrapper(queryBar()))

Thank you in advance for your consideration and response.

Solution 1

If some function is blocking and not async by nature, only proper way to run it inside asyncio event loop is to run it inside thread using run_in_executor:

# Our example blocking functions
import time


def queryFoo():
    time.sleep(3)
    return 'foo'


def queryBar():
    time.sleep(3)
    return 'bar'


# Run them using asyncio
import asyncio
from concurrent.futures import ThreadPoolExecutor


_executor = ThreadPoolExecutor(10)


async def in_thread(func):
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(_executor, func)


async def main():
    results = await asyncio.gather(
        in_thread(queryFoo), 
        in_thread(queryBar),
    )

    print(results)


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(main())
    finally:
        loop.run_until_complete(loop.shutdown_asyncgens())
        loop.close()

It does job.

If you however want to avoid using threads only way to do it - is to rewrite queryFoo/queryBar to be async by nature.

Solution 2

I presume you are after concurrency and hopefully do not insist on using asyncio module itself in which case this little example could be helpful:

import asyncio
import time
from concurrent.futures import ThreadPoolExecutor

def queryFoo():
    time.sleep(2)
    return "FOO"

def queryBar():
    time.sleep(4)
    return "BAR"

with ThreadPoolExecutor(max_workers=2) as executor:
    foo = executor.submit(queryFoo)
    bar = executor.submit(queryBar)
    results = [foo.result(), bar.result()]

print(results)

It runs both queryFoo() and queryBar() in parallel and collects their results in a list in order in which they've been mentioned in an assignment to results.

Comments

[Emerson Xu]

Coming from here, the example you showed above will still need async function for run_in_executor(), while I want to make sure the async function converted to sync such that I don't have nested async calls that have nested event loops. One of the solutions I can think of is to spawn new event loop for the inner async calls and block until its result got returned, and continue to next outer loops. Will that work?

[Mikhail Gerasimov]

@EmersonXu 1) if you want to cast some async function to sync, you just have to do loop.run_until_complete(asyncfunc()) inside sync funtions. But once you did it, you've done with async world at all: from now on all your outer code is sync. 2) If you want to cast some sync function to async, use await loop.run_in_executor(_executor, syncfunc) inside other async functions. 3) If you want to do something else, I'm afraid it's hard to me to understand it without reproducible code example.

[Gabriel G.]

Looks like he updated the question to say he does need it to be an "asyncio wrapper".