asyncio.run() cannot be called from a running event loop

Question

I would like to use asyncio to get webpage html.

I run the following code in jupyter notebook:

import aiofiles
import aiohttp
from aiohttp import ClientSession
async def get_info(url, session):
    resp = await session.request(method="GET", url=url)
    resp.raise_for_status()
    html = await resp.text(encoding='GB18030')
    with open('test_asyncio.html', 'w', encoding='utf-8-sig') as f:
        f.write(html)
    return html
async def main(urls):
    async with ClientSession() as session:
        tasks = [get_info(url, session) for url in urls]
        return await asyncio.gather(*tasks)
if __name__ == "__main__":
    url = ['http://huanyuntianxiazh.fang.com/house/1010123799/housedetail.htm', 'http://zhaoshangyonghefu010.fang.com/house/1010126863/housedetail.htm']
    result = asyncio.run(main(url))

However, it returns RuntimeError: asyncio.run() cannot be called from a running event loop

What is the problem?

How to solve it?

Solution 1

The asyncio.run() documentation says:

This function cannot be called when another asyncio event loop is running in the same thread.

In your case, jupyter (IPython ≥ 7.0) is already running an event loop:

You can now use async/await at the top level in the IPython terminal and in the notebook, it should — in most of the cases — “just work”. Update IPython to version 7+, IPykernel to version 5+, and you’re off to the races.

Therefore you don't need to start the event loop yourself and can instead call await main(url) directly, even if your code lies outside any asynchronous function.

Jupyter / IPython

async def main():
    print(1)
await main()

Python (≥ 3.7)

import asyncio
async def main():
    print(1)
asyncio.run(main())

In your code that would give:

url = ['url1', 'url2']
result = await main(url)
for text in result:
    pass # text contains your html (text) response

Caution

There is a slight difference on how Jupyter uses the loop compared to IPython.

Solution 2

To add to cglacet's answer - if one wants to detect whether a loop is running and adjust automatically (ie run main() on the existing loop, otherwise asyncio.run()), here is a snippet that may prove useful:

# async def main():
#     ...
try:
    loop = asyncio.get_running_loop()
except RuntimeError:  # 'RuntimeError: There is no current event loop...'
    loop = None
if loop and loop.is_running():
    print('Async event loop already running. Adding coroutine to the event loop.')
    tsk = loop.create_task(main())
    # ^-- https://docs.python.org/3/library/asyncio-task.html#task-object
    # Optionally, a callback function can be executed when the coroutine completes
    tsk.add_done_callback(
        lambda t: print(f'Task done with result={t.result()}  << return val of main()'))
else:
    print('Starting new event loop')
    asyncio.run(main())

Solution 3

Just use this:

https://github.com/erdewit/nest_asyncio

import nest_asyncio
nest_asyncio.apply()

Solution 4

Combining the methods from Pankaj Sharma and Jean Monet, I wrote the following snippet that acts as asyncio.run (with slightly different syntax), but also works within a Jupyter notebook.

class RunThread(threading.Thread):
    def __init__(self, func, args, kwargs):
        self.func = func
        self.args = args
        self.kwargs = kwargs
        super().__init__()
    def run(self):
        self.result = asyncio.run(self.func(*self.args, **self.kwargs))
def run_async(func, *args, **kwargs):
    try:
        loop = asyncio.get_running_loop()
    except RuntimeError:
        loop = None
    if loop and loop.is_running():
        thread = RunThread(func, args, kwargs)
        thread.start()
        thread.join()
        return thread.result
    else:
        return asyncio.run(func(*args, **kwargs))

Usage:

async def test(name):
    await asyncio.sleep(5)
    return f"hello {name}"
run_async(test, "user")  # blocks for 5 seconds and returns "hello user"

Solution 5

I found the unsync package useful for writing code that behaves the same way in a Python script and the Jupyter REPL.

import asyncio
from unsync import unsync
@unsync
async def demo_async_fn():
    await asyncio.sleep(0.1)
    return "done!"
print(demo_async_fn().result())

View more solutions

Comments

[Chan]

Thank you, cglacet. However, there is a warning : c:\program files\python37\lib\site-packages\ipykernel_launcher.py:29: RuntimeWarning: coroutine 'main' was never awaited

[cglacet]

That's probably because you called main(url) instead of await main(url).

[Foad S. Farimani]

Is there a way to have a code snippet that works both inside and outside of Jupyter?

[cglacet]

I don't know if it's possible, but I'm not sure you would really want that anyway. Maybe ask this as a separate question and see if people have ideas on whether it's possible or desirable.

[cglacet]

Nice, indeed adding a callback here is probably the only way of getting the result back. That would probably be interesting to have this code in a function so you would only have to call something like run(main(*args, **kwargs)). The function could also make sure you get your output back, not sure that's really simple tho (because of the callback).

[Jean Monet]

After some experimeting, it would seem that the only way to handle results is within the async code (if needed by further nesting and structuring async functions and callbacks). Any attempt to await on the result outside the async functions (= in synchronous code) would block the whole thing (if it is running in the same process).

[Luca]

Interestingly, when I run the above in jupyter, I get: SyntaxError: 'await' outside function

[Gregor Müllegger]

The docs for nest_asyncio mention a issue report with asyncio where its explicitly stated that this is no intended behaviour for asyncio. So I would consider nest_asyncio a big fat hack which I wouldn't trust with in my codebase to not break everything with a new Python version.

[mknote]

Another issue is that nest_asyncio requires Python 3.5 or above, which is not helpful to me as I am stuck on Python 2.7.

[cglacet]

Maybe try to upgrade jupyter, older versions do not work the same way.

[Alex S.]

this allowed me to solve the issue for the imported code (where you can't change it yourself)

[Hakan]

thank you for this clever solution. Do you have any suggestion for "asyncio attached to a different loop" error? Inside child loops i need to use 2 coroutines from main loop.

[Mauro Abrantes]

@Luca i used the wait it and it worked for me ( in a very simple program to just sign in and receive a token from telethon )

[Mika C.]

This worked so well fixing a imported asyncio issue, thanks alot