How to not await in a loop with asyncio?

Question

Here is a toy example that downloads the home page from several websites using asyncio and aiohttp:

import asyncio
import aiohttp

sites = [
    "http://google.com",
    "http://reddit.com",
    "http://wikipedia.com",
    "http://afpy.org",
    "http://httpbin.org",
    "http://stackoverflow.com",
    "http://reddit.com"
]


async def main(sites):
    for site in sites:
        download(site)


async def download(site):
    response = await client.get(site)
    content = await response.read()
    print(site, len(content))


loop = asyncio.get_event_loop()
client = aiohttp.ClientSession(loop=loop)
content = loop.run_until_complete(main(sites))
client.close()

If I run it, I get:

RuntimeWarning: coroutine 'download' was never awaited

But I don't want to await it.

In twisted I can do:

for site in sites:
    download(site)

And If I don't explicitly "yield" or add a callback to the returned Deferred, it just runs without blocking nor complaining. I can't access the result, but in this case I don't need it.

In JS I can do:

site.forEarch(site){
    donwload(site)
}

And again, it doesn't block nor does it requires anything from my part.

I found a way to do:

async def main(sites):
    await asyncio.wait([download(site) for site in sites])

But:

• this is really not obvious to find it out. I it's hard to remember.
• it's hard to understand what it does. "waits" seems to say "i block", but does not convey clearly it block for the entire list of coroutine to finish.
• you can't pass in a generator, it needs to be a real list, which i feels really unatural in Python.
• what if I have only ONE awaitable ?
• what if I don't want to wait at all on my tasks, and just schedule them for execution then carry on with the rest of my code ?
• it's way more verbose thant twisted and JS solution.

It there a better way ?

Solution 1

• this is really not obvious to find it out. I it's hard to remember.

The documentation on coroutines does make it pretty clear what asyncio.wait's purpose is.

• it's hard to understand what it does. "waits" seems to say "i block", but does not convey clearly it block for the entire list of coroutine to finish.

Again, see the documentation.

• you can't pass in a generator, it needs to be a real list, which i feels really unatural in Python.

Again, see the documentation, specifically asyncio.as_completed

• what if I have only ONE awaitable ?

It should still work.

• what if I don't want to wait at all on my tasks, and just schedule them for execution then carry on with the rest of my code ?

Then you can use asyncio.ensure_furture. In fact, asyncio.wait is a convenience function around asyncio.ensure_future (and some other logic).

• it's way more verbose thant twisted and JS solution.

Maybe, but that's not a bad thing (from my perspective).

Solution 2

In order to schedule a coroutine as a task, use asyncio.ensure_future:

for site in sites:
    coro = download(site)
    future = asyncio.ensure_future(coro)

It replaces the deprecated function asyncio.async in version 3.4.4.

Then you can manage those futures using await, asyncio.wait or asyncio.gather.

Comments

[e-satis]

Accepted and upvoted this it does solve my problem. However : the docs are not that obvious (even Guido aknowledge that). And having to go to the doc for such a simple problem is not a sign of good ergonomics. What's more, being more verbose than Twisted and JS, 2 very verbose tech is definitly not a badge of honor. Espcially since rigth now you can run only one eventloop, so not using the default event loop automatically seems overkill. But thanks, it's way more obvious now.

[Jashandeep Sohi]

Fair enough; it's probably obvious to me because I've been using it. I can definitely understand people getting lost in the docs. There are so many sections. The asyncio docs need a tutorial like quick guide, like many of the other modules' doc have.

[Labo]

Nowadays, [create_task}(docs.python.org/3/library/asyncio-task.html#as‌​yncio.create_task) is recommended instead.