How to schedule a task in asyncio so it runs at a certain date?

Question

My program is supposed to run 24/7 and i want to be able to run some tasks at a certain hour/date.

I have already tried to work with aiocron but it only supports scheduling functions (not coroutines) and i have read that is not a really good library. My program is built so most if not all the tasks that i would want to schedule are built in coroutines.

Is there any other library that allows for such kind of task scheduling?

Or if not, any way of warping coroutines so they run of a normal function?

Solution

I have already tried to work with aiocron but it only supports scheduling functions (not coroutines)

According to the examples at the link you provided, that does not appear to be the case. The functions decorated with @asyncio.coroutine are equivalent to coroutines defined with async def, and you can use them interchangeably.

However, if you want to avoid aiocron, it is straightforward to use asyncio.sleep to postpone running a coroutine until an arbitrary point in time. For example:

import asyncio, datetime

async def wait_until(dt):
    # sleep until the specified datetime
    now = datetime.datetime.now()
    await asyncio.sleep((dt - now).total_seconds())

async def run_at(dt, coro):
    await wait_until(dt)
    return await coro

Example usage:

async def hello():
    print('hello')

loop = asyncio.get_event_loop()
# print hello ten years after this answer was written
loop.create_task(run_at(datetime.datetime(2028, 7, 11, 23, 36),
                        hello()))
loop.run_forever()

---

Note: Python versions before 3.8 didn't support sleeping intervals longer than 24 days, so wait_until had to work around the limitation. The original version of this answer defined it like this:

async def wait_until(dt):
    # sleep until the specified datetime
    while True:
        now = datetime.datetime.now()
        remaining = (dt - now).total_seconds()
        if remaining < 86400:
            break
        # pre-3.7.1 asyncio doesn't like long sleeps, so don't sleep
        # for more than one day at a time
        await asyncio.sleep(86400)
    await asyncio.sleep(remaining)

The limitation was removed in Python 3.8 and the fix was backported to 3.6.7 and 3.7.1.

Comments

[Vincent]

Why not use loop.call_at?

[user4815162342]

@Vincent loop.call_at accepts a time reference relative to the time returned by loop.time(), so you still have to calculate and cannot just give it an e.g. Unix timestamp. The second problem is that call_at accepts a function, not a coroutine, so you'd still need a trampoline that calls create_task. Using call_at (and call_later and call_soon) you cannot easily obtain the return value of the coroutine, as done in the edited answer.

[user4815162342]

The final issue is that I believe call_at would have the same problem with long sleeps that asyncio.sleep has, because the error comes from the underlying poll. With call_at it would be even harder to work around because you cannot give it absolute time too far in the future, so the code would have to revert to call_later with similar logic to what's done in the answer.

[Vincent]

Thanks for the info. I always assumed the loop to use time.time(), but time.monotonic() makes a lot more sense since you don't want asyncio to break after a change in the system time-of-day clock. Still, I don't see why long sleeps would be a problem, do you have some references to share?

[user4815162342]

@Vincent I discovered the timeout error while testing the code by scheduling an event a month from now. Looking at the source, the limitation seems to be that the nearest timeout in milliseconds must not exceed 2**31-1, which means that you cannot sleep for more than 24.8 days without a workaround like the loop in the answer. For most practical uses of asyncio this is not an issue, but for a scheduler that runs tasks at particular dates it must be worked around.

[Vincent]

Interesting, I found this related issue (bpo-20423) on the python bug tracker. Another way to work around this issue is to schedule a background task that wakes up every day.

[user4815162342]

@Vincent That is close to what the code in my answer does, except handles the remainder of the sleep precisely. This allows it to handle arbitrary points in the future. It would be nice if asyncio handled this automatically, but at this point it doesn't.

[Syranolic]

in the doc I see this: Changed in version 3.8: In Python 3.7 and earlier timeouts (relative delay or absolute when) should not exceed one day. This has been fixed in Python 3.8. So we can get back to sleep.

[user4815162342]

@Syranolic Indeed they did, and the fix is (on their side) nothing short of trivial. They even backported it to 3.7.1 and 3.6.x - I've now amended the answer to mention it.

[Syranolic]

@user4815162342 wait what? their "fix" is to just clamp the delay? I suppose I still need a loop with multiple sleeps, then... It is literally the kind of issue you don't have time to test for :)

[user4815162342]

@Syranolic Don't worry, asyncio devs are not that incompetent. :) I shouldn't have called the fix trivial, but something along the lines of deceptively simple.. The event loop waits for IO events and the earliest timeout at the same time. When select returns, the loop doesn't assume that it has slept up to the earliest timeout, it checks the current time to see which timeouts are ready to fire. (It could have slept less due to IO or spurious wakeups, but it could also have slept longer due to the CPU being busy with other work.) See the while self._scheduled loop under the fixed line.

[user4815162342]

So sleeping less is never a problem, those timeouts simply won't fire and will be left for the next wait. In other words, the loop is right there, in the event loop. (puts on Horatio glasses)

[Ali Husham]

is it safe to use this instead of celery with django?

[Philip Couling]

Just note that this may not be accurate in the way you expect. I believe asyncio.sleep() is based on the monotonic clock. The monotonic clock isn't always adjusted when the system updates it's time (eg: via NTP). This means it's prone to drift away from any calculation based on the UTC time if/when the system clock is "stepped". I don't think this is solvable. But it's worth noting that setting a time days in the future might not precisely hit the time you expected.

[user4815162342]

@PhilipCouling While you're technically right, I wonder if this is a problem in practice. NTP adjustments are typically in milliseconds, and even the multi-month sleeps will still be accurate to within a fraction of a second. Although it will not "precisely" hit the expected time, it will be very close, at least when compared to the length of the sleep interval.

[Philip Couling]

@user4815162342 Depends entirely on the system. If the (S)NTP client is prone to stepping the clock and not slewing it, then the drift between the monotonic and system clock will be the same as the system clock drift against UTC before NTP adjustment. I've had that be 2 minutes per day on some PCs. If you are using NTP not SNTP then slewing the clock is more likely. However many Linux distros are shipping with SNTP by default these days.

[Philip Couling]

@user4815162342 ... and then I've seen home WiFi routers which, when reset, manage reset almost every system clock in the building back to the year 2017 for a few minutes. Oh the fun of hard-coded DHCP settings with badly written NTP relays! I made my first comment because I've had too many hours debugging funky system clocks screwing with my timing loops.