How to run asynchronous tasks in Python GObject Introspection apps
Solution 1
Your problem is a very common one, therefore there are tons of solutions (sheds, queues with multiprocessing or threading, worker pools, ...)
Since it is so common, there is also a python build-in solution (in 3.2, but backported here: http://pypi.python.org/pypi/futures) called concurrent.futures. 'Futures' are available in many languages, therefore python calls them the same. Here are the typical calls (and here is your full example, however, the db part is replaced by sleep, see below why).
from concurrent import futures
executor = futures.ProcessPoolExecutor(max_workers=1)
#executor = futures.ThreadPoolExecutor(max_workers=1)
future = executor.submit(slow_load)
future.add_done_callback(self.on_complete)
Now to your problem, which is much more complicated than your simple example suggests. In general you have threads or processes to solve this, but here is why your example is so complicated:
- Most Python implementations have a GIL, which makes threads not fully utilize multicores. So: do not use threads with python!
- The objects you want to return in
slow_load
from the DB are not pickelable, which means that they can not simply be passed between processes. So: no multiprocessing with softwarecenter results! - The library you call (softwarecenter.db) is not threadsafe (seems to include gtk or similar), therefore calling these methods in a thread results in strange behaviour (in my test, everything from 'it works' over 'core dump' to simple quitting without results). So: no threads with softwarecenter.
- Every asynchronous callback in gtk should not do anything except sheduling a callback which will be called in the glib mainloop. So: no
print
, no gtk state changes, except adding a callback! - Gtk and alike does not work with threads out of the box. You need to do
threads_init
, and if you call a gtk or alike method, you have to protect that method (in earlier versions this wasgtk.gdk.threads_enter()
,gtk.gdk.threads_leave()
. see for example gstreamer: http://pygstdocs.berlios.de/pygst-tutorial/playbin.html).
I can give you the following suggestion:
- Rewrite your
slow_load
to return pickelable results and use futures with processes. - Switch from softwarecenter to python-apt or similar (you probably don't like that). But since your employed by Canonical, you could ask the softwarecenter developers directly to add documention to their software (e.g. stating that it is not thread safe) and even better, making softwarecenter threadsafe.
As a note: the solutions given by the others (Gio.io_scheduler_push_job
, async_call
) do work with time.sleep
but not with softwarecenter.db
. This is, because it all boils down to threads or processes and threads to not work with gtk and softwarecenter
.
Solution 2
Here's another option using GIO's I/O Scheduler (I've never used it before from Python, but the example below seems to run fine).
from gi.repository import GLib, Gio, GObject
import time
def slow_stuff(job, cancellable, user_data):
print "Slow!"
for i in xrange(5):
print "doing slow stuff..."
time.sleep(0.5)
print "finished doing slow stuff!"
return False # job completed
def main():
GObject.threads_init()
print "Starting..."
Gio.io_scheduler_push_job(slow_stuff, None, GLib.PRIORITY_DEFAULT, None)
print "It's running async..."
GLib.idle_add(ui_stuff)
GLib.MainLoop().run()
def ui_stuff():
print "This is the UI doing stuff..."
time.sleep(1)
return True
if __name__ == '__main__':
main()
Solution 3
Use the introspected Gio
API to read a file, with its asynchronous methods, and when making the initial call, do it as a timeout with GLib.timeout_add_seconds(3, call_the_gio_stuff)
where call_the_gio_stuff
is a function which returns False
.
The timeout here is necessary to add (a different number of seconds may be required, though), because while the Gio async calls are asynchronous, they are not non-blocking, meaning that the heavy disk activity of reading a large file, or large number of files, can result in blocked UI, as the UI and I/O are still in the same (main) thread.
If you want to write your own functions to be async, and integrate with the main loop, using Python's file I/O APIs, you'll have to write the code as a GObject, or to pass callbacks around, or use python-defer
to help you do it. But it's best to use Gio here, as it can bring you a lot of nice features, especially if you're doing file open/save stuff in the UX.
Solution 4
You can also use GLib.idle_add(callback) to call the long running task once the GLib Mainloop finishes all it's higher priority events (which I believe includes building the UI).
Solution 5
I think it bears noting that this is a convoluted way to do what @mhall suggested.
Essentially, you've got a run this then run that function of async_call.
If you want to see how it works, you can play with the sleep timer and keep clicking the button. It's essentially the same as @mhall's answer except that there's example code.
Based on this which is not my work.
import threading
import time
from gi.repository import Gtk, GObject
# calls f on another thread
def async_call(f, on_done):
if not on_done:
on_done = lambda r, e: None
def do_call():
result = None
error = None
try:
result = f()
except Exception, err:
error = err
GObject.idle_add(lambda: on_done(result, error))
thread = threading.Thread(target = do_call)
thread.start()
class SlowLoad(Gtk.Window):
def __init__(self):
Gtk.Window.__init__(self, title="Hello World")
GObject.threads_init()
self.connect("delete-event", Gtk.main_quit)
self.button = Gtk.Button(label="Click Here")
self.button.connect("clicked", self.on_button_clicked)
self.add(self.button)
self.file_contents = 'Slow load pending'
async_call(self.slow_load, self.slow_complete)
def on_button_clicked(self, widget):
print self.file_contents
def slow_complete(self, results, errors):
'''
'''
self.file_contents = results
self.button.set_label(self.file_contents)
self.button.show_all()
def slow_load(self):
'''
'''
time.sleep(5)
self.file_contents = "Slow load in progress..."
time.sleep(5)
return 'Slow load complete'
if __name__ == '__main__':
win = SlowLoad()
win.show_all()
#time.sleep(10)
Gtk.main()
Additional note, you have to let the other thread finish before it will terminate properly or check for a file.lock in your child thread.
Edit to address comment:
Initially I forgot GObject.threads_init()
. Evidently when the button fired, it initialized threading for me. This masked the mistake for me.
Generally the flow is create the window in memory, immediately launch the other thread, when the thread completes update the button. I added an additional sleep before I even called Gtk.main to verify that the full update COULD run before the window was even drawn. I also commented it out to verify that the thread launch doesn't impede window drawing at all.
Related videos on Youtube
David Planella
I work at GitLab as Director of Community Relations. Before, I worked for Canonical as the former Ubuntu Community Team Manager. As an Open Source contributor, I am mostly involved in app development and localization: I'm the developer of Qreator, former lead the Ubuntu Catalan Translators team and also a GNOME translator. In the past I've contributed to other projects, such as Debian or Mozilla. Blog Google+ Twitter
Updated on September 18, 2022Comments
-
David Planella over 1 year
I'm writing a Python + GObject app that needs to read a non-trivial amount of data from disk upon start. The data is read synchronously and it takes about 10 seconds to finish the read operation, during which time the loading of the UI is delayed.
I'd like to run the task asynchronously, and get a notification when it's ready, without blocking the UI, more or less like:
def take_ages(): read_a_huge_file_from_disk() def on_finished_long_task(): print "Finished!" run_long_task(task=take_ages, callback=on_finished_long_task) load_the_UI_without_blocking_on_long_task()
I've used GTask in the past for this sort of thing, but I'm concerned that its code hasn't been touched in 3 years, let alone been ported to GObject Introspection. Most importantly, it's no longer available in Ubuntu 12.04. So I'm looking for an easy way to run tasks asynchronously, either in a standard Python way or in a GObject/GTK+ standard way.
Edit: here's some code with an example of what I'm trying to do. I've tried
python-defer
as suggested in the comments, but I could not manage to run the long task asynchronously and let the UI load without having to wait for it to finish. Browse the test code.Is there an easy and widely used way of running asynchronous tasks and get notified when they're finished?
-
RobotHumans almost 12 yearsIt's not a pretty example, but I'm pretty sure this is what you're looking for: raw.github.com/gist/1132418/…
-
David Planella almost 12 yearsCool, I think your
async_call
function might be what I need. Would you mind expanding on it a bit and adding an answer, so that I can accept it and credit you after I test it? Thanks! -
Rafał Cieślak almost 12 yearsGreat question, very useful! ;-)
-
-
David Planella almost 12 yearsThanks @dobey. I'm not actually reading a file from disk directly, I should have probably made that clearer in the original post. The long-running task that I'm running is reading the Software Center database as per the answer to askubuntu.com/questions/139032/…, so I'm not sure I can use the
Gio
API. What I was wondering is whether there is a way to run any generic long-running task asynchronously in the same way GTask used to do. -
David Planella almost 12 yearsThanks Mike. Yes, that would definitely help with starting the task when the UI is ready. But on the other hand, I understand that when
callback
is called, that would be done synchronously, thus blocking the UI, right? -
dobey almost 12 yearsI don't know what GTask is exactly, but if you mean gtask.sourceforge.net then I don't think you should be using that. If it's something else, then I don't know what it is. But it sounds like you will have to take the second route I mentioned, and implement some asynchronous API to wrap that code, or just do it all in a thread.
-
dobey almost 12 yearsThe idle_add doesn't quite work like that. Making blocking calls in an idle_add is still a bad thing to do, and it will prevent updates to the UI from happening. And even asynchronous API can still be blocking, where the only way to avoid blocking the UI and other tasks, is to do it in a background thread.
-
David Planella almost 12 yearsThere's a link to it in the question. GTask is (was): chergert.github.com/gtask
-
David Planella almost 12 yearsThanks. I'm not sure I can follow it. For one, I would have expected
slow_load
to be executed shortly after the UI starts, but it never seems to be called, unless the button is clicked, which confuses me a bit, as I thought the purpose of the button was just to provide visual indication of the state of the task. -
RobotHumans almost 12 yearsSorry, I missed one line. That did it. I forgot to tell GObject to get ready for threads.
-
Siegfried Gevatter almost 12 yearsIdeally you'd split your slow task into chunks, so you can run a bit of it in an idle callback, return (and let other stuff like UI callbacks run), continue doing some more work once your callback is called again, and so on.
-
Siegfried Gevatter almost 12 yearsSee also GIO.io_scheduler_job_send_to_mainloop(), if you want to run something in the main thread once slow_stuff finishes.
-
dobey almost 12 yearsBut you're calling into the main loop from a thread, which can cause problems, though they may not be easily exposed in your trivial example which doesn't do any real work.
-
RobotHumans almost 12 yearsValid point, but I didn't think a trivial example merited sending the notification via DBus(which I think a non-trivial app should be doing)
-
dobey almost 12 yearsAh, that looks very similar to the API provided by python-defer (and twisted's deferred API). Perhaps you should look at using python-defer?
-
David Planella almost 12 yearsI had a look at it when you first mentioned it, but I'm afraid I might be to thick to use it... I probably need to paste complete code, but the first try using it in this fragment lead to
slow_load
being aparently executed synchronously and delaying the start of the UI.deferred = defer.defer(self.slow_load) deferred.add_callback(self.on_complete)
-
David Planella almost 12 yearsHere's some very simple code I've used to test
python-defer
, but as I say, theslow_load
function seems to be called synchronously followed by the callback before the UI is loaded: bazaar.launchpad.net/~dpm/+junk/slowdefer/view/head:/… -
David Planella almost 12 yearsHm, running
async_call
in this example works for me, but it brings mayhem when I port it to my app and I add the realslow_load
function I've got. -
David Planella almost 12 yearsThanks! I'm going to accept your answer as it points me in very much detail to why it is not doable. Unfortunately, I cannot use software that is not packaged for Ubuntu 12.04 in my app (it is for Quantal, though launchpad.net/ubuntu/+source/python-concurrent.futures), so I guess I'm stuck with not being able to run my task asynchronously. Regarding the note to talk to the Software Center developers, I'm in the same position as any volunteer to contribute changes to the code and documentation or to talk to them :-)
-
David Planella almost 12 yearsThanks Sigfried for the answer and the example. Unfortunately, it seems that with my current task I have no chance to use the Gio API to make it run asynchronously.
-
dobey almost 12 yearsYou still need to delay that being called, until after the main priority events have happened, by using GLib.idle_add() for example. Like this: pastebin.ubuntu.com/1011660
-
sil almost 12 yearsThis was really useful, but as far as I can tell Gio.io_scheduler_job_send_to_mainloop doesn't exist in Python :(
-
jfs over 11 yearsGIL is released during IO so it is perfectly fine to use threads. Though it is not necessary if async IO is used.
-
Flimm about 11 yearsA gotcha with
idle_add
is that the return value of the callback matters. If it's true, it will be called again.