better way to get results from multiple threads

Question

what i want to do is be able to call a function with multiple threads and get their results.

i have the following code:

(it is an example, the actual code doesn't simply convert str to int)

from threading import Thread
import time
import Queue

#an example - actual code connects to a server
def str_to_int(arg, queue):
    result = 0
    result = int(arg)
    #sleep to check that they happen at once.
    time.sleep(10)
    queue.put(result)

def combine():
    q1 = Queue.Queue()
    q2 = Queue.Queue()
    q3 = Queue.Queue()

    t1 = Thread(target = str_to_int, args=("111", q1))
    t2 = Thread(target = str_to_int, args=("222", q2))
    t3 = Thread(target = str_to_int, args=("333", q3))

    t1.start()
    t2.start()
    t3.start()

    t1.join()
    t2.join()
    t3.join()

    return (q1.get(),q2.get(),q3.get())

print combine()

this code works. and i get the expected results:

>>> 
(111, 222, 333)

however, there must be a better way to do this. i plan on having many more threads than 3, but even if i was only staying with 3 - it seems very ugly.

EDIT: i need to be able to know which result came from which thread (ie: from which parameters/arguments that i gave the function)

Solution

Here are some advices:

1. Queue's are thread-safe, so use 1 queue to pass results.
2. You can create all threads in a cycle and use your queue to pass results. You don't need to have explicit variable for each thread.

So here's what your code might look like:

def str_to_int(arg, queue):
    result = int(arg)
    queue.put({arg: result})

def combine():
    arguments = ('111', '222', '333')
    q = Queue.Queue()
    threads = []

    for argument in arguments:
        t = Thread(target=str_to_int, args=(argument, q))
        t.start()
        threads.append(t)

    for t in threads:
        t.join()

    return [q.get() for _ in xrange(len(arguments))]

Comments

[Austin Phillips]

You may want to move the t.join()'s into their own loop, otherwise the threads will run serially.

[Inbar Rose]

this does not work. the threads are running after eachother not simultaneously, as @AustinPhillips said.

[Rostyslav Dzinko]

Yeah, my mistake, threads has to be joined in a separate loop, because join causes main thread blocking.

[Inbar Rose]

hmm - sorry i marked it as accepted, but then i realized this causes a problem for me. how to i now get the different results and be able to know which one i am working on?

[Rostyslav Dzinko]

You can put not only result into queue, but something that will point to arguments, e.g. queue.put({argument: resut}) in your worker function. I've updated post with modified str_to_int function

[Inbar Rose]

so then i could read all of the q.get() into a dictionary, and then i could access the results for each function by the argument?

[Rostyslav Dzinko]

It's a bad idea to read all results:values into one dictionary unless you know that your arguments are repeating, but in general yes, you can read all queue and build a dictionary of it

[Austin Phillips]

You could pass a unique thread number as part of the args when invoking the thread, which would be posted to the result queue. Once you have the results you can order them according to thread number of you need to.