Python queue block timeout does not timeout - any idea why?

Question

I expect the following python code will print "Timeout:" in the console output.

It has a thread that produce objects. The consumer thread will get the queued objects and print it out.

The expected Queue Get() timeout is not happening. Any idea on why?

The output is: (No expected "Timeout: " printout.)

1390521788.42  Outputting: o={'test': 2, 'sName': 't1'} 
1390521791.42  Outputting: o={'test': 3, 'sName': 't1'}
1390521794.42  Outputting: o={'test': 4, 'sName': 't1'}
1390521797.42  Outputting: o={'test': 5, 'sName': 't1'}
end while sName=t1 

This is tested with Python 2.7 in Linux.

import threading, Queue, time

class ProduceThread(threading.Thread):
    def __init__ (self, start_num, end, q, sName, nSleep=1):
        self.num = start_num
        self.q = q
        threading.Thread.__init__ (self)
        self.m_end = end;
        self.m_sName = sName;
        self.m_nSleep = nSleep;

    def run(self):
        o = {};
        o['sName'] = self.m_sName;
        while True:
            if self.num != self.m_end:
                self.num += 1
                o['test'] = self.num;
                # self.q.put(self.num)
                self.q.put(o)
                time.sleep(self.m_nSleep)
            else:
                break
        print "end while sName=%s" % (self.m_sName);

myQueue = Queue.Queue()
myThread = ProduceThread(1, 5, myQueue, 't1', 3); myThread.start()
# myThread2 = ProduceThread(1, 5, myQueue, 't2', 3); myThread2.start()
# myThread3 = ProduceThread(1, 5, myQueue, 't3', 3); myThread3.start()

def Log(s):
    t = time.time();
    print "%s  %s" %(t, s)

################################################################
#  Consumer Loop
while True:
    if not myQueue.empty():
        try:
            o = myQueue.get(block=True, timeout=1)
            Log( "Outputting: o=%s" % (o));
        except:
            ###### I expect the Timeout to happen here. But it is not.
            Log( "Timeout: " );
            pass;
    # time.sleep(1)

Solution

Well, think about this:

if not myQueue.empty():
    try:
        o = myQueue.get(block=True, timeout=2)
        Log( "Outputting: o=%s" % (o));

Leave aside that you should never rely on the Queue.empty() method. See the docs:

If empty() returns True it doesn’t guarantee that a subsequent call to put() will not block. Similarly, if empty() returns False it doesn’t guarantee that a subsequent call to get() will not block.

However, in a context this simple, it's "pretty reliable" ;-) Now how could your timeout possibly occur? If and only if the .get() attempt is made when your queue is empty. But you never execute your .get() when your queue is empty, because of your:

if not myQueue.empty():

test! In effect, you're asking this:

I only try to do .get() when I'm sure something is on the queue. So I'm sure .get() will succeed immediatey. So why doesn't it ever time out?

Remove the

if not myQueue.empty():

statement entirely and then it will eventually time out.

Comments

[abarnert]

Pretty clever; instead of code that works except when there's a very rare race condition, he's written code that fails except when there's a very rare race condition, which is a lot easier to debug. :)