Get length of Queue in Python's multiprocessing library

Question

I have a multiprocessing.Manager object that contains a multiprocessing.Queue to manage all of the work that I want a group of processes to do. I would like to get the number of elements left in this queue and was wondering how to do this?

Python's inbuilt len() function does not work.

Solution

If the queue you are talking about is multiprocessing.Queue, try to use qsize() method for multiprocessing.Queue objects, but be careful:

qsize()

Return the approximate size of the queue. Because of multithreading/multiprocessing semantics, this number is not reliable.

Note that this may raise NotImplementedError on Unix platforms like Mac OS X where sem_getvalue() is not implemented.

Comments

[CodeGuru]

What is the solution to NotImplementedError

[Fanchen Bao]

One way to work around NotImplementedError is to define your custom Queue class that allows .qsize() method. Here is one working example in Python3: gist.github.com/FanchenBao/d8577599c46eab1238a81857bb7277c9