Python Interpreter only using 12% CPU Power

Question

I'm using python on ubuntu for text analysis. Despite the severe amount of work the program is doing the CPU usage as shown in the system monitor stays consistently at 12%.

I changed the priority of the program from Normal to Very High but that had no effect.

What is limiting the amount of CPU usage which my python program can get and how can I change that, so the program can utilize more cpu power?

Solution 1

I assume you have a CPU with 8 virtual cores (quad-core with hyper-threading probably)? That means one fully loaded CPU thread/virtual core equals 12.5% total load.

The Python interpreter is an application which only runs as one single process by default and is therefore not able to take advantage of more than one virtual core. Even if the code you run with it uses multithreading, it will still only use one CPU thread/virtual core, because of the GIL (global interpreter lock).

Only if your Python program uses multiprocessing, which in fact starts up multiple instances of the Python interpreter and lets them perform your tasks truly parallel, you can take advantage of multiple virtual cores/CPU threads. (As @SargeBorsch pointed out in his comment, there are also some advanced ways to achieve this without multiprocessing, but that's normally not something you quickly write yourself.)

Solution 2

Another possibility, less likely in this case, is that the program is disk-bound, i.e. it is reading and writing to/from the disk which is slow, and the CPU is waiting for the disk.

Comments

[Matthias Herrmann]

That actually makes a lot of sense. Yes I'm having a quad-core with 4 cores (8 virtual cores). Ty

[WinEunuuchs2Unix]

@MatthiasHerrmann You could consider a system monitor to show you what percentage each CPU is operating at. That way you could have seen only 1 of 8 CPUs was at 100%. Here is one thread in AU on the subject: Windows “gadgets” equivalent (for wifi and cpu usage)?

[cat]

try iotop to monitor programs bound to iowait

[Display Name]

Not true, it's possible to use all cores from single python process just fine. One just needs to call C code and release the GIL. And many existing libraries do exactly that (numpy for example).

[OrangeDog]

Or use Jython or IronPython, which don't have a GIL.

[Zydnar]

Or the code itself is synchronous and blocking.