Python – How to maximize the CPU usage of a Python subprocess

cpuprocesspython

I have a Python script launching a C++ executable.

The C++ executable is a multithreaded program that usually takes several hours to run. The way the C++ code is written, it will run on all the cores of the CPU if possible.

However, when I run the Python script and check my task manager, I read that the Python script is only using 30% of my CPU. I think this 30% includes the CPU usage of the subprocess running the executable as well, because the executable appears in the task manager with 0% CPU usage (but it's indeed running and producing results). Sometimes the executable will spike to 7% CPU usage and go right back down to 0%.

Is there a way I can increase the CPU usage of this subprocess to save time?

EDIT:
Maybe i should specify that this Python script is communicating with the C++ subprocess through a pipe. While it's running, the subprocess sends back a lot of information through the pipe that the Python script uploads to a database which might explain the high CPU usage of the script.

Best Answer

The way the C++ code is written, it will run on all the cores of the CPU if possible

Well, did you actually test this, or do you just assume this? What happens if your C++ is started directly from the command line?

However, when I run the Python script and check my task manager, I read that the Python script is only using 30% of my CPU. I think this 30% includes the CPU usage of the subprocess running the executable as well

That is definitely wrong. If your Python script uses 30% of the CPU, it is doing the work by itself, and not the subprocess. Impossible to say what your script does without seeing the code, but my best guess is it probably wastes CPU cycles by waiting for the end of the subprocess in a very inefficient manner.

the executable appears in the task manager with 0% CPU usage (but it's indeed running and producing results

This behaviour might happen when the executable is running with a low priority, whilst your Python process has a higher priority and blocks the execution of the C++ program. Other things which could impose blocking behaviour are shared file access or other shared resource access. I would check for these kind of things.

Related Topic