How to Debug Multiprocessing in Python

True multiprocessing Python programs (as opposed to multi-threaded Python programs which have to deal with the dreaded GIL) are no different from those in any other language. They all have the same basic challenges:

1. Task assignment and result reporting. Even if they are mostly working on independent datasets, they normally have to go back to the master thread to report results and get new data to work on. This can be a choke point.
2. Race conditions. Processes are trying to use a one-at-a-time resource and they need to use mutex (or something similar) to keep from stepping all over each other's data. Failure to protect these kinds of resources can lead to really, really painful debug sessions.
3. Sequentiality. Sometimes you are trying to make something parallel that isn't. The various processes end up waiting for each other to do something and the end result is that you have, for all intents and purposes, taken a sequential program, made it parallel, and it still ends up executing in linear time (or worse).

Although there are development methods that try to avoid each of these problems, at the end of the day you really need to think about what you are doing. I recommend heavy stress testing — far beyond anything you think might happen in real life — so that you have a good chance of hitting those Windows of Opportunity and blowing up in development as opposed to in the middle of a major demo or during production.

We used to use microsecond-timestamped log files and then created a color-coded log viewing app so we could try to visualize what was happening between N process running on M processors. We also tried (and mostly succeeded) in creating a system that would drive off the log files to recreate the context of the crash.

But the best tool is good design and really wicked, nasty people who try to blow your app out of the water. (Hi, dbell!)