Python – How to Hide Python Code from Users

I kind of hit this wall myself when I took a full-time Python programming job a couple years ago. I love Python, I really do, but when I started to do some performance tuning, I had some rude shocks.

The strict Pythonistas can correct me, but here are the things I found, painted in very broad strokes.

• Python memory usage is kind of scary. Python represents everything as a dict -- which is extremely powerful, but has a result that even simple data types are gigantic. I remember the character "a" took 28 bytes of memory. If you're using big data structures in Python, make sure to rely on numpy or scipy, because they are backed by direct byte-array implementation.

That has a performance impact, because it means there are extra levels of indirection at run time, in addition to slogging around huge amounts of memory compared to other languages.

• Python does have a global interpreter lock, which means that for the most part, processes are running single-threaded. There may be libraries that distribute tasks across processes, but we were spinning up 32 or so instances of our python script and running each single threaded.

Others can talk to the execution model, but Python is a compile-at-runtime and then interpreted, which means it doesn't go all the way to machine code. That also has a performance impact. You can easily link in C or C++ modules, or find them, but if you just run straight up Python, it's going to have a performance hit.

Now, in web service benchmarks, Python compares favorably to the other compile-at-runtime languages like Ruby or PHP. But it's pretty far behind most of the compiled languages. Even the languages that compile to intermediate language and run in a VM (like Java or C#) do much, much better.

Here is a really interesting set of benchmark tests that I refer to occasionally:

http://www.techempower.com/benchmarks/

(All that said, I still love Python dearly, and if I get the chance to choose the language I'm working in, it's my first choice. Most of the time, I'm not constrained by crazy throughput requirements anyway.)

How do I know if my code is running fast enough?

That very much depends on your use case -- your program runs for 1.4 hours which might or might not be fast enough. If this is a one-time process 1.4 hours is not that much - spending any time on optimization is hardly worth the investment. On the other hand, if this is a process that should run e.g. once every hour, clearly it is worth finding a less time-consuming approach

Is there a measurable way to test the speed & performance of my code?

yes, profiling - and you've already done that. That's a good start.

what do I do next?

Best practices include:

1. measure baseline performance (before any optimization)
2. analyze the parts where the program spends most of its time
3. reduce run-time complexity (the Big-O type)
4. check for the potential of parallel computation
5. compare against baseline performance

You have already done 1. So let's move to 2.

Analysis

In your case the program spends most of it's time in line OBSparser.py:48, of which a third is spent calculating the mean 7638018 times.

As the profiler output shows, this is on an ndarray, i.e. using numpy, and it doesn't look like it's taking a lot of time on a per-call basis. A quick calculation confirms that:

179' / 7.638.018 = 23.6 microseconds per call

Since that's already implemented in C-code (numpy), there is likely not much you can do to improve the per-call performance by changing the actual mean code (or using another library).

However, ask yourself several questions:

1. How can the number of calls to .mean() be reduced?
2. Can the calls to .mean() be implemented more efficiently?
3. Could the data be grouped and each group be processed independently?
4. ask more questions

Other calls worth looking at are to .astype() and reduce, I focused on .mean()simply for illustration.

Reducing complexity

Not knowing what your code actually does, here's my 5cents on the specifics, anyway:

On 2., a quick check on my i7 core reveals that for ndarray.mean() to take 20-odd microseconds, this takes around 50 values. So I'm guessing your are grouping values and then calling .mean() on every group. There might be more efficient ways - a search on numpy group aggregate performance or some variant of that might find you some helpful pointers.

Parallel computation

On 3. I'm guessing multi-processing is unlikely to be a solution here, since your computations seem mostly CPU-bound and the overhead of launching seperate tasks and exchanging data probably outweighs the benefits.

However there might be some use of SIMD-approach, i.e. vectorization. Again, just a hunch.

Compare against baseline performance

To reduce the time it takes to re-profile, consider subsetting your data such that the performance behavior is still visible (i.e. 23 us per call to .mean()) but where the total running time is under maybe 1-2 minutes, or even less. This will help you evaluate several approaches before applying them to your program in full. There is no use in running the full process over and over again just to test some small optimization.