Python – Are exceptions for flow control best practice in Python

exceptionsprogramming practicespython

I'm reading "Learning Python" and have come across the following:

User-defined exceptions can also signal nonerror conditions. For
instance, a search routine can be coded to raise an exception when a
match is found instead of returning a status flag for the caller to
interpret. In the following, the try/except/else exception handler
does the work of an if/else return-value tester:

class Found(Exception): pass

def searcher():
    if ...success...:
        raise Found()            # Raise exceptions instead of returning flags
    else:
        return

Because Python is dynamically typed and polymorphic to the core,
exceptions, rather than sentinel return values, are the generally
preferred way to signal such conditions.

I've seen this sort of thing discussed multiple times on various forums, and
references to Python using StopIteration to end loops, but I can't find much
in the official style guides (PEP 8 has one offhand reference to exceptions
for flow control) or statements from developers. Is there anything official
that states this is best practice for Python?

This (Are exceptions as control flow considered a serious antipattern? If so, Why?) also has several commenters state that this style is Pythonic. What is this based on?

TIA

Best Answer

The general consensus “don't use exceptions!” mostly comes from other languages and even there is sometimes outdated.

  • In C++, throwing an exception is very costly due to “stack unwinding”. Every local variable declaration is like a with statement in Python, and the object in that variable may run destructors. These destructors are executed when an exception is thrown, but also when returning from a function. This “RAII idiom” is an integral language feature and is super important to write robust, correct code – so RAII versus cheap exceptions was a tradeoff that C++ decided towards RAII.

  • In early C++, a lot of code was not written in an exception-safe manner: unless you actually use RAII, it is easy to leak memory and other resources. So throwing exceptions would render that code incorrect. This is no longer reasonable since even the C++ standard library uses exceptions: you can't pretend exceptions don't exist. However, exceptions are still an issue when combining C code with C++.

  • In Java, every exception has an associated stack trace. The stack trace is very valuable when debugging errors, but is wasted effort when the exception is never printed, e.g. because it was only used for control flow.

So in those languages exceptions are “too expensive” to be used as control flow. In Python this is less of an issue and exceptions are a lot cheaper. Additionally, the Python language already suffers from some overhead that makes the cost of exceptions unnoticeable compared to other control flow constructs: e.g. checking if a dict entry exists with an explicit membership test if key in the_dict: ... is generally exactly as fast as simply accessing the entry the_dict[key]; ... and checking if you get a KeyError. Some integral language features (e.g. generators) are designed in terms of exceptions.

So while there is no technical reason to specifically avoid exceptions in Python, there is still the question whether you should use them instead of return values. The design-level problems with exceptions are:

  • they are not at all obvious. You can't easily look at a function and see which exceptions it may throw, so you don't always know what to catch. The return value tends to be more well-defined.

  • exceptions are non-local control flow which complicates your code. When you throw an exception, you don't know where the control flow will resume. For errors that can't be immediately handled this is probably a good idea, when notifying your caller of a condition this is entirely unnecessary.

Python culture is generally slanted in favour of exceptions, but it's easy to go overboard. Imagine a list_contains(the_list, item) function that checks whether the list contains an item equal to that item. If the result is communicated via exceptions that is absolutely annoying, because we have to call it like this:

try:
  list_contains(invited_guests, person_at_door)
except Found:
  print("Oh, hello {}!".format(person_at_door))
except NotFound:
  print("Who are you?")

Returning a bool would be much clearer:

if list_contains(invited_guests, person_at_door):
  print("Oh, hello {}!".format(person_at_door))
else:
  print("Who are you?")

If the function is already supposed to return a value, then returning a special value for special conditions is rather error-prone, because people will forget to check this value (that's probably the cause of 1/3 of the problems in C). An exception is usually more correct.

A good example is a pos = find_string(haystack, needle) function that searches for the first occurrence of the needle string in the `haystack string, and returns the start position. But what if they haystack-string does not contain the needle-string?

The solution by C and mimicked by Python is to return a special value. In C this is a null pointer, in Python this is -1. This will lead to surprising results when the position is used as a string index without checking, especially as -1 is a valid index in Python. In C, your NULL pointer will at least give you a segfault.

In PHP, a special value of a different type is returned: the boolean FALSE instead of an integer. As it turns out this isn't actually any better due to the implicit conversion rules of the language (but note that in Python as well booleans can be used as ints!). Functions that do not return a consistent type are generally considered very confusing.

A more robust variant would have been to throw an exception when the string can't be found, which makes sure that during normal control flow it is impossible to accidentally use the special value in place of an ordinary value:

 try:
   pos = find_string(haystack, needle)
   do_something_with(pos)
 except NotFound:
   ...

Alternatively, always returning a type that can't be used directly but must first be unwrapped can be used, e.g. a result-bool tuple where the boolean indicates whether an exception occurred or if the result is usable. Then:

pos, ok = find_string(haystack, needle)
if not ok:
  ...
do_something_with(pos)

This forces you to handle problems immediately, but it gets annoying very quickly. It also prevents you from chaining function easily. Every function call now needs three lines of code. Golang is a language that thinks this nuisance is worth the safety.

So to summarize, exceptions are not entirely without problems and can definitively be overused, especially when they replace a “normal” return value. But when used to signal special conditions (not necessarily just errors), then exceptions can help you to develop APIs that are clean, intuitive, easy to use, and difficult to misuse.

Related Topic