Python Packages – Best Way to Set sys.path for ‘Hot Library’ Development

packagespython

I have my Python source structured as follows:

+-branchname/
  +-dst/
  +-src/
  | +-library/
  | | +-cleese/
  | | | +-test/
  | | | | +-__init__.py
  | | | | +-test_cleese.py
  | | | +-__init__.py
  | | | +-cleese.py
  | | +-palin/
  | |   +-test/
  | |   | +-__init__.py 
  | |   | +-test_palin.py 
  | |   +-__init__.py
  | |   +-palin.py
  | +-productline/
  |    +-circus/
  |    | +-test/
  |    | | +-__init__.py
  |    | | +-test_circus.py
  |    | +-__init__.py
  |    | +-circus.py
  |    +-grail/
  |      +-test/
  |      | +-__init__.py
  |      | +-test_grail.py
  |      +-__init__.py
  |      +-grail.py
  +-branch_root_marker

Entry points are in circus/ and grail/, as well as (possibly) each of the test/ directories, depending on how testing is implemented.

I have multiple copies of this source tree present on local storage at any one point in time (corresponding to various maintenance and feature branches), so I cannot set PYTHONPATH in my shell without some pain. (I would need to remember to change it each time I switched to work on a different branch, and I am very forgetful)

Instead, I have some logic that walks up the file tree, starting at the "current" file location, moving from leaf towards root, looking for branch_root_marker. Once the root directory of the current working copy is found, it adds library/ and productline/ to sys.path. I call this function from each entry-point in the system.

"""Add working copy (branch) to sys.path"""

import os
import sys


def _setpath():
    """Add working copy (branch) to sys.path"""

    markerfile = "branch_root_marker"
    path = ""
    if ("__file__" in locals() or globals()) and __file__ != "__main__":
        path = __file__
    elif sys.argv and sys.argv[0] and not sys.argv[0] == "-c":
        path = sys.argv[0]
    path = os.path.realpath(os.path.expandvars(path))
    while os.path.exists(path):
        if os.path.exists(os.path.join(path, markerfile)):
            break
        else:
            path = os.path.dirname(path)
        errormsg = " ".join(("Could not find", markerfile))
        assert os.path.exists(path) and path != os.path.dirname(path), errormsg

    path = os.path.join(path, "src")
    (_, subdir_list, _) = os.walk(path).next()
    for subdir in subdir_list:
        if subdir.startswith("."):
            continue
        subdir_path = os.path.join(path, subdir)
        if subdir_path in sys.path:
            continue
        sys.path.append(subdir_path)
_setpath()

Currently, I need to keep a separate but identical copy of this function in each entry point. Even though it is quite a short function, I am quite distressed by how fragrantly the DRY principle is being violated by this approach, and would love to find a way to keep the sys.path modification logic in one place. Any ideas?

Note:- One thing that springs to mind is to install the sys.path modifying logic into a common location that is always on PYTHONPATH. Whilst this is not a terrible idea, it means introducing an installation step that needs to be carried out each time I move to a fresh environment; another thing to remember (or, more likely, forget), so I would like to avoid this if at all possible.

Best Answer

OK, I started to craft a solution that involved PEP 302 import hooks. Way too complicated.

I think you answered your own question with:

this "gateway" script could implement all manner of bootstrapping functionality - for example, ensuring that virtualenv is set up correctly and activated for the current branch.

if you are already using virtualenv, put the logic you have in the "~/venv/or-whatever-path/bin/activate" script that is already there. I'd suggest listing of possible entry points and prompting for a response (list with numbers and a raw_input() statement will do) if you are not specifying one.

I'd also put the name of your branch in to $PS1 so that is shows up in the tab name or window title of your terminal. I so something similar and change the color of the prompt so I don't do the right thing in the wrong place.