Python – Best way to handle dev/test/prod variables in Python

python

I'm creating a python module at work, and I have a config file with variables that change for dev or prod (i.e. S3 buckets). What's the best way to do this? Right now I have dicts in config with DEV or PROD options, but selecting this every time in the classes seems tedious (i.e. S3_BUCKET[DEV] etc).

Best Answer

When you need different values depending of the environment, there are several things you can rely on:

  • A command line argument such as --db-path=...
  • An system environment variable such as $MYAPP_DB_PATH
  • A fixed path local file (on unix systems, they are usually located at /etc).

Usually, I go for a fixed path local file, they are easier to handle in the program and can be extended easily. It's also always possible to add later on the flexibility to specify the path of that file by command line or system environment variable. The file should contain the variables of your logical environment. The code you run should never know which environment it runs in, because that would then mean you can't test your production code (which is a very bad idea).

This is a general advice and not particularily tied to Python, but this is an example of how you would do it the simple way in that language (i don't really know S3 buckets, I will just use plain json files for clarity purpose):

Code.py:

import json
with open('mycfg.json') as f:
    env_vars = json.loads(f.read())
do_something(env_vars)

mycfg_dev.json:

{
    "path": "/tmp",
    "ip": "127.0.0.1",
    "login": "root",
    "password": "root"
}

mycfg_prod.json:

{
    "path": "/var/www",
    "ip": "192.168.1.44",
    "login": "myapp",
    "password": "aunriseqvoa"
}

You would version the two files somewhere, encrypted if necessary (for example in an ansible vault). Be very careful of not commiting plain text production credentials in your repo since it will be hard to erase from git later on.

When you deploy, simply rename the one you need when running your module, or provide it to the run command, based on your implementation. Note that if that is practical, the cfg file can redirect to another data container (such as S3) if it brings any advantage over a plain file.

Generally speaking this is how I handle environment variables, but maybe I understood something wrong about your requirements.

Related Topic