Python Configuration Files – How to Use Python Files for Configuration Management

configurationconfiguration-managementjsonpythonpython-3.x

I've always used JSON files for configuration of my applications. I started using them from when I coded a lot of Java, and now I'm working mainly on server-side and data science Python development and am not sure if JSON is the right way to go any more.

I've seen Celery use actual Python files for configuration. Initially I was skeptical about it. But the idea of using simple Python data structures for configuration is starting to grow on me. Some pros:

  • The data structures will be the same as I'm normally coding in. So, I don't need to change frame of mind.
  • My IDE (PyCharm) understands the connection between configuration and code. Ctrl + B makes it possible to jump between configuration and code easily.
  • I don't need to work with IMO unnecessary strict JSON. I'm looking at you double quotes, no trailing commas and no comments.
  • I can write testing configurations in the application I'm working on, then easily port them to a configuration file without having to do any conversion and JSON parsing.
  • It is possible to do very simple scripting in the configuration file if really necessary. (Although this should be very, very limited.)

So, my question is: If I switch, how am I shooting myself in the foot?

No unskilled end user will be using the configuration files. Any changes to the configuration files are currently committed to Git and are rolled out to our servers as part of continuous deployment. There are no manual configuration changes, unless there is an emergency or it is in development.

(I've considered YAML, but something about it irks me. So, for now it is off the table.)

Best Answer

Using a scripting language in place of a config file looks great at first glance: you have the full power of that language available and can simply eval() or import it. In practice, there are a few gotchas:

  • it is a programming language, which needs to be learnt. To edit the config, you need to know this language sufficiently well. Configuration files typically have a simpler format that is more difficult to get wrong.

  • it is a programming language, which means that the config can get difficult to debug. With a normal config file you look at it and see what values are provided for each property. With a script, you potentially need to execute it first to see the values.

  • it is a programming language, which makes it difficult to maintain a clear separation between the configuration and the actual program. Sometimes you do want this kind of extensibility, but at that point you are probably rather looking for a real plugin system.

  • it is a programming language, which means that the config can do anything that the programming language can do. So either you are using a sandbox solution which negates much of the flexibility of the language, or you are placing high trust in the config author.

So using a script for configuration is likely OK if the audience of your tool is developers, e.g. Sphinx config or the setup.py in Python projects. Other programs with executable configuration are shells like Bash, and editors like Vim.

Using a programming language for configuration is necessary if the config contains many conditional sections, or if it provides callbacks/plugins. Using a script directly instead of eval()-ing some config field tends to be more debuggable (think of the stack traces and line numbers!).

Directly using a programming language may also be a good idea if your config is so repetitive that you are writing scripts to autogenerate the config. But perhaps a better data model for the config could remove the need for such explicit configuration? For example, it may be helpful if the config file can contain placeholders that you later expand. Another feature sometimes seen is multiple config files with different precedence that can override each other, though that introduces some problems of its own.

In the majority of cases, INI files, Java property files, or YAML documents are much better suited for configuration. For complex data models, XML may also be applicable. As you've noted, JSON has some aspects that make it unsuitable as a human-editable configuration file, although it is a fine data exchange format.

Related Topic