I'm creating a python module at work, and I have a config file with variables that change for dev or prod (i.e. S3 buckets). What's the best way to do this? Right now I have dicts in config with DEV or PROD options, but selecting this every time in the classes seems tedious (i.e. S3_BUCKET[DEV] etc).
Python – Best way to handle dev/test/prod variables in Python
python
Related Solutions
I think it is a bad idea.
Its a very odd use of meta classes which will contribute to confusion of anyone trying to follow your code. In order to make it worth using this technique, you need the advantages to outweigh that confusion.
As for your reasons:
- Creating an accesible Foo class that can be instantiated, but doesn't actually work like a typical object is much more prone to misuse than an internal class that you know you aren't supposed to call.
- Don't name the base class with an underscore. If its intended to be inherited by other modules, it shouldn't be named that way.
- There already is a method to implement static classes in python,
staticmethod
orclassmethod
. As noted, its not hard to implement astaticproperty
In general, there is a theme in your approach of trying to prevent client code from doing the wrong thing. That's not a pythonic attitude. The pythonic approach is to trust your client code. If they want to do something crazy, let them. Don't try to prevent client from instantiating your internal object, there may well be a good reason to do that.
I'm a bit of a purist and think that you shouldn't be storing state like this. So the very fact that you are asking the question means you're already doing it wrong. If you have state, I think it should be in an object. I think storing any sort of state at the module level should be avoided. I think its only a good idea to implement something like this if you have no changing state.
To close off, I'd like to plug the Code Review Stack Exchange, where I'm a moderator. If you post your code there you can get suggestions on how to make it better. I think you may get some helpful suggestions for how to restructure your code to avoid this question from even coming up.
There are two problems come together here:
- Your package is a singleton although there exist many different configurations.
- Where to store the data and how to migrate - which database to use.
Singleton
I did a fun project with lots of modules which needed configuration themselves. The project was intended to run once per computer.
The alternative is to put everything into classes and instantiate them with a configuration. Tist would allow many different configurations to exist within one program. When you may have this need you should restructure your whole code.
Database
Maybe your project is for a company and shall be developed for a longer time and the configuration shall not be thrown away. Then you may need to keep in mind many previous configurations when changing the default values and updating the configuration. Proper databases have solved that problem.
My tradeoff solution
In my case
- The configuration can be thrown away if my model changes.
- The package runs not only once per process but also once per computer.
- Only one thread accesses the configuration.
There is a module called config.py with the following methods:
import config
config.load()
config.save()
You use it like this (Example1):
config.load()
config.my_value = 'test'
config.save()
There is also a file called constants.py which should better be called default_config.py
. It has the functions
import constants
constants.default_configuration()
constants.config_file_name() # where to store the config data
And for (Example1) the constants.py should look like this:
def default_configuration():
return {'my_value' : 'default'} # to avoid attribute errors
The config module saves and loads the configuration using pickle. If no configuration is found for a variable the default configuration is used.
You will need a coding style that always fetches the configuration from the config.py. It shall not be stored in a local variable or attribute over a longer time since it can change.
My previous version is called runningConfiguration. It has no explicit load and save and also no default attributes.
Best Answer
When you need different values depending of the environment, there are several things you can rely on:
--db-path=...
$MYAPP_DB_PATH
/etc
).Usually, I go for a fixed path local file, they are easier to handle in the program and can be extended easily. It's also always possible to add later on the flexibility to specify the path of that file by command line or system environment variable. The file should contain the variables of your logical environment. The code you run should never know which environment it runs in, because that would then mean you can't test your production code (which is a very bad idea).
This is a general advice and not particularily tied to Python, but this is an example of how you would do it the simple way in that language (i don't really know S3 buckets, I will just use plain json files for clarity purpose):
Code.py:
mycfg_dev.json:
mycfg_prod.json:
You would version the two files somewhere, encrypted if necessary (for example in an ansible vault). Be very careful of not commiting plain text production credentials in your repo since it will be hard to erase from git later on.
When you deploy, simply rename the one you need when running your module, or provide it to the run command, based on your implementation. Note that if that is practical, the cfg file can redirect to another data container (such as S3) if it brings any advantage over a plain file.
Generally speaking this is how I handle environment variables, but maybe I understood something wrong about your requirements.