Managing configurations for different environments

Question

I've had a discussion with some people at work and we couldn't come to a conclusion.
We've faced a dilemma - how do you manage different configuration values for different environments?

We've come up with some options, but none of them seemed to satisfy us:
- Separate config files (i.e. config.test, config.prod etc.), and having a file pointing to the selected one (at ~/env for instance), or an environment variable pointing to it.
- Using a single DB to store all configurations (you query it with your environment and get the corresponding configuration values)
- Creating configuration files on deploy (Using CI/CD system like Atlassian Bamboo)

Which is the more widely used option? Is there a better way?
Should config file be kept in the git repository along with the rest of the code? Our systems are written in python (both 2.7 and 3)

Solution 1

We ended up using a method similar to this one. We had a base settings file, and environment specific files that simply imported everything from the base file base.py:

SAMPLE_CONFIG_VARIABLE = SAMPLE_CONFIG_VALUE

prod.py:

from base.py import *

So all we had to do when accessing values from the configuration is read an environment variable and creating the file corresponding to it.

Solution 2

You can put all these configuration in single config file config.py.

class Base():
    DEBUG = False
    TESTING = False

class DevelopmentConfig(Base):
    DEBUG = True
    DEVELOPMENT = True
    DATABASE_URI = "mysql+mysqldb://root:root@localhost/demo"

class TestingConfig(Base):
    DEBUG = False
    TESTING = True
    DATABASE_URI = "mysql+mysqldb://root:root@test_server_host_name/demo_test"

class ProductionConfig(Base):
    DEBUG = False
    TESTING = False
    DATABASE_URI = "mysql+mysqldb://root:root@prod_host_name/demo_prod"

on the shell set environment variable like

APP_SETTINGS = config.DevelopmentConfig

In your main application app.py, load this environment variable (flask app as example)

from flask import Flask
import os

app = Flask(__name__)
app.config.from_object(os.environ['APP_SETTINGS'])

I hope this can help

Solution 3

It's usually a bad idea to commit your configuration settings to source control, particularly when those settings include passwords or other secrets. I prefer using environment variables to pass those values to the program. The most flexible way I've found is to use the argparse module, and use the environment variables as the defaults. That way, you can override the environment variables on the command line. Be careful about putting passwords on the command line, though, because other users can probably see your command line arguments in the process list.

Here's an example that uses argparse and environment variables:

def parse_args(argv=None):
    parser = ArgumentParser(description='Watch the raw data folder for new runs.',
                            formatter_class=ArgumentDefaultsHelpFormatter)
    parser.add_argument(
        '--kive_server',
        default=os.environ.get('MICALL_KIVE_SERVER', 'http://localhost:8000'),
        help='server to send runs to')
    parser.add_argument(
        '--kive_user',
        default=os.environ.get('MICALL_KIVE_USER', 'kive'),
        help='user name for Kive server')
    parser.add_argument(
        '--kive_password',
        default=SUPPRESS,
        help='password for Kive server (default not shown)')

    args = parser.parse_args(argv)
    if not hasattr(args, 'kive_password'):
        args.kive_password = os.environ.get('MICALL_KIVE_PASSWORD', 'kive')
    return args

Setting those environment variables can be a bit confusing, particularly for system services. If you're using systemd, look at the service unit, and be careful to use EnvironmentFile instead of Environment for any secrets. Environment values can be viewed by any user with systemctl show.

I usually make the default values useful for a developer running on their workstation, so they can start development without changing any configuration.

Another option is to put the configuration settings in a settings.py file, and just be careful not to commit that file to source control. I have often committed a settings_template.py file that users can copy.