Manage configuration files across environments

Question

How do you (your company) manage the config-files of the apps/systems you build? Let me tell you how we do it, and what the problem is.

I'm working at a company where we develop software with about 15 developers. We build line-of-business web apps that are deployed at our managed hosting provider. One of our main apps consists of one web site and about ten WCF services. Some of the services are connected to each other.

I don't know if this is a big system, or small, but my opinion is that is takes us way too long to get things up and running in our different environments (test, acceptance and production).

We have config-files per environment in our Visual Studio projects. So a web.test.config, a web.acc.config, a web.prod.config and a web.config for development. They all have the same keys in them, but the values can be different, depending on the environment they are meant for.

If I do a quick count of the appsettings in the web.config for the webapp I count 32. And I count 5 endpoints. We have four environments (dev, test, acc and prod) this means 128 appsettings and 20 endpoints in total for one web app. We can easily make mistakes, especially when deadlines are closing in.

We are all humans, so things like this may happen to anyone:

• We make a change in one of config files, but forget to check in before we build and deploy.
• Or we make a change on the WebServer and forget to update accordingly in four other web.configs.
• Or we Change only three of four config files. And so on.

Then we have the infrastructure at our managed hosting provider. By default every port is closed. So if one of WCF services needs to talk to other one of WCF services located on a different server, a firewall protected port has to be open.

We do this in Test, but in Acceptance we have to do it again, and we have forgotten which ports have to be opened, so it's more like trial-and-error: Oh my service can't connect to the database, probably the port is closed. The same problem may happen in production as well.

Our managed hosting provider can take a few days to open a port in a firewall, according to the SLA. So, this quickly becomes a pretty long process. And in the end it takes us about two months to have Test, Acceptance and production up and running.

So, my question is: how do you manage the configurations and the infrastructure and the process around it?

Solution 1

The Config4* project (disclaimer: I am its primary developer) does not have an out-of-the-box integration with .Net or WCF, so it is probably not useful to you. However, one of the features in Config4* is relevant to your question: it is the ability to embed if-then-else statements in a configuration file, so that the file can "adapt" itself for different environments (such as development, testing, acceptance and production).

You may be able to modify that concept to work with whatever configuration syntax you are using in your .Net/WCF-based project (I'm not familiar with those technologies, but I'm guessing they probably use XML-based configuration files). In particular, you could write a script using, say, Python, that uses if-then-else statements to set environment-specific name=value pairs in a map, and then use some print statements to generate a set of configuration files tailored for an environment. A pseudo-code outline of such a script is:

#--------
# Set up configuration variables suitable for a specified environment
#--------
cfg["variable1"] = "default value";
cfg["variable2"] = "another default value";
if (environment == "testing") {
  cfg["variable1"] = "override default value for this environment";
  cfg["variable3"] = "value suitable for this environment";
  ...
} else if (environment == "production") {
  ...
}
#--------
# Now use print statements to generate configuration files
# Alternatively, use the _name=value_ pairs in the map to
# perform global search-and-replace on template versions of
# configuration files.
#--------
...

For bonus points, the script could also generate a checklist of tests that need to be performed for the environment, for example, "Check if a firewall port needs to be opened between the following endpoints: ..."

Solution 2

We are developing a distributed storage system and we catch many problems like this with our unit and integration tests that run with every build. Also, we are using ReviewBoard, for other developers to have a look at any changes before they get committed. Next step is the continuous integration server (jenkins) that auto deploys and test the artifacts in different environments. Finally, our last line of defense is our stress test testbed that runs against any new version, before it is published. Of course it is essential to have a testbed that mimics your production environments as good as possible, but if you always deploy at the same hosting provider, that shouldn't be too hard.

• Concerning your special case, where changes in one file might have been forgotten in the others, etc.: I imagine this would be quite easy to auto check with a test script.

• Uncommitted changes should be as easy to detect as "git diff master", or whatever vcs you use.

• To know what ports need to be open for the services seems to be more like a question of proper documentation than of configuration management.

Many of the sites that use our system also use puppet to manage the configuration of their different nodes. I am not sure if that would be an option for you, but it might be worth having a look at.

Solution 3

This may help, it talks about a central location for config files and services referencing from the one source.

http://blogs.msdn.com/b/youssefm/archive/2010/09/02/loading-wcf-client-configuration-from-different-files-with-configurationchannelfactory.aspx

Solution 4

It sounds like you have a small enough environment you could easily manage all the config files using Ansible templates or SaltStack.

Solution 5

Take a look at Config at http://www.configapp.com. The way it will work is you will import a base configuration file called web.config. Then from within the webapp, you will create 4 environments, dev, test, acc and prod. Go to the dev environment, create your environment variables, and set the values appropriate to the environment. You will be maintaining just one configuration file with environment variables. You can see all the environment variables, and easily view the differences between the environments.

You will make less mistakes because you only have 1 configuration file to manage. When you add a new common/static configuration, all environments will magically have that new setting. When you add a variable configuration, just go the correct environment tab and set the value there. You can view a specific configuration value for all environments so it's a quick way to check if you missed something. You can also eyeball each configuration file per environment, since they are all right in front of you. You don't need to RDP/SSH to each server.

You won't forget to check in since Config will be the master copy, and you won't edit configuration files directly anymore. If you do edit them directly, we have a diff functionality so you can see the differences between the master and the local copy. You can deploy the master copy to your local server using various deployment models that suits your team. You can push, manual pull, auto pull or export/copy.

We will support custom types, and one thing we can add in the future is a port data type. Part of port validation would be checking if the port is open. This will only work using the on-premises plan since it needs access to the internal network. Or you can just check it manually. Go to the port environment variable, and display all the ports currently configured. Check each port in the list. If the port looks good, add a comment in Config that it's open and it was checked on a certain date. Config is designed for searching and documentation.

I'm part of the Config team, by the way.

View more solutions