At what point does a config file become a programming language?

Question

I have been mulling over config files and their relationship to code for a while now and depending on the day and direction of the wind my opinions seem to change. More and more though I keep coming back to the realization I first had while learning Lisp: there is little difference between data and code. This seems doubly true for config files. When looked at in the right light a Perl script is little more than a config file for perl. This tends to have fairly heavy consequences for tasks such as QA and divisions of labor like who should be responsible for changing config files.

The creep from config file to full fledged language is generally slow and seems to be driven by the desire to have a generic system. Most projects seem to start out small with a few config items like where to write logs, where to look for data, user names and passwords, etc. But then they start to grow: features start to be able to be turned on or off, the timings and order of operations start to be controlled, and, inevitably, someone wants to start adding logic to it (e.g. use 10 if the machine is X and 15 if the machine is Y). At a certain point the config file becomes a domain specific language, and a poorly written one at that.

Now that I have rambled on to set the stage, here are my questions:

1. What is the true purpose of a config file?
2. Should an attempt be made to keep config files simple?
3. Who should be responsible for making changes to them (developers, users, admins, etc.)?
4. Should they be source controlled (see question 3)?

As I said earlier my answers to these questions shift constantly, but right now I am thinking:

1. to allow a non-programmers to change large chunks of behaviour quickly
2. yes, anything that is not coarsely grained should be in code
3. users should be responsible for config files and programmers should be responsible for a configuration layer between config files and code that gives more fine grained control of the application
4. no, but the finer grained middle layer should be

Solution 1

Very interesting questions!

I tend to limit my config files to a very simple "key=value" format, because I fully agree with you that config files can very quickly become full-blown programs. For example, anyone who has ever tried to "configure" OpenSER knows the feeling you are talking about: it's not configuration, it's (painful) programming.

When you need your application to be very "configurable" in ways that you cannot imagine today, then what you really need is a plugins system. You need to develop your application in a way that someone else can code a new plugin and hook it into your application in the future.

So, to answer your questions:

1. What is the true purpose of a config file?

   I would say, to allow the people who will install your application to be able to tweek some deployment-related parameters, such as host name, number of threads, names of the plugins you need, and the deployment-parameters for those plugins (check out FreeRadius's configuration for an example of this principle), etc.. Definitely not the place to express business logic.

2. Should an attempt be made to keep config files simple?

   Definitely. As you suggested, "programming" in a config file is horrible. I believe it should be avoided.

3. Who should be responsible for making changes to them (developers, users, admins, etc.)?

   In general, I would say admins, who deploy the application.

4. Should they be source controlled (see question 3)?

   I usually don't source-control the configuration files themselves, but I do source-control a template configuration file, with all the parameters and their default values, and comments describing what they do. For example, if a configuration file is named database.conf, I usually source-control a file named database.conf.template. Now of course I am talking about what I do as a developer. As an admin, I may want to source-control the actual settings that I chose for each installation. For example, we manage a few hundred servers remotely, and we need to keep track of their configurations: we chose to do this with source-control.

---

Edit: Although I believe the above to be true for most applications, there are always exceptions, of course. Your application may allow its users to dynamically configure complex rules, for example. Most email clients allow the users to define rules for the management of their emails (for example, "all emails coming from 'john doe' and not having me in the To: field should be discarded"). Another example is an application that allows the user to define a new complex commercial offer. You may also think about applications like Cognos which allow their users to build complex database reports. The email client will probably offer the user a simple interface to define the rules, and this will generate a complex configuration file (or even perhaps a bit of code). On the other hand, the user-defined configuration for the commercial offers might be saved in a database, in a structured way (neither a simple key=value structure nor a portion of code). And some other applications might even allow the user to code in python or VB, or some other automation-capable language. In other words... your mileage may vary.

Solution 2

Ok. You will have some users which want a really simple config, you should give it to them. At the same time, you will have constant requests of "Can you add this? How do I do in the config file?", I don't see why you can't support both groups.

The project I am currently working on uses Lua for its configuration file. Lua is a scripting language, and it works quite well in this scenario. There is available an example of our default configuration.

You'll note that it is mainly key=value statements, where value can be any of Lua's built-in types. The most complicated thing there are lists, and they aren't really complicated (it's just a matter of syntax).

Now I'm just waiting for someone to ask how to set their server's port to a random value every time they start it up...

Solution 3

Recently I was working upon a project and I realised that I wanted to have conditionals inside my configuration file - which had previously just been a pretty simple one of the form:

key = val
key2 = val
name = `hostname`

I didn't want to write a mini-language, because unless I did it very carefully I couldn't allow the flexibility that would be useful.

Instead I decided that I'd have two forms:

1. If the file started with "#!" and was executable I'd parse the result of running it.

2. Otherwise I'd read it as-is

This means that I can now allow people to write "configuration files" that look like this:

 #!/usr/bin/perl
if ( -x /bin/foo ) 
{
   print <<EOF;
foo=me
bar=you
EOF
}
else
{
   print <<EOF;
foo=bar
bar=foo
EOF
}

This way I get the power of a dynamic configuration file if the user wants to use it, and the simplicity of not having to write my own mini-language.

Solution 4

Every (sufficiently-long-lived) config file schema eventually becomes a programming language. Due to all the implications you describe, it is wise for the config-file designer to realize she is authoring a programming language and plan accordingly, lest she burden future users with bad legacy.

Solution 5

I have a different philosophy about config files. Data about how an application should be run is still data, and therefore belongs in a data store, not in code (a config file IMO is code). If end users need to be able to change the data, then the application should provide an interface to do so.

I only use config files to point at data stores.

View more solutions

Comments

[aib]

When they become Turing-complete, of course!

[Chas. Owens]

Regular Expressions are not Turing-complete, but are still considered a computer language.

[Ali Afshar]

"Files" are not really adequate for some config situations. Hence the existence of systems like gconf.

[Chas. Owens]

There is no real difference between gconf and a file. Gconf is really just a series of directories with files in them with an in memory representation. Even if you were to bring up an RDBMS, it could be represented by a single file. The issue is how much complexity is safe/good in a config file.

[Ali Afshar]

Chas. It's how you access the "file" that is the difference. And how you handle changes to config data when multiple clients are connected. Yes Gconf is represented as files on a disk, but behaves differently. If you mean "complexity of config data in a config system", sure.

[Chas. Owens]

I could write a gconf backend that used a single file and you wouldn't notice a difference unless you looked in ~/.gconf. Conceptually gconf is very simple (just key/value pairs). The question is about systems that become more complex than simple key/value pairs and if that is a bad thing.

[Neil McGuigan]

OpenBSD's pf (firewall) configuration format is its own language: man.openbsd.org/OpenBSD-current/man5/pf.conf.5#GRAMMAR

[Benjamin Confino]

+1 for useing an actual programming language, far neater than just letting one grow from a config file

[Andrew Y]

+1 - it's the right approach. Lua does have a very "config-like" syntax for those who are new. And allows powerful manipulations for those who are not.

[Hugh Perkins]

ANT config files are xml, and have complex structures such as if and for. Writing config files in xml is no guarantee that the config files will be compact, and comfortable for a human to read.

[Supreeth Padavala]

The next response in the conversation flow should have been: You: "Yeah, welcome to the club, buddy"

[Ti Strga]

Amusingly, one of Lua's predecessor languages essentially started as a configuration file format, and grew to encompass basic if-then-else and minimal looping. When it came time to design Lua, the authors knew that the immediate user base would not necessarily be programmers by profession, and so the syntax had to be friendly. Wonderful, excellent language!