Parse config files, environment, and command-line arguments, to get a single collection of options

Question

Python's standard library has modules for configuration file parsing (configparser), environment variable reading (os.environ), and command-line argument parsing (argparse). I want to write a program that does all those, and also:

• Has a cascade of option values:

  • default option values, overridden by
  • config file options, overridden by
  • environment variables, overridden by
  • command-line options.

• Allows one or more configuration file locations specified on the command line with e.g. --config-file foo.conf, and reads that (either instead of, or additional to, the usual configuration file). This must still obey the above cascade.

• Allows option definitions in a single place to determine the parsing behaviour for configuration files and the command line.

• Unifies the parsed options into a single collection of option values for the rest of the program to access without caring where they came from.

Everything I need is apparently in the Python standard library, but they don't work together smoothly.

How can I achieve this with minimum deviation from the Python standard library?

Solution 1

It seems the standard library doesn't address this, leaving each programmer to cobble configparser and argparse and os.environ all together in clunky ways.

Solution 2

UPDATE: I finally got around to putting this on pypi. Install latest version via:

   pip install configargparser

Full help and instructions are here.

Original post

Here's a little something that I hacked together. Feel free suggest improvements/bug-reports in the comments:

import argparse
import ConfigParser
import os

def _identity(x):
    return x

_SENTINEL = object()


class AddConfigFile(argparse.Action):
    def __call__(self,parser,namespace,values,option_string=None):
        # I can never remember if `values` is a list all the time or if it
        # can be a scalar string; this takes care of both.
        if isinstance(values,basestring):
            parser.config_files.append(values)
        else:
            parser.config_files.extend(values)


class ArgumentConfigEnvParser(argparse.ArgumentParser):
    def __init__(self,*args,**kwargs):
        """
        Added 2 new keyword arguments to the ArgumentParser constructor:

           config --> List of filenames to parse for config goodness
           default_section --> name of the default section in the config file
        """
        self.config_files = kwargs.pop('config',[])  #Must be a list
        self.default_section = kwargs.pop('default_section','MAIN')
        self._action_defaults = {}
        argparse.ArgumentParser.__init__(self,*args,**kwargs)


    def add_argument(self,*args,**kwargs):
        """
        Works like `ArgumentParser.add_argument`, except that we've added an action:

           config: add a config file to the parser

        This also adds the ability to specify which section of the config file to pull the 
        data from, via the `section` keyword.  This relies on the (undocumented) fact that
        `ArgumentParser.add_argument` actually returns the `Action` object that it creates.
        We need this to reliably get `dest` (although we could probably write a simple
        function to do this for us).
        """

        if 'action' in kwargs and kwargs['action'] == 'config':
            kwargs['action'] = AddConfigFile
            kwargs['default'] = argparse.SUPPRESS

        # argparse won't know what to do with the section, so 
        # we'll pop it out and add it back in later.
        #
        # We also have to prevent argparse from doing any type conversion,
        # which is done explicitly in parse_known_args.  
        #
        # This way, we can reliably check whether argparse has replaced the default.
        #
        section = kwargs.pop('section', self.default_section)
        type = kwargs.pop('type', _identity)
        default = kwargs.pop('default', _SENTINEL)

        if default is not argparse.SUPPRESS:
            kwargs.update(default=_SENTINEL)
        else:  
            kwargs.update(default=argparse.SUPPRESS)

        action = argparse.ArgumentParser.add_argument(self,*args,**kwargs)
        kwargs.update(section=section, type=type, default=default)
        self._action_defaults[action.dest] = (args,kwargs)
        return action

    def parse_known_args(self,args=None, namespace=None):
        # `parse_args` calls `parse_known_args`, so we should be okay with this...
        ns, argv = argparse.ArgumentParser.parse_known_args(self, args=args, namespace=namespace)
        config_parser = ConfigParser.SafeConfigParser()
        config_files = [os.path.expanduser(os.path.expandvars(x)) for x in self.config_files]
        config_parser.read(config_files)

        for dest,(args,init_dict) in self._action_defaults.items():
            type_converter = init_dict['type']
            default = init_dict['default']
            obj = default

            if getattr(ns,dest,_SENTINEL) is not _SENTINEL: # found on command line
                obj = getattr(ns,dest)
            else: # not found on commandline
                try:  # get from config file
                    obj = config_parser.get(init_dict['section'],dest)
                except (ConfigParser.NoSectionError, ConfigParser.NoOptionError): # Nope, not in config file
                    try: # get from environment
                        obj = os.environ[dest.upper()]
                    except KeyError:
                        pass

            if obj is _SENTINEL:
                setattr(ns,dest,None)
            elif obj is argparse.SUPPRESS:
                pass
            else:
                setattr(ns,dest,type_converter(obj))

        return ns, argv


if __name__ == '__main__':
    fake_config = """
[MAIN]
foo:bar
bar:1
"""
    with open('_config.file','w') as fout:
        fout.write(fake_config)

    parser = ArgumentConfigEnvParser()
    parser.add_argument('--config-file', action='config', help="location of config file")
    parser.add_argument('--foo', type=str, action='store', default="grape", help="don't know what foo does ...")
    parser.add_argument('--bar', type=int, default=7, action='store', help="This is an integer (I hope)")
    parser.add_argument('--baz', type=float, action='store', help="This is an float(I hope)")
    parser.add_argument('--qux', type=int, default='6', action='store', help="this is another int")
    ns = parser.parse_args([])

    parser_defaults = {'foo':"grape",'bar':7,'baz':None,'qux':6}
    config_defaults = {'foo':'bar','bar':1}
    env_defaults = {"baz":3.14159}

    # This should be the defaults we gave the parser
    print ns
    assert ns.__dict__ == parser_defaults

    # This should be the defaults we gave the parser + config defaults
    d = parser_defaults.copy()
    d.update(config_defaults)
    ns = parser.parse_args(['--config-file','_config.file'])
    print ns
    assert ns.__dict__ == d

    os.environ['BAZ'] = "3.14159"

    # This should be the parser defaults + config defaults + env_defaults
    d = parser_defaults.copy()
    d.update(config_defaults)
    d.update(env_defaults)
    ns = parser.parse_args(['--config-file','_config.file'])
    print ns
    assert ns.__dict__ == d

    # This should be the parser defaults + config defaults + env_defaults + commandline
    commandline = {'foo':'3','qux':4} 
    d = parser_defaults.copy()
    d.update(config_defaults)
    d.update(env_defaults)
    d.update(commandline)
    ns = parser.parse_args(['--config-file','_config.file','--foo=3','--qux=4'])
    print ns
    assert ns.__dict__ == d

    os.remove('_config.file')

TODO

This implementation is still incomplete. Here's a partial TODO list:

• (easy) Interaction with parser defaults
• (easy) If type conversion doesn't work, check against how argparse handles error messages

Conform to documented behavior

• (easy) Write a function that figures out dest from args in add_argument, instead of relying on the Action object
• (trivial) Write a parse_args function which uses parse_known_args. (e.g. copy parse_args from the cpython implementation to guarantee it calls parse_known_args.)

Less Easy Stuff…

I haven't tried any of this yet. It's unlikely—but still possible!—that it could just work…

• (hard?) Mutual Exclusion
• (hard?) Argument Groups (If implemented, these groups should get a section in the config file.)
• (hard?) Sub Commands (Sub-commands should also get a section in the config file.)

Solution 3

The argparse module makes this not nuts, as long as you're happy with a config file that looks like command line. (I think this is an advantage, because users will only have to learn one syntax.) Setting fromfile_prefix_chars to, for example, @, makes it so that,

my_prog --foo=bar

is equivalent to

my_prog @baz.conf

if @baz.conf is,

--foo
bar

You can even have your code look for foo.conf automatically by modifying argv

if os.path.exists('foo.conf'):
    argv = ['@foo.conf'] + argv
args = argparser.parse_args(argv)

The format of these configuration files is modifiable by making a subclass of ArgumentParser and adding a convert_arg_line_to_args method.

Solution 4

There's library that does exactly this called configglue.

configglue is a library that glues together python's optparse.OptionParser and ConfigParser.ConfigParser, so that you don't have to repeat yourself when you want to export the same options to a configuration file and a commandline interface.

It also supports environment variables.

There's also another library called ConfigArgParse which is

A drop-in replacement for argparse that allows options to also be set via config files and/or environment variables.

You might be interested in PyCon talk about configuration by Łukasz Langa - Let Them Configure!

Solution 5

While I haven't tried it by my own, there is ConfigArgParse library which states that it does most of things that you want:

A drop-in replacement for argparse that allows options to also be set via config files and/or environment variables.

View more solutions

Comments

[bignose]

This is rather further from “minimum deviation from the Python standard library” than I'd hoped for.

[Lemur]

Until someone provides a better alternative, this is the right answer. I've been using argparse, and didn't even look at this feature. Nice!

[jterrace]

but this doesn't have an answer for environment variables?

[Alex Szatmary]

@jterrace: This SO answer may work for you: stackoverflow.com/a/10551190/400793

[MarioVilas]

but still useful if you want to remain compatible with older versions of Python

[Piotr Dobrogost]

I asked if there are any plans to support argparse module.

[brent.payne]

do you mind throwing this into a github repo so everyone can improve on this?

[mgilson]

@brent.payne -- github.com/mgilson/configargparser -- If I'm gonna release this as real code, I decided to take a little time tonight to clean it up a bit. :-)

[mgilson]

FWIW, I finally got around to putting this on pypi -- You should be able to install it via pip install configargparser

[maxschlepzig]

I tried it, ConfigArgParse is very convenient and indeed a drop-in replacement.

[ErichBSchulz]

@mgilson - I updated your post. This package deserves more use!

[bignose]

> sorry, it's not argparse, but click is better and much more advanced […] Regardless of the merits of a third-party library, that makes this not an answer to the question.

[betontalpfa]

stackoverflow.com/a/14743159/2506522 answer is the same