What's the best practice using a settings file in Python?

Question

I have a command line script that I run with a lot of arguments. I have now come to a point where I have too many arguments, and I want to have some arguments in dictionary form too.

So in order to simplify things I would like to run the script with a settings file instead. I don't really know what libraries to use for the parsing of the file. What's the best practice for doing this? I could of course hammer something out myself, but if there is some library for this, I'm all ears.

A few 'demands':

• Rather than using pickle I would like it to be a straight forward text file that can easily be read and edited.
• I want to be able to add dictionary-like data in it, i.e., some form of nesting should be supported.

A simplified pseudo example file:

truck:
    color: blue
    brand: ford
city: new york
cabriolet:
    color: black
    engine:
        cylinders: 8
        placement: mid
    doors: 2

Solution 1

You can have a regular Python module, say config.py, like this:

truck = dict(
    color = 'blue',
    brand = 'ford',
)
city = 'new york'
cabriolet = dict(
    color = 'black',
    engine = dict(
        cylinders = 8,
        placement = 'mid',
    ),
    doors = 2,
)

and use it like this:

import config
print(config.truck['color'])  

Solution 2

The sample config you provided is actually valid YAML. In fact, YAML meets all of your demands, is implemented in a large number of languages, and is extremely human friendly. I would highly recommend you use it. The PyYAML project provides a nice python module, that implements YAML.

To use the yaml module is extremely simple:

import yaml
config = yaml.safe_load(open("path/to/config.yml"))

Solution 3

I Found this the most useful and easy to use https://wiki.python.org/moin/ConfigParserExamples

You just create a "myfile.ini" like:

[SectionOne]
Status: Single
Name: Derek
Value: Yes
Age: 30
Single: True

[SectionTwo]
FavoriteColor=Green
[SectionThree]
FamilyName: Johnson

[Others]
Route: 66

And retrieve the data like:

>>> import ConfigParser
>>> Config = ConfigParser.ConfigParser()
>>> Config
<ConfigParser.ConfigParser instance at 0x00BA9B20>
>>> Config.read("myfile.ini")
['c:\\tomorrow.ini']
>>> Config.sections()
['Others', 'SectionThree', 'SectionOne', 'SectionTwo']
>>> Config.options('SectionOne')
['Status', 'Name', 'Value', 'Age', 'Single']
>>> Config.get('SectionOne', 'Status')
'Single'

Solution 4

Yaml and Json are the simplest and most commonly used file formats to store settings/config. PyYaml can be used to parse yaml. Json is already part of python from 2.5. Yaml is a superset of Json. Json will solve most uses cases except multi line strings where escaping is required. Yaml takes care of these cases too.

>>> import json
>>> config = {'handler' : 'adminhandler.py', 'timeoutsec' : 5 }
>>> json.dump(config, open('/tmp/config.json', 'w'))
>>> json.load(open('/tmp/config.json'))   
{u'handler': u'adminhandler.py', u'timeoutsec': 5}

View more solutions

Comments

[Benson]

While more or less equivalent, json isn't nearly as human readable as yaml. Since his sample config is actually valid yaml, I'd stress that instead of json.

[nikolay]

This is a pretty bad idea as if you want to allow low-privileged users to be able to change configuration files only, this way you're essentially allowing them to sneak in privileged code.

[XTL]

Allowing "low-privileged" users to change config for a more privileged program is probably a questionable setup anyway.

[phobie]

Using "json.dump(config, fp, sort_keys=True, indent=4)" improves readability.

[Javier Castellanos]

This gives no protection against "low-privileged" users changing the configuration. If you import the module at the beginning of a script, change the value of one of the variables and the import the rest of the modules you can modify the configuration values even if you don't have permission to write the configuration file.

[Chris]

Perhaps it is not in the current working directory (cwd). In that case, you have to make it visible to Python by either changing the cwd with os.chdir (use os.getcwd() to know where you are) or adding the config file location to the PYTHONPATH (not recommended). Hope this helps.

[Chris]

Otherwise, if the config file is in a subdir, you can turn that subdir into a package by placing an empty __init__.py file in it. Then you could import your config file with import subdir.config

[bschwagg]

You may also run into issues packaging your project for deployment using a tool such as py2app. The user may not be able to edit the configuration file once it's distributed since it would invalidate the app's signature.

[Todor Minakov]

yaml is always something I turn to; the format can be from dead simple to supporting embedded python code, and the standard library does the heavy lifting of parsing and sanitation for you.

[Jeremy]

If using python as a config file, you can use exec() to load it into a dictionary or imp.new_module to turn it into a module. This way the configuration is not in the package and can be placed in a system-standard config location if you prefer (like /etc). For more advanced usage you can also prepopulate the dict you pass to exec with objects that your config file can use as a simple DSL.

[Esteis]

Agreed. For you or users writing YAML, here is the best YAML reference that I know of. The official documentation is unfortunately a spec aimed at implementers, and nothing else, but Eevee's guide is fantastic.

[Apalala]

The main disadvantage with this (otherwise very convenient option) is that .py files are executable, so any kind of code could be run while trying to load the configuration through import. That's unacceptable from a security standpoint.

[André C. Andersen]

Can't a version of this be done safely with ast.literal_eval? docs.python.org/3/library/ast.html#ast.literal_eval

[Admin]

For Python 3 use the configparser module instead (all lowercase)

[Aleksandar]

This is the fastest, clearest and easiest to implement solution, since there is no implementation, just usage. :) Thank You!

[Hope]

A simple python class loading its data members from a JSON file in RAII manner

[Jakobovski]

(1) security is not always in issue, it really depends on the project. (2) A problem I found with config files like this is if you need to create them programaticlly. Then it is difficult

[user8675309]

For us uninitiated, that's pip3 install pyyaml to get it ready to import into python scripts.

[Gringo Suave]

Beware, yaml is only friendly if you keep it very simple, it by default has tons of problematic, bordering on unsafe features. Try hitchdev.com/strictyaml instead as a safe-by-default lite alternative.

[Jakub Bláha]

Would this support nested dictionaries as asked in the question though?

[Hans Ginzel]

See Munch, stackoverflow.com/questions/52570869/… import yaml; from munch import munchify; f = munchify(yaml.load(…)); print(fo.d.try)

[cambunctious]

@JakubBláha no.

[Begoodpy]

@HansGinzel You should make an answer on its own, as what you are suggesting is way easier to use

[Albert]

Note, there is the Python-like (Python dialect/subset) Starlark language, which is intended for this use case of config files (and used by Bazel). The main feature is hermetic execution, i.e. execution cannot access the file system, network, system clock. It is safe to execute untrusted code.

[user202729]

Actually import config might not always work, see python - How to import a module given the full path? - Stack Overflow

[JeopardyTempest]

As others sort of allude to, this seems to become a bit more complex if you're having multiple directories of scripts within your package and you want them all to reference the settings file, as there are plenty of topics about the complexities of relative imports in Python