Adding and reading a config.ini file inside python package

Question

I am writing my first python package which I want to upload on PyPI. I structured my code based on this blog post.

I want to store user setting in a config.ini file. Read it once(every time the package is run) in separate python module in same package and save user setting in global variables of that module. Later import those in other modules.

To recreate the error I just edited few lines of code, in the template described in the blog post. (Please refer to it since it would take too much typing to recreate entire thing here in question.)

The only difference is that my stuff.py reads from config file like this:

from ConfigParser import SafeConfigParser

config = SafeConfigParser()
config.read('config.ini')

TEST_KEY = config.get('main', 'test_key')

Here are the contents of config.ini (placed in same dir as stuff.py):

[main]
test_key=value

And my bootstrap.py just imports and print the TEST_KEY

from .stuff import TEST_KEY

def main():
    print(TEST_KEY)

But on executing the package, the import fails give this error

Traceback (most recent call last):
    File "D:\Coding\bootstrap\bootstrap-runner.py", line 8, in <module>
    from bootstrap.bootstrap import main
    File "D:\Coding\bootstrap\bootstrap\bootstrap.py", line 11, in <module>
    from .stuff import TEST_KEY
    File "D:\Coding\bootstrap\bootstrap\stuff.py", line 14, in <module>
    TEST_KEY = config.get('main', 'test_key')
    File "C:\Python27\Lib\ConfigParser.py", line 607, in get
    raise NoSectionError(section)
ConfigParser.NoSectionError: No section: 'main'

Import keeps giving ConfigParser.NoSectionError, but if you build/run only stuff.py(I use sublime3), the module gives no errors and printing TEST_KEY gives value as output.

Also, this method of import does work when I just use 3 files(config, stuff, main) in a dir and just execute the main as a script. But there I had to import it like this

from stuff import TEST_KEY

I'm just using the explicit relative imports as described in that post but don't have enough understanding of those. I guess the error is due to project structure and import, since running stuff.py as standalone script raises no ConfigParser.NoSectionError.

Other method to read the config file once and then use data in other modules will be really helpful as well.

Solution 1

There are two aspects to this question. First is the weird behavior of ConfigParser. When ConfigParser is unable to locate the .ini file; it never gives, for some annoying reason, an IOError or an error which indicates that it is unable to read the file.

In my case it keeps giving ConfigParser.NoSectionError when the section is clearly present. When I caught the ConfigParser.NoSectionError error it gave an ImportError! But it never tells you that it is simply unable to read the file.

Second is how to safely read the data files that are included in your package. The only way I found to do this was to use the __file__ parameter. This is how you would safely read the config.ini in the above question, for Python27 and Python3:

import os

try:
    # >3.2
    from configparser import ConfigParser
except ImportError:
    # python27
    # Refer to the older SafeConfigParser as ConfigParser
    from ConfigParser import SafeConfigParser as ConfigParser

config = ConfigParser()

# get the path to config.ini
config_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'config.ini')

# check if the path is to a valid file
if not os.path.isfile(config_path):
    raise BadConfigError # not a standard python exception

config.read(config_path)

TEST_KEY = config.get('main', 'test_key') # value

This relies on the fact that config.ini is located inside our package bootstrap and is expected to be shipped with it.

The important bit is how you get the config_path:

config_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'config.ini')

__file__ refers to the location of current script that is being executed. In my question that means the location of stuff.py, that is inside the bootstrap folder, and so is config.ini.

The above line of code then means; get the absolute path to stuff.py; from that get path to the directory containing it; and join that with config.ini (since it is in same directory) to give absolute path to the config.ini. Then you can proceed to read it and raise an exception just in case.

This will work even when you release your package on pip and a user installs it from there.

As a bonus, and digressing from the question a bit, if you are releasing your package on pip with data files inside your package, you must tell setuptools to include them inside you package when you build sdist and bdists. So to include the config.ini in above package add the following lines to setup class call in setup.py:

include_package_data = True,
package_data = {
    # If any package contains *.ini files, include them
    '': ['*.ini'],
},

But it still may not work in some cases eg. building wheels etc. So you also do the same in your MANIFEST.IN file:

include LICENSE
include bootstrap/*.ini

Solution 2

abhimanyuPathania : The issue is with path of config.ini in stuff.py. Change config.read('config.ini') to config.read('./bootstrap/config.ini') in stuff.py. I tried the solution. It works for me.

Enjoying Pythoning...

Comments

[Paul Cornelius]

I'm not sure if this is your problem, but the ConfigParser.read() function does not raise an Exception if the file can't be found or there is some kind of IO error (see the docs). If it can't read the file, it will fail silently and just initialize to an empty dictionary,

[Abhimanyu Pathania]

@PaulCornelius But if you run the stuff.py alone it does read the ini file and gives not errors. Only when I'm importing TEST_KEY like that, it raises that error.

[Paul Cornelius]

That's my point. I thought it might be useful to isolate the reason why the program is failing, which you can do by inspecting the returned value from ConfigParser.read(). I can't follow what you're saying about standalone versus import versus files, etc. I don't know what you mean by "when I execute the package."

[Abhimanyu Pathania]

yes it does work. But now if I build stuff.py standalone, it gives the same error. So issue is definitely with the path. Can you please edit your answer and explain a bit more?

[Srikanth Lankapalli]

Sure... What steps are you doing to build in stand alone and how are you executing them... Please elaborate

[Abhimanyu Pathania]

By build I mean just open stuff.py in sublime and (Ctrl + B) to run it. Or you can open cmd prompt, cd to inner bootstrap dir and then: python stuff.py

[Srikanth Lankapalli]

@abhimanyuPathania The path is relative ..it depends on who is calling .... if you are executing python stuff.py then it is config.ini if you are executing python -m bootstrap then it is ./bootstrap/config.ini

[Abhimanyu Pathania]

ok. Thanks. I will dig a bit more into relative and absolute paths and will mark your answer as accepted later.