Import constants from .h file into python

Question

I've been looking for a simple answer to this question, but it seems that I can't find one. I would prefer to stay away from any external libraries that aren't already included in Python 2.6/2.7.

I have 2 c header files that resemble the following:

//constants_a.h
const double constant1 = 2.25;
const double constant2 = -0.173;
const int constant3 = 13;

...

//constants_b.h
const double constant1 = 123.25;
const double constant2 = -0.12373;
const int constant3 = 14;

...

And I have a python class that I want to import these constants into:

#pythonclass.py
class MyObject(object):
    def __init(self, mode):
        if mode is "a":
            # import from constants_a.h, like:
            # self.constant1 = constant1
            # self.constant2 = constant2
        elif mode is "b":
            # import from constants_b.h, like:
            # self.constant1 = constant1
            # self.constant2 = constant2

...

I have c code which uses the constants as well, and resembles this:

//computations.c
#include <stdio.h>
#include <math.h>
#include "constants_a.h"

// do some calculations, blah blah blah

How can I import the constants from the header file into the Python class?

The reason for the header files constants_a.h and constants_b.h is that I am using python to do most of the calculations using the constants, but at one point I need to use C to do more optimized calculations. At this point I am using ctypes to wrap the c code into Python. I want to keep the constants away from the code just in case I need to update or change them, and make my code much cleaner as well. I don't know if it helps to note I am also using NumPy, but other than that, no other non-standard Python extensions. I am also open to any suggestions regarding the design or architecture of this program.

Solution 1

I recommend using regular expressions (re module) to parse the information you want out of the files.

Building a full C parser would be huge, but if you only use the variables and the file is reasonably simple/predictable/under control, then what you need to write is straightforward.

Just watch out for 'gotcha' artifacts such as commented-out code!

Solution 2

In general, defining variables in C header file is poor style. The header file should only declare objects, leaving their definition for the appropriate ".c" source code file.

One thing you may want to do is to declare the library-global constants like extern const whatever_type_t foo; and define (or "implement") them (i.e. assigning values to them) somewhere in your C code (make sure you do this only once).

Anyway, let's ignore how you do it. Just suppose you've already defined the constants and made their symbols visible in your shared object file "libfoo.so". Let us suppose you want to access the symbol pi, defined as extern const double pi = 3.1415926; in libfoo, from your Python code.

Now you typically load your object file in Python using ctypes like this:

>>> import ctypes
>>> libfoo = ctypes.CDLL("path/to/libfoo.so")

But then you'll see ctypes thinks libfoo.pi is a function, not a symbol for constant data!

>>> libfoo.pi
<_FuncPtr object at 0x1c9c6d0>

To access its value, you have to do something rather awkward -- casting what ctypes thinks is a function back to a number.

>>> pi = ctypes.cast(foo.pi, ctypes.POINTER(ctypes.c_double))
>>> pi.contents.value
3.1415926

In C jargon, this vaguely corresponds to the following thing happening: You have a const double pi, but someone forces you to use it only via a function pointer:

typedef int (*view_anything_as_a_function_t)(void);
view_anyting_as_a_function_t pi_view = π

What do you do with the pointer pi_view in order to use the value of pi? You cast it back as a const double * and dereference it: *(const double *)(pi_view).

So this is all very awkward. Maybe I'm missing something but this I believe is by design of the ctypes module -- it's there chiefly for making foreign function calls, not for accessing "foreign" data. And exporting pure data symbol in a loadable library is arguably rare.

And this will not work if the constants are only C macro definitions. There's in general no way you can access macro-defined data externally. They're macro-expanded at compile time, leaving no visible symbol in the generated library file, unless you also export their macro values in your C code.

Solution 3

I would recommend using some kind of configuration file readable by both Python and C program, rather than storing constant values in headers. E.g. a simple csv, ini-file, or even your own simple format of 'key:value' pairs. And there will be no need to recompile the C program every time you'd like to change one of the values :)

Solution 4

I'd up-vote emilio, but I'm lacking rep!

Although you have requested to avoid other non-standard libraries, you may wish to take a look at Cython (Cython: C-Extensions for Python www.cython.org/), which offers the flexibility of Python coding and the raw speed of execution of C/C++-compiled code.

This way you can use regular Python for everything, but handle the expensive elements of code using its built-in C-types. You can then convert your Python code into .c files too (or just wrap external C-libraries themselves. ), which can then be compiled into a binary. I've achieved up to 10x speed-ups doing so for numerical routines. I also believe NumPy uses it.

View more solutions

Comments

[Eryk Sun]

Have a look at the Tools/Scripts/h2py.py script, revised since 1992.

[Cong Ma]

Well I just found out the non-awkward way to do this: python.net/crew/theller/ctypes/… You can try pi = ctypes.c_double(libfoo, "pi") in Python code; this returns a ctypes.c_double instance, and you can access its value by pi.value.

[Manila Thrilla]

Thanks Cong for your very detailed answer. I think accessing the values of the instances would be suitable for my work. It's a scientific endeavor rather than an actual software release, so I'm going for ease and clarity. I am curious, though, as to what you would consider a better style for defining constants. Keep in mind that these constants should be in a separate file so that it is a simple matter to quickly edit or change their values.

[Manila Thrilla]

This sounds great, but I think then I would have to either create a parser for the C program, which I don't want to do, or use a library like LibConfig, which I also do not want to do. It would be nice, though, to not have to recompile the C program every time. Do you possibly have an example for this configuration file?

[Manila Thrilla]

Well I originally implemented my code in Cython, but like I stated in my question, I am using ctypes now to maintain portability. The problem is that Cython is not available in the standard Python library, but ctypes is. But you're right, Cython is awesome and very easy to use. I did find ctypes to be a few microseconds faster than cython though.

[Manila Thrilla]

Thanks Emilio and eryksun. Until I saw Cong's answer, I was believing this was the way to go. It would be a good exercise to write a small parser for this purpose too.

[Emilio M Bumachar]

@ManilaThrilla: You are very welcome. I have read and upvoted Cong's answer, it is very informative. Since a better answer is unlikely to come up after all this time, I suggest you mark Cong's as accepted.

[Zaur Nasibov]

A JSON-formatted config as an example. Supported by Python's standard library and lots of implementations for C are available on json.org

[Manila Thrilla]

I actually used this implementation rather than Cong's. I found this to be a bit more adaptive to changing/adding constants and parameters. Therefore, I have chosen this as the best answer now.