Embedding Python in C++ and calling methods from the C++ code with Boost.Python

Question

I try to embed a Python script into my C++ program. After reading some things about embedding and extending I understand how to open my own python script and how to pass some integers to it. But now I'm at a point a do not understand how to resolve my problem. I have to do both, calling Python functions from C++ and calling C++ functions from my embedded Python script. But I do not know where I have to start. I know I have to compile a .so file to expose my C++ functions to Python but this is nothing I can do, because I have to embed my Python file and control it by using C++ code (I have to extend a large software with a script language, to make some logic easy to edit).

So, is there any way to do both things? Calling Python functions from C++ and calling C++ functions from Python?

This is my C++ code

#include <Python.h>
#include <boost/python.hpp>
using namespace boost::python;


// <----------I want to use this struct in my python file---------
struct World
{
    void set(std::string msg) { this->msg = msg; }
    std::string greet() { return msg; }
    std::string msg;
};


// Exposing the function like its explained in the boost.python manual
// but this needs to be compiled to a .so to be read from the multiply.py
BOOST_PYTHON_MODULE(hello)
{
    class_<World>("World")
        .def("greet", &World::greet)
        .def("set", &World::set)
    ;
}
// <---------------------------------------------------------------


int
main(int argc, char *argv[]) // in the main function is only code for embedding the python file, its not relevant to this question
{
    setenv("PYTHONPATH",".",1);
    PyObject *pName, *pModule, *pDict, *pFunc;
    PyObject *pArgs, *pValue;
    int i;

    if (argc < 3) {
        fprintf(stderr,"Usage: call pythonfile funcname [args]\n");
        return 1;
    }

    Py_Initialize();
    pName = PyString_FromString(argv[1]);
    /* Error checking of pName left out */

    pModule = PyImport_Import(pName);
    Py_DECREF(pName);

    if (pModule != NULL) {
        pFunc = PyObject_GetAttrString(pModule, argv[2]);
        /* pFunc is a new reference */

        if (pFunc && PyCallable_Check(pFunc)) {
            pArgs = PyTuple_New(argc - 3);
            for (i = 0; i < argc - 3; ++i) {
                pValue = PyInt_FromLong(atoi(argv[i + 3]));
                if (!pValue) {
                    Py_DECREF(pArgs);
                    Py_DECREF(pModule);
                    fprintf(stderr, "Cannot convert argument\n");
                    return 1;
                }
                /* pValue reference stolen here: */
                PyTuple_SetItem(pArgs, i, pValue);
            }
            pValue = PyObject_CallObject(pFunc, pArgs);
            Py_DECREF(pArgs);
            if (pValue != NULL) {
                printf("Result of call: %ld\n", PyInt_AsLong(pValue));
                Py_DECREF(pValue);
            }
            else {
                Py_DECREF(pFunc);
                Py_DECREF(pModule);
                PyErr_Print();
                fprintf(stderr,"Call failed\n");
                return 1;
            }
        }
        else {
            if (PyErr_Occurred())
                PyErr_Print();
            fprintf(stderr, "Cannot find function \"%s\"\n", argv[2]);
        }
        Py_XDECREF(pFunc);
        Py_DECREF(pModule);
    }
    else {
        PyErr_Print();
        fprintf(stderr, "Failed to load \"%s\"\n", argv[1]);
        return 1;
    }
    Py_Finalize();
    return 0;
}

and this is my Python file

import hello_ext #importing the C++ file works only if its compiled as a .so
planet = hello.World() #this class should be exposed to python
planet.set('foo')

def multiply(a,b):
    planet.greet()
    print "Will compute", a, "times", b
    c = 0
    for i in range(0, a):
        c = c + b
    return c

Solution

In short, Python extensions that are statically linked with embedded Python need to have their module initializer function explicitly added to the initialization table before the interpreter is initialized.

PyImport_AppendInittab("hello", &inithello);
Py_Initialize();

---

Boost.Python uses the BOOST_PYTHON_MODULE macro to define a Python module initializer. The resulting function is not the module importer. This difference is similar to that of creating a example.py module and calling import example.

When importing a module, Python will first check if the module is a built-in module. If the module is not there, then Python will then search the module search path trying to find a python file or library based on the module name. If a library is found, then Python expects the library to provide a function that will initialize the module. Once found, the import will create an empty module in the modules table, then initialize it. For statically linked modules, such as hello in the original code, the module search path will not be helpful, as there is no library for it to find.

For embedding, the module table and initialization function documentation states that for static modules, the module initializer function will not be automatically called unless there is an entry in the initialization table. For Python 2 and Python 3, one can accomplish this by calling PyImport_AppendInittab() before Py_Initialize():

BOOST_PYTHON_MODULE(hello)
{
  // ...
}

PyImport_AppendInittab("hello", &inithello);
Py_Initialize();
// ...
boost::python::object hello = boost::python::import("hello");

Also note that the Python's C API for embedding changed naming conventions for module initialization functions between Python 2 and 3, so for BOOST_PYTHON_MODULE(hello), one may need to use &inithello for Python 2 and &PyInit_hello for Python 3.

---

Here is a complete example demonstrating having an embedded Python import a demo user module, that will then import a statically linked hello module. It also invokes a function in the user module demo.multiply, that will then invoke a method exposed through the statically linked module.

#include <cstdlib>  // setenv, atoi
#include <iostream> // cerr, cout, endl
#include <boost/python.hpp>

struct World
{
  void set(std::string msg) { this->msg = msg; }
  std::string greet()       { return msg;      }
  std::string msg;
};

/// Staticly linking a Python extension for embedded Python.
BOOST_PYTHON_MODULE(hello)
{
  namespace python = boost::python;
  python::class_<World>("World")
    .def("greet", &World::greet)
    .def("set", &World::set)
    ;
}

int main(int argc, char *argv[])
{
  if (argc < 3)
  {
    std::cerr << "Usage: call pythonfile funcname [args]" << std::endl;
    return 1;
  }
  char* module_name   = argv[1];
  char* function_name = argv[2];

  // Explicitly add initializers for staticly linked modules.
  PyImport_AppendInittab("hello", &inithello);

  // Initialize Python.
  setenv("PYTHONPATH", ".", 1);
  Py_Initialize();

  namespace python = boost::python;
  try
  {
    // Convert remaining args into a Python list of integers.
    python::list args;
    for (int i=3; i < argc; ++i)
    {
      args.append(std::atoi(argv[i]));
    }

    // Import the user requested module.
    // >>> import module
    python::object module = python::import(module_name);

    // Invoke the user requested function with the provided arguments.
    // >>> result = module.fn(*args)
    python::object result = module.attr(function_name)(*python::tuple(args));

    // Print the result.
    std::cout << python::extract<int>(result)() << std::endl;
  }
  catch (const python::error_already_set&)
  {
    PyErr_Print();
    return 1;
  }

  // Do not call Py_Finalize() with Boost.Python.
}

Contents of demo.py:

import hello
planet = hello.World()
planet.set('foo')

def multiply(a,b):
    print planet.greet()
    print "Will compute", a, "times", b
    c = 0
    for i in range(0, a):
        c = c + b
    return c

Usage:

$ ./a.out demo multiply 21 2
foo
Will compute 21 times 2
42

In the above code, I opted to use Boost.Python instead of the Python/C API, with the C++ comments annotated with the equivalent Python code. I find it to be much more succinct and far less error prone. If a Python error occurs, Boost.Python will throw an exception and all reference counting will be handled appropriately.

Also, when using Boost.Python, do not invoke Py_Finalize(). Per the Embedding - Getting started section:

Note that at this time you must not call Py_Finalize() to stop the interpreter. This may be fixed in a future version of boost.python.