Calling python method from C++ (or C) callback

Question

I am trying to call methods in a python class from C++. The C++ method from which this is called is a C++ callback.

Within this method when I am trying to call python method, it was giving segmentation fault.

I have saved an instance of python function in a global variable like

// (pFunc is global variable of type PyObject*)
pFunc = PyDict_GetItemString(pDict, "PlxMsgWrapper");

where PlxMsgWrapper is a python method, which will be used in the callback.

In the callback, the arguments are created as

PyObject* args = PyTuple_Pack(2, PyString_FromString(header.c_str()),
                                 PyString_FromString(payload.c_str()));

When creating the

PyObject * pInstance = PyObject_CallObject(pFunc, args);

In this line its giving segmentation fault. After this the actual python method is called as

PyObject* recv_msg_func = PyObject_GetAttrString(module, (char *)"recvCallback");
args = PyTuple_Pack(1, pInstance);
PyObject_CallObject(recv_msg_func, args);

Solution 1

There are a few things you need to do if you are invoking a Python function from a C/C++ callback. First when you save off your python function object, you need to increment the reference count with:

Py_INCREF(pFunc)

Otherwise Python has no idea you are holding onto an object reference, and it may garbage collect it, resulting in a segmentation fault when you try to use it from your callback.

Then next thing you need to be concerned about is what thread is running when your C/C++ callback is invoked. If you are getting called back from another non-Python created thread (i.e. a C/C++ thread receiving data on a socket), then you MUST acquire Python's Global Interpreter Lock (GIL) before calling any Python API functions. Otherwise your program's behavior is undefined. To acquire the GIL you do:

void callback() {
    PyGILState_STATE gstate;
    gstate = PyGILState_Ensure();

    // Get args, etc.

    // Call your Python function object
    PyObject * pInstance = PyObject_CallObject(pFunc, args);

    // Do any other needed Python API operations

    // Release the thread. No Python API allowed beyond this point.
    PyGILState_Release(gstate);
}

Also, in your extension module's init function, you should do the following to ensure that threading is properly initialized:

// Make sure the GIL has been created since we need to acquire it in our
// callback to safely call into the python application.
if (! PyEval_ThreadsInitialized()) {
    PyEval_InitThreads();
}

Otherwise, crashes and strange behavior may ensue when you attempt to acquire the GIL from a non-Python thread.

See Non-Python Created Threads for more detail on this.

Solution 2

Python should look for a module in the directory where it is being run from, however, if you think that the issue is that python is not finding your file, you can add an arbitrary directory on your computer to the module search path within your program:

// Initialize the Python Interpreter
Py_Initialize();

// The following two lines to the trick:
// add path to your module to python's search paths
PyRun_SimpleString("import sys");
PyRun_SimpleString("sys.path.append(\"/path/to/python/module/here\")");

// Build the name object
pName = PyString_FromString("your_module");

// Load the module object
pModule = PyImport_Import(pName);

// pDict is a borrowed reference 
pDict = PyModule_GetDict(pModule);

// pFunc is also a borrowed reference 
pFunc = PyDict_GetItemString(pDict, "PlxMsgWrapper");

pArgs = ... 

if (PyCallable_Check(pFunc)) 
{
   PyObject_CallObject(pFunc, pArgs);
} else {
   PyErr_Print();
}

Solution 3

This doesn't exactly answer your question, but you can greatly simplify your code and avoid reference count issues with Boost::Python.

#include "boost/python.hpp"

using namespace boost::python;

int main()
{
  Py_Initialize();

  object pyFunPlxMsgWrapper = import("your_module").attr("PlxMsgWrapper");
  pyFunPlxMsgWrapper(2, "string", "data");
  return 0;
}

Comments

[Chaitanya]

Thanks a lot. :) That was the exact problem.

[eddiewould]

Thanks a lot - hadn't considered I might be in the wrong thread.