How to fix "SystemError: <built-in function 'name'> returned NULL without setting an error" in Python C Extension

Question

Tools: Python3.7 (64 bit), Visual C++ 10.0 I am trying to create a C extension for Python. To start, I am testing a simple C code which prints a string and invokes the Sleep() function inside a for loop. However, when I make a simple call to this C function, named gen_nums, from Python, I get the following error:

"SystemError: built-in function gen_nums returned NULL without setting an error"

I think the problem is with the Sleep() function; deleting the "Sleep(1000)" part or placing it before "printf("Printed from C thread...\n")" eliminates this error. I looked over the documentation for Sleep() but couldn't find anything useful.

C Code:

#include <Python.h>

static void gen_nums() {
    int i;
    for(i = 0; i < 10; i++) {
        printf("Printed from C thread...\n");
        Sleep(1000);
    }
}
static PyMethodDef gen_numsmethods[] = {
    {"gen_nums", gen_nums, METH_VARARGS, "This is a threading test"},
    {NULL, NULL, 0, NULL}
};
static struct PyModuleDef threadmod = {
    PyModuleDef_HEAD_INIT,
    "threadrun",
    "This is a thread test module",
    -1,
    gen_numsmethods
};

PyMODINIT_FUNC PyInit_threadrun(void) {
    return PyModule_Create(&threadmod);
}

Python Call:

threadrun.gen_nums() \\ the C module is called threadrun

The result should be: "Printed from C thread..." 10 times, with a 1 second interval between each statement.

However, the program prints the statement 10 times and then displays the aforementioned error.

Solution

The reason for the error is this: Python extension functions must have a certain C prototype:

PyObject *func(PyObject *self, PyObject *args)

The method slots contain function pointers of type

PyObject *(*)(Pyobject *, PyObject *)

The old way was to forcibly cast the function to this pointer type to be stored into the method slot. The explicit cast will silence the error of conversion of void (*)() to PyObject *(*)(Pyobject *, PyObject *). The conversion is valid, but needs an explicit cast. If an explicit cast is not there, then a C compiler must issue a diagnostics message.

Your code does not have an explicit cast, hence you must get a warning for

{"gen_nums", gen_nums, METH_VARARGS, "This is a threading test"},

In any case, if there were an explicit cast, the program would still be a correct program up until when Python tries to invoke your function gen_nums(), because Python will do so as if its prototype were

PyObject *gen_nums(PyObject *, PyObject *);

Now the C standard says that while everything was fine up to this point, from now on the behaviour of the program is undefined, because C11 6.3.2.3:

8. A pointer to a function of one type may be converted to a pointer to a function of another type and back again; the result shall compare equal to the original pointer. If a converted pointer is used to call a function whose type is not compatible with the referenced type, the behavior is undefined.

Your function returns void, i.e. nothing at all, yet you ask "why it returns NULL only when Sleep() is in there. The reason is "undefined behaviour".

---

As for how to fix this, please do read and understand the Chapter 1 of 1. Extending Python with C or C++ - there are plenty of details in there, but everything needed to fix this simple function is detailed in there. If you get stuck please do ask further questions but do refer to the documentation in questions.

The fix for that function would be to write it as

static PyObject *gen_nums(PyObject *self, PyObject *args) {
    int i;
    for(i = 0; i < 10; i++) {
        printf("Printed from C thread...\n");
        Sleep(1000);
    }
    Py_RETURN_NONE;
}

Comments

[Antti Haapala -- Слава Україні]

that's not at all how a C extension function needs to be written. Please read the tutorial again and refer to it in further questions.

[Antti Haapala -- Слава Україні]

Most relevant information to answer this question is in the section 1.1 and 1.2, I cannot write a more perfect answer than to rip the entire text of sections 1.1 and 1.2 here verbatim, so it doesn't help much now.

[Wyatt T.]

I had actually properly created a C extension, just uploaded the entire C code now. However, what I don't understand is why the "gen_nums" returns NULL only when Sleep() is included.

[Antti Haapala -- Слава Україні]

Ok then please do ask it! You asked "how to fix"

[Wyatt T.]

I read over the documentation and it makes sense now. Thank you very much!