Cython with Visual Studio (Windoes SDK package) compiler cl.exe to create a .pyd

Question

I am trying to compile a .c file create by Cython to a .pyd file using cl.exe, which comes with the Microsoft Windows SDK.

Since this is the recommended package to build Cython under Windows 7 64 bit for Python 2.7 64 bit, I thought it would be a good idea to use the same compiler to create the .pyd files.

My current call uses a cythoncc.bat file with:

set CLPATH="C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin\amd64\cl.exe"
SET INCLUDE="C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\include";"C:\Program Files\Microsoft SDKs\Windows\v7.0\Include";"C:\Python27\include";"C:\Python27\Lib\site-packages\numpy\core\include";
SET LIB="C:\Python27\libs";"C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\lib\amd64";"C:\Program Files\Microsoft SDKs\Windows\v7.0\Lib\x64";
SET CFLAGS="/O2 /favor:<INTEL64>"
%CLPATH% %CFLAGS% /Fo%1.pyd %2.c

I think the problem is in the last line, but not sure yet... it gives me the error:

/out:_conecyl.exe
_conecyl.pyd
   Creating library _conecyl.lib and object _conecyl.exp
LIBCMT.lib(crt0.obj) : error LNK2019: unresolved external symbol main referenced
 in function __tmainCRTStartup
_conecyl.exe : fatal error LNK1120: 1 unresolved externals

And creates a .pyd plus a .lib file.

Any help will be very appreciated! Thank you!

Solution 1

Use the setup.py recipe explained in the documentation and here

---

EDIT by Saullo:

the setup.py looked just like this:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules = [Extension("_conecyl", ["_conecyl.pyx"])]

setup(
  name = '_conecyl',
  cmdclass = {'build_ext': build_ext},
  ext_modules = ext_modules
)

And with the following calls it worked just perfect:

setenv /x64 / release
set INCLUDE=%INCLUDE%;C:\Python27\include;C:\Python27\Lib\site-packages\numpy\core\include
set LIB=%LIB%;C:\Python27\libs
set DISTUTILS_USE_SDK=1
python setup.py install

Solution 2

I have been battling with this Cython pyd creation on Windows and have a workable solution. This post comes up high on a search for 'build cython pyd visual studio windows', so I thought I'd add my method to save people some rage & tears.

My requirements are as follows:

• Most of my code written in Python with PyCharm
• I use Cython to speed-up certain calculations involving fixed-size arrays of floats
• Debugging Cython (which is another topic!) is doable but not user friendly. I find the best way is to continually add/remove log/print statements in my pyx and rerun my dev Python script. Obviously I want a quick turnaround from changing the pyx file to having a linked pyd file to use in Python again.

Here's what I do. Say we want to create a new Cython module:

1. Create a python file with build and architecture info. e.g. functions_362_64.py. I now write a good skeleton of functionality in Python. I would advise writing some tests and get them to pass - this will be much harder once in Cython space. Your tests use 'from xxx.yyy.zzz.functions_362_64.py import my_function1' etcetera. Now change your file name to 'functions_362_64.pyx'.

2. Run 'cythonize.exe -a functions_362_64.pyx' to create 2 files. The html explains what Cython did to your code in a human-readable form - it is very useful for debugging. Also you get your C code in 'functions_362_64.c'.

3. Now we use Visual Studio to create the pyd. Read all the stuff about getting the matching versions. I currently use MSVS2017 with Python32. Create a blank solution. Create one C++ dll project for each pyd file. So here we create a project called 'functions_362_64'. Make the following kind of changes to get a useful compile and link:

   • include directory from your Python venv
   • libs directory from your Python env
   • change the output name to pyd
   • link debug symbols in Release build
   • no precompiled headers
   • no short-builds or optimization until you have finished testing
   • post-build step to copy your pyd/pbd to Python import directory This is all pretty standard stuff (you can find it in many MSVS posts).

You should now have 'functions_362_64.pyd' with 'functions_362_64.pdb'. Check it works in place of your original py file - the import doesn't change.

4. Now we want a batch file to automate everything - mine has 8 lines. What you need to do is get the build commands that visual studio uses in your project. Go to Tools->Options->Projects and Solutions->Build and Run->Output verbosity and set to Detailed. Build the project and search the successful output for PATH, LIB, LIBPATH, INCLUDE, cl.exe and copy. Then paste the correct lines into your batch file, and add "" and 'SET ' where needed. Mine looks like this:
   • line-1: "C:...your_path...\Scripts\cythonize.exe" -a "C:...your_path...\functions_362_64.pyx"
   • line-2: SET PATH=C:..... paste from build output .....
   • line-3: SET LIB=C:..... paste from build output .....
   • line-4: SET LIBPATH=C:..... paste from build output .....
   • line-5: SET INCLUDE=C:..... paste from build output .....
   • line-6: "C:...microsoft_path...\CL.exe" /c ..... paste from build output .....
   • line-7: copy /Y "C:..... post-build copy pyd .....
   • line-8: copy /Y "C:..... post-build copy pdb .....

Put this .bat in your Visual Studio project directory. Get a command prompt to sit in there.

Now you have it all:

• change pyx file
• run .bat
• run Python again

It all very quick. You can even get PyCharm or whatever IDE you use to run this script as an external tool. I find it easier to just keep running the .bat file.

Well I hope this saves you guys some time and trouble. I was almost considering swicthing to linux before getting this process working ....... :)

Thanks, LJHW

REVISION BY AUTHOR: 23/08/2017

I now use an x86 or x64 Visual Studio Native Tools Command Prompt - I think these come with an SDK install as well as full MSVS. This takes care of all the ENV issues, which is what seems to cause most people problems. By doing this everything works out-of-the-box in a cross-platform way so that one is not implementing specific windows hacks. I also do a bit of tidying, so I use a .bat file.

• Add windows_setup.bat to package directory
• Run this in the VS Native prompt

In this .bat file add:

1. del ...\functions.pyd
2. python.exe setup.py build_ext --inplace
3. del ...\functions.c
4. rename functions.cp36-win_amd64.pyd functions.pyd

Now, I have simply converted pyx to pyd and have no intermediary files. The architecture of python.exe must match cmd.exe. The beauty of this is that you can run all the linux-based examples out there on the web - you just update your setup.py file accordingly and setuptools/disutils will take care of the windows specific stuff because you have the correct ENV settings in the native prompt.

So to develop, I go:

• change pyx file
• run windows_setup.bat
• develop in python code again

If I need to debug, I don't delete the c-file, load functions.c in Visual Studio and build the pdb as described above in the original post. I don't see why this system wouldn't work with say Python32 & MSVC10 by just using the 10.0 native prompt.

Comments

[LJHW]

I have revised my view on this since using it practically. It is too 'hacky' for my liking and proj build doesn't always detect updates to the c file. I am now using the following solution, which is much more 'out of the box'.