Python build using wrong version of GCC on OS X

Question

I am attempting to build the python package pycrypto. OS X has gcc-4.2 installed and not gcc-4.0, but python continues to attempt to use gcc-4.0. How can I get it to use gcc-4.2? Or should I go about this a different way.

I am getting the following error:

bash-3.2$ 
bash-3.2$ sudo python setup.py build
running build
running build_py
running build_ext
warning: GMP library not found; Not building Crypto.PublicKey._fastmath.
building 'Crypto.Hash.MD2' extension
gcc-4.0 -fno-strict-aliasing -fno-common -dynamic -arch ppc -arch i386 -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c src/MD2.c -o build/temp.macosx-10.3-fat-2.6/src/MD2.o
unable to execute gcc-4.0: No such file or directory
error: command 'gcc-4.0' failed with exit status 1
bash-3.2$ 
bash-3.2$ 
bash-3.2$ 

I am using Mac OS X 10.6.7 with python 2.6.6 and XCode is installed.

EDIT: If I add CC=gcc-4.2, then I still get the error:

bash-3.2$ 
bash-3.2$ export CC=gcc-4.2
bash-3.2$ python setup.py build
running build
running build_py
running build_ext
warning: GMP library not found; Not building Crypto.PublicKey._fastmath.
building 'Crypto.Hash.MD2' extension
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -std=c99 -O3 -fomit-frame-pointer -arch i386 -arch x86_64 -Isrc/ -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c src/MD2.c -o build/temp.macosx-10.3-intel-2.6/src/MD2.o
gcc-4.0 -g -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 build/temp.macosx-10.3-intel-2.6/src/MD2.o -o build/lib.macosx-10.3-intel-2.6/Crypto/Hash/MD2.so
unable to execute gcc-4.0: No such file or directory
error: command 'gcc-4.0' failed with exit status 1
bash-3.2$ 

EDIT: It seems that using sudo makes a difference here.

I tried using both CC and CXX as suggest by Adam, and I get the following error without sudo:

bash-3.2$ python setup.py build
running build
running build_py
creating build/lib.macosx-10.3-fat-2.6
creating build/lib.macosx-10.3-fat-2.6/Crypto
copying lib/Crypto/__init__.py -> build/lib.macosx-10.3-fat-2.6/Crypto
copying lib/Crypto/pct_warnings.py -> build/lib.macosx-10.3-fat-2.6/Crypto
...
warning: GMP library not found; Not building Crypto.PublicKey._fastmath.
building 'Crypto.Hash.MD2' extension
creating build/temp.macosx-10.3-fat-2.6
creating build/temp.macosx-10.3-fat-2.6/src
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -arch ppc -arch i386 -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c src/MD2.c -o build/temp.macosx-10.3-fat-2.6/src/MD2.o
/usr/libexec/gcc/powerpc-apple-darwin10/4.2.1/as: assembler (/usr/bin/../libexec/gcc/darwin/ppc/as or /usr/bin/../local/libexec/gcc/darwin/ppc/as) for architecture ppc not installed
Installed assemblers are:
/usr/bin/../libexec/gcc/darwin/x86_64/as for architecture x86_64
/usr/bin/../libexec/gcc/darwin/i386/as for architecture i386
lipo: can't open input file: /var/tmp//ccxan625.out (No such file or directory)
error: command 'gcc-4.2' failed with exit status 1

If I use sudo, I get the following error where it tries to use 4.0:

bash-3.2$ sudo python setup.py build
Password:
running build
running build_py
running build_ext
warning: GMP library not found; Not building Crypto.PublicKey._fastmath.
building 'Crypto.Hash.MD2' extension
gcc-4.0 -fno-strict-aliasing -fno-common -dynamic -arch ppc -arch i386 -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c src/MD2.c -o build/temp.macosx-10.3-fat-2.6/src/MD2.o
unable to execute gcc-4.0: No such file or directory
error: command 'gcc-4.0' failed with exit status 1
bash-3.2$ 
bash-3.2$ 

Does this extra info make it more obvious what is going on and how to fix it? Any idea why the call without sudo is getting that other error?

Solution 1

Based on the path shown (/Library/Frameworks/Python.framework/Versions/2.6), it appears you have installed a 32-bit-only Python 2.6, perhaps using a python.org installer. When you build a Python package that includes a C extension module, the Python Distutils included with that Python instance will attempt to use the same version of gcc and the same CPU architectures that that Python was built with. Apparently you have installed the new cutting-edge Xcode 4 which no longer includes gcc-4.0 or ppc support. The Python versions you are using was built with the Xcode 3 tools included with Mac OS X 10.6. You may be able to work around it by overriding the compiler choice with:

export CC=gcc-4.2
python setup.py build
sudo python setup.py install

EDIT:

It looks like that is not going to work for pycrypto; its build is too compex. If you don't mind using the Apple-supplied Python 2.6 in OS X 10.6, this should work:

export ARCHFLAGS='-arch i386 -arch x86_64'
/usr/bin/python2.6 setup.py build

Another option would be to install the 64-bit/32-bit Python 2.7 installer from python.org. That is built with gcc-4.2 and is Intel-only so there shouldn't be any problems when using it with Xcode 4.

UPDATE:

Here are the exact steps I used with Xcode 3. They should work as well with Xcode 4 installed:

$ mkdir p
$ cd p
$ curl -O http://ftp.dlitz.net/pub/dlitz/crypto/pycrypto/pycrypto-2.3.tar.gz
$ tar xf pycrypto-2.3.tar.gz 
$ cd pycrypto-2.3/
$ export ARCHFLAGS='-arch i386 -arch x86_64'
$ /usr/bin/python2.6 setup.py build
running build
running build_py
creating build
creating build/lib.macosx-10.6-universal-2.6
creating build/lib.macosx-10.6-universal-2.6/Crypto
copying lib/Crypto/__init__.py -> build/lib.macosx-10.6-universal-2.6/Crypto
[...]
running build_ext
warning: GMP library not found; Not building Crypto.PublicKey._fastmath.
building 'Crypto.Hash.MD2' extension
creating build/temp.macosx-10.6-universal-2.6
creating build/temp.macosx-10.6-universal-2.6/src
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -fwrapv -Wall -Wstrict-prototypes -DENABLE_DTRACE -pipe -std=c99 -O3 -fomit-frame-pointer -arch i386 -arch x86_64 -Isrc/ -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c src/MD2.c -o build/temp.macosx-10.6-universal-2.6/src/MD2.o
gcc-4.2 -Wl,-F. -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 build/temp.macosx-10.6-universal-2.6/src/MD2.o -o build/lib.macosx-10.6-universal-2.6/Crypto/Hash/MD2.so
[...]
building 'Crypto.Util._counter' extension
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -fwrapv -Wall -Wstrict-prototypes -DENABLE_DTRACE -pipe -std=c99 -O3 -fomit-frame-pointer -arch i386 -arch x86_64 -Isrc/ -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c src/_counter.c -o build/temp.macosx-10.6-universal-2.6/src/_counter.o
gcc-4.2 -Wl,-F. -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 build/temp.macosx-10.6-universal-2.6/src/_counter.o -o build/lib.macosx-10.6-universal-2.6/Crypto/Util/_counter.so
$ /usr/bin/python2.6 setup.py install
running install
running build
running build_py
running build_ext
warning: GMP library not found; Not building Crypto.PublicKey._fastmath.
running install_lib
creating /Library/Python/2.6/site-packages/Crypto
[...]
byte-compiling /Library/Python/2.6/site-packages/Crypto/pct_warnings.py to pct_warnings.pyc
running install_egg_info
Writing /Library/Python/2.6/site-packages/pycrypto-2.3-py2.6.egg-info
$ /usr/bin/python2.6 setup.py test
running test
............................................................................................[...]
----------------------------------------------------------------------
Ran 902 tests in 42.612s

OK
$ cd 
$ /usr/bin/python2.6
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from Crypto.Hash import MD5
>>> m = MD5.new()
>>> m.update('abc')
>>> m.digest()
'\x90\x01P\x98<\xd2O\xb0\xd6\x96?}(\xe1\x7fr'
>>> m.hexdigest()
'900150983cd24fb0d6963f7d28e17f72'
>>> ^D

Solution 2

BTW, this is a distutils problem, Python itself doesn't compile anything.

As you've discovered, you can override the compiler with the CC environment variable. You can override the linker used with the CXX environment variable. I would also like to know why distutils acts this way, but so it does.

Solution 3

Focus on this error:

unable to execute gcc-4.0: No such file or directory
error: command 'gcc-4.0' failed with exit status 1

The compiler is telling you EXACTLY what the problem is. gcc-4.0 (and gcc-4.2) is not in your PATH. HINT: which gcc-4.2, e.g.:

% which gcc-4.2
/usr/bin/gcc-4.2

Assuming yours is in the same location, I'm not sure why /usr/bin isn't in your PATH, but there you have it!

Find out what your PATH actually is:

% echo $PATH
/opt/local/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/bin/X11:/sw/bin:/sw/sbin

Fix your path and you'll be on your path to enlightenment.

Comments

[Mark Tomlin]

If I add CC=gcc-4.2, then I still get the error. See the "EDIT" added to the question above. Any other ideas? Thanks in advance.

[Ned Deily]

Yeah, that simple solution is not going to work for pycrypto. See the updated answer for other options.

[Ned Deily]

The problem here is that the new Xcode 4 recently released by Apple no longer includes gcc-4.0 nor legacy ppc support.

[Ned Deily]

Distutils tries to ensure that any C extension modules are compiled and linked it a way that will be compatible with the way that the C modules in Python itself was compiled and linked. There are differences among platforms as to what is deemed necessary. On OS X, Distutils tries to ensure that a compatible set of CPU architectures are used, that the same OS X ABI is used (MACOSX_DEPLOYMENT_TARGET), and that the same GCC version is used (for example, the gcc-4.2 in OS X 10.6 does not support the ppc G3 architecture which is still supported by the 32-bit python.org installers), etc.

[Adam]

I understand that, but overriding the default behavior is very wonky. Note here that TWO environment variables are used, for no good reason that I can see. I also don't understand why env vars are used at all, instead of providing a plain old command line switch. (The --compiler switch only changes the flavor of compiler, so to speak, i.e. "unix"/vc/borland/etc. It doesn't let you specify a command). The documentation for all this is frustratingly incomplete, as I learned when trying to use mpicc as the compiler.

[Ned Deily]

No argument about the frustratingly incomplete Distutils documentation. As you may know, there is a Distutils2 project nearing completion that aims to replace Distutils in the Python 3.3 Standard Library and will also be installable in existing Pythons (2.4 -> 3.2). packages.python.org/Distutils2 would be the place to suggest or contribute improvements in the code and/or docs.

[Adam]

Actually, that is news to me. Hopefully they allow distributing the distutils2 package with the source package as my library is made for running on clusters with frighteningly old versions of software on them.

[Mark Tomlin]

That location is in my path. I have 4.2 installed, but gcc-4.0 is not on my machine. I don't know why it is looking for 4.0, and none of the solution for getting it to look for 4.2 have worked (such as setting CC and CXX environment vars).

[Mark Tomlin]

The ARCHFLAGS solution does not work for me. Also, I need to use python 2.6 because of one of the libraries I want to use. I installed python using the Mac Installer disk image found here: python.org/download/releases/2.6.6

[Mark Tomlin]

It seems that using CXX and not using sudo makes a difference. See my most recent edit. Any ideas what this other error is from? Thanks!

[Mark Tomlin]

It seems that using CXX and not using sudo makes a difference. See my most recent edit. Any ideas what this other error is from? Thanks!

[Ned Deily]

Mark: I explained the difference sudo makes in my answer to your other similar question. Generally for security reasons, a sudo process does not inherit most environment variables from its parent s,o if you define CC or CXX prior to the sudo steps, they won't be seen within the sudoed command. That's why I showed you how to get around that by doing sudo bash etc.

[Ned Deily]

The ARCHFLAGS option works for me using the system Python /usr/bin/python2.6. Are you properly cleaning the build area when you try it? Try starting from an empty directory and source tarball and show exactly how it fails.

[Adam]

@Mark, the difference between the sudo and non-sudo is that the sudo didn't pick up the compiler change. It's ok to build without sudo and just install with sudo. Your non-sudo issue is these switches -arch ppc -arch i386. It's trying to compile for PPC but you don't have a PPC assembler, so that fails. Distutils is using that switch because your Python was originally compiled with that switch. Unfortunately the only way to change that, that I know of, is to recompile Python. See if the --dry-run switch will print out all the commands and you can remove -arch ppc and run them manually.

[Adam]

Actually, maybe now Ned's ARCHFLAGS comment might help.

[ChrisCantrell]

Thank you so much. I typed these commands in as-is and it worked perfectly.