gcc running out of memory in Ubuntu 12.10

Question

I'm trying to compile my C project in Ubuntu (32bit) running in VirtualBox. Compiling the code in question requires quite a bit of memory, at least 3 gig. So I gave the VM 2 gig. Along with 2 gig of swap space available, it should be enough. For some reason though gcc is failing with an out -of-memory error after it has allocated 900 meg or so. Bumping the amount of memory to 2.7 gig (the maximum allowed by VirtualBox) didn't help. There seems to be a limit on how much memory a process can use. But when I run ulimit, it shows "unlimited".

UPDATE - Here's the make log:

libtool: compile: cc -msse2 -I. -I/home/cleong/qb -DPHP_ATOM_INC -I/home/cleong/qb/include -I/home/cleong/qb/main -I/home/cleong/qb -I/usr/include/php5 -I/usr/include/php5/main -I/usr/include/php5/TSRM -I/usr/include/php5/Zend -I/usr/include/php5/ext -I/usr/include/php5/ext/date/lib -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DHAVE_CONFIG_H -g -O2 -c /home/cleong/qb/qb_interpreter_gcc.c -fPIC -DPIC -o .libs/qb_interpreter_gcc.o

cc1: out of memory allocating 408 bytes after a total of 924852224 bytes

make: *** [qb_interpreter_gcc.lo] Error 1

Output from /bin/time -v:

Command exited with non-zero status 2
    Command being timed: "make"
    User time (seconds): 62.09
    System time (seconds): 11.28
    Percent of CPU this job got: 64%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 1:53.02
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
    Maximum resident set size (kbytes): 1848592
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 9433
    Minor (reclaiming a frame) page faults: 1391779
    Voluntary context switches: 5642
    Involuntary context switches: 6069
    Swaps: 0
    File system inputs: 630360
    File system outputs: 1376
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 2

The version of gcc is 4.7.2.

The source code is here:

https://github.com/chung-leong/qb

It's a PHP extension. The build process uses phpize.

UPDATE - I'm encountering the same problem in 32-bit Mageia 3, which also includes gcc 4.7.2. The failure happens at around 2.7 gig instead. In a 64-bit environment, it doesn't happen.

If I install gcc 4.6.3 and build using that, it works.

Solution

The fact that with optimization level of -O1 or lower, 64-bit Ubuntu manages to compile the project, means that the problem lies with the optimization phase of gcc.

I would guess that some source-file in your project happens upon a bug or inefficiency in the gcc optimizer introduced in version 4.7.2. which causes the use of an unreasonable amount of memory.

So what I would suggest is :

1. A 32-bit version of gcc can only access 2-3 GB of memory (whatever size is the swap).
2. Enlarge the swap space, but keep on reading if this doesn't help. In any case, for a 32-bit gcc there will be a limit to the amount of memory that it can use.
3. By compiling the source files one-by-one, find the one that is responsible for the problem. If this is all of them, then the problem is with some include file(s).
4. If the problem is not with an include file, further break up the problematic source file into several parts, until you either stop getting the error or you locate the very function that causes it.
5. As a temporary workaround, modify the make file so as to compile that function with -O1. The problem is in this case with gcc itself, and you can submit that function in a bug report (together with all your include files).
6. You can of course also opt to compile the entire project with -O1 which should be sufficient for a PHP extension, or stay with gcc 4.6.3 for the moment.
7. Keep on trying out new updates to gcc, since the bug could be fixed (or the inefficiency removed) unrelatedly to your bug report.

Comments

[terdon]

Are you sure there is no problem in your code? I am no expert but 3gig of memory to compile a C program seems excessive.

[Ярослав Рахматуллин]

Include gcc version and relevant parts of the build log (make output or similar).

[harrymc]

What happens if you compile without optimization ("-O0" ) ? Did you use a 64-bit version of gcc ?

[cleong]

It works when it's -O1 or lower. In the 64 bit version of Ubuntu, I was able to get it to compile.

[Breakthrough]

@cleong try adding more swap space as per this question on Stack Overflow. It's clear that the 2GB RAM + 2GB Swap isn't going to cut it, try with an 8GB swap partition.

[cleong]

I've managed to compile the project on Mageia, CentOS, and three version of Debian, in both 32-bit and 64-bit. It compiled fine in Ubuntu Server 12.04. I think it's something specific to Ubuntu 12.10.

[terdon]

@cleong on these other systems did the compilation need so much memory?

[cleong]

It does. Just the nature of the code.