How to speed up Compile Time of my CMake enabled C++ Project?

Question

I came across several SO questions regarding specific aspects of improving the turn-around time of CMake enabled C++ projects lately (like "At what level should I distribute my build process?" or "cmake rebuild_cache for just a subdirectory?"), I was wondering if there is a more general guidance utilizing the specific possibilities CMake offers. If there is probably no cross-platform compile time optimization, I'm mainly interested in Visual Studio or GNU toochain based approaches.

And I'm already aware of and investing into the generally recommended areas to speed up C++ builds:

1. Change/Optimize/fine-tune the toolchain

2. Optimize your code base/software architecture (e.g by reducing the dependencies and use well-defined sub-projects - unit tests)

3. Invest in a better hardware (SSD, CPU, memory)

like recommended here, here or here. So my focus in this question is on the first point.

Plus I know of the recommendations to be found in CMake's Wiki:

• CMake: building with all your cores
• CMake Performance Tips

The former just handles the basics (parallel make), the later handles mostly how to speed-up parsing CMake files.

Just to make this a little more concrete, if I take my CMake example from here with 100 libraries using MSYS/GNU I got the following time measurement results:

$ cmake --version
cmake version 3.5.2
CMake suite maintained and supported by Kitware (kitware.com/cmake).

$ time -p cmake -G "MSYS Makefiles" ..
-- The CXX compiler identification is GNU 4.8.1
...
-- Configuring done
-- Generating done
-- Build files have been written to: [...]
real 27.03
user 0.01
sys 0.03        

$ time -p make -j8
...
[100%] Built target CMakeTest
real 113.11
user 8.82
sys 33.08

So I have a total of ~140 seconds and my goal - for this admittedly very simple example - would be to get this down to about 10-20% of what I get with the standard settings/tools.

Solution

Here's what I had good results with using CMake and Visual Studio or GNU toolchains:

1. Exchange GNU make with Ninja. It's faster, makes use of all available CPU cores automatically and has a good dependency management. Just be aware of

   a.) You need to setup the target dependencies in CMake correctly. If you get to a point where the build has a dependency to another artifact, it has to wait until those are compiled (synchronization points).

   $ time -p cmake -G "Ninja" ..
   -- The CXX compiler identification is GNU 4.8.1
   ...
   real 11.06
   user 0.00
   sys 0.00
   
   $ time -p ninja
   ...
   [202/202] Linking CXX executable CMakeTest.exe
   real 40.31
   user 0.01
   sys 0.01
   

   b.) Linking is always such a synchronization point. So you can make more use of CMake's Object Libraries to reduce those, but it makes your CMake code a little bit uglier.

   $ time -p ninja
   ...
   [102/102] Linking CXX executable CMakeTest.exe
   real 27.62
   user 0.00
   sys 0.04
   

2. Split less frequently changed or stable code parts into separate CMake projects and use CMake's ExternalProject_Add() or - if you e.g. switch to binary delivery of some libraries - find_library().

3. Think of a different set of compiler/linker options for your daily work (but only if you also have some test time/experience with the final release build options).

   a.) Skip the optimization parts

   b.) Try incremental linking

4. If you often do changes to the CMake code itself, think about rebuilding CMake from sources optimized for your machine's architecture. CMake's officially distributed binaries are just a compromise to work on every possible CPU architecture.

   When I use MinGW64/MSYS to rebuild CMake 3.5.2 with e.g.

   cmake -DCMAKE_BUILD_TYPE:STRING="Release"
         -DCMAKE_CXX_FLAGS:STRING="-march=native -m64 -Ofast -flto" 
         -DCMAKE_EXE_LINKER_FLAGS:STRING="-Wl,--allow-multiple-definition"
         -G "MSYS Makefiles" .. 
   

   I can accelerate the first part:

   $ time -p [...]/MSYS64/bin/cmake.exe -G "Ninja" ..
   real 6.46
   user 0.03
   sys 0.01
   

5. If your file I/O is very slow and since CMake works with dedicated binary output directories, make use of a RAM disk. If you still use a hard drive, consider switching to a solid state disk.

6. Depending of your final output file, exchange the GNU standard linker with the Gold Linker. Even faster than Gold Linker is lld from the LLVM project. You have to check whether it supports already the needed features on your platform.

7. Use Clang/c2 instead of Visual C++ compiler. For the Visual C++ compiler performance recommendations are provided from the Visual C++ team, see https://blogs.msdn.microsoft.com/vcblog/2016/10/26/recommendations-to-speed-c-builds-in-visual-studio/

8. Increadibuild can boost the compilation time.

References

• CMake: How to setup Source, Library and CMakeLists.txt dependencies?
• Replacing ld with gold - any experience?
• Is the lld linker a drop-in replacement for ld and gold?

Comments

[csl]

Also, use ccache. It makes recompilations insanely fast.

[Chris Kitching]

If your build system is doing incremental rebuilds properly, ccache should be of little to no help. On well-written cmake build which does incremental builds right, ccache actually slows you down with its overhead. I have in the past had success with ccache on projects with sucky build systems that didn't do incremental compilation, however (firefox, years ago)