Unit testing as part of the build

Question

I have a CMake project separated in three parts:

1. Compilation of my libraries
2. Compilation of many unit tests programs testing each precise subpart of those libraries
3. Compilation of program samples using those libraries

My question is about the 2^nd part. My unit tests executables are short binary programs with a main returning 0 on success, 1 on fail. I would like to integrate their running as part of the build.

Is it possible to use CMake to do one of those two solutions:

• Run every unit test program and check return values and generate an error if any fail.
• Generate a test script that would run and check every unit test program and check return values to yield an error if any fail.

I'm not looking for a complete CMake script code, a simple indication on what is possible with a link to corresponding documentation would be enough.

Solution 1

See a similar problem and my answer here.

Mainly my recommendation is to add a POST_BUILD step to your unit test targets that runs ctest. If a POST_BUILD step does fail (return code is not 0), the build will fail.

Something like:

set(UNIT_TEST MyLibUnitTestTargetName)
add_test(NAME ${UNIT_TEST} COMMAND ${UNIT_TEST})
add_custom_command(
     TARGET ${UNIT_TEST}
     COMMENT "Run tests"
     POST_BUILD 
     WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
     COMMAND ${CMAKE_CTEST_COMMAND} -C $<CONFIGURATION> -R "^${UNIT_TEST}$" --output-on-failures
)

This has the advantage that it runs like you would call ctest.

The short version - without add_test() / ctest - would be:

set(UNIT_TEST MyLibUnitTestTargetName)
add_custom_command(
     TARGET ${UNIT_TEST}
     COMMENT "Run tests"
     POST_BUILD 
     COMMAND ${UNIT_TEST}
)

References:

• add_test()
• add_custom_command()
• How to run ctest after building my project with cmake
• CMake: Running unit test as part of the build

Solution 2

If you want your test binary to be preserved, even if tests fail, you can try something like this:

set(UNIT_TEST MyLibUnitTestTargetName)
add_test(NAME ${UNIT_TEST} COMMAND ${UNIT_TEST})
add_custom_target(run_unit_test ALL
    COMMAND ${CMAKE_CTEST_COMMAND} --output-on-failure
    DEPENDS ${UNIT_TEST})

This will create a target that always runs your unit tests. However, because it's a separate target failure of that command will result in build failure, but not in removing the binary produced by your original UNIT_TEST target.

Comments

[Angew is no longer proud of SO]

There's no need for the genex in add_test. Target names are automatically understood to mean the executable produced by the target.

[Florian]

@Angew Thanks for the hint. Changed it to directly use the target name as COMMAND for add_test() and the short add_custom_command() version.

[ivaigult]

This code causes binary removal, in case of tests failure for make and ninja

[jacobq]

I'm not sure why, but I had to add a space like this: -R "^${UNIT_TEST}$ ". Otherwise --output-on-failures was getting treated as part of the regex resulting in "No tests were found!!!"