How do I make Jenkins build fail when Maven unit tests fail?

Question

I'm using Jenkins, Maven 3.1, and Java 1.6. I have the following Maven job set up in Jenkins with the following goals and options ...

clean install -U -P cloudbees -P qa

below is my pom.xml surefire configuration ...

<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.17</version>
        <configuration>
                <reuseForks>true</reuseForks>
                <argLine>-Xmx2048m -XX:MaxPermSize=512M </argLine>
                <skipTests>false</skipTests>
        </configuration>
</plugin>

However, when my unit tests fail, the Jenkins console output still says "BUILD SUCCESS" and the build is marked as "unstable" instead of outright failing. How do I configure things in Jenkins (or Maven if taht's what it turns out to be) so that my build fails (not becomes unstable or passes) if any of the unit tests fail?

Below is what the console output says

17:08:04   MyProjectOrganizationControllerTest.testRecoverFromError » IllegalState Failed to...
17:08:04   MyProjectOrganizationControllerTest.testVerifyDistrictListPopulated » IllegalState
17:08:04   MyProjectOrganizationControllerTest.testUpdateSchool » IllegalState Failed to loa...
17:08:04   MyProjectOrganizationControllerTest.testDeleteSchool » IllegalState Failed to loa...
17:08:04   MyProjectOrganizationControllerTest.testVerifyOrgListPopulatedPrivateSchoolOrgType » IllegalState
17:08:04   MyProjectOrganizationControllerTest.testSubmitMultipleParams » IllegalState Faile...
17:08:04 
17:08:04 Tests run: 155, Failures: 0, Errors: 154, Skipped: 1
17:08:04 
17:08:04 [ERROR] There are test failures.
17:08:04 
17:08:04 Please refer to /scratch/jenkins/workspace/MyProject/MyProject/target/surefire-reports for the individual test results.
17:08:04 [JENKINS] Recording test results
17:08:07 log4j:WARN No appenders could be found for logger (org.apache.commons.beanutils.converters.BooleanConverter).
17:08:07 log4j:WARN Please initialize the log4j system properly.
17:08:14 [INFO] 
17:08:14 [INFO] --- maven-war-plugin:2.4:war (default-war) @ MyProject ---
17:08:15 [INFO] Packaging webapp
17:08:15 [INFO] Assembling webapp [MyProject] in [/scratch/jenkins/workspace/MyProject/MyProject/target/MyProject]
17:08:15 [INFO] Processing war project
17:08:15 [INFO] Copying webapp resources [/scratch/jenkins/workspace/MyProject/MyProject/src/main/webapp]
17:08:15 [INFO] Webapp assembled in [662 msecs]
17:08:15 [INFO] Building war: /scratch/jenkins/workspace/MyProject/MyProject/target/MyProject.war
17:08:20 [INFO] 
17:08:20 [INFO] --- maven-failsafe-plugin:2.17:integration-test (default) @ MyProject ---
17:08:20 [JENKINS] Recording test results
17:08:25 [INFO] 
17:08:25 [INFO] --- maven-failsafe-plugin:2.17:verify (default) @ MyProject ---
17:08:25 [INFO] Failsafe report directory: /scratch/jenkins/workspace/MyProject/MyProject/target/failsafe-reports
17:08:25 [JENKINS] Recording test results[INFO] 
17:08:25 [INFO] --- maven-install-plugin:2.4:install (default-install) @ MyProject ---
17:08:25 
17:08:25 [INFO] Installing /scratch/jenkins/workspace/MyProject/MyProject/target/MyProject.war to /home/jenkins/.m2/repository/org/mainco/subco/MyProject/76.0.0-SNAPSHOT/MyProject-76.0.0-SNAPSHOT.war
17:08:25 [INFO] Installing /scratch/jenkins/workspace/MyProject/MyProject/pom.xml to /home/jenkins/.m2/repository/org/mainco/subco/MyProject/76.0.0-SNAPSHOT/MyProject-76.0.0-SNAPSHOT.pom
17:08:26 Started calculate disk usage of build
17:08:26 Finished Calculation of disk usage of build in 0 seconds
17:08:26 Started calculate disk usage of workspace
17:08:26 Finished Calculation of disk usage of workspace in 0 seconds
17:08:26 [INFO] ------------------------------------------------------------------------
17:08:26 [INFO] BUILD SUCCESS
17:08:26 [INFO] ------------------------------------------------------------------------
17:08:26 [INFO] Total time: 11:00.616s
17:08:26 [INFO] Finished at: Mon Feb 23 17:08:26 UTC 2015
17:08:27 [INFO] Final Memory: 90M/414M
17:08:27 [INFO] ------------------------------------------------------------------------
17:08:27 Waiting for Jenkins to finish collecting data
17:08:28 [JENKINS] Archiving /scratch/jenkins/workspace/MyProject/MyProject/pom.xml to org.mainco.subco/MyProject/76.0.0-SNAPSHOT/MyProject-76.0.0-SNAPSHOT.pom
17:08:28 [JENKINS] Archiving /scratch/jenkins/workspace/MyProject/MyProject/target/MyProject.war to org.mainco.subco/MyProject/76.0.0-  SNAPSHOT/MyProject-76.0.0-SNAPSHOT.war
17:08:31 channel stopped
17:08:31 Started calculate disk usage of build
17:08:31 Finished Calculation of disk usage of build in 0 seconds
17:08:31 Started calculate disk usage of workspace
17:08:31 Finished Calculation of disk usage of workspace in 0 seconds
17:08:31 Finished: UNSTABLE

Solution 1

You can add -Dmaven.test.failure.ignore=false to the MAVEN_OPTS if you click on Advanced button in the Build section of your Jenkins Job.

See Maven Surefire Plugin - surefire:test options for reference.

Solution 2

Use Text Finder plugin. Configure it to look for There are test failures and downgrade the build to FAILED

Comments

[Paresh]

But do we really have to specify this flag as false ? As per my understanding if we don't specify this flat then default value is "FALSE" only. Please suggest. Thanks!

[Torsten]

Jenkins specifies this value to true per default. Have a look at issues.jenkins-ci.org/browse/JENKINS-24655

[Alexey]

Nice hack, but a hack none the less.

[bdeniker]

In addition to being a hack, I found it failed to downgrade an UNSTABLE to a FAILED build for me. I thought it might be something to do with the documented "the reclassification only applies to builds which returned an overall exit status of zero".

[Kai]

Keep in mind the documentation tells "Its use is NOT RECOMMENDED".

[Meadowlark Bradsher]

@Kai To be clear, setting -Dmaven.test.failure.ignore=true is what is not recommended based upon the documentation in your link. If Jenkins sets this to a value not recommended then going about changing it back to false complies with the spirit of the warning in the documentation.

[Terran]

For Jenkins pipelines I found this hack to be the only manageable solution: Run mvn with -Dmaven.test.failure.ignore=true in a try-finally. In the finally have findText regexp: 'There are test failures', alsoCheckConsoleOutput: true, unstableIfFound: true and junit allowEmptyResults: true, testResults: 'target/surefire-reports/*.xml'

[vitro]

But this does not show you which tests have failed in the Jenkins UI.