Why is a junit test that is skipped because of an assumption failure is not reported as skipped?

Question

I use junit assumptions to decide whether to run a test or not.
Tests where the assumption fails are ignored/skipped by the junit framework.

I wonder why skipped tests are not reported as 'skipped'?

Please have a look at the example:

import static org.junit.Assert.fail;
import org.junit.Assume;
import org.junit.Test;

public class AssumptionTest {

    @Test
    public void skipAssumptionFailureTest() {
        Assume.assumeTrue("foo".equals("bar"));
        fail("This should not be reached!");
    }
}

Running this test in a maven project results in:

Running AssumptionTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec

I would prefer to have this test reported as 'skipped'. Is there any chance to achieve this?

(junit 4.8.1; maven 2.2.1; java 1.6.0_21)

Solution 1

You can't do this out-of-the-box as the only way to skip tests is with the @Ignore annotation. However, I found a blog post which might just be what you are looking for:

Solution 2

While previous answers are good on their own, I'd like to address the "WHY" question.

The problem stemmed from the fact how tests failed / ignored were counted. When assumptions were added, usual counts were kinda skewered (assumption can IGNORE the tests half-way through it's run).

Tools runners had problems with that as well, causing messy reporting. See here for Surefire dev conversation with JUnit people, and here for related Eclipse bug.

Right now this ain't a problem in Maven, and it should not be in newer Eclipses. I use STS based on eclipse.buildId = 2.9.2.201205071000-RELEASE and I still experience this behaviour.

EDIT: reference type links did not work.

Solution 3

Spring brings another variant of an Ignore annotation, one that is able to decide at runtime whether to ignore a test: @IfProfileValue. In combination with a user supplied ProfileValueSource in a @ProfileValueSourceConfiguration, you can do some limited tricks.

See Spring Test Documentation.

But it is not the same as being able to perform some test code and in the middle of the test method decide to ignore the test, as you can with assumptions.

Comments

[FrVaBe]

Thanks for the response. Too bad that this does not work out of the box. Implementing a custom runner will not work for me - I normally already use a custom runner (SpringJUnit4ClassRunner) in my tests which I will not extend. I would have preferred a build in solution.

[raksja]

Note: Spring @IfProfileValue only skips/ignores the test if the name and value doesn't match and it doesn't not-run it. If you want to implement JUnit Categories type of implementation with this you may need to end up extending SpringJUnit4ClassRunner#runChild to not-run instead of ignoring. This mainly matters if its results are in jenkins.

[Simon Forsberg]

PLEASE add the important parts from the blog post as an answer, otherwise this answer will be quite useless as soon as that blog post disappears. (I'm impressed no one has mentioned this for the last 7 years)

[Donal Fellows]

Indeed, while the blog post is there, the code it references is now long gone.

[Marcono1234]

@DonalFellows, the code can still be found in the repository linked from the blog post: code.google.com/archive/p/programaticallyspeaking/source/… There you have to click "Download" in the sentence "Download the code for this repo.". In case that site is shut down, you can also download it from the Internet Archive.