JUnit 5 - Empty test suite in IntelliJ IDEA when using JUnit Jupiter engine

Question

How to execute All Suite tests with JUnit 5 in IntelliJ IDEA v2016.2.2?

I get Empty test suite running this code:

import org.junit.platform.runner.IncludeEngines;
import org.junit.platform.runner.JUnitPlatform;
import org.junit.platform.runner.SelectPackages;
import org.junit.runner.RunWith;

@RunWith(JUnitPlatform.class)
@IncludeEngines("junit-jupiter")
@SelectPackages("<eu...package>") //I confirm that <eu...package> is ok.
public class AllTests {
}

I receive:

INFORMAZIONI: Discovered TestEngines with IDs: [junit-jupiter, junit-vintage]
Empty test suite.
Empty test suite.

[root]
JUnit Jupiter
JUnit Vintage

OR

import eu.....services.ServiceTest;
import eu.....repository.DAOTest;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;

@RunWith(Suite.class)
@Suite.SuiteClasses({
        ServiceTest.class,
        DAOTest.class
})
public class AllTests {
}

I receive:

INFORMAZIONI: Discovered TestEngines with IDs: [junit-jupiter, junit-vintage]
Empty test suite.

[root]
|+--JUnit Vintage
|   +--eu.....AllTests
|+--JUnit Jupiter

I was able to run suite with JUnit 4, but it doesn't work with JUnit 5.

Solution 1

Short Answer

If you are using IntelliJ IDEA 2016.2, it is currently not possible to execute a test class annotated with @RunWith(JUnitPlatform.class) within the IDE.

Long Answer

Based on the behavior you reported, after some painstaking investigative work, I believe I have the answer to your question...

If you are using IntelliJ IDEA 2016.2 which has built-in support for JUnit 5, then the following is what is happening.

1. IDEA launches the JUnit Platform via the Launcher API, selecting the test class annotated with @RunWith(JUnitPlatform.class) (let's call it TestSuite).
2. The Launcher detects both the junit-jupiter and junit-vintage TestEngine implementations.
3. The JUnit Jupiter engine ignores TestSuite since it is technically not a JUnit Jupiter test class.
4. The JUnit Vintage engine also ignores TestSuite since it is annotated with @RunWith(JUnitPlatform.class).
5. The end result is that neither registered test engine claims that it can run the TestSuite class.

The non-intuitive part is that the JUnit Vintage engine ignores TestSuite, when it in fact looks like a JUnit 4 based test class since it's annotated with @RunWith(). The reason it is ignored is to avoid infinite recursion, which is explained in the source code for DefensiveAllDefaultPossibilitiesBuilder:

if ("org.junit.platform.runner.JUnitPlatform".equals(runnerClass.getName())) {
    return null;
}

The fact that the above code returns null in such scenarios results in an empty suite.

Of course, it would certainly be better if the user were informed of such scenarios -- for example, via a log statement. I have therefore opened issues for both JUnit 5 and IntelliJ to improve the usability in such scenarios.

On the plus side, since you're using IntelliJ IDEA 2016.2, you don't need to use the test suite support. Instead, you can simply right-click on src/test/java in the project view in IDEA and select Run 'All Tests', and that will run all your tests.

Regards,

Sam (JUnit 5 core committer)

Solution 2

Additional to Sam Brannen's answer. I needed a TestSuit to setup the backend before running all test classes in a package. This is probably not possible via Run 'All Tests'.

But I think I found a good workaround for it. I came up with this:

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Nested;
import your.other.package.test.classes.*;

public class TestSuit {
    @BeforeAll
    public static void setup(){}

    @Nested
    public class TestExtender extends MyTestClass {}
}

You could extend each test class from another package and add the @Nested annotation to it. It is not the best solution, but it is a workaround until IDEA or JUnit 5 find another solution for this cases.

Comments

[Davideas]

@SamBrannen, the methods are currently annotated with org.junit.jupiter.api.Test. If I put org.junit.Test than it is Vintage, it works if I put Suite.class, but it is Vintage. I asked suite execution with JUnit5 (Jupiter). As confirmation, test suite is empty also in your library when you execute the test class SpringExtensionTestSuite.

[Sam Brannen]

After further investigation, I now assume that this is perhaps a bug in IDEA. I will post back here with additional information later.

[Davideas]

it doesn't work your solution. On the methods of the test classes I have org.junit.jupiter.api.Test and if execute tests for that class only it runs under engine Jupiter. Also, if I put org.junit.Test, it means Vintage and indeed Suite execution is performed in Vintage! But this is not what we all want. I asked JUnit5 so with engine Jupiter, I think it's obvious, otherwise why to upgrade to JUnit5? I mean [4.4.4. Test Suite] in the official website documentation.

[Davideas]

OK, in the end you found out the reason. I couldn't imagine that it was the IDE. Good work Sam. I will run all tests from the folder then.

[mitch]

Thanks. My suite wasn't running. Right clicking to run all tests worked great.