Running JUnit4 Test classes in specified order

Question

I've written a number of tests, divided not only into separate classes but, depending on what area of my application they're testing, into separate sub-packages. So, my package structure looks a bit like this:

my.package.tests
my.package.tests.four
my.package.tests.one
my.package.tests.three
my.package.tests.two

In the my.package.tests package I have a parent class that all tests in the sub-packages one through four extend. Now, I would like to choose the order in which the tests are run; not within the classes (which seems to be possible with the FixMethodOrder annotation) but the order of the classes or sub-packages themselves (So those in sub-package one first, then those in two, ect.). Some of the test classes use the Parameterized runner, just in case that makes a difference. Choosing the order is not required for the tests to succeed, they are independent of each other; it would however be helpful to sort them to reflect the order in which the various parts of the program are normally used as it makes the analysis a bit easier.

Now, ideally I would like to have some configuration file which tells JUnit which order the tests should be executed in; I'm guessing however that it won't be so easy. Which options do I have here and what would be the easiest one? I'd also prefer to only have to list the sub-packages, not the various classes in the packages or even the test functions in the classes.

I'm using Java 1.6.0_26 (I don't have a choice here) and JUnit 4.11.

Solution

You can do this with test suites. (you can "nest" these, too)

@SuiteClasses({SuiteOne.class, SuiteTwo.class})
@RunWith(Suite.class)
public class TopLevelSuite {}

@SuiteClasses({Test1.class, Test2.class})
@RunWith(Suite.class)
public class SuiteOne {}


@SuiteClasses({Test4.class, Test3.class})
@RunWith(Suite.class)
public class SuiteTwo {}

... And so on. Use the "toplevelsuite" as an entry point, and the tests will execute in the order in which you define the @SuiteClasses array.

As for the methods within a class, you can specify the order they execute in with @FixMethodOrder (as you have already mentioned in your question.

Comments

[blalasaadri]

Thanks, that works like a charm! I guess there isn't the option to somehow tell it "run all tests in the following package" though, right?

[blgt]

No, you can't automatically tell it to include all classes in a package (AFAIK, at least).

[blgt]

The general idea here is that you can place one Suite class in each package, which would include all the tests in that package. You then place your "parent" suite in the parent package and include those in it. This way you mimic the package hierarchy in your Suite classes.

[blalasaadri]

Yes, I've done that. Certainly better than what I had before. :-)

[testtamas]

This a great solution indeed, but one more thing. You can use @RunWith annotation only once. If you use a specific framework to run test cases, you cannot apply this solution.