Force JUnit to run one test case at a time

Question

I have a problematic situation with some quite advanced unit tests (using PowerMock for mocking and JUnit 4.5). Without going into too much detail, the first test case of a test class will always succeed, but any following test cases in the same test class fails. However, if I select to only run test case 5 out of 10, for example, it will pass. So all tests pass when being run individually. Is there any way to force JUnit to run one test case at a time? I call JUnit from an ant-script.

I am aware of the problem of dependant test cases, but I can't pinpoint why this is. There are no saved variables across the test cases, so nothing to do at @Before annotation. That's why I'm looking for an emergency solution like forcing JUnit to run tests individually.

Solution 1

Your problem is not that JUnit runs all the tests at once, you problem is that you don't see why a test fails. Solutions:

1. Add more asserts to the tests to make sure that every variable actually contains what you think
2. Download an IDE from the Internet and use the built-in debugger to look at the various variables
3. Dump the state of your objects just before the point where the test fails.
4. Use the "message" part of the asserts to output more information why it fails (see below)
5. Disable all but a handful of tests (in JUnit 3: replace all strings "void test" with "void dtest" in your source; in JUnit 4: Replace "@Test" with "//D@TEST").

Example:

assertEquals(list.toString(), 5, list.size());

Solution 2

I am aware of all the recommendations, but to finally answer your question here is a simple way to achieve what you want. Just put this code inside your test case:

Lock sequential = new ReentrantLock();

@Override
protected void setUp() throws Exception {
    super.setUp();
    sequential.lock();
}

@Override
protected void tearDown() throws Exception {
    sequential.unlock();
    super.tearDown();
}

With this, no test can start until the lock is acquired, and only one lock can be acquired at a time.

Solution 3

It seems that your test cases are dependent, that is: the execution of case-X affects the execution of case-Y. Such a testing system should be avoided (for instance: there's no guarantee on the order at which JUnit will run your cases).

You should refactor your cases to make them independent of each other. Many times the use of @Before and @After methods can help you untangle such dependencies.

Solution 4

Congratulations. You have found a bug. ;-)

If the tests "shouldn't" effect each other, then you may have uncovered a situation where your code can enter a broken state. Try adding asserts and logging to figure out where the code goes wrong. You may even need to run the tests in a debugger and check your code's internal values after the first test.

Solution 5

Excuse me if I dont answer your question directly, but isn't your problem exactly what TestCase.setUp() and TestCase.tearDown() are supposed to solve? These are methods that the JUnit framework will always call before and after each test case, and are typically used to ensure you begin each test case in the same state.

See also the JavaDoc for TestCase.

View more solutions

Comments

[tddmonkey]

+1, I wouldnt say "should" be avoided but "must" be avoided to avoid just this scenario

[Ciryon]

I am aware of the problem of dependant test cases, but I can't pinpoint why this is. There are no saved variables across the test cases, so nothing to do at @Before annotation. That's why I'm looking for an emergency solution like forcing JUnit to run tests individually.

[Itay Maman]

Dependency has many faces: Databases, files, system properties, etc. You should reduce your suite to two conflicting tests. Start commenting code in one test until they succeed. Then uncomment and do the same in the second test. This will give you some insights regarding the cause of your problem.

[Kalecser]

"There are no saved variables across the test cases" are you sure about that? What about static variables?

[guerda]

Okay, it's strongly not recommended.