Concurrent JUnit testing

Question

I have a large JUnit test suite, where I'd quite like to run all the tests concurrently for two reasons:

• Exploit multiple cores to run the whole test suite faster
• Hopefully detect some errors due to non-thread-safe global objects

I recognise that this will force me to refactor some code to make it thread-safe, but I consider that to be a good thing :-)

What's the best way to get JUnit to run all the tests concurrently?

Solution 1

Are you fixed to JUnit? TestNG provides good multi thread testing out of the box and it's compatible with JUnit tests (you need to make a few changes). For example you could run a test like this:

@Test(threadPoolSize = 3, invocationCount = 9,  timeOut = 10000)
public void doSomething() {
...
}

This would mean that the doSomething() method will be invoked 9 times by 3 different threads.

I highly recommend TestNG.

Solution 2

I was looking for an answer to exactly this question, and based on the answers here, and what I read elsewhere, it appears as if there isn't currently an easy out-of-the-box way to run existing tests in parallel using JUnit. Or if there is I didn't find it. So I wrote a simple JUnit Runner that accomplishes that. Please feel free to use it; see http://falutin.net/2012/12/30/multithreaded-testing-with-junit/ for a complete explanation and the source code of the MultiThreadedRunner class. With this class, you can just annotate your existing test class(es) like this:

@RunWith(MultiThreadedRunner.class)

Solution 3

The following code should achieve your requirements which was taken from the German book JUnit Profiwissen which contains some hints to either tests stuff in parallel or on how to reduce execution time through using multiple cores instead of only a single core.

JUnit 4.6 introduced a ParallelComputer class which offered the parallel execution of tests. However, this functionality was not publicly accessible until JUnit 4.7 which provided the possibility to set a custom scheduler for the parent runner.

public class ParallelScheduler implements RunnerScheduler {

    private ExecutorService threadPool = Executors.newFixedThreadPool(
        Runtime.getRuntime().availableProcessors());

    @Override
    public void schedule(Runnable childStatement) {
        threadPool.submit(childStatement);
    }

    @Override
    public void finished() {
        try {
            threadPool.shutdown();
            threadPool.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException("Got interrupted", e);
        }
    }
}

public class ParallelRunner extends BlockJUnit4ClassRunner {

    public ParallelRunner(Class<?> klass) throws InitializationError {
        super(klass);
        setScheduler(new ParallelScheduler());
    }
}

If you now annotate a test-class with @RunWith(ParallelRunner.class) each method will run within its own thread. Furthermore, there will be as many active threads (only) as CPU-cores are available at the executing machine.

If multiple classes should be executed in parallel you can define a customized suite like this:

public class ParallelSuite extends Suite {

    public ParallelSuite(Class<?> klass, RunnerBuilder builder) 
      throws InitializationError {
        super(klass, builder);
        setScheduler(new ParallelScheduler());
    }
}

and then change @RunWith(Suite.class) with @RunWith(ParallelSuite.class)

You can even leverage the functionality of f.e. the WildcardPatternSuite by extending directly from that suite instead of Suite as in the example before. This enables you furthermore to filter unit-tests f.e. by any @Category - a TestSuite which only executes UnitTest annotated categories in parallel could look like this:

public interface UnitTest {

}

@RunWith(ParallelSuite.class)
@SuiteClasses("**/*Test.class")
@IncludeCategories(UnitTest.class)
public class UnitTestSuite {

}

A simple test case could now look like this:

@Category(UnitTest.class)
@RunWith(MockitoJUnitRunner.class)
public class SomeClassTest {

    @Test
    public void testSomething() {
        ...
    }
}

The UnitTestSuite will execute each class found in subdirecotries that ends with Test and has a @Category(UnitTest.class) specified in parallel - depending on the number of CPU cores available.

I'm not sure if it can get simpler than that :)

Solution 4

Apparently in his post "Concurrent JUnit Tests With RunnerScheduler" Mathieu Carbou has done implementation for concurrence that could helps!

http://dzone.com/articles/concurrent-junit-tests

OneJunit

@RunWith(ConcurrentJunitRunner.class)
@Concurrent(threads = 6)
public final class ATest {
 
    @Test public void test0() throws Throwable { printAndWait(); }
    @Test public void test1() throws Throwable { printAndWait(); }
    @Test public void test2() throws Throwable { printAndWait(); }
    @Test public void test3() throws Throwable { printAndWait(); }
    @Test public void test4() throws Throwable { printAndWait(); }
    @Test public void test5() throws Throwable { printAndWait(); }
    @Test public void test6() throws Throwable { printAndWait(); }
    @Test public void test7() throws Throwable { printAndWait(); }
    @Test public void test8() throws Throwable { printAndWait(); }
    @Test public void test9() throws Throwable { printAndWait(); }

    void printAndWait() throws Throwable {
        int w = new Random().nextInt(1000);
        System.out.println(String.format("[%s] %s %s %s",Thread.currentThread().getName(), getClass().getName(), new Throwable       ().getStackTrace()[1].getMethodName(), w));
        Thread.sleep(w);
    }
}

Multiple JUnits:

@RunWith(ConcurrentSuite.class)
@Suite.SuiteClasses({ATest.class, ATest2.class, ATest3.class})
public class MySuite {
}

Solution 5

You can also try HavaRunner. It is a JUnit runner that runs tests in parallel by default.

HavaRunner also has handy suites: you can declare a test to be a member of a suite by adding the annotation @PartOf(YourIntegrationTestSuite.class) onto the class. This approach differs from JUnit's, where you declare the suite memberships in the suite class.

In addition, HavaRunner suites may intialise heavy-weight objects such as an embedded web application container. HavaRunner then passes this heavy-weight object to the constructor of each suite member. This removes the need for the @BeforeClass and @AfterClass annotations, which are problematic, because they promote global mutable state, which in turn makes parallelisation difficult.

Lastly, HavaRunner has scenarios – a way to run the same test against different data. Scenarios reduce the need to duplicate test code.

HavaRunner has been battle-tested in two mid-size Java projects.

Ps. I'm the author of HavaRunner, and I'd appreciate your feedback on it.

View more solutions

Comments

[Admin]

+1 for TestNG, I use it for all my thread safety tests. It also has nice paramaterized tests.

[tddmonkey]

Downvoted due to not actually answering the OPs question. He specifically states he wants to run his entire suite across threads and run a single method multiple times

[Stéphane Piette]

Have you an idea on how to implement a concurrent Suite runner, and not only a concurrent test runner ? Could be helpful too :p

[Mike Sokolov]

I haven't looked at that, Stéphane. But making the test runner was so easy, I bet the suite runner wouldn't be hard either. If you solve the problem, I'll gladly add your code to my post :)

[Rafael Winterhalter]

@TobyHobson This is a terrible way of testing for thread safety. And by definition not a unit tests since it is not deterministic. And TestNG contains numerous bugs. I fixed some of them lately but they do not get pulled (no suggestion does), TestNG is fairly inactive. Rather try Thread Weaver (code.google.com/p/thread-weaver)

[Splaktar]

This is pretty nice but doesn't seem to work with tests that include an @ BeforeClass and @ Before methods. It tries to run the @ BeforeClass methods for every thread instead of just once. For our setup, this means 5 threads dropping and rebuilding the DB at the same time :(

[Stefan Birkner]

You don't have to rewrite this code. The author published a library named JUnit Toolbox: code.google.com/p/junit-toolbox

[Roman Vottner]

@Splaktar One way to solve this is to maintain a static reference to f.e. an AtomicBoolean isInitialized variable in the test class which is set after the before class method was executed once. Any additional invocations of the init-method will just return if this boolean variable is true. This prevents multiple threads from dropping already initialized data

[Brain]

The link is dead.

[Mike Sokolov]

It lives again!