Running JUnit Test in parallel on Suite Level?

Question

I have a bunch of tests that are organized in JUnit test suites. These tests are greatly utilizing selenium to test a web application. So, naturaly for selenium, the runtime of these tests is quite long. Since the test classes in the suites can not run parallel due some overlaps in the test database, i would like to run the suites parallel.

The JUnit ParallelComputer can only execute tests on class or method level in parallel, are there any standard ways for JUnit to do that with suites?

If i just pass suite classes to the junit runner and configure the computer to parallelize on class level, it picks the test classes itself, not the suites.

br Frank

Solution 1

Here is some code that worked for me. I did not write this. If you use @RunWith(ConcurrentSuite.class) instead of @RunWith(Suite.class) it should work. There is an annotation that is also needed which is found below.

package utilities.runners;

import org.junit.internal.builders.AllDefaultPossibilitiesBuilder;
import org.junit.runner.Runner;
import org.junit.runners.Suite;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.RunnerBuilder;
import org.junit.runners.model.RunnerScheduler;

import utilities.annotations.Concurrent;

import java.util.Arrays;
import java.util.LinkedList;
import java.util.List;
import java.util.Queue;
import java.util.concurrent.CompletionService;
import java.util.concurrent.ExecutorCompletionService;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.atomic.AtomicInteger;

/**
 * @author Mathieu Carbou ([email protected])
 */
public final class ConcurrentSuite extends Suite {
    public ConcurrentSuite(final Class<?> klass) throws InitializationError {
        super(klass, new AllDefaultPossibilitiesBuilder(true) {
            @Override
            public Runner runnerForClass(Class<?> testClass) throws Throwable {
                List<RunnerBuilder> builders = Arrays.asList(
                        new RunnerBuilder() {
                            @Override
                            public Runner runnerForClass(Class<?> testClass) throws Throwable {
                                Concurrent annotation = testClass.getAnnotation(Concurrent.class);
                                if (annotation != null)
                                    return new ConcurrentJunitRunner(testClass);
                                return null;
                            }
                        },
                        ignoredBuilder(),
                        annotatedBuilder(),
                        suiteMethodBuilder(),
                        junit3Builder(),
                        junit4Builder());
                for (RunnerBuilder each : builders) {
                    Runner runner = each.safeRunnerForClass(testClass);
                    if (runner != null)
                        return runner;
                }
                return null;
            }
        });
        setScheduler(new RunnerScheduler() {
            ExecutorService executorService = Executors.newFixedThreadPool(
                    klass.isAnnotationPresent(Concurrent.class) ?
                            klass.getAnnotation(Concurrent.class).threads() :
                            (int) (Runtime.getRuntime().availableProcessors() * 1.5),
                    new NamedThreadFactory(klass.getSimpleName()));
            CompletionService<Void> completionService = new ExecutorCompletionService<Void>(executorService);
            Queue<Future<Void>> tasks = new LinkedList<Future<Void>>();

            @Override
            public void schedule(Runnable childStatement) {
                tasks.offer(completionService.submit(childStatement, null));
            }

            @Override
            public void finished() {
                try {
                    while (!tasks.isEmpty())
                        tasks.remove(completionService.take());
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                } finally {
                    while (!tasks.isEmpty())
                        tasks.poll().cancel(true);
                    executorService.shutdownNow();
                }
            }
        });
    }

    static final class NamedThreadFactory implements ThreadFactory {
        static final AtomicInteger poolNumber = new AtomicInteger(1);
        final AtomicInteger threadNumber = new AtomicInteger(1);
        final ThreadGroup group;

        NamedThreadFactory(String poolName) {
            group = new ThreadGroup(poolName + "-" + poolNumber.getAndIncrement());
        }

        @Override
        public Thread newThread(Runnable r) {
            return new Thread(group, r, group.getName() + "-thread-" + threadNumber.getAndIncrement(), 0);
        }
    }

}

And the annotation is as follows.

package utilities.annotations;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
 * @author Mathieu Carbou ([email protected])
 */
@Retention(RetentionPolicy.RUNTIME)
@Target({ ElementType.TYPE })
public @interface Concurrent {
    int threads() default 5;
}

Solution 2

Since Suite is used to annotate a Class, so run the Suite-annotated class in JUnitCore.runClasses(ParallelComputer.classes(), cls) way. cls are Suite-annotated classes.

@RunWith(Suite.class)
@Suite.SuiteClasses({
Test1.class,
Test2.class})
public class Suite1 {
}

@RunWith(Suite.class)
@Suite.SuiteClasses({
Test3.class,
Test4.class})
public class Suite2 {
}
...
JUnitCore.runClasses(ParallelComputer.classes(), new Class[]{Suite1.class, Suite2.class})

Comments

[Frank]

Hi, thanks for the quik answer. Sadly this doesn't run the test in the way I intended. Like I wrote at the end of my question, this runs the tests classes inside of the suite parallel, not the suites itself.

[Frank]

Your example does execute the following way: Test1 and Test2 run parallel, exactly the behavior I must prevent.

[卢声远 Shengyuan Lu]

Welcome! If my code cannot help you, you may write multi thread to run your Suites(if no better solution).

[Frank]

I was thinking about this, but if I do this, I have to aggregate the junit result myself, because i need that for test report. But thanks for your quick help!

[Lucas Holt]

Looks like a copy of this code is available at this URL mycila.googlecode.com/svn/sandbox/src/main/java/com/mycila/…

[Reid Mac]

Yes, I should have included the link. Thanks.

[Todd Flanders]

This worked after adding an additional constructor: public ConcurrentSuite(Class<?> klass, RunnerBuilder builder) throws InitializationError { this(klass); }

[Simon Sobisch]

As the link is dead now - here's the pre-compiled jar and sources on Maven: mvnrepository.com/artifact/com.mycila/mycila-junit and a random sample