How to run all JUnit tests in a category/suite with Ant?

Question

I'm using JUnit Categories and ClassPathSuite in a setup similar to that described in this answer. To recap:

public interface FastTests {
}

@RunWith(Categories.class)
@Categories.IncludeCategory(FastTests.class)
@Suite.SuiteClasses(AllTests.class)
public class FastTestSuite {
}

@RunWith(ClasspathSuite.class) 
public class AllTests {
}

...where AllTests makes use of the ClasspathSuite library.

A test class that's part of the FastTests category would look like this:

@Category(FastTests.class)
public class StringUtilsTest {
    //  ...
}

When I run "FastTestSuite" in my IDE, all tests with the FastTests annotation are executed, nice & smooth:

Now, I want to do the same thing with Ant. (To my surprise, I couldn't easily find instructions for this on SO.) In other words, I need an Ant target that runs all tests with the FastTests annotation.

I've tried some simplistic approaches using <test> or <batchtest>...

 <junit showoutput="true" printsummary="yes">
     <test name="fi.foobar.FastTestSuite"/>
     <formatter type="xml"/>
     <classpath refid="test.classpath"/>
 </junit>

... but no luck, so far.

Edit: Besides the IDE, it works fine with JUnitCore on the command line:

$ java -classpath "classes:WebContent/WEB-INF/lib/*" org.junit.runner.JUnitCore fi.foobar.FastTestSuite
.............
Time: 0.189

OK (13 tests)

Solution 1

Right, I got it working with <batchtest> quite simply:

<junit showoutput="true" printsummary="yes" fork="yes">
    <formatter type="xml"/>
    <classpath refid="test.classpath"/>
    <batchtest todir="${test.reports}">
        <fileset dir="${classes}">
            <include name="**/FastTestSuite.class"/>
        </fileset>
    </batchtest>
</junit>

I had tried <batchtest> earlier, but had made the silly mistake of using "**/FastTestSuite.java" instead of "**/FastTestSuite.class" in the <include> element... Sorry about that :-)

NB: it's necessary to set fork="yes" (i.e., run the tests in a separate VM); otherwise this will also produce "initializationError" at java.lang.reflect.Constructor.newInstance(Constructor.java:513) like <test> (see comments on the question). However, I couldn't get <test> working even with fork="yes".

The only shortcoming is that this produces just one JUnit XML report file (TEST-fi.foobar.FastTestSuite.xml) which makes it look like all the (hundreds) of tests are in one class (FastTestSuite). If anyone knows how to tweak this to show the tests in their original classes & packages, please let me know.

Solution 2

I found a workaround to use ant's junit task to run test of a specific Category:

<target name="test" description="Run Selenium tests by category">
    <fail unless="category.name" message="Please specify 'category.name' property"/>

    <junit showoutput="true" printsummary="true" fork="true">
        <formatter type="xml"/>
        <classpath refid="test.classpath"/>

        <batchtest todir="${test.reports}" haltonfailure="false">
            <fileset dir="${classes}">
                <!-- regex '^\s*@Category' part added to ensure @Category annotation is not commented out -->
                <containsregexp expression="^\s*@Category.*${category.name}"/>
            </fileset>
        </batchtest>
    </junit>
</target>

Execute test by supplying category.name property to ant like this:

ant -Dcategory.name=FastTests test

Using batchtest will also produce separate JUnit XML report files per test (e.g. TEST-fi.foobar.FastTestClassN.xml).

Comments

[oers]

is name="..." a path like org/test/TestSuite or in the package annotation org.test.TestSuite? What is the error?

[Jonik]

@oers: Latter; fully qualified classname. No real error message, it just... fails. unittest-fast: [junit] Running fi.foobar.FastTestSuite [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0.053 sec [junit] Test fi.foobar.FastTestSuite FAILED

[oers]

change the formatter to plain and change printsummary to withOutAndErr . That should show the reason for the errors. I think there is an exception thrown in there. Your setup is definitely the right way to execute Suites.

[Jonik]

@oers, thanks! Actually the error was there even without "withOutAnderr" (I forgot I need to look at the separate report files): Testcase: initializationError took 0.002 sec Caused an ERROR null at java.lang.reflect.Constructor.newInstance(Constructor.java:5‌​13) at java.lang.reflect.Constructor.newInstance(Constructor.java:5‌​13) Btw, as I edited on the question, this works with JUnitCore, which seems encouraging...

[amphibient]

Hi. I have a question that is related. feel free to chime in: stackoverflow.com/questions/15776718/…

[Scott Langley]

Using the junit4 Ant Task bundled inside the randomized testing project from carrotsearch: [labs.carrotsearch.com/randomizedtesting.html] I am able to run tests using Categories and ClassPathSuite - but not required to be inside a <batchtest>. Their JSON-type report generates output reports that have the tests broken down by their original packages - and look pretty nice, too.

[Illidanek]

Is there a way of doing this by specifying the category name (class) itself in the build.xml? Like in maven where you can say 'mvn test -Dgroups=test.classpath.FastTests'

[Ilie Daniel Stefan]

A workaround for your shortcoming is to apply a xslt before junitreport. You just need to split the initial xml in smaller ones

[R. Oosterholt]

Added another answer which fixes your 'shortcomming' because it will produce separate JUnit XML report files.

[Brian Riehman]

I am not able to reproduce this because the containsregexp appears to be searching the class file for the text of the category. However, the text is not in the binary file.