JUnit test report enrichment with JavaDoc

Question

For a customer we need to generate detailed test reports for integration tests which not only show, that everything is green, but also what the test did. My colleagues and I are lazy guys and we do not want to hack spreadsheets or text documents.

For that, I think about a way to document the more complex integration tests with JavaDoc comments on each @Test annotated method and each test class. For the test guys it is a good help to see to which requirement, Jira ticket or whatever the test is linked to and what the test actually tries to do. We want to provide this information to our customer, too.

The big question now is: How can we put the JavaDoc for each method and each test class into the JUnit reports? We use JUnit 4.9 and Maven.

I know, that there is a description for each assertXXX(), but we really would need a nice HTML list as result or a PDF document which lists all classes and there documentation and below that all @Test methods and their description, the testing time, the result and if failed, the reason why.

Or is there another alternative to generate fancy test scripts? (Or should we start an OpenSource project on this!? ;-) )

Update: I asked another question on how to add a RunListener to Eclipse to have it also report in Eclipse when started there. The proposed solution with a custom TestRunner is another possibility to have the test results report. Have a look: How can I use a JUnit RunListener in Eclipse?

Solution 1

One way to achieve this would be to use a custom RunListener, with the caveat that it would be easier to use an annotation rather than javadoc. You would need to have a custom annotation such as:

@TestDoc(text="tests for XXX-342, fixes customer issue blahblah")
@Test
public void testForReallyBigThings() {
    // stuff
}

RunListener listens to test events, such as test start, test end, test failure, test success etc.

public class RunListener {
    public void testRunStarted(Description description) throws Exception {}
    public void testRunFinished(Result result) throws Exception {}
    public void testStarted(Description description) throws Exception {}
    public void testFinished(Description description) throws Exception {}
    public void testFailure(Failure failure) throws Exception {}
    public void testAssumptionFailure(Failure failure) {}
    public void testIgnored(Description description) throws Exception {}
}

Description contains the list of annotations applied to the test method, so using the example above you can get the Annotation TestDoc using:

description.getAnnotation(TestDoc.class);

and extract the text as normal.

You can then use the RunListener to generate the files you want, with the text specific to this test, whether the test passed or failed, or was ignored, the time taken etc. This would be your custom report.

Then, in surefire, you can specify a custom listener, using:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.10</version>
  <configuration>
    <properties>
      <property>
        <name>listener</name>
        <value>com.mycompany.MyResultListener,com.mycompany.MyResultListener2</value>
      </property>
  </configuration>
</plugin>

This is from Maven Surefire Plugin, Using JUnit, Using custom listeners and reporters

This solution has the disadvantage that you don't have the flexibility of javadoc as far as carriage returns, formatting is concerned, but it does have the advantage that the documentation is in one specific place, the annotation TestDoc.

Solution 2

Have you looked at Maven Sure-fire reports?

You can generate a HTML report from your JUnit Tests.

http://maven.apache.org/plugins/maven-surefire-report-plugin/

I'm not sure how customizable it is though. But it's a good starting point.

I also know that TestNG ( alternative to JUnit ) has some report generating capabilities. http://testng.org/doc/documentation-main.html#logging-junitreports

I would also recommend log4j http://logging.apache.org/log4j/1.2/manual.html

Solution 3

you can use jt-report an excellent framework for test reporting.

Solution 4

I have created a program using testNG and iText which outputs the test results in a nice pdf report. You can put a description of your test in the @Test tag, and that can be included in the .pdf report also. It provides the run times of the tests, and for the entire suite. It is currently being used to test webapps with selenium, but that part could be ignored. It also allows you to run multiple test suites in one run, and if tests fail, it allows you to re-run only those tests without having to re-run the entire suite, and those results will be appended to the original results PDF. See below the image for a link to the source if you are interested. I wouldn't mind this becoming an opensource project as I have a good start on it, though I'm not sure how to go about doing that. Here's a screenshot

So I figured out how to create a project on sourceforge. Here's the link sourceforge link

View more solutions

Comments

[Rick-Rainer Ludwig]

I do not now any possibility to put any other documentation into the surefire report plugin. Do you know any? The only way is to put code cross references in, but that is not what I need, because this would only work for failed tests, as far as I know. TestNG is not an option. We are bound to JUnit here. We have a lot of legacy code. Log4J is in use, but it's nothing we could put into reports nicely, I am afraid. But, thanks for you answer anyway.

[Patrick Magee]

I do not, sorry. I can defintialy say there is a lack of a good open source report generator for use with Unit Testing. Maybe you are after something more BDD. Like JBehave: jbehave.org Where you can write full user storys and have them integrated into testing your code. Then generate huge reports based on the user stories. blog.codecentric.de/en/2011/03/… Certainly not an answer, but something you could look into. Hope you find what you are looking for.

[Rick-Rainer Ludwig]

JBehave looks fine, but for another purpose. It gives me another idea to think about.That's something for the project people for sprint planning and monitoring. (+1 for that ;-)) I work in testing and QA and I look for something to satisfy the wish of our customers for nice reports for testing and for our internal customers and developers to have a quick look on failed test with a hint what went wrong. I am very short before writing a script myself to enrich the XML output of surefire. It just depends on the answers coming in on this questions within the next three days... ;-)

[Rick-Rainer Ludwig]

This looks promising, but I can not have a closer look now. I check details tomorrow. But, I am not an expert in TestNG: Is is compatible with JUnit? Can it work with Maven, too? This would be the optimum. ;-) I forgot to mention Maven in my question before. :-( I edited the question for that.

[Reid Mac]

That's fine, TestNG is compatible with Maven. TestNG and Maven. I am currently running my tests using eclipse. It is compatible with JUnit, the plugin for eclipse actually provides an option to convert JUnit tests to TestNG tests. They essentially have the same format, except they use different libraries for the Annotations, but the actual annotation itself is the same except for the Before/After tags, but there is an example in the project on sourceforge which you can take a look at. Hope this helps.

[Rick-Rainer Ludwig]

This idea is actual quite good. For writing something new, the amount of work is quite small. I checked the idea and tried some things out. It is better than nothing. ;-) One could create some text documents first and later one could extend it to write HTML or something.

[Matthew Farwell]

You can write text documents, or xml or html, anything. In your case it may be best to write simple XML and then transform it to something prettier as a later part of the process. That way the RunListener stays simple.

[Rick-Rainer Ludwig]

I already know Maven and I use it, too. But, the Surefire reports look very simple and do not contain enough information. I would really like to see some human readable and interpretable descriptions there for people which are not directly involved in software development. They should also understand what a test tries to tackle...

[Patrick Magee]

TestNG is like JUnit, it is a unit testing framework, similar to JUnit 4. But more customizable and it can also be used with Junit for reports.

[havexz]

I looked into the maven plugins. surefire-report just uses the Test-*.xml files to generate the report. Now those xml files contain no extra info. So to achieve what you want you have to first customize the surefire plugin to add some extra fields (fields can be part of annotation) in the xmls and then have your surefire-report to accept those fields. OR...the solution above (by Matthew Farwell) is also good enough.

[havexz]

snippet manually edited to add description field. <testcase time="0.071" classname="com.param.acdm.coffeemaker.CoffeeDespenserTest" name="getAvalaibleBeverages" description="hello"/> <testcase time="0.001" classname="com.param.acdm.coffeemaker.CoffeeDespenserTest" name="despense" description="hello2"/> <testcase time="0.001" classname="com.param.acdm.coffeemaker.CoffeeDespenserTest" name="reStockInventory" description="hello3"/> once this is done we have to modify the surefire-report

[Rick-Rainer Ludwig]

I do not like the idea to patch the surefire report plugin. With every new version of the plugin we would also need to apply our changes. Maybe, someday, the guys from the plugin change something more essential (e.g. the architecture), we would need to make also some greater adoptions... That's not really a way to go for us. At the moment the idea by Matthew Farwell is my favorite.

[havexz]

I agree ...I also dont like patching...I was just wondering if they provide some hooks without opening their source code. Like one mentioned above for Listeners...that can help. BTW one thing to add if you use Listener approach you might loose the aggregation of test results that the default one does.

[Rick-Rainer Ludwig]

This is the way we are going to go at first. The amount of time to be implemented is not so much and it is simple and clear to understand for others.

[Patrick Magee]

Glad you found what you needed in the end...for now :P

[Marcel Stör]

@MatthewFarwell, thanks for the valuable information. I wish there was a way a custom RunListener could contribute to the XML created by the surefire plugin :( Because it cannot a listener will have to put its output "somewhere" and a post-processing task will have to merge it with the XML.

[Rick-Rainer Ludwig]

There is another solution with a custom JUnit TestRunner. Have a look here: stackoverflow.com/questions/10537495/…