JUnit @Ignore usefulness?

Question

In JUnit you can use @Ignore before methods to tell the test runner to automatically skip those tests. From what I can gather, this is really only a convenient way to document/mark off incomplete/no longer functional tests that you want to get back to later.

Am I correct in saying then, that at runtime there is no difference between an @Ignore test, a method with no annotation, and a commented out method? (Assuming these tests are all self contained.) Is there some way to obtain a list of the ignored test cases in JUnit on Netbeans? If not, how much usefulness does the @Ignore tag really have, since it might be more useful to fail the test so it's not overlooked?

Solution 1

Am I correct in saying then, that at runtime there is no difference between an @Ignore test, a method with no annotation, and a commented out method?

An @Ignored method can be found via reflection. A method with no annotation can't (or, to be precise, it can't be identified with certainty as an ignored test method), and a commented out method does not even get into the bytecode.

Albeit I don't think there would be much practical value in finding @Ignored methods runtime, it may be useful to generate statistics / reports.

how much usefulness does the @Ignore tag really have

One thing I can think of is searchability. You can easily identify all @Ignore annotations in the source code, while unannotated or commented out tests are not so simple to find.

it might be more useful to fail the test so it's not overlooked?

If you want (and can) fix it right away, it is fine to have it fail. There are cases when you can't, but you still want to method to be around, precisely so that it does not get forgotten. Then @Ignore makes sense.

Solution 2

• A method with no annotation most likely is a utility method, likely used by other methods in the test.

• A commented-out method is one that is completely hidden.

• But you might use @Ignore to indicate that you are temporarily disabling the test. If you have a large and complicated piece of code that is in transition, you might want to temporarily disable some tests while that code is failing, especially if the mechanism running that code is run as part of some larger regression suite. The @Ignore attribute would be a reminder to you (or your QA team) that there is still more to do.

  Remember that @Ignore'd tests are still live code, and can be seen and even executed via reflection, which commented-out code cannot be. And it also means when you refactor other code, they are also updated, and syntax breakage in an API will show there. So even tests with @Ignore set can actively contribute to quality of code.

• JUnit 4 TestRunners will show @Ignore'd tests as "skipped" so you can see in the end how many tests passed/failed/skipped.

Although your copy of Netbeans may not take full advantage of @Ignore, I can assure you that it is not the only consumer of JUnit tests.

Solution 3

@Ignored tests are more maintainable (and maintained) than commented-out ones. They are subject to compiler errors as things evolve, structured searching via IDE's, and IDE-assisted refactoring.

Solution 4

FYI, TestNG supports this slightly differently:

• You can include a description of why the test is being ignored:

  @Test(enabled = false, description = "Disabled until BUG-1234 gets fixed")
  

  • The Eclipse TestNG plug-in and the HTML reports will give you a list of all the tests that have been disabled, along with the reason why:

Solution 5

Using @Ignore instead of commenting out @Test has the advantage that Hudson/Jenkins recognizes this as an ignored test and displays it in the test results accordingly. So you get reminded about that ignored Test which is a big benefit!

View more solutions

Comments

[donnyton]

true, those are the implications of the different ways to disable a test. but at runtime they all become equivalent, right? (assuming it is not a utility method)

[donnyton]

I've heard this, but where do I find the reports? Netbeans does not report it in the visual JUnit runner.

[lavinio]

Except for two things: 1. @Ignore'd tests are visible via reflection, so there is some additional information conveyed there. 2. If you comment out a test, you'll never know that you broke your API causing a potential syntax error. Keeping the code there and @Ignore'd means at least the syntax is still correct during code changes.

[Jihed Amine]

I don't know about Netbeans, I use maven commands in a terminal (mvn test / integration-test). The reports are generated in target/failsafe-reports or surefire-reports by default but the destination folder can be customized. But in Netbeans, don't you get a summary with X Tests, X1 Failures, X2 Errors, X3 Skipped ? The number of skipped tests is inferred from the @Ignore annotations.

[Nate]

@Ignore tests are also (or should be) reported by JUnit4 TestRunners as "skipped" tests - which is still useful information in reports.

[vkraemer]

@lavinio : you may want to integrate your 'except for two things' comment into the text of your answer... it would improve the answer.

[matbrgz]

junit 4.11 supports @Ignore("Description"). I do not know when that was introduced.