Exclude individual JUnit Test methods without modifying the Test class?

Question

I'm currently re-using JUnit 4 tests from another project against my code. I obtain them directly from the other project's repository as part of my automated Ant build. This is great, as it ensures I keep my code green against the very latest version of the tests.

However, there is a subset of tests that I never expect to pass on my code. But if I start adding @Ignore annotations to those tests, I will have to maintain my own separate copy of the test implementation, which I really don't want to do.

Is there a way of excluding individual tests without modifying the Test source? Here's what I have looked at so far:

• As far as I can see, the Ant JUnit task only allows you to exclude entire Test classes, not individual test methods - so that's no good for me, I need method granularity.

• I considered putting together a TestSuite that uses reflection to dynamically find and add all of the original tests, then add code to explicitly remove the tests I don't want to run. But I ditched that idea when I noticed that the TestSuite API doesn't provide a method for removing tests.

• I can create my own Test classes that extend the original Test classes, override the specific tests I don't want to run, and annotate them with @Ignore. I then run JUnit on my subclasses. The downside here is that if new Test classes are added to the original project, I won't pick them up automatically. I'll have to monitor for new Test classes as they are added to the original project. This is my best option so far, but doesn't feel ideal.

• The only other option I can think of is to run the bad tests anyway and ignore the failures. However, these tests take a while to run (and fail!) so I'd prefer to not run them at all. Additionally, I can't see a way of telling the Ant task to ignore failures on specific test methods (again - I see how you can do it for individual Test classes, but not methods).

Solution 1

If you can't touch the original test at all you are going to have some serious limitations. Your overriding sounds like the best bet, but with a couple of changes:

Build the Ant tests specifically excluding the super classes, so that additional classes that you don't know about get run.

You can use the @Rule annotation (new to JUnit 4.7) to know what test is being run and abort it (by returning an empty Statement implementation) rather than overriding specific methods, giving you more flexibility in knowing whether or not to avoid the test. The only problem with this method is that you can't stop the @Before methods from running using this method, which may be slow. If that is a problem (and you really can't touch the tests) then @Ignore in the overridden method is the only thing I can think of.

If, however, you can touch those tests, some additional options open up:

You could run them with a custom runner by specifying the @RunWith tag on the class. This runner would just pass over execution to the standard runner (JUnit4.class) in that project, but in your project (via a system property or some other mechanism) would inspect the test name and not run a test. This has the advantage of being the least intrusive, but the most difficult to implement (runners are hairy beasts, one of the stated goals of @Rule was to eliminate most of the need to make them).

Another is to make an assumeThat statement on the test that would check some configuration setting that would be true if that test should run. That would actually involve injecting right into the test, which is most likely a deal breaker in anything remotely labeled a "separate project."

Solution 2

It doesn't help you now, but TestNG supports this sort of ability.

Solution 3

OK, this is a rather heavyweight solution, but don't throw things at me if it sounds ridiculous.

The core of Junit4 is the org.junit.runner.Runner class, and its various subclasses, most importantly org.junit.runners.Suite. These runners determine what the tests are for a given test class, using things like @Test and @Ignore.

It's quite easy to create custom implementations of a runner, and normally you would hook them up by using the @RunWith annotation on your test classes, but obviously that's not an option for you.

However, in theory you could write your own Ant task, perhaps based upon the standard Ant Junit task, which takes your custom test runner and uses it directly, passing each test class to it in turn. Your runner implementation could use an external config file which specifies which test methods to ignore.

It'd be quite a lot of work, and you'd have to spend time digging around in the prehistoric Ant Junit codebase to find out how it works. The investment in time may be worth it, however.

It's just a shame that the Junit Ant task doesn't provide a mechanism to specify the test Runner, that would be ideal.

Solution 4

A possibility I can think of to achieve what you want with the stated constraints is to use bytecode modification. You could keep a list of classes and methods to ignore in a separate file, and patch the bytecode of the test classes as you load them to remove this methods altogether.

If I am not mistaken, JUnit uses reflection to find the test methods to execute. A method rename operation would then allow you to remove these methods before JUnit finds them. Or the method can be modified to return immediately, without performing any operation.

A library like BCEL can be used to modify the classes when loaded.

Solution 5

If you want to run only a subset of the tests it sounds like that class has more than one responsibility and should be refactored down. Alternately the test class could be broken apart so that the original project had all the tests but on one or more classes(I'm guessing some of the tests are really integration tests and touch the database or network) and you could exclude the class(es) you didn't want.

If you can't do any of that, your option of overriding is probably best. Take the process of whenever you need to ignore some methods you extend that class and add it to your Ant exclude list. That way you can exclude what you can't pass and will still pull in all new tests (methods you didn't override and new test classes) without modifying your build.

View more solutions

Comments

[Romczyk]

No, unfortunately I need to exclude specific test methods. That is, I want to run some tests in a given Test class but not others.

[Yishai]

I would say implementing a runner is deceptively simple. I found myself copying and pasting code from other parts of JUnit to get a custom one to work properly, and it was very brittle - pretty much guaranteed to break on a new release.

[Yishai]

JUnit 4 uses annotations instead of method names to determine the tests to run.

[Mario Ortegón]

It would be still valid to modify the method to return immediately. But thanks, I was thinking about JUnit3

[Romczyk]

Presumably bytecode modification could also be used to add @Ignore annotations to methods in compiled classes.

[skaffman]

All true. I never claimed it was elegant :)