JUnit vs TestNG

Question

At work we are currently still using JUnit 3 to run our tests. We have been considering switching over to JUnit 4 for new tests being written but I have been keeping an eye on TestNG for a while now. What experiences have you all had with either JUnit 4 or TestNG, and which seems to work better for very large numbers of tests? Having flexibility in writing tests is also important to us since our functional tests cover a wide aspect and need to be written in a variety of ways to get results.

Old tests will not be re-written as they do their job just fine. What I would like to see in new tests though is flexibility in the way the test can be written, natural assertions, grouping, and easily distributed test executions.

Solution 1

I've used both, but I have to agree with Justin Standard that you shouldn't really consider rewriting your existing tests to any new format. Regardless of the decision, it is pretty trivial to run both. TestNG strives to be much more configurable than JUnit, but in the end they both work equally well.

TestNG has a neat feature where you can mark tests as a particular group, and then easily run all tests of a specific group, or exclude tests of a particular group. Thus you can mark tests that run slowly as in the "slow" group and then ignore them when you want quick results. A suggestion from their documentation is to mark some subset as "checkin" tests which should be run whenever you check new files in. I never saw such a feature in JUnit, but then again, if you don't have it, you don't REALLY miss it.

For all its claims of high configuration, I did run into a corner case the a couple weeks ago where I couldn't do what I wanted to do... I wish I could remember what it is, but I wanted to bring it up so you know that it's not perfect.

The biggest advantage TestNG has is annotations... which JUnit added in version 4 anyways.

Solution 2

First I would say, don't rewrite all your tests just to suit the latest fad. Junit3 works perfectly well, and the introduction of annotations in 4 doesn't buy you very much (in my opinion). It is much more important that you guys write tests, and it sounds like you do.

Use whatever seems most natural and helps you get your work done.

I can't comment on TestNG b/c I haven't used it. But I would recommend unitils, a great wrapper for JUnit/TestNG/DBUnit/EasyMock, regardless of which route you take. (It supports all the flavors mentioned above)

Solution 3

TestNG's biggest draw cards for me include its support test groups, and more importantly - test group dependencies (marking a test as being dependent of a group causes the tests to simply skip running when the dependent group fails).

TestNG's other big draw cards for me include test parameters, data providers, annotation transformers, and more than anything - the vibrant and responsive user community.

Whilst on the surface one might not think all of TestNGs features above might not be needed, once you start to understand the flexibility bring to your tests, you'll wonder how you coped with JUnit.

(disclaimer - I've not used JUnit 4.x at all, so am unable to really comment on advances or new features there).

Solution 4

About a year ago, we had the same problem. I spent sometime considering which move was better, and eventually we realized that TestNG has no 'killer features'. It's nice, and has some features JUnit 4 doesn't have, but we don't need them.
We didn't want people to feel uncomfortable writing tests while getting to know TestNG because we wanted them to keep writing a lot of tests.
Also, JUnit is pretty much the de-facto standard in the Java world. There's no decent tool that doesn't support it from the box, you can find a lot of help on the web and they added a lot of new features in the past year which shows it's alive.

We decided to stick with JUnit and never looked back.

Solution 5

Cheers to all the above. Some other things I've personally found I like more in TestNG are:

1. The @BeforeClass for TestNG takes place after class creation, so you aren't constrained by only being able to call static methods of your class in it.

2. Parallel and parameterized tests, maybe I just don't have enough of a life... but I just get a kick writing one set of Selenium tests, accepting a driver name as a parameter. Then defining 3 parallel test groups, 1 each for the IE, FF and Chrome drivers, and watching the race! I originally did 4, but way too many of the pages I've worked on break the HtmlUnit driver for one reason or another.

Yeah, probably need to find that life. ;)

View more solutions

Comments

[Justin Standard]

JUnit can do the grouping thing you are talking about by defining a test suite and then adding the tests in the desired group to that suite. You can then set up a target in your ant script that only runs that suite, and set up your source control to run that target upon checkin.

[davetron5000]

The biggest advantage TestNG has over JUnit is the ability to dynanmically generate test data for parameterized tests. Each test data element is a different "test", so it makes it really easy to create data-driven tests testng.org/doc/documentation-main.html#parameters

[Mnementh]

Parametrized tests are easy done with Theories, that are integrated in newer versions of Junit (but are experimental at the moment).

[Kaitsu]

TestNG groups can be done in JUnit 4.8 with Categories: kentbeck.github.com/junit/doc/ReleaseNotes4.8.html.

[Chinasaur]

Unitils doesn't look like it has been updated in a while; will it work with newer versions of JUnit/TestNG?

[Ogre Psalm33]

Just for anyone finding this answer in 2011, I checked, and the latest unitils version (3.2), has a release date of: 2011-09-29. So it is being actively maintained.

[hidralisk]

I am using both JUnit4 and TestNG, TestNG has better support for spring spring-test integration. Makes testing spring based application a lot easier.

[user2412398]

finally some meaty comment after all the ummm they are both equal ..and awwhing

[djangofan]

Yeah, TestNG rules: I load driver instances via test args from the TestNG DataProvider as well. here is how I did it: github.com/djangofan/yet-another-selenium-framework/blob/mas‌​ter/…

[djangofan]

Some things not mentioned: I think TestNG's best advantage is being able to pass the same parameters that you passed into each test method, also into the before/after test configuration methods: this is great help in setup and teardown, IMHO very powerful. If you dont understand that, you might think you dont need it. Also, I like how you can define test suites in code.

[djangofan]

IMHO, TestNG's killer feature is the ability to pass context args in via Before and After setup methods. You can do a lot of neat magic tricks there that Junit cannot do. 95% of people don't bother to learn TestNG that well and are ignorant of it. Also, TestNG has a neat way of threading via DataProvider, but only if you take advantage of it. Also, testng.xml can contain Beanshell.

[djangofan]

You didn't list TestNG's most distinct feature, IHMO: the ability to pass context args into Before and After methods, which enables you do do some magic. The DataProvider works this way, as an example.