How do you measure the quality of your unit tests?

Question

If you (or your organization) aspires to thoroughly unit test your code, how do you measure the success or quality of your efforts?

• Do you use code coverage, what percentage do you aim for?
• Do you find that philosophies like TDD have a better impact than metrics?

Solution 1

My tip is not a way to determine whether you have good unit tests per se, but it's a way to grow a good test suite over time.

Whenever you encounter a bug, either in your development or reported by someone else, fix it twice. You first create a unit test that reproduces the problem. When you've got a failing test, then you go and fix the problem.

If a problem was there in the first place it's a hint about a subtlety about the code or the domain. Adding a test for it lets you make sure it's never going to be reintroduced in the future.

Another interesting aspect about this approach is that it'll help you understand the problem from a higher level before you actually go and look at the intricacies of the code.

Also, +1 for the value and pitfalls of test coverage already mentioned by others.

Solution 2

Code coverage is a useful metric but should be used carefully. Some people take code coverage, specially the percentage covered, a bit too seriously and see it as THE metric for good unit testing.

My experience tells me that more important than trying to get 100% coverage, which is not that easy, people should focus on checking the critical sections are covered. But even then you may get false positives.

Solution 3

I am very much pro-TDD, but I don't place much importance in coverage stats. To me, the success and usefulness of unit tests is felt over a period of development time by the development team, as the tests (a) uncover bugs up front, (b) enable refactoring and change without regression, (c) help flesh out modular, decoupled design, (d) and whatnot.

Or, as Martin Fowler put it, the anecdotal evidence in support of unit tests and TDD is overwhelming, but you cannot measure productivity. Read more on his bliki here: http://www.martinfowler.com/bliki/CannotMeasureProductivity.html

Solution 4

To attain a full measure of confidence in your code you need different levels of testing: unit, integration and functional. I agree with the advice given above that states that testing should be automated (continuous integration) and that unit testing should cover all branches with a variety of edge case datasets. Code coverage tools (e.g. Cobertura, Clover, EMMA etc) can identify holes in your branches, but not in the quality of your test datasets. Static code analysis such as FindBugs, PMD, CPD can identify problem areas in your code before they become an issue and go a long way towards promoting better development practices.

Testing should attempt to replicate the overall environment that the application will be running in as much as possible. It should start from the simplest possible case (unit) to the most complex (functional). In the case of a web application, getting an automated process to run through all the use cases of your website with a variety of browsers is a must so something like SeleniumRC should be in your toolkit.

However, software exists to meet a business need so there is also testing against requirements. This tends to be more of a manual process based on functional (web) tests. Essentially, you'll need to build a traceability matrix against each requirement in the specification and the corresponding functional test. As functional tests are created they are matched up against one or more requirements (e.g. Login as Fred, update account details for password, logout again). This addresses the issue of whether or not the deliverable matches the needs of the business.

Overall, I would advocate a test driven development approach based on some flavour of automated unit testing (JUnit, nUnit etc). For integration testing I would recommend having a test database that is automatically populated at each build with a known dataset that illustrates common use cases but allows for other tests to build on. For functional testing you'll need some kind of user interface robot (SeleniumRC for web, Abbot for Swing etc). Metrics about each can easily be gathered during the build process and displayed on the CI server (eg Hudson) for all developers to see.

Solution 5

If it can break, it should be tested. If it can be tested, it should be automated.

View more solutions

Comments

[Bill the Lizard]

Right. Hitting 100% code coverage doesn't mean you're done testing. There could still be a lot wrong with your code.

[Bill the Lizard]

When is it not possible to get 100% code coverage? Is it simply a time constraint?

[vfilby]

Joel phrases this as fix twice, I always liked that philosophy. +1

[Wedge]

@Bill, sometimes it's very difficult to mock everything, and in those cases it may be impossible to test some bits of code anywhere but in a live production environment.

[quamrana]

I realise now that 100% code coverage doesn't mean you're done testing, it just means that code coverage has ceased to be any further use. Its like the compiler saying 'Yup, that compiles', you know that's not the end of the story.

[vfilby]

It sounds like what you are saying is, "Test important things first." I am not sure I agree with the 100%/0% division. I do agree with testing key components first then working outwards though.

[quamrana]

That's OK, you don't have to agree with the 100%/0% thing. However, you might like to try it and see what happens.

[K J]

Essential: martinfowler.com/bliki/TestCoverage.html

[K J]

Thanks. Could you explain a little bit "the way" with checking critical sections? How do you derive this, based on architecture or language or wisdom or ... ?

[K J]

Good! But can you, so we can learn, elaborate on the "Test quality is determined by how well a test does its job in testing." How do you do that? What should a strategy look like? How do you remove bias from "Guru 1" vs "Guru 2" opinions etc.? Thanks

[K J]

Agree, but is that enough? What's behind the trend? More critical parts being tested? How to "inspect" unit tests? Ideas? Thanks!