How does google test make test sequence

Question

How google-test makes test sequence (or order of test case execution) for testing test cases?

Suppose I have 5 test cases.

TEST(First, first)
TEST(Secnd, secnd)
TEST(Third, third)
...
TEST(Fifth, fifth)

How google-test test above test cases? I mean in what sequence? Or can we provide any test sequence?

Solution 1

The advanced reference pages for googletest in the chapter Shuffling the Tests tells :

By default, Google Test uses a random seed calculated from the current time. Therefore you'll get a different order every time.

This is actually a good way of unit testing, since tests should not depend on the order of execution.

As far as I know, there are no ways of setting the order of tests execution. The only parameter you can set is the seed, used to set the same order of execution.

Solution 2

By default it will test them in the order it finds them at link time, which will depend upon your tools.

You can select which tests to run, such as a subset, or a single test.

There is also an option to run them in a random order.

Solution 3

By default they run in the declaration order. As said by other, you have to provide the flag --gtest_shuffle to shuffle them.

Solution 4

Even if you can guess some pattern for the order of execution (as written, or linked) you shouldn't depend on that.

It would be repeated on different executions, though. If you don't want it to happen, you can use --gtest_shuffle. That runs the test on a random order, according to a random seed.

In case of failure, you can use --gtest_random_seed= with that number and repeat the exact sequence (to investigate why it failed).

That said, the randomness is not complete:

• Test suites will be run in a random order
• Tests within a test suite will be run in a random order

If not run in this way, SetUpTestSuite and TearDownTestSuite methods would be mixed. You don't need a fixture for this grouping to happen, though.

View more solutions

Comments

[user1284631]

That (random order of execution) is true only if you specify the --gtest_shuffle flag.

[NuPagadi]

it states, they don't: Remember that the test order is undefined, so your code can't depend on a test preceding or following another.

[Simon Pickup]

The given reference doesn't appear to say that tests are run in declaration order. More likely it is dependent on tool order, as per another answer.

[Kevin]

"tests should not depend on the order of execution" What if one small test failing means many other tests fail? In that case it's faster to run the small test first and abort when it fails.

[BЈовић]

@Kevin no, it is better not to abort, because in that case you wouldn't see what other tests are failing. You can fix more tests at once. Executing unit tests should be very very fast, therefore it doesn't matter if a test is big or small. Time it takes me to execute 600 unit tests for one of my libraries takes me 0.24 seconds. My PC is middle class.

[Cort Ammon]

There are times where it is useful to have an order of execution. My current project had a handful of "bootstrap" tests which test the underlying functionality, along with dozens of higher-level tests which unit test higher level functionality. If something goes wrong in the bootstrap, it's nice to know then, rather than having to wade through dozens of higher level tests that all failed.

[yaobin]

On Jul 17, 2018, they updated the doc so now (2020-08-26) it is "By default, googletest uses a random seed calculated from the current time. Therefore you'll get a different order every time." The random order has become a default behavior, although --gtest_shuffle surely still works.