How to diagnose "TestFixtureSetUp Failed"

Question

We use TeamCity as our CI server, and I've just started seeing "TestFixtureSetUp Failed" in the test failure window.

Any idea how I go about debugging this problem? The tests run fine on my workstation (R# test runner in VS2008).

Solution 1

It is a bit of a flaw in the implementation of TestFixtureSetUp (and TestFixtureTearDown) that any exceptions are not well reported. I wrote the first implementation of them and I never got it to work the way it was supposed to. At the time the concepts in the NUnit code were tightly coupled to the idea that actions were directly related to a single test. So the reporting of everything was related to a test result. There wasn't really a space for reporting something that happened at the suite level without a huge re-write (it isn't a refactoring when you change a sheep into an escalator).

Because of that bit of history it's hard to find out what really happened in a TestFixtureSetUp. There isn't a good place to attach the error. The TestFixtureSetUp call is a side effect of running a test instead of being directly related to it.

@TrueWill has the right idea. Check the logs and then modify the test to add more logging if necessary. You might want to put at try/catch inside the TestFixtureSetup and log a lot in the catch block. I just thought I could add some background to it (in other words it's kind of my fault).

Solution 2

I'd check the Build Log first.

If it's not obvious from that, you could try including Console.WriteLines in the tests - I'm not positive, but I think those are written to the Build Log. Alternately you could log to a file (even using log4net if you wanted to get fancy).

If you have Visual Studio installed on the CI server, you could try running the build/tests from there. If it's a connectivity issue, that might resolve it.

I've seen path issues, though, where relative paths to files were no longer correct or absolute paths were used. These are harder to debug, and might require logging the paths and then checking if they exist on the build server.

Solution 3

I ran into this today when creating some integration tests that have long running setup that I don't want to duplicate. I ended up wrapping all the test fixture setup logic in a try/catch. I then add a SetUp method whose sole purpose is to see if a failure occurred during fixture setup and provide better logging.

Exception testFixtureSetupException = null;

[TestFixtureSetUp]
public void FixtureSetup()
{
    try
    {
        // DoTestFixtureSetup
    }
    catch (Exception ex)
    {
        testFixtureSetupException = ex;
    }
}

[SetUp]
// NUnit doesn't support very useful logging of failures from a TestFixtureSetUp method. We'll do the logging here.
public void CheckForTestFixturefailure()
{         
    if (testFixtureSetupException != null)
    {
        string msg = string.Format("There was a failure during test fixture setup, resulting in a {1} exception. You should check the state of the storage accounts in Azure before re-running the RenewStorageAccountE2ETests. {0}Exception Message: {3}{0}Stack Trace:{4}",
            Environment.NewLine, testFixtureSetupException.GetType(), accountNamePrefix, testFixtureSetupException.Message, testFixtureSetupException.StackTrace);
        Assert.Fail(msg);
    }
}

Solution 4

I was getting the same error while running any test with SpecFlow using Visual NUnit. When I tried doing the same from the Unit Test Explorer(provided by Resharper), it gave a little more helpful message: Binding methods with more than 10 parameters are not supported. I realized I can't have a SpecFlow method with more than 10 params, had to remove the test.

Solution 5

I was able to see that I was not creating my test database correctly by doing a quick switch to VS Unit Testing. In my Case it was able to return a better response to the reason why it failed. I usually use NUnit. "Unable to create instance of class X. Error: System.Data.SqlClient.SqlException: A file activation error occurred. The physical file name '\DbTest.mdf' may be incorrect. Diagnose and correct additional errors, and retry the operation. CREATE DATABASE failed. Some file names listed could not be created. Check related errors.. "

View more solutions

Comments

[TrueWill]

@Mike - thanks! BTW, I have few complaints about NUnit - I've been using it happily for as long as I've been working in .NET.

[Dhir Pratap]

fyi you can use a ReadOnlyDictionary

[Charlie]

Nunit itself reports the information these days. So he could simply run the test under NUnit-Console to see what is causing the error. The real problem is that the VS Test Window IDE doesn't provide any place to report errors that are not directly associated with a specific test - reminiscent of the early days of NUnit itself, in fact.

[Christian Hagelid]

rep boost for "it isn't a refactoring when you change a sheep into an escalator"