Unit Test fails on Run All but not on Run Selected Tests with all selected

Question

I'm getting some weird behavior. If I click "Run All" in Test Explorer then 1 of my unit tests fails, but if I select all tests and click "Run Selected Tests" then the unit test passes.

The test that is failing is throwing a reflection error: System.Reflection.TargetException: Non-static method requires a target. on a type which is defined in the dll code which I'm testing. There doesn't appear to be anything odd with the class - there are plenty of other classes defined in the dll which reflection is happy with. I've included the test stack trace below.

NB It is a complex test - it reads inputs & expected answers from an .xlsx file, populates a LocalDb with data from the xlsx, performs a calculation using the data in the LocalDB and then compares the calculated to expected values. However as I say it is working and works when I run all tests (using select all > Run Selected Test).

What is different about Run All? Any insights would be appreciated.

I have tried a clean and rebuild with no luck. Catching and logging reflection error indicates that the GetValue call is throwing for every property I attempt to access on the type - but only when run by "Run All" and only on this one type? (if I catch the error then all GetValues succeed on all other types).

Stack Trace

Test Name:  IT_CheckCashOnly1DepositOutputValues
Test FullName:  Lib.AE.Tests.Integration.CalculationTests.IT_CheckCashOnly1DepositOutputValues
Test Source:    c:\netreturn.co.za\Main\NetReturn\Lib.AE.Tests\IntegrationTests\CalculationTest.cs : line 23
Test Outcome:   Failed
Test Duration:  0:00:00.1661906

Result Message: 
Test method Lib.AE.Tests.Integration.CalculationTests.IT_CheckCashOnly1DepositOutputValues threw exception: 
System.Reflection.TargetException: Non-static method requires a target.
Result StackTrace:  
at System.Reflection.RuntimeMethodInfo.CheckConsistency(Object target)
at System.Reflection.RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, BindingFlags invokeAttr, Binder binder, Object[] index, CultureInfo culture)
at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, Object[] index)
at Lib.AE.Xlsx.XlsxHelper.Compare[T](T expected, T calculated, ExcelWorksheet ws, Int32 r, Int32 colStart, Boolean& valid) in c:\netreturn.co.za\Main\NetReturn\Lib.AE\Xlsx\XlsxHelper.cs:line 101
at Lib.AE.Xlsx.XlsxWorkSheet_SharePNL.CompareXlsx(ExcelPackage pck, List`1 expectedXlsx, ValuationCalculation calc) in c:\netreturn.co.za\Main\NetReturn\Lib.AE\Xlsx\XlsxSharePNL.cs:line 143
at Lib.AE.Tests.Integration.CalculationTests.CheckCalculationResults(String xlsxDocToLoad, WorkSheets testingScenarios) in c:\netreturn.co.za\Main\NetReturn\Lib.AE.Tests\IntegrationTests\CalculationTest.cs:line 64
at Lib.AE.Tests.Integration.CalculationTests.IT_CheckCashOnly1DepositOutputValues() in c:\netreturn.co.za\Main\NetReturn\Lib.AE.Tests\IntegrationTests\CalculationTest.cs:line 23

---

Solution

So this turned out to be (a) my problem - my unit test shared state with another unit test and (b) an order problem. Be aware that it's not obvious which order TestExplorer will run your tests in. I created a new UnitTestProject with 2 x UnitTest .cs files and three TestMethods each i.e.:

UnitTest1.cs

    [TestMethod]
    public void ONE_AAA() {}

    [TestMethod]
    public void ONE_BBB() {}

    [TestMethod]
    public void ONE_CCC() {}

UnitTest2.cs

    [TestMethod]
    public void TWO_CCC() {}

    [TestMethod]
    public void TWO_BBB() {}

    [TestMethod]
    public void TWO_AAA() {}

Then ran these tests via two methods i.e. (1) Run All (2) Select all and Run Selected Tests, and logged the order in which the tests were started by TestExplorer. The results for Run Selected Tests are fairly unintuitive:

-- Run All
2013-01-16 11:53:47.4062 INFO TestInitialize: ONE_AAA
2013-01-16 11:53:47.4122 INFO TestCleanup: ONE_AAA
2013-01-16 11:53:47.4122 INFO TestInitialize: ONE_BBB
2013-01-16 11:53:47.4122 INFO TestCleanup: ONE_BBB
2013-01-16 11:53:47.4122 INFO TestInitialize: ONE_CCC
2013-01-16 11:53:47.4282 INFO TestCleanup: ONE_CCC
2013-01-16 11:53:47.4282 INFO TestInitialize: TWO_CCC
2013-01-16 11:53:47.4282 INFO TestCleanup: TWO_CCC
2013-01-16 11:53:47.4282 INFO TestInitialize: TWO_BBB
2013-01-16 11:53:47.4282 INFO TestCleanup: TWO_BBB
2013-01-16 11:53:47.4282 INFO TestInitialize: TWO_AAA
2013-01-16 11:53:47.4282 INFO TestCleanup: TWO_AAA

-- Select All > Run Selected
2013-01-16 11:55:26.0139 INFO TestInitialize: TWO_BBB
2013-01-16 11:55:26.0139 INFO TestCleanup: TWO_BBB
2013-01-16 11:55:26.0249 INFO TestInitialize: ONE_BBB
2013-01-16 11:55:26.0249 INFO TestCleanup: ONE_BBB
2013-01-16 11:55:26.0249 INFO TestInitialize: TWO_AAA
2013-01-16 11:55:26.0249 INFO TestCleanup: TWO_AAA
2013-01-16 11:55:26.0249 INFO TestInitialize: TWO_CCC
2013-01-16 11:55:26.0249 INFO TestCleanup: TWO_CCC
2013-01-16 11:55:26.0249 INFO TestInitialize: ONE_CCC
2013-01-16 11:55:26.0249 INFO TestCleanup: ONE_CCC
2013-01-16 11:55:26.0249 INFO TestInitialize: ONE_AAA
2013-01-16 11:55:26.0249 INFO TestCleanup: ONE_AAA

Solution 1

I have been caught out in the past with running multiple unit tests in that the order in which the tests run may not be in the order in which they are declared in the test class, and may in fact be in order of test method name. e.g. if I have

[Test]
public void PreviousTest()
{
}

[Test]
public void LaterTest()
{
}

Then LaterTest gets run first as its name appears before PreviousTest when ordered alphabetically.

This won't matter if all of your tests are entirely independent, but if they are modifying shared resources then you may get unusual behaviour if you were expecting that LaterTest's changes wouldn't have any effect on PreviousTest due to it being declared second.

Solution 2

Make sure your members are being initialized in the setup scope, otherwise they can be reused.

[TestFixture]
public class RegistrationInteractorTests
{
    private  IRegistrationService _registrationService;
    private  IRegistrationValidationService _registrationValidationService;
  ......

    [SetUp]
    public void Init()
    {
        _registrationService = A.Fake<IRegistrationService>();
        _registrationValidationService = A.Fake<IRegistrationValidationService>();
    }

....

}

// The wrong way would be like this:

 [TestFixture]
    public class RegistrationInteractorTests
    {
        protected readonly IRegistrationService _registrationService = A.Fake<IRegistrationService>();

        protected readonly IRegistrationValidationService _registrationValidationService =
            A.Fake<IRegistrationValidationService>();



        [SetUp]
        public void Init()
        {

        }

Solution 3

We tried to investigate this issue for a week now. Make sure that all your TEST Projects are using the same version of DLLs, especially the Nuget Packages.

The thing is that, when you execute the "Run All", all the used libraries in all your TEST Projects are placed in one folder (something like this MyApp\TestResults\Deploy_user1 2018-11-20 15_52_15\Out). and because of these Dll libraries conflicted.

Comments

[Codeman]

Please post your test code.

[Ilan]

It's too big to post - it's my entire codebase (the problem only occurs when I run all). The even the single unit test which fails is large. I'm trying to resolve this now - i.e. seeing whether I can separate out a smaller unit test project which still exhibits the same behaviour, but it may take a while.

[DeanOC]

+1 for posting the detailed solution

[dano]

I had this problem as well. I realized I had a static variable in a class that wasn't being reset every [Test]

[TamaMcGlinn]

How did you log the order in which the tests were run?

[Ilan]

I don't think it is a problem relating to multiple tests (i.e. another unit test changing state which affects the running of the failing test) - because it works when I run multiple tests. It something specific with "Run All" and perhaps reflection. Any idea what "Run All" does differently? Hmmm.. actually it be quicker for me to log the order in which unit tests are running, to verify whether 'select all > run selected' runs tests in the same order as 'run all'. I'll try that first before trying to cause the same behaviour in a smaller unit tests (as per my comment above).

[Ilan]

Thanks Dean, it was in fact an order problem after all. Selecting tests and 'Run Selected' can use a different order to 'Run all'. The reflection exception was a red herring (brought on by the order problem and the dependency between the failing test and a previous test. I'm going to write some additional notes in my question to help others troubleshoot this type of problem