Unit Tests failing when I Run All Tests but pass when I Debug

c# visual-studio unit-testing nunit nunit-3.0

14,575

Solution 1

The failing tests share a resource that affects them all when tested together. Recheck the affected tests and their subjects.

You should also look into static fields or properties in the subjects. They tends to cause issues if not used properly when designing your classes.

Solution 2

Some subtle differences might occur. For instance if a first test change a state which affects the behavior of a second test, then the outcome of this 2nd test may not be the same if I run it alone.

An idea to help understand a test failure when a breakpoint can't be used, could be to add logging.

Anyway, to answer your questions:

This ever happen to anyone else?

Yes

Am I doing something stupid or could there be a bug in VS2017 or NUnit or something?

I bet that it's neither: just a case a bit more subtle

14,575

Michael Hennigan

I'm an IT consultant working in the IT department of a large bank that has outsourced all of the actual technical IT work to even larger Indian IT companies. This means that I mostly do admin but also that I don't have enough work to do for most of the day. This is not what I wanted when I signed up to be an IT consultant. I am self-studying software development, learning through content freely available online. On a personal note, I am mad about sports especially the national sports of Ireland (I'm an extremely patriotic Irishman), Hurling and Gaelic Football.

Updated on June 04, 2022

Comments

Michael Hennigan almost 2 years

I'm using NUnit3 in Visual Studio 2017 and doing TDD. Something really strange is happening since I updated my code to make my latest test pass.

Now, 3 of my other tests are failing when I click Run All Tests, as below:

It is telling me that the actual and expected values in my Assert method are not equal.

However, when I put a breakpoint at the line where the Assert method is and start debugging, the stacktrace is showing that expected and actual are the same value and then the test passes, as below:

Am I doing something stupid or could there be a bug in VS2017 or NUnit or something?

This ever happen to anyone else?

[Edit: I should probably add that I have written each test as a separate class]
- Zenima almost 7 years
  
  Running one test at a time works, right?
- Michael Hennigan almost 7 years
  
  @Zenima Yeah, all three pass when run individually but fail when I click 'Run All Tests' which I don't understand because they are still unit tests, not integration tests.
- Nkosi almost 7 years
  
  @MichaelHennigan The failing tests share a resource that affects them all when tested together. Recheck the affected tests and their subjects.
- Michael Hennigan almost 7 years
  
  @Nkosi So could this happen because I use the same variable names in different tests?
- Nkosi almost 7 years
  
  @MichaelHennigan if that variable is global to the test class yes. but can't see if that is your case without seeing the offending code
- Michael Hennigan almost 7 years
  
  Never Mind, I tried giving the variables in each test class different names but this didn't work.
- Nkosi almost 7 years
  
  @MichaelHennigan You should also look into static fields or properties in the subjects. They tends to cause issues when not designed properly.
- Michael Hennigan almost 7 years
  
  @Nkosi Okay, thanks for the advice. I did introduce a static field in a production class when I did the code change that caused this problem so maybe that's it
Michael Hennigan almost 7 years

Thanks for getting back to my question. I thought there might be some odd but logical reason due to my code change but I have spent ages trawling through both my test code and production code and can't find a reason why the tests fail. What happens you click 'Run All Tests'? I don't understand why it is different than running each test one by one
Michael Hennigan almost 7 years

Thanks @Nkosi. When I commented out the static field and hit Run All Tests, the two older tests passed. However, my latest test is failing. I was using a static int nextID to give new user objects a UserID of NextID and then incrementing NextID. How can I achieve this functionality without messing up my unit tests?
Nkosi almost 7 years

@MichaelHennigan Ok that is a new issue. Ask a new question that properly explains the issue and I'll see how best I can help. I have some ideas but it would only be a guess without seeing the actual problem you are encountering. This is most probably a design issue.
Michael Hennigan almost 7 years

Or does it even matter? Should I just ignore the fact that my tests don't all pass when I Run All?
Nkosi almost 7 years

Do not ignore tests. designing your classes properly will fix the problem. Never ignore tests. they will come back to bite you (technical debt) in the future.
Michael Hennigan almost 7 years

Okay - new question here: stackoverflow.com/questions/43958823/…
Charlie almost 7 years

Think of it this way: One of your tests interferes with the others. They can only interfere if you run them all together.