Simulation of Service using Mockito 2 leads to stubbing error

Question

I try to simulate the behaviour of a class, using Mockito. This worked using Mockito 1.x. Migrating to JUnit 5 and Mockito 2 it seems not to work anymore.

@ExtendWith(MockitoExtension.class)
public class MockitoExample {

  static abstract class TestClass {
    public abstract int booleanMethod(boolean arg);
  }

  @Mock
  TestClass testClass;

  @BeforeEach
  public void beforeEach() {
    when(testClass.booleanMethod(eq(true))).thenReturn(1);
    when(testClass.booleanMethod(eq(false))).thenReturn(2);
  }

  @Test
  public void test() {
    assertEquals(1,testClass.booleanMethod(true));
    assertEquals(2,testClass.booleanMethod(false));
  }
}

The expectation is, that the mocked TestClass shows the behaviour as tested in the test-method.

The error I get is:

org.mockito.exceptions.misusing.PotentialStubbingProblem: 

  Strict stubbing argument mismatch. Please check:
   - this invocation of 'booleanMethod' method:
      testClass.booleanMethod(false);
      -> at org.oneandone.ejbcdiunit.mockito_example.MockitoExample.beforeEach(MockitoExample.java:30)
   - has following stubbing(s) with different arguments:
      1. testClass.booleanMethod(false);
        -> at org.oneandone.ejbcdiunit.mockito_example.MockitoExample.beforeEach(MockitoExample.java:29)
  Typically, stubbing argument mismatch indicates user mistake when writing tests.
  Mockito fails early so that you can debug potential problem easily.
  However, there are legit scenarios when this exception generates false negative signal:
    - stubbing the same method multiple times using 'given().will()' or 'when().then()' API
      Please use 'will().given()' or 'doReturn().when()' API for stubbing.
    - stubbed method is intentionally invoked with different arguments by code under test
      Please use default or 'silent' JUnit Rule (equivalent of Strictness.LENIENT).
  For more information see javadoc for PotentialStubbingProblem class.

In both cases, the argument false seems to be matched, though I clearly matched with true.

Is that a bug in Mockito 2.17 or a misunderstanding. How should/can I use Mockito 2.x to simulate calls with different boolean arguments?

The example can also be found on github. But surefire will start the test only using

mvn test -Dtest=MockitoExample

Executing the test using Mockito 2.21 leads to the same results.

Solution 1

Since Mockito 2.20 it is also possible, to add lenient() locally

@ExtendWith(MockitoExtension.class)
public class MockitoExample {

  static abstract class TestClass {
    public abstract int booleanMethod(boolean arg);
  }

  @Mock
  TestClass testClass;

  @BeforeEach
  public void beforeEach() {
    lenient().when(testClass.booleanMethod(eq(true))).thenReturn(1);
    lenient().when(testClass.booleanMethod(eq(false))).thenReturn(2);
  }

  @Test
  public void test() {
    assertEquals(1,testClass.booleanMethod(true));
    assertEquals(2,testClass.booleanMethod(false));
  }
}

Solution 2

With strict stubs (the default behavior of Mockito) calling several whens on the same method will reset that mock. The solution is to call when once and have the logic in an Answer:

@BeforeEach
public void beforeEach() {
    when(testClass.booleanMethod(anyBoolean())).thenAnswer(invocationOnMock -> {
        if ((boolean) invocationOnMock.getArguments()[0]) {
            return 1;
        }
        return 2;
    });
}

Alternatively, you can use lenient mocking, but that's not always a good idea - lenient mocking allows redundant stubbing, and makes it easier for you to make mistakes in your test, which may lead to unnoticed bugs in the "production" code:

@ExtendWith(MockitoExtension.class)
@MockitoSettings(strictness = Strictness.LENIENT)
public class MockitoExample {

Solution 3

Mockito 1 and 2 don't have the same "strictness" level.
Besides by using Mockito 2 with JUnit 4 or 5 the default level will be still different.

To sum up :

3 levels of strictness :

• LENIENT : minimum strictness
• WARN : extra warnings emitted to the console
• STRICT_STUBS : ensures clean tests by throwing exception if potential misuse but may also produce some false positives.

Default effective level according to the APIs used :

• Mockito 1 : LENIENT
• Mockito 2 with JUnit 4 : WARN
• Mockito 2 with JUnit 5 (MockitoExtension.class) : STRICT_STUBS
• Mockito 3 : planned to be STRICT_STUBS.

More details

The actual Mockito documentation is very clear about that :

The Strictness javadoc states :

Configures the "strictness" of Mockito during a mocking session.A session typically maps to a single test method invocation. Strictness drives cleaner tests and better productivity.The easiest way to leverage enhanced Strictness is usingMockito's JUnit support (MockitoRule or MockitoJUnitRunner).If you cannot use JUnit support MockitoSession is the way to go.

How strictness level influences the behavior of the test (mocking session)?

1.Strictness.LENIENT - no added behavior.The default of Mockito 1.x.Recommended only if you cannot use STRICT_STUBS nor WARN.

2.Strictness.WARN - helps keeping tests clean and improves debuggability.Reports console warnings about unused stubsand stubbing argument mismatch (see org.mockito.quality.MockitoHint).The default behavior of Mockito 2.x when JUnitRule or MockitoJUnitRunner are used. Recommended if you cannot use STRICT_STUBS.

3.Strictness.STRICT_STUBS - ensures clean tests, reduces test code duplication, improves debuggability.Best combination of flexibility and productivity. Highly recommended.Planned as default for Mockito v3.See STRICT_STUBS for the details.

But whatever the thrown exception associated to the message

"has following stubbing(s) with different arguments"

seems to be a excessively strict check. The exception message proves that in a some way :

However, there are legit scenarios when this exception generates false negative signal:

...

• stubbed method is intentionally invoked with different arguments by code under test

So forbidding it by default seems to be too much.
So if you use JUnit 5, as alternative to STRICT_STUBS you could use WARNING but you generally want to avoid LENIENT that is too quiet.

In addition to MockitoExtension, the mockito-junit-jupiter library provides @MockitoSettings that may be used at the method level as well as at the class level.

Here is an example :

import java.util.List;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mockito;
import org.mockito.junit.jupiter.MockitoExtension;
import org.mockito.junit.jupiter.MockitoSettings;
import org.mockito.quality.Strictness;

@ExtendWith(MockitoExtension.class)
public class FooTest {

    @MockitoSettings(strictness = Strictness.WARN)
    @Test
    void foo() throws Exception {
        List<String> strings = Mockito.mock(List.class);
        Mockito.when(strings.add("a"))
               .thenReturn(true);
        Mockito.when(strings.add("b"))
               .thenReturn(false);
    }

    @Test
    void fooKo() throws Exception {
        List<String> strings = Mockito.mock(List.class);
        Mockito.when(strings.add("a"))
               .thenReturn(true);
        Mockito.when(strings.add("b"))
               .thenReturn(false);

    }

}

fooKo() throws the misuse Mockito exception while foo() is successful but provides helpful warnings :

[MockitoHint] FooTest (see javadoc for MockitoHint):
[MockitoHint] 1. Unused -> at FooTest.foo(FooTest.java:19)
[MockitoHint] 2. Unused -> at FooTest.foo(FooTest.java:21)

As other alternative you can also use Mockito.lenient() very well described by aschoerk to apply the lenient strictness for a specific invocation. As well as you can set every mock invocations as lenient at the mock instantiation :

@Test
void foo() throws Exception {
    List<String> strings = Mockito.mock(List.class, Mockito.withSettings()
                                                           .lenient());
     ....
}

Comments

[aschoerk]

Really? I have to use Answers, interpret the arguments there for which originally a very fine DSL was implemented and make the stubs the more prone for mistakes? In that case it seems to be better to drop mockito usage and inject real simulations (at least, if you are using spring or weld)

[aschoerk]

sorry, thank you very much for the answer. That seems to be a way to go forward.

[Draken]

If you have a new question, please ask it by clicking the Ask Question button. Include a link to this question if it helps provide context. - From Review

[johanneslink]

@Draken It would have been better as a comment but comments don't allow code snippets. And no, I don't have an additional question since everything works as I expect it to.

[johanneslink]

@aschoerk I cloned your repo, CDed to mockito-example, removed the pom dependency to ejb-cdi-unit since I don't have it and voila: mvn test -> BUILD SUCCESS. But it fails in IntelliJ. Hm.

[Draken]

@johanneslink I think you've misunderstood how StackOverflow works, it's not a forum. This area is only for answering the OP's question, and not asking further questions. You've asked a question as you've used the word, why? If it's meant to be an answer to the question, please update it so it takes on that form. Answers shouldn't generally end with a question in the final line. If you're trying to get the OP to debug the question and find out through self discovery, then you need to be clearer about it.

[aschoerk]

@johanneslink Perhaps you did not use "mvn clean install -Dtest=MockitoExample" or you used an older version using Lenient.

[johanneslink]

@Draken Don't think I've misunderstood it. Sometimes you need a conversation to clear up a question. But do as you please, delete my answer.

[johanneslink]

@aschoerk It's failing for me in IntelliJ, too. I'll find out, but it will take some time.

[Draken]

@johanneslink Please read the meta on this topic and here. This section is only for answering and not for conversation. If you need to ask the OP for further clarification, then use the comments. You shouldn't just use the answer area because it suits your needs better.

[Draken]

They do, You need to use the `

[johanneslink]

@aschoerk It's the extension that makes the difference. If I remove @ExtendWith(MockitoExtension.class) test1 works. Maybe the MockitoExtensions changes some Mockito configuration.

[johanneslink]

@Draken interface Poops { String get(boolean is); } does look really good. yes. @Test void test1() { Poops a = mock(Poops.class); when(a.get(eq(true))).thenReturn("1"); when(a.get(eq(false))).thenReturn("2"); Assertions.assertEquals("1", a.get(true)); Assertions.assertEquals("2", a.get(false)); }

[aschoerk]

the extension sets to strict. The code is clear. I also want to use @Mock-Annotations, since that is used in the Mockit1.x-code.

[johanneslink]

@aschoerk I find it strange that the extension changes the default setting. But well, I get perfectly along without it.

[davidxxx]

Your answer appears to give the better advice. I wrote an answer to complete your approach (+1). I lost more 1 hour today to guess the issue with a very complex code...

[amdev]

org.mockito.quality.Strictness.LENIENT

[Paweł Sosnowski]

I also agree this should be accepted one, maybe with some explanation why its required

[amoe]

If you can live with the doReturn().when() syntax, porting to that instead is a decent approach.