Mockito Tests Pass individually but fail as part of a suite

Question

I have a case where my mockito tests all pass individually but can fail when run as part of a testing suite. I am using mockito version 2.9.0

My test class is as follows

import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;

import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.BDDMockito;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import org.springframework.test.context.web.WebAppConfiguration;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
import org.springframework.web.context.WebApplicationContext;

import com.cache.CacheServices;

@RunWith(SpringJUnit4ClassRunner.class)
@WebAppConfiguration
@ContextConfiguration(value={"classpath:generalConfig/generalConfigMocks-context.xml"})
public class GeneralConfigAPIClientTest {

    private MockMvc mockMvc;

    @Autowired
    CacheServices cacheServices; 

    @Autowired
    IGeneralConfigServices generalConfigServices;

    @Autowired 
    private WebApplicationContext wac;

    /**
     * Set up the test context, initialize the mockMvc.
     * */
    @Before 
    public void setUp() {
        this.mockMvc = MockMvcBuilders.webAppContextSetup(this.wac).build();       
    }

    @Test
    public void testEmptyGeneralConfigCache_FailureFalseReturned() throws Exception {

        BDDMockito.given(cacheServices.emptyCacheContents(BDDMockito.anyString())).willReturn(false);

        this.mockMvc.perform(get("/generalConfig/emptyGeneralConfigCache"))
                .andExpect(status().isOk())     
                .andReturn();

        BDDMockito.verify(cacheServices, BDDMockito.times(1)).emptyCacheContents(BDDMockito.anyString());
    }

    @Test
    public void testEmptyGeneralConfigCache_SuccessTrueReturned() throws Exception {
        BDDMockito.given(cacheServices.emptyCacheContents(BDDMockito.anyString())).willReturn(true);

        this.mockMvc.perform(get("/generalConfig/emptyGeneralConfigCache"))
                .andExpect(status().isOk())     
                .andReturn();

        BDDMockito.verify(cacheServices, BDDMockito.times(1)).emptyCacheContents(BDDMockito.anyString());
    }

    }

The contents of generalConfig/generalConfigMocks-context.xml are as follows

 <?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:mvc="http://www.springframework.org/schema/mvc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:aop="http://www.springframework.org/schema/aop" xmlns:oxm="http://www.springframework.org/schema/oxm"
    xmlns:util="http://www.springframework.org/schema/util" xmlns:context="http://www.springframework.org/schema/context"
    xmlns:p="http://www.springframework.org/schema/p"
    xmlns:ehcache="http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring"
    xmlns:cache="http://www.springframework.org/schema/cache"
    xsi:schemaLocation="
   http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd
   http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
   http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
   http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-3.1.xsd
   http://www.springframework.org/schema/oxm http://www.springframework.org/schema/oxm/spring-oxm-3.1.xsd
   http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring
   http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring/ehcache-spring-1.1.xsd
   http://www.springframework.org/schema/cache http://www.springframework.org/schema/cache/spring-cache-4.1.xsd
   http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc-3.0.xsd">

    <mvc:annotation-driven />

    <context:annotation-config />

    <context:component-scan base-package="com..config" />

    <bean id="generalConfigServices" class="org.mockito.BDDMockito" factory-method="mock">
        <constructor-arg value="com.config.IGeneralConfigServices"/>
    </bean>


    <bean id="cacheServices" class="org.mockito.BDDMockito" factory-method="mock">
        <constructor-arg value="com.cache.CacheServices"/>
    </bean>


</beans>

A sample error response I get from running my tests is as follows

    org.mockito.exceptions.verification.TooManyActualInvocations: 
cacheServices.emptyCacheContents(
    <any string>
);
Wanted 1 time:
-> at com.config.GeneralConfigAPIClientTest.testEmptyGeneralConfigCache_SuccessTrueReturned(GeneralConfigAPIClientTest.java:65)
But was 2 times. Undesired invocation:
-> at com.config.GeneralConfigAPIClient.callEmptyGeneralConfigCache(GeneralConfigAPIClient.java:34)

    at com.config.GeneralConfigAPIClientTest.testEmptyGeneralConfigCache_SuccessTrueReturned(GeneralConfigAPIClientTest.java:65)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
    at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
    at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
    at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:539)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:761)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:461)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:207)

Can anyone offer any insights as to why my tests are passing individually but failing as a group

Solution

Reason: your test load one application context with bean mock inside. The bean mocks are dirty after any test. You have a dirty application context also.

Solution: you have to clean bean mock. Using Mockito.reset(mocks) after each test. I've found sth similar: How to clean up mocks in spring tests when using Mockito

Comments

[pvpkiran]

Tr using @DirtiesContext(classMode=ClassMode.AFTER_EACH_TEST_METHOD)

[Rob Obdeijn]

You're doing verify(cacheServices, BDDMockito.times(1)) in both methods meaning it can only be invoked once. It is the same object so it gets invoked twice when you run the full suite.

[Florian Schaetz]

@DirtiesContext produces quite a bit of overhead, I would suggest solving the problem without that, for example by resetting the offending mocks after the test. See the answer that Chi Coung Le posted below for more possibilities.