JUnit5: How to repeat failed test?

Question

One of the practice many companies follow is to repeat unstable test until is passes x times (in a row or in total). If it is executed n times and fail to pass at least x times it is marked as failed.

TestNG supports that with the following annotation:

@Test(invocationCount = 5, successPercentage = 40)

How do I realize similar functionality with JUnit5?

There's similar annotation in JUnit5, called @RepeatedTest(5) but it is not executed conditionally.

Solution 1

Ok, I took a little bit of time to whip together a little example of how to do this using the TestTemplateInvocationContextProvider, ExecutionCondition, and TestExecutionExceptionHandler extension points.

The way I was able to handle failing tests was to mark them as "aborted" rather than let them flat out fail (so that the entire test execution does not consider it a failure) and only fail tests when we can't get the minimum amount of successful runs. If the minimum amount of tests has already succeeded, then we also mark the remaining tests as "disabled". The test failures are tracked in a ExtensionContext.Store so that the state can be looked up at each place.

This is a very rough example that definitely has a few problems but can hopefully serve as an example of how to compose different annotations. I ended up writing it in Kotlin:

@Retry-esque annotation loosely based on the TestNG example:

import org.junit.jupiter.api.TestTemplate
import org.junit.jupiter.api.extension.ExtendWith

@TestTemplate
@Target(AnnotationTarget.FUNCTION)
@ExtendWith(RetryTestExtension::class)
annotation class Retry(val invocationCount: Int, val minSuccess: Int)

TestTemplateInvocationContext used by templatized tests:

import org.junit.jupiter.api.extension.Extension
import org.junit.jupiter.api.extension.TestTemplateInvocationContext

class RetryTemplateContext(
  private val invocation: Int,
  private val maxInvocations: Int,
  private val minSuccess: Int
) : TestTemplateInvocationContext {
  override fun getDisplayName(invocationIndex: Int): String {
    return "Invocation number $invocationIndex (requires $minSuccess success)"
  }

  override fun getAdditionalExtensions(): MutableList<Extension> {
    return mutableListOf(
      RetryingTestExecutionExtension(invocation, maxInvocations, minSuccess)
    )
  }
}

TestTemplateInvocationContextProvider extension for the @Retry annotation:

import org.junit.jupiter.api.extension.ExtensionContext
import org.junit.jupiter.api.extension.ExtensionContextException
import org.junit.jupiter.api.extension.TestTemplateInvocationContext
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider
import org.junit.platform.commons.support.AnnotationSupport
import java.util.stream.IntStream
import java.util.stream.Stream

class RetryTestExtension : TestTemplateInvocationContextProvider {
  override fun supportsTestTemplate(context: ExtensionContext): Boolean {
    return context.testMethod.map { it.isAnnotationPresent(Retry::class.java) }.orElse(false)
  }

  override fun provideTestTemplateInvocationContexts(context: ExtensionContext): Stream<TestTemplateInvocationContext> {
    val annotation = AnnotationSupport.findAnnotation(
        context.testMethod.orElseThrow { ExtensionContextException("Must be annotated on method") },
        Retry::class.java
    ).orElseThrow { ExtensionContextException("${Retry::class.java} not found on method") }

    checkValidRetry(annotation)

    return IntStream.rangeClosed(1, annotation.invocationCount)
        .mapToObj { RetryTemplateContext(it, annotation.invocationCount, annotation.minSuccess) }
  }

  private fun checkValidRetry(annotation: Retry) {
    if (annotation.invocationCount < 1) {
      throw ExtensionContextException("${annotation.invocationCount} must be greater than or equal to 1")
    }
    if (annotation.minSuccess < 1 || annotation.minSuccess > annotation.invocationCount) {
      throw ExtensionContextException("Invalid ${annotation.minSuccess}")
    }
  }
}

Simple data class representing the retry (injected into test cases in this example using ParameterResolver).

data class RetryInfo(val invocation: Int, val maxInvocations: Int)

Exception used for representing failed retries:

import java.lang.Exception

internal class RetryingTestFailure(invocation: Int, cause: Throwable) : Exception("Failed test execution at invocation #$invocation", cause)

Main extension implementing ExecutionCondition, ParameterResolver, and TestExecutionExceptionHandler.

import org.junit.jupiter.api.extension.ConditionEvaluationResult
import org.junit.jupiter.api.extension.ExecutionCondition
import org.junit.jupiter.api.extension.ExtensionContext
import org.junit.jupiter.api.extension.ParameterContext
import org.junit.jupiter.api.extension.ParameterResolver
import org.junit.jupiter.api.extension.TestExecutionExceptionHandler
import org.opentest4j.TestAbortedException

internal class RetryingTestExecutionExtension(
  private val invocation: Int,
  private val maxInvocations: Int,
  private val minSuccess: Int
) : ExecutionCondition, ParameterResolver, TestExecutionExceptionHandler {
  override fun evaluateExecutionCondition(
    context: ExtensionContext
  ): ConditionEvaluationResult {
    val failureCount = getFailures(context).size
    // Shift -1 because this happens before test
    val successCount = (invocation - 1) - failureCount
    when {
      (maxInvocations - failureCount) < minSuccess -> // Case when we cannot hit our minimum success
        return ConditionEvaluationResult.disabled("Cannot hit minimum success rate of $minSuccess/$maxInvocations - $failureCount failures already")
      successCount < minSuccess -> // Case when we haven't hit success threshold yet
        return ConditionEvaluationResult.enabled("Have not ran $minSuccess/$maxInvocations successful executions")
      else -> return ConditionEvaluationResult.disabled("$minSuccess/$maxInvocations successful runs have already ran. Skipping run $invocation")
    }
  }

  override fun supportsParameter(
    parameterContext: ParameterContext,
    extensionContext: ExtensionContext
  ): Boolean = parameterContext.parameter.type == RetryInfo::class.java

  override fun resolveParameter(
    parameterContext: ParameterContext,
    extensionContext: ExtensionContext
  ): Any = RetryInfo(invocation, maxInvocations)

  override fun handleTestExecutionException(
    context: ExtensionContext,
    throwable: Throwable
  ) {

    val testFailure = RetryingTestFailure(invocation, throwable)
    val failures: MutableList<RetryingTestFailure> = getFailures(context)
    failures.add(testFailure)
    val failureCount = failures.size
    val successCount = invocation - failureCount
    if ((maxInvocations - failureCount) < minSuccess) {
      throw testFailure
    } else if (successCount < minSuccess) {
      // Case when we have still have retries left
      throw TestAbortedException("Aborting test #$invocation/$maxInvocations- still have retries left",
        testFailure)
    }
  }

  private fun getFailures(context: ExtensionContext): MutableList<RetryingTestFailure> {
    val namespace = ExtensionContext.Namespace.create(
      RetryingTestExecutionExtension::class.java)
    val store = context.parent.get().getStore(namespace)
    @Suppress("UNCHECKED_CAST")
    return store.getOrComputeIfAbsent(context.requiredTestMethod.name, { mutableListOf<RetryingTestFailure>() }, MutableList::class.java) as MutableList<RetryingTestFailure>
  }
}

And then, the test consumer:

import org.junit.jupiter.api.DisplayName

internal class MyRetryableTest {
  @DisplayName("Fail all retries")
  @Retry(invocationCount = 5, minSuccess = 3)
  internal fun failAllRetries(retryInfo: RetryInfo) {
    println(retryInfo)
    throw Exception("Failed at $retryInfo")
  }

  @DisplayName("Only fail once")
  @Retry(invocationCount = 5, minSuccess = 4)
  internal fun succeedOnRetry(retryInfo: RetryInfo) {
    if (retryInfo.invocation == 1) {
      throw Exception("Failed at ${retryInfo.invocation}")
    }
  }

  @DisplayName("Only requires single success and is first execution")
  @Retry(invocationCount = 5, minSuccess = 1)
  internal fun firstSuccess(retryInfo: RetryInfo) {
    println("Running: $retryInfo")
  }

  @DisplayName("Only requires single success and is last execution")
  @Retry(invocationCount = 5, minSuccess = 1)
  internal fun lastSuccess(retryInfo: RetryInfo) {
    if (retryInfo.invocation < 5) {
      throw Exception("Failed at ${retryInfo.invocation}")
    }
  }

  @DisplayName("All required all succeed")
  @Retry(invocationCount = 5, minSuccess = 5)
  internal fun allRequiredAllSucceed(retryInfo: RetryInfo) {
    println("Running: $retryInfo")
  }

  @DisplayName("Fail early and disable")
  @Retry(invocationCount = 5, minSuccess = 4)
  internal fun failEarly(retryInfo: RetryInfo) {
    throw Exception("Failed at ${retryInfo.invocation}")
  }
}

And the test output in IntelliJ looks like:

I don't know if throwing a TestAbortedException from the TestExecutionExceptionHandler.handleTestExecutionException is supposed to abort the test, but I am using it here.

Solution 2

U can try this extension for junit 5.

<dependency>
    <groupId>io.github.artsok</groupId>
    <artifactId>rerunner-jupiter</artifactId>
    <version>LATEST</version>
</dependency> 

Examples:

     /** 
        * Repeated three times if test failed.
        * By default Exception.class will be handled in test
        */
       @RepeatedIfExceptionsTest(repeats = 3)
       void reRunTest() throws IOException {
           throw new IOException("Error in Test");
       }


       /**
        * Repeated two times if test failed. Set IOException.class that will be handled in test
        * @throws IOException - error occurred
        */
       @RepeatedIfExceptionsTest(repeats = 2, exceptions = IOException.class)
       void reRunTest2() throws IOException {
           throw new IOException("Exception in I/O operation");
       }


       /**
        * Repeated ten times if test failed. Set IOException.class that will be handled in test
        * Set formatter for test. Like behavior as at {@link org.junit.jupiter.api.RepeatedTest}
        * @throws IOException - error occurred
        */
       @RepeatedIfExceptionsTest(repeats = 10, exceptions = IOException.class, 
       name = "Rerun failed test. Attempt {currentRepetition} of {totalRepetitions}")
       void reRunTest3() throws IOException {
           throw new IOException("Exception in I/O operation");
       }

       /**
       * Repeated 100 times with minimum success four times, then disabled all remaining repeats.
       * See image below how it works. Default exception is Exception.class
       */
       @DisplayName("Test Case Name")
       @RepeatedIfExceptionsTest(repeats = 100, minSuccess = 4)
       void reRunTest4() {
            if(random.nextInt() % 2 == 0) {
                throw new RuntimeException("Error in Test");
            }
       }

View at IDEA:

With minimum success four times then disables all other:

You can also mix @RepeatedIfExceptionsTest with @DisplayName

source -> github

Solution 3

if you are running tests via Maven, with Surefire you care re-run failing tests automatically by using rerunFailingTestsCount.

However, as of 2.21.0, that does not work for JUnit 5 (only 4.x). But hopefully it will be supported in the next releases.

Solution 4

If you happen to be running your tests using the Gradle build tool, you can use the Test Retry Gradle plugin. This will rerun each failed test a certain number of times, with the option of failing the build if too many failures have occurred overall.

plugins {
    id 'org.gradle.test-retry' version '1.2.0'
}

test {
    retry {
        maxRetries = 3
        maxFailures = 20 // Optional attribute
    }
}

View more solutions

Comments

[Naman]

You might have to write your custom runner which acts on the attributes of both @Test and @RepeatedTest attributes like currentRepetition, totalRepetitions and success

[dzieciou]

@nullpointer. Runner is a concept from JUnit4. Junit5 does not have custom runners.

[mkobit]

Do you expect it to always execute 5 times, or do you expect it to run until there are at least 40% successful executions?

[seanf]

Thanks @mkobit. FWIW, I put the seven separate Kotlin files into a Gist: gist.github.com/seanf/c6d16b00713ef3fdcd7f3371b4c5798a Have you got a licence in mind?

[mkobit]

@seanf I'm not sure if Stack Overflow itself has a license for answers, so I would defer to that. If not, consider it to be released under the Unlicense.

[vanangelov]

There is a feature request for that - issues.apache.org/jira/browse/SUREFIRE-1584

[Buzz]

Can your extension work with parallel executed tests? And does it work with ParameterizedTests?

[InfernalRapture]

Thanks for this, I have an unstable test suit and this is exactly what I need.

[user1207289]

When I am including this dependency rerunner-jupiter, I am receiving this error on running the tests, without even using @RepeatedIfExceptionsTest in tests java.lang.NoClassDefFoundError: org/junit/jupiter/api/extension/ScriptEvaluationException . Any idea , how to resolve this?

[Hakanai]

This solution very nearly works for us but we wanted to try and get it working without it being a TestTemplate, because TestTemplate is Testable, and if one of the tests you want to mark flaky like this is also a ParameterizedTest, all hell breaks loose.

[Abhilash Mandaliya]

@Buzz the answer is no as per this blog: swtestacademy.com/junit-5-how-to-repeat-failed-test