Rerunning failed cucumber tests using cucumber-jvm

Question

I have a Cucumber-JVM, JUnit, Selenium setup. I initiate the run by running RunSmokeTests.java using JUnit within Eclipse. I have also set up a maven profile to run the tests from command line, and possibly Jenkins in the future.

When the tests are run then some of them may fail sometimes, mainly due to the application taking longer than expected. I would then have to re-run these scenarios. At the moment I run them by manually attaching @rerun tag to the ones that failed and then running RunReruns.java, which is similar to RunSmokeTest.java but with @rerun tag.

With the increasing number of automated tests it is time consuming to tag the tests and start the run and clear the tags. Is there a automated way with Cucumber-JVM to re-run failed tests?

RunSmokeTests.java

package testGlueClasses;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@Cucumber.Options(features = "src/test/java", strict = true, format = {
        "html:target/CucumberReport", "json:target/JSON/Cucumber.json",
        "FrameworkCore.CustomTestReporter" }, tags = { "@SmokeTest" }, glue = {
        "FrameworkCore", "MyApp.Utils", "MyApp.StepDefinitions" })
public class RunSmokeTests {

} 

Maven snippet:

    <profile>
        <id>smoke</id>
        <properties>
            <include.tests>
                **/RunSmokeTests.java
            </include.tests>
        </properties>
    </profile>

Solution 1

I came up with another solution to rerun just failed test using maven & cucumber.

1) Record test failures using a RunNotifier

public class RerunningCucumber extends Cucumber {

    private final String className;

    @SuppressWarnings("rawtypes")
    public RerunningCucumber(Class clazz) throws InitializationError, IOException {
        super(clazz);
        className = clazz.getSimpleName();
    }


    @Override
    public void run(RunNotifier notifier) {
        notifier.addListener(new RunListener(){

            public void testFailure(Failure failure) throws Exception {

                Throwable error = failure.getException();
                if (error instanceof AssertionError){
                    //Nothing. This is a normal failure. Continue
                    return;
                }

                //No! A wild exception has appeared!
                //Let's run this test again.
                RerunningCucumber.addFile(className);
            }

        });
        super.run(notifier);
    }


    private static final String filename = "target/rerun.properties";
    private static final Set<String> addedClasses = new HashSet<String>();
    public static synchronized void addFile(String className) throws IOException{
        //First find the file

        if (addedClasses.contains(className)){
            return;
        }

        File file = new File(filename);
        if (!file.exists()){
            //Need to create the file
            PrintWriter writer = new PrintWriter(file, "UTF-8");
            writer.print("retryclasses=**/"+className+".class");
            writer.close();
        }
        else {
            PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter(file, true)));
            out.print(",**/"+className+".class");
            out.close();
        }

        addedClasses.add(className);
    }
}

2) Use custom class as a runner for the cucumber tests.

This will run the tests, and whenever there is a failure, output the failed class to a file. Trick is to keep features short and create a lot of test classes to avoid repeating tests.

@RunWith(RerunningCucumber.class)
@CucumberOptions(features = {"classpath:features/testFeature.feature}, format = {
        "html:target/cucumber-html-report/testFeature.html",
        "json:target/cucumber-json-report/testFeature.json"},
        tags = {"@testFeature"})

public class RunTestFeature {
}

3) Add a Rerun profile to maven.

This does three things: 1) it loads the failed classes into memory, 2) cleans JUST the failed classes properties file, and 3) reruns ONLY the failed tests as loaded from the properties file:

    <profile>
        <id>retry</id>
        <build>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>properties-maven-plugin</artifactId>
                    <version>1.0-alpha-2</version>
                    <executions>
                        <!-- Associate the read-project-properties goal with the initialize 
                            phase, to read the properties file. -->
                        <execution>
                            <phase>pre-clean</phase>
                            <goals>
                                <goal>read-project-properties</goal>
                            </goals>
                            <configuration>
                                <files>
                                    <file>target/rerun.properties</file>
                                </files>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-clean-plugin</artifactId>
                    <version>2.6.1</version>
                    <configuration>
                        <filesets>
                            <fileset>
                                <directory>target</directory>
                                <includes>
                                    <include>rerun.properties</include>
                                </includes>
                            </fileset>
                        </filesets>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-antrun-plugin</artifactId>
                    <version>1.6</version>
                    <executions>
                        <execution>
                            <phase>compile</phase>
                            <goals>
                                <goal>run</goal>
                            </goals>
                            <configuration>
                                <target>
                                    <echo>Retrying the following classes: "${retryclasses}"</echo>
                                </target>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>2.17</version>
                    <configuration>
                        <includes>
                            <include>${retryclasses}</include>
                        </includes>
                        <testFailureIgnore>true</testFailureIgnore>
                    </configuration>
                    <executions>
                        <execution>
                            <phase>test</phase>
                            <goals>
                                <goal>test</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>
    </profile>

4) Usage

First test run:

mvn clean test

Next test runs:

mvn clean test -Pretry
mvn clean test -Pretry
mvn clean test -Pretry
...

You can repeat as many times as you want until there are no errors.

Solution 2

I don't have an executable example at hand, but you can do this also on the jvm. There is a RerunFormatter that writes a text file listing the file and line numbers of failed scenarios:

@CucumberOptions(format = {"rerun:target/rerun.txt"})

You should be able to specify this file as input for another test class by prefixing it with @:

@CucumberOptions(features = {"@target/rerun.txt"})

Solution 3

You can pass cucumber options to mvn as below

 mvn clean verify  -Dcucumber.options="@rerun.txt"

Note there is a tricky part here. If you are using the same test runner for both first run and rerun (and I believe that's what you want), then the test runner would contains something like

@CucumberOptions(plugin = { "rerun:target/rerun.txt"})

If you fire your rerun with maven using the same rerun file name as below

 mvn clean verify  -Dcucumber.options="@target/rerun.txt"

then cucumber will complain it could not find the rerun file. Why? Because the plugin "rerun:target/rerun.txt" will delete the file first with this test runner.

Workaround is copy/rename the file first, then kick off the mvn run like

mv target/rerun.txt rerun.txt &&  mvn clean verify  -Dcucumber.options="@rerun.txt"

And this is actually what you want. Because say if there are 5 failed scenarios in file target/rerun.txt. And with the rerun after some fix, 2 of them passed. Now the target/rerun.txt will contain the remaining 3 failed scenarios only, which would be your new start point along the debugging way.

Comments

[Jason]

We create a new test class and then add the @CucumberOptions(features = {"@target/rerun.txt"})? Do you know a way in maven to get this file to be run last?

[Jörn Horstmann]

I was thinking of specifying the rerun test in a separate maven profile that you could execute when you see test failures. That would not be fully automated though. I don't think junit or maven guarantee the order that classes are executed, but you could probably group them in a test suite: junit.sourceforge.net/javadoc/org/junit/runners/Suite.html

[Jörn Horstmann]

Interesting solution, but requiring one class per feature seems a bit cumbersome. Can you explain again why the rerun feature of cucumber did not work for you?

[Jason]

I think it was two things: 1) when I specified the rerun.txt file as the input feature, it didn't rerun those tests cases; it was just confused. 2) I couldn't specify the test run order to guarantee the rerunner was run last.

[Sugat Mankar]

what is ${retryclasses} ?

[Isabelle T.]

@Jason, the io.cucumber.junit.Cucumber class is final. I'm assuming you were doing this at the times people used the cukes cucumber class. Ever managed to achieve something like this with the new cucumber class? I don't seem to be able to use the rerun.txt because when the tests are run in parallel, not all failed scenarios get logged to rerun.txt. So I would love to see how I could get this to work. Thanks