Is it possible to skip a scenario with Cucumber-JVM at run-time

Question

I want to add a tag @skiponchrome to a scenario, this should skip the scenario when running a Selenium test with the Chrome browser. The reason to-do this is because some scenario's work in some environments and not in others, this might not even be browser testing specific and could be applied in other situation for example OS platforms.

Example hook:

@Before("@skiponchrome") // this works
public void beforeScenario() {
  if(currentBrowser == 'chrome') { // this works
    // Skip scenario code here
  }
}

I know it is possible to define ~@skiponchrome in the cucumber tags to skip the tag, but I would like to skip a tag at run-time. This way I don't have to think about which steps to skip in advance when I starting a test run on a certain environment.

I would like to create a hook that catches the tag and skips the scenario without reporting a fail/error. Is this possible?

Solution 1

I too had the same challenge, where in I need to skip a scenario from running based on a flag which I obtain from the application dynamically in run-time, which tells whether the feature to be tested is enabled on the application or not..

so this is how I wrote my logic in the scenarios file, where we have the glue code for each step.

I have used a unique tag '@Feature-01AXX' to mark my scenarios that need to be run only when that feature(code) is available on the application.

so for every scenario, the tag '@Feature-01XX' is checked first, if its present then the check for the availability of the feature is made, only then the scenario will be picked for running. Else it will be merely skipped, and Junit will not mark this as failure, instead it will me marked as Pass. So the final result if these tests did not run due to the un-availability of the feature will be pass, that's cool...

@Before
public void before(final Scenario scenario) throws Exception {
    /*
        my other pre-setup tasks for each scenario.
    */

    // get all the scenario tags from the scenario head.
    final ArrayList<String> scenarioTags = new ArrayList<>();
    scenarioTags.addAll(scenario.getSourceTagNames());

    // check if the feature is enabled on the appliance, so that the tests can be run.
    if (checkForSkipScenario(scenarioTags)) {
        throw new AssumptionViolatedException("The feature 'Feature-01AXX' is not enabled on this appliance, so skipping");
    }
}

private boolean checkForSkipScenario(final ArrayList<String> scenarioTags) {
    // I use a tag "@Feature-01AXX" on the scenarios which needs to be run when the feature is enabled on the appliance/application
    if (scenarioTags.contains("@Feature-01AXX") && !isTheFeatureEnabled()) { // if feature is not enabled, then we need to skip the scenario.
        return true;
    }
    return false;
}

private boolean isTheFeatureEnabled(){
    /*
        my logic to check if the feature is available/enabled on the application.
        in my case its an REST api call, I parse the JSON and check if the feature is enabled.
        if it is enabled return 'true', else return 'false'
    */
}

Solution 2

I realized that this is a late update to an already answered question, but I want to add one more option directly supported by cucumber-jvm:

@Before //(cucumber one)
public void setup(){
    Assume.assumeTrue(weAreInPreProductionEnvironment);
}

"and the scenario will be marked as ignored (but the test will pass) if weAreInPreProductionEnvironment is false."

You will need to add

import org.junit.Assume;

The major difference with the accepted answer is that JUnit assume failures behave just like pending

Important Because of a bug fix you will need cucumber-jvm release 1.2.5 which as of this writing is the latest. For example, the above will generate a failure instead of a pending in cucumber-java8-1.2.3.jar

Solution 3

I really prefer to be explicit about which tests are being run, by having separate run configurations defined for each environment. I also like to keep the number of tags I use to a minimum, to keep the number of configurations manageable.

I don't think it's possible to achieve what you want with tags alone. You would need to write a custom jUnit test runner to use in place of @RunWith(Cucumber.class). Take a look at the Cucumber implementation to see how things work. You would need to alter the RuntimeOptions created by the RuntimeOptionsFactory to include/exclude tags depending on the browser, or other runtime condition.

Alternatively, you could consider writing a small script which invokes your test suite, building up a list of tags to include/exclude dynamically, depending on the environment you're running in. I would consider this to be a more maintainable, cleaner solution.

Solution 4

It's actually really easy. If you dig though the Cucumber-JVM and JUnit 4 source code, you'll find that JUnit makes skipping during runtime very easy (just undocumented).

Take a look at the following source code for JUnit 4's ParentRunner, which Cucumber-JVM's FeatureRunner (which is used in Cucumber, the default Cucumber runner):

@Override
public void run(final RunNotifier notifier) {
    EachTestNotifier testNotifier = new EachTestNotifier(notifier,
            getDescription());
    try {
        Statement statement = classBlock(notifier);
        statement.evaluate();
    } catch (AssumptionViolatedException e) {
        testNotifier.fireTestIgnored();
    } catch (StoppedByUserException e) {
        throw e;
    } catch (Throwable e) {
        testNotifier.addFailure(e);
    }
}

This is how JUnit decides what result to show. If it's successful it will show a pass, but it's possible to @Ignore in JUnit, so what happens in that case? Well, an AssumptionViolatedException is thrown by the RunNotifier (or Cucumber FeatureRunner in this case).

So your example becomes:

@Before("@skiponchrome") // this works
public void beforeScenario() {
  if(currentBrowser == 'chrome') { // this works
    throw new AssumptionViolatedException("Not supported on Chrome")
  }
}

If you've used vanilla JUnit 4 before, you'd remember that @Ignore takes an optional message that is displayed when a test is ignored by the runner. AssumptionViolatedException carries the message, so you should see it in your test output after a test is skipped this way without having to write your own custom runner.

Solution 5

I've implemented a customized junit runner as below. The idea is to add tags during runtime.

So say for a scenario we need new users, we tag the scenarios as "@requires_new_user". Then if we run our test in an environment (say production environment which dose not allow you to register new user easily), then we will figure out that we are not able to get new user. Then the ""not @requires_new_user" will be added to cucumber options to skip the scenario.

This is the most clean solution I can imagine now.

public class WebuiCucumberRunner extends ParentRunner<FeatureRunner> {
    private final JUnitReporter jUnitReporter;
    private final List<FeatureRunner> children = new ArrayList<FeatureRunner>();
    private final Runtime runtime;
    private final Formatter formatter;

    /**
     * Constructor called by JUnit.
     *
     * @param clazz the class with the @RunWith annotation.
     * @throws java.io.IOException                         if there is a problem
     * @throws org.junit.runners.model.InitializationError if there is another problem
     */
    public WebuiCucumberRunner(Class clazz) throws InitializationError, IOException {
        super(clazz);
        ClassLoader classLoader = clazz.getClassLoader();
        Assertions.assertNoCucumberAnnotatedMethods(clazz);

        RuntimeOptionsFactory runtimeOptionsFactory = new RuntimeOptionsFactory(clazz);
        RuntimeOptions runtimeOptions = runtimeOptionsFactory.create();

        addTagFiltersAsPerTestRuntimeEnvironment(runtimeOptions);

        ResourceLoader resourceLoader = new MultiLoader(classLoader);
        runtime = createRuntime(resourceLoader, classLoader, runtimeOptions);
        formatter = runtimeOptions.formatter(classLoader);
        final JUnitOptions junitOptions = new JUnitOptions(runtimeOptions.getJunitOptions());
        final List<CucumberFeature> cucumberFeatures = runtimeOptions.cucumberFeatures(resourceLoader, runtime.getEventBus());
        jUnitReporter = new JUnitReporter(runtime.getEventBus(), runtimeOptions.isStrict(), junitOptions);
        addChildren(cucumberFeatures);
    }

    private void addTagFiltersAsPerTestRuntimeEnvironment(RuntimeOptions runtimeOptions) 
    {
        String channel = Configuration.TENANT_NAME.getValue().toLowerCase();
        runtimeOptions.getTagFilters().add("@" + channel);

        if (!TestEnvironment.getEnvironment().isNewUserAvailable()) {
            runtimeOptions.getTagFilters().add("not @requires_new_user");
        }

    }
...
}

Or you can extends the official Cucumber Junit test runner cucumber.api.junit.Cucumber and override method

    /**
     * Create the Runtime. Can be overridden to customize the runtime or backend.
     *
     * @param resourceLoader used to load resources
     * @param classLoader    used to load classes
     * @param runtimeOptions configuration
     * @return a new runtime
     * @throws InitializationError if a JUnit error occurred
     * @throws IOException         if a class or resource could not be loaded
     * @deprecated Neither the runtime nor the backend or any of the classes involved in their construction are part of
     * the public API. As such they should not be  exposed. The recommended way to observe the cucumber process is to
     * listen to events by using a plugin. For example the JSONFormatter.
     */
    @Deprecated
    protected Runtime createRuntime(ResourceLoader resourceLoader, ClassLoader classLoader,
                                    RuntimeOptions runtimeOptions) throws InitializationError, IOException {
        ClassFinder classFinder = new ResourceLoaderClassFinder(resourceLoader, classLoader);
        return new Runtime(resourceLoader, classFinder, classLoader, runtimeOptions);
    }

You can manipulate runtimeOptions here as you wish. But the method is marked as deprecated, so use it with caution.

View more solutions

Comments

[cbaldan]

How can I overwrite the run method? I looked into Cucumber project structure but I couldn't find a way to overwride it from my framework code. Can you give some pointer?

[Mike Holler]

You don't need to. That is the source code of the run method as it is; it's already there. All you need is in the second code block.

[cbaldan]

I'm using cucumber-java 1.2.4, and it reports AssumptionViolationExceptions as failures. On pure JUnit it works though.

[Niels van Reijmersdal]

Looks good, although I havent tested this I will make this the accepted answer.

[George Pantazes]

I have tested throwing AssumptionViolatedException in the @Before hook and it fails the test, skipping it by causing the failure early. The failed test shows up in the Tests in Error in the after test summary. How can these tests be skipped completely? Do you use a custom Cucumber plugin or fork?

[Rahal Kanishka]

@cbaidanyou can copy from the maven repo and use it after customize.

[Karthic.K]

please use org.junit.internal.AssumptionViolatedException;

[YuraHoy]

It works great for the steps so the steps are marked as Pending and the build is marked as successful. But cucumber-reports Jenkins plugin marks scenarios containing peding tests as Failed. Do you know how to mark such scenarios as Successsul?