How do I implement a retry option for failed stages in Jenkins pipelines?

Question

I have a Jenkinsfile with multiple stages and one of them is in fact another job (the deploy one) which can fail in some cases.

I know that I can made prompts using Jenkinsfile but I don't really know how to implement a retry mechanism for this job.

I want to be able to click on the failed stage and choose to retry it.

Solution 1

You should be able to combine retry + input to do that Something like that

stage('deploy-test') {
   try {
     build 'yourJob'
   } catch(error) {
     echo "First build failed, let's retry if accepted"
     retry(2) {
        input "Retry the job ?"
        build 'yourJob'
     }
   }
}

you could also use timeout for the input if you want it to finish if nobody validates. There is also waitUntil that might be useful but i haven't used it yet

Edit : WaitUntil seems definitely the best, you should play with it a bit but something like that is cleaner :

stage('deploy-test') {
   waitUntil {
     try {
       build 'yourJob'
     } catch(error) {
        input "Retry the job ?"
        false
     }
   }
}

By the way, there is doc all of the steps here https://jenkins.io/doc/pipeline/steps

Solution 2

This one with a nice incremental wait

stage('deploy-test') {
 def retryAttempt = 0
 retry(2) {
    if (retryAttempt > 0) {
       sleep(1000 * 2 + 2000 * retryAttempt)
    }

    retryAttempt = retryAttempt + 1
    input "Retry the job ?"
    build 'yourJob'
 }
}

Solution 3

This gist (not mine) was one of the better options that I found while trying to implement this functionality too. https://gist.github.com/beercan1989/b66b7643b48434f5bdf7e1c87094acb9

Changed it to a method in a shared library that just did retry or abort for my needs. Also added a max retries and made the timeout variable so that we could change it depending on the job or stage that needs it.

package com.foo.bar.jenkins

def class PipelineHelper {
    def steps

    PipelineHelper(steps) {
        this.steps = steps
    }

    void retryOrAbort(final Closure<?> action, int maxAttempts, int timeoutSeconds, final int count = 0) {
        steps.echo "Trying action, attempt count is: ${count}"
        try {
            action.call();
        } catch (final exception) {
            steps.echo "${exception.toString()}"
            steps.timeout(time: timeoutSeconds, unit: 'SECONDS') {
                def userChoice = false
                try {
                    userChoice = steps.input(message: 'Retry?', ok: 'Ok', parameters: [
                            [$class: 'BooleanParameterDefinition', defaultValue: true, description: '', name: 'Check to retry from failed stage']])
                } catch (org.jenkinsci.plugins.workflow.steps.FlowInterruptedException e) {
                    userChoice = false
                }
                if (userChoice) {
                    if (count <= maxAttempts) {
                        steps.echo "Retrying from failed stage."
                        return retryOrAbort(action, maxAttempts, timeoutMinutes, count + 1)
                    } else {
                        steps.echo "Max attempts reached. Will not retry."
                        throw exception
                    }
                } else {
                    steps.echo 'Aborting'
                    throw exception;
                }
            }
        }
    }
}

Example usage with a max of 2 retries that waits for 60s for input.

def pipelineHelper = new PipelineHelper(this)

stage ('Retry Example'){
    pipelineHelper.retryOrAbort({
        node{
            echo 'Here is an example'
            throw new RuntimeException('This example will fail.')
        }
    }, 2, 60)
}

Just remember to put nodes inside of the closure so that waiting for an input doesn't block an executor.

If you have the paid jenkins enterprise Cloudbees has a Checkpoint plugin that can better handle this, but it is not planned to be release for open source Jenkins (JENKINS-33846).

Comments

[sorin]

Is it going to add a retry prompt? I doubt.

[fchaillou]

Oh no you're right. i'll update my answer for that !

[sorin]

Is it possible to enable the timeout only for the retry part? I may want to have a different timeout for the job. I didn't accepted the answer yet as I don't find a blocking job as a good solution. Ideally the retry option should be after the job already finished. Imagine that this job is triggered by a GitHub hook on a PR. I would prefer to see the failure on GitHub instead of no-answer in case of an error.

[Slack Flag]

In my testing with waitUntil{} with the build pipeline step, I found I needed to explicitly return true, because that step doesn't return a boolean type.

[Peter McIntyre]

no longer works as posted, with error: Unknown stage section "waitUntil". Starting with version 0.5, steps in a stage must be in a ‘steps’ block.

[Carl Walsh]

sleep() has default units of seconds, so unless you want the first wait to be over an hour, specify sleep(..., unit:"MILLISECONDS") or use less seconds.

[jcollum]

I think you can no longer put a retry at the top of a stage block: Expected one of "steps", "stages", or "parallel" for stage "Code Coverage" @ line 423, column 17.

[jcollum]

It only worked for me if I put it after the steps declaration.

[xpmatteo]

It's generally better to take advantage of the Jenkins build SDL facilities as other answers show

[Chris Maggiulli]

I believe you can put it outside of steps if you use stage('test') { options { retry(2) } steps { echo "hello" } } for a declarative pipeline only