Jenkins: Pipeline sh bad substitution error

Question

A step in my pipeline uploads a .tar to an artifactory server. I am getting a Bad substitution error when passing in env.BUILD_NUMBER, but the same commands works when the number is hard coded. The script is written in groovy through jenkins and is running in the jenkins workspace.

sh 'curl -v --user user:password --data-binary ${buildDir}package${env.BUILD_NUMBER}.tar -X PUT "http://artifactory.mydomain.com/artifactory/release-packages/package${env.BUILD_NUMBER}.tar"'

returns the errors:

[Pipeline] sh
[Package_Deploy_Pipeline] Running shell script
/var/lib/jenkins/workspace/Package_Deploy_Pipeline@tmp/durable-4c8b7958/script.sh: 2: 
/var/lib/jenkins/workspace/Package_Deploy_Pipeline@tmp/durable-4c8b7958/script.sh: Bad substitution
[Pipeline] } //node
[Pipeline] Allocate node : End
[Pipeline] End of Pipeline
ERROR: script returned exit code 2

If hard code in a build number and swap out ${env.BUILD_NUMBER} I get no errors and the code runs successfully.

sh 'curl -v --user user:password --data-binary ${buildDir}package113.tar -X PUT "http://artifactory.mydomain.com/artifactory/release-packages/package113.tar"'

I use ${env.BUILD_NUMBER} within other sh commands within the same script and have no issues in any other places.

Solution 1

This turned out to be a syntax issue. Wrapping the command in ''s caused ${env.BUILD_NUMBER to be passed instead of its value. I wrapped the whole command in "s and escaped the nested. Works fine now.

sh "curl -v --user user:password --data-binary ${buildDir}package${env.BUILD_NUMBER}.tar -X PUT \"http://artifactory.mydomain.com/artifactory/release-packages/package${env.BUILD_NUMBER}.tar\""

Solution 2

In order to Pass groovy parameters into bash scripts in Jenkins pipelines (causing sometimes bad substitions) You got 2 options:

The triple double quotes way [ " " " ] OR the triple single quotes way [ ' ' ' ]

1. In triple double quotes you can render the normal parameter from groovy using ${someVariable} ,if it's environment variable ${env.someVariable} , if it's parameters injected into your job ${params.someVariable}

example:

     def YOUR_APPLICATION_PATH= "${WORKSPACE}/myApp/"

      sh """#!/bin/bash
      cd ${YOUR_APPLICATION_PATH}
      npm install
      """

2. In triple single quotes things getting little bit tricky, you can pass the parameter to environment parameter and using it by "\${someVaraiable}" or concating the groovy parameter using ''' + someVaraiable + '''

examples:

   def YOUR_APPLICATION_PATH= "${WORKSPACE}/myApp/"

   sh '''#!/bin/bash
          cd ''' + YOUR_APPLICATION_PATH + '''
          npm install
    '''

OR

   pipeline{
     agent { node { label "test" } }
     environment {
       YOUR_APPLICATION_PATH = "${WORKSPACE}/myapp/"
     }

     continue...
     continue...
     continue...

     sh '''#!/bin/bash
          cd "\${YOUR_APPLICATION_PATH}"
          npm install
    '''

    //OR
    sh '''#!/bin/bash
          cd "\${env.YOUR_APPLICATION_PATH}"
          npm install
    '''

Solution 3

Actually, you seem to have misunderstood the env variable. In your sh block, you should access ${BUILD_NUMBER} directly.

Reason/Explanation: env represents the environment inside the script. This environment is used/available directly to anything that is executed, e.g. shell scripts.

Please also pay attention to not write anything to env.*, but use withEnv{} blocks instead.

Solution 4

Usually the most common issue for:

Bad substitution

error is to use sh instead of bash.

Especially when using Jenkins, if you're using Execute shell, make sure your Command starts with shebang, e.g. #!/bin/bash -xe or #!/usr/bin/env bash.

Solution 5

I can definitely tell you, it's all about sh shell and bash shell. I fixed this problem by specifying #!/bin/bash -xe as follows:

node {
    stage("Preparing"){
        sh'''#!/bin/bash -xe
            colls=( col1 col2 col3 )
            for eachCol in ${colls[@]}
            do
              echo $eachCol
            done
        '''
    }      
}

View more solutions

Comments

[Stephen Nichols]

Yea you nailed it, thanks for all the help. I missed your original comment somehow though, wish I would have read it sooner.

[Eyal Ben Moshe]

Not directly related to this issue, but please consider using the Artifactpry Pipeline DSL. It provides a more convenient Groovy DSL than curl cmd executions, reducing the chance for syntax mistakes: wiki.jenkins-ci.org/display/JENKINS/…

[Omer Tuchfeld]

@BW use """ """

[tripleee]

As long as any variables in the URL get substituted by Groovy and not by the shell, you can use single quotes instead of escaped double quotes around the URL.