File not found in Jenkins pipeline script

Question

I'm trying to have a pipeline script currently running on our Jenkins master, execute on a remote Jenkins node. But I'm getting a strange FileNotFound exception.

The most basic version of the pipeline I've been able to reproduce the issue with is this:

node("remoteNode") {
env.SERVICE_VERSIONS_FILE = pwd() + '/service_versions.csv'
stage('Read file') {
  git credentialsId: '***', url: '***'      
  sh "cat $env.SERVICE_VERSIONS_FILE"
  new File(env.SERVICE_VERSIONS_FILE).each { line ->
    echo "$line"
   }
  }
}

Which results in:

>java.io.FileNotFoundException: /home/***/workspace/DeploymentPipelines/test-deployer/service_versions.csv
> (No such file or directory)   at java.io.FileInputStream.open0(Native
> Method)   at java.io.FileInputStream.open(FileInputStream.java:195)   at
> java.io.FileInputStream.<init>(FileInputStream.java:138)  at
> groovy.util.CharsetToolkit.<init>(CharsetToolkit.java:71)     at
> org.codehaus.groovy.runtime.ResourceGroovyMethods.newReader(ResourceGroovyMethods.java:1572)
>   at
> org.codehaus.groovy.runtime.ResourceGroovyMethods.readLines(ResourceGroovyMethods.java:533)
>   at
> org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.asCollection(DefaultTypeTransformation.java:461)
>   at
> org.codehaus.groovy.runtime.DefaultGroovyMethods.iterator(DefaultGroovyMethods.java:15955)
>   at org.codehaus.groovy.runtime.dgm$367.doMethodInvoke(Unknown Source)
>   at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
>   at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
>   at
> org.codehaus.groovy.runtime.InvokerHelper.invokePojoMethod(InvokerHelper.java:913)
>   at
> org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:904)
>   at
> org.codehaus.groovy.runtime.InvokerHelper.asIterator(InvokerHelper.java:573)
>   at org.codehaus.groovy.runtime.InvokerHelper$asIterator.call(Unknown
> Source)   at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
>   at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
>   at
> com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
>   at
> com.cloudbees.groovy.cps.CpsDefaultGroovyMethods.each(CpsDefaultGroovyMethods:1890)
>   at WorkflowScript.run(WorkflowScript:8)     at
> ___cps.transform___(Native Method)    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
>   at
> com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
>   at
> com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
>   at sun.reflect.GeneratedMethodAccessor324.invoke(Unknown Source)    at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)     at
> com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
>   at
> com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:39)
>   at
> com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
>   at
> com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:28)
>   at
> com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
>   at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)   at
> com.cloudbees.groovy.cps.Next.step(Next.java:83)  at
> com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)     at
> com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)     at
> org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
>   at
> org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
>   at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
>   at
> org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
>   at
> org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
>   at
> org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
>   at
> org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
>   at
> org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
>   at
> org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)     at
> hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
>   at
> jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
>   at
> jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
>   at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)     at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)   Finished: FAILURE

The sh "cat $env.SERVICE_VERSIONS_FILE" returns the correct results (i.e. - it prints the contents of the file).

The pipeline works fine when executed on the master. It feels like I might be missing something catastrophically simple? Or is it a bug?

Solution

I'm not really sure how or why it works, but I found this https://stackoverflow.com/a/38679858/985291 and managed to solve the issue by using the readFile step https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#readfile-read-file-from-workspace mentioned.

so basically, changing

 new File(env.SERVICE_VERSIONS_FILE).each { line ->

to

 readFile(env.SERVICE_VERSIONS_FILE).split("\n").each { line ->

and it works.

EDIT As @zett42 mentions in their comment, this is apparently by design, and you should just use the build in steps (readFile,writeFile) when dealing with files. (https://issues.jenkins-ci.org/browse/JENKINS-37577?focusedCommentId=267445&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-267445)

Comments

[vruum]

I checked, and the file is there. That's also why I included the sh "cat..." command in the pipeline. To verify that the file is there before attempting to read it.

[zett42]

The root cause is explained on a comment at this Jira issue: "The behavior is "as designed". Low-level Java commands will always access files on the master. There are Pipeline commands like readFile() and writeFile(), which allow accessing files within node workspaces."

[vruum]

hi @zett42 thank you, I think you're right, that would explain it. Feel free to post your comment as an answer, and I'll accept it, otherwise I'll update my "answer" to reference your comment.

[zett42]

I don't think we need another answer, this answer has all info that is needed now. Cheers!

[herm]

The error message of file not found it really misleading and causing people to waste a lot of time.