Jenkins Pipeline NotSerializableException: groovy.json.internal.LazyMap

Question

Solved: Thanks to below answer from S.Richmond. I needed to unset all stored maps of the groovy.json.internal.LazyMap type which meant nullifying the variables envServers and object after use.

Additional: People searching for this error might be interested to use the Jenkins pipeline step readJSON instead - find more info here.

---

I am trying to use Jenkins Pipeline to take input from the user which is passed to the job as json string. Pipeline then parses this using the slurper and I pick out the important information. It will then use that information to run 1 job multiple times in parallel with differeing job parameters.

Up until I add the code below "## Error when below here is added" the script will run fine. Even the code below that point will run on its own. But when combined I get the below error.

I should note that the triggered job is called and does run succesfully but the below error occurs and fails the main job. Because of this the main job does not wait for the return of the triggered job. I could try/catch around the build job: however I want the main job to wait for the triggered job to finish.

Can anyone assist here? If you need anymore information let me know.

Cheers

def slurpJSON() {
return new groovy.json.JsonSlurper().parseText(BUILD_CHOICES);
}

node {
  stage 'Prepare';
  echo 'Loading choices as build properties';
  def object = slurpJSON();

  def serverChoices = [];
  def serverChoicesStr = '';

  for (env in object) {
     envName = env.name;
     envServers = env.servers;

     for (server in envServers) {
        if (server.Select) {
            serverChoicesStr += server.Server;
            serverChoicesStr += ',';
        }
     }
  }
  serverChoicesStr = serverChoicesStr[0..-2];

  println("Server choices: " + serverChoicesStr);

  ## Error when below here is added

  stage 'Jobs'
  build job: 'Dummy Start App', parameters: [[$class: 'StringParameterValue', name: 'SERVER_NAME', value: 'TestServer'], [$class: 'StringParameterValue', name: 'SERVER_DOMAIN', value: 'domain.uk'], [$class: 'StringParameterValue', name: 'APP', value: 'application1']]

}

Error:

java.io.NotSerializableException: groovy.json.internal.LazyMap
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:569)
    at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
    at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
    at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
    at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
    at java.io.ObjectOutputStream.writeObject(Unknown Source)
    at java.util.LinkedHashMap.internalWriteEntries(Unknown Source)
    at java.util.HashMap.writeObject(Unknown Source)
...
...
Caused by: an exception which occurred:
    in field delegate
    in field closures
    in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@5288c

Solution 1

I ran into this myself today and through some bruteforce I've figured out both how to resolve it and potentially why.

Probably best to start with the why:

Jenkins has a paradigm where all jobs can be interrupted, paused and resumable through server reboots. To achieve this the pipeline and its data must be fully serializable - IE it needs to be able to save the state of everything. Similarly, it needs to be able to serialize the state of global variables between nodes and sub-jobs in the build, which is what I think is happening for you and I and why it only occurs if you add that additional build step.

For whatever reason JSONObject's aren't serializable by default. I'm not a Java dev so I cannot say much more on the topic sadly. There are plenty of answers out there about how one may fix this properly though I do not know how applicable they are to Groovy and Jenkins. See this post for a little more info.

How you fix it:

If you know how, you can possibly make the JSONObject serializable somehow. Otherwise you can resolve it by ensuring no global variables are of that type.

Try unsetting your object var or wrapping it in a method so its scope isn't node global.

Solution 2

Use JsonSlurperClassic instead.

Since Groovy 2.3 (note: Jenkins 2.7.1 uses Groovy 2.4.7) JsonSlurper returns LazyMap instead of HashMap. This makes new implementation of JsonSlurper not thread safe and not serializable. This makes it unusable outside of @NonDSL functions in pipeline DSL scripts.

However you can fall-back to groovy.json.JsonSlurperClassic which supports old behavior and could be safely used within pipeline scripts.

Example

import groovy.json.JsonSlurperClassic 


@NonCPS
def jsonParse(def json) {
    new groovy.json.JsonSlurperClassic().parseText(json)
}

node('master') {
    def config =  jsonParse(readFile("config.json"))

    def db = config["database"]["address"]
    ...
}    

ps. You still will need to approve JsonSlurperClassic before it could be called.

Solution 3

EDIT: As pointed out by @Sunvic in the comments, the below solution does not work as-is for JSON Arrays.

I dealt with this by using JsonSlurper and then creating a new HashMap from the lazy results. HashMap is Serializable.

I believe that this required whitelisting of both the new HashMap(Map) and the JsonSlurper.

@NonCPS
def parseJsonText(String jsonText) {
  final slurper = new JsonSlurper()
  return new HashMap<>(slurper.parseText(jsonText))
}

Overall, I would recommend just using the Pipeline Utility Steps plugin, as it has a readJSON step that can support either files in the workspace or text.

Solution 4

I want to upvote one of the answer: I would recommend just using the Pipeline Utility Steps plugin, as it has a readJSON step that can support either files in the workspace or text: https://jenkins.io/doc/pipeline/steps/pipeline-utility-steps/#readjson-read-json-from-files-in-the-workspace

script{
  def foo_json = sh(returnStdout:true, script: "aws --output json XXX").trim()
  def foo = readJSON text: foo_json
}

This does NOT require any whitelisting or additional stuff.

Solution 5

This is the detailed answer that was asked for.

The unset worked for me:

String res = sh(script: "curl --header 'X-Vault-Token: ${token}' --request POST --data '${payload}' ${url}", returnStdout: true)
def response = new JsonSlurper().parseText(res)
String value1 = response.data.value1
String value2 = response.data.value2

// unset response because it's not serializable and Jenkins throws NotSerializableException.
response = null

I read the values from the parsed response and when I don't need the object anymore I unset it.

View more solutions

Comments

[S.Richmond]

Just ran into this myself. Did you make any further progress yet?

[Christophe Roussy]

github.com/jenkinsci/pipeline-plugin/blob/master/…

[Rhubarb]

nice question - upvoted, but just a tip: you don't need to edit your question to link to the answer: that's what the checkmark is for (it also distracts from the highest-voted answer, which worked best for me)

[Sunvic]

Thanks, That's the clue I've needed to solve this. Whilst I had tried your suggesion already It made me look again and I hadn't considered that I was storing parts of the map in other variables - these were causing the errors. So I needed to unset them as well. Will amend my question to include the correct changes to the code. Cheers

[Jordan Stefanelli]

This is viewed ~8 times a day. Would you guys mind providing a more detailed example of how to implement this solution?

[S.Richmond]

There is no simple solution as it depends on what you've done. The information provided here as well as the solution @Sunvic added at the top of his post aught to be enough to led one to a solution for their own code.

[mybecks]

Could you please tell me how to approve JsonSlurperClassic?

[luka5z]

Jenkins administrator will need to navigate to Manage Jenkins » In-process Script Approval.

[Sunvic]

Didn't work me - kept getting an error Could not find matching constructor for: java.util.HashMap(java.util.ArrayList). Documentation suggests it should spit out a list or a map - how do you configure to return a map?

[mkobit]

@Sunvic Good catch, the data we have been parsing are always objects, never JSON arrays. Are you trying to parse a JSON array?

[Sunvic]

Ah yes, it's a JSON array, that'll be it.

[yiwen]

Both this answer and below, on Jenkins, raised an RejectedEception because Jenkins runs groovy in sandbox env

[mkobit]

@yiwen I mentioned that it requires administrator whitelisting, but maybe the answer could be clarified as to what that means?

[yiwen]

The thing is when using HashMap or JsonSlurperClassic. The Jenkins in-process approval UI is supposed to ask an admin if you approve the use of them. But the UI didn't display that so there is no way to approve it

[dvtoever]

Unfortunately I only get hudson.remoting.ProxyException: org.codehaus.groovy.control.MultipleCompilationErrorsExcepti‌​on: startup failed: Script1.groovy: 24: unable to resolve class groovy.json.JsonSlurperClassic

[Marcos Brigante]

JsonSluperClassic.. This name tells a lot about current state of software development

[Sunvic]

Would recommend to use Jenkins pipeline step readJSON instead - find more info here.

[Quartz]

The solution below, using JsonSlurperClassic fixed the exact same issue I had, should probably be the approved choice here. This answer has merits, but it's not the right solution for this particular problem.

[Nils Rommelfanger]

@JordanStefanelli I posted the code for my solution. See my answer below

[Wojtas.Zet]

JsonSluperClassic() messes up with the order of Json elements...

[Sathish Prakasam]

No idea why this post still is a approved answer for this question. Though it gives the cause of the problem, it doesnt give the difference between LazyMap and HashMap (we assume are getting HashMap but we are getting not a thread safe lazyMap). And so the error. Changing it to JsonSlurperClassic fixed my issue.

[Sathish Prakasam]

Thank you so much for this detailed explanation. You saved a lot of time of mine. This solution works like a charm in my jenkins pipeline.

[Paulo Oliveira]

I was retrieving the Junit results from currentBuild within a Stage block. With the help of @S.Richmond I was able to understand how the pipeline works and what was triggering the Exception.Then I moved the currentBuild code to outside of the pipeline block into a @NonCPS independent method. Start working immediately without exceptions. (I know I'm not supposed to still want to thank @S.Richmond for the very good explanation) So I moved the

[byOnti]

thanks, this clue "Try unsetting your object var " save my day. In my case I just set the variable to null after use it.