Docker->Maven->Failsafe->Surefire starting fork fails with "The forked VM terminated without properly saying goodbye. VM crash or System.exit called?"

Question

As per title: I'm trying to run Maven automated test from Jenkins slave that is containerized and after battling this for a week now I'm running out of ideas. It works as is on AWS instance with 4G of RAM but in unrestricted (on RAM and CPU) container it fails with error like below. The only circumstances when it runs is when I disable forking for Failsafe plugin but that is not an option going forward.

I tried all sorts of Java/Maven/Failsafe/Surefire options I could have found using Google but no luck (like adding global Java -Xmx options and also per plugin in pom.xml).

Has anyone ran it successfully like this?

It would seem this should be a lot easier to deal with but I'd would have pulled by now all hair from my head should I have any. I still don't like the idea of admitting the defeat. Please help!

These are the dumps the plugin creates after failure:

failsafe-summary.xml:

<?xml version="1.0" encoding="UTF-8"?>
<failsafe-summary xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-sure
fire-plugin/xsd/failsafe-summary.xsd" result="254" timeout="false">
    <completed>0</completed>
    <errors>0</errors>
    <failures>0</failures>
    <skipped>0</skipped>
    <failureMessage>org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM cras
h or System.exit called?
Command was /bin/sh -c cd /var/lib/jenkins/workspace/ui_acceptance_test_chrome_docker_freestyle &amp;&amp; /usr/lib/jvm/java-1.8-openjdk/jre/bin/ja
va -Dfile.encoding=UTF-8 -jar /var/lib/jenkins/workspace/ui_acceptance_test_chrome_docker_freestyle/target/surefire/surefirebooter81206735832436906
05.jar /var/lib/jenkins/workspace/ui_acceptance_test_chrome_docker_freestyle/target/surefire 2017-10-10T15-02-35_189-jvmRun1 surefire59539140137458
58339tmp surefire_03559885505222114015tmp
Error occurred in starting fork, check output in log
Process Exit Code: 1
       at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:686)
       at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:535)
       at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:280)
       at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:245)
       at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1124)
       at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:954)
       at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:832)
       at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
       at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
       at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
       at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
       at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
       at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
       at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
       at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
       at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
       at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
       at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
       at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
       at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
       at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
       at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
       at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
       at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
</failureMessage>
</failsafe-summary>

2017-10-10T15-02-35_189-jvmRun1.dump:

# Created on 2017-10-10T15:02:36.303
Killing self fork JVM. Maven process died.

Solution 1

Try downgrading to Surefire 1.18.1. I ran into this issue tonight and spent a couple hours on it, so far it's not easily apparent why newer builds of Surefire break under Docker.

* Update *

I was having an issue with Alpine linux, but when using an Ubuntu or Debian base image everything was fine. So something within 1.21 is breaking compatibility with certain operating systems.

Solution 2

I know it's been a while but will add my solution to the problem which took me more than a day to FIX.

Problem: I'm running my integration tests in the PaaS in a docker container and have no control on the memory allocation for my process. the default behavior is for failsafe/surfire to fork a JVM and run the tests on it. I could not find a way to control the memory allocation for that forked JVM and tests kept failing with the error "Error occurred in starting fork" also saw "The forked VM terminated without saying properly goodbye docker" in the logs depending on which version of failsafe I was trying.

Solution: My solution was to disable forking of the JVM and have all the tests run in the same JVM as the main maven process, now this may not be the ideal solution for many but it would work if you can only control the max memory allocation of the main maven process.

To disable forking it's as simple as setting the forkMode in the config:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <configuration>
        <forkCount>0</forkCount>
    </configuration>

.....

UPDATE: Since giving this solution it seems like you can now pass params to the forked JVM so shouldn't need the earlier solution.

From the maven docs under Forked Test Execution:

With the argLine property, you can specify additional parameters to be passed to the forked JVM process, such as memory settings. System property variables from the main maven process are passed to the forked process as well. Additionally, you can use the element systemPropertyVariables to specify variables and values to be added to the system properties during the test execution.

https://maven.apache.org/surefire/maven-failsafe-plugin/examples/fork-options-and-parallel-execution.html

Solution 3

I have encountered the same issue in the following environment: docker image from alpine 3.7, maven surefire plugin version 2.21.0.

The root cause of that is described at SUREFIRE-1422: surefire tries to use ps -p to check forked process. The solution for me was to add procps:

RUN apk add --no-cache procps

Solution 4

We suddenly experienced the exact same issue, but only on our CI/CD pipeline (gitlab). The error message was:

error occurred in starting fork, check output in log
Process Exit Code: 1
org.apache.maven.surefire.booter.SurefireBooterForkException: 
The forked VM terminated without properly saying goodbye. VM crash or System.exit called?

It turned out that it was related to a new version of the OpenJDK docker image.

Setting the useSystemClassLoader property to false solved the issue:

   <plugin>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${maven-surefire-plugin.version}</version>
    <configuration>
      <includes>
        <include>**/*Test.java</include>
      </includes>
      <useSystemClassLoader>false</useSystemClassLoader>
    </configuration>
  </plugin>

Solution 5

We had the same issue while using spring boot with surefire version 2.21.0 on alpine with docker (zenika/alpine-maven). Like mentioned before, downgrading to 2.18.1 might be an option and solved the forked vm termination issue, but triggered new issues with incompatibilities between different slf4j versions. So we did an explicit upgrade to surefire version 2.22.1, which solved the issue in our case.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.22.1</version>
</plugin>

View more solutions

Comments

[Lars]

Up-voting, though I only downgraded to 2.18.1 (...I assume that is actually what you meant?)

[Brian Ploetz]

This did the trick for me as well (downgrading to surefire 2.18.1) for a build that was failing on CircleCI

[md_5]

2.20 works, 2.21 does not. There is a bug report here issues.apache.org/jira/browse/SUREFIRE-1444 that looks similar, I'm going to add some details.

[Jitsusama]

Using version 2.20.1 of maven-surefire-plugin with version 1.2.0 of junit-platform-surefire-provider with version 5.2.0 of junit-jupiter-engine does NOT work for me on an Alpine Linux openjdk:8-jdk-alpine image as of today. Switching to the openjdk:8-jdk image works just fine.

[mmelnik]

Thank you for this answer. Spent several hours solving this issue!

[Ali Dehghani]

One more reason to hate Alpine! Wasted a lot of time on this :( Thanks 🙏

[Uwe Plonus]

This also occured to me using a debian stretch image as base... So don't blame Alpine alone but also the creators of the surefire plugin.

[vmaldosan]

Setting the forkCount to 0 in maven-surefire-plugin did the trick for me. Thanks!