Jenkins not showing maven compiler errors

Question

When building our multi-module maven 3 project in Jenkins, if there's a build error we get this cryptic message that the maven compiler plugin failed. This only just started happening within the last week:

[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:22.340s
[INFO] Finished at: Fri Feb 10 09:44:02 CET 2012
[INFO] Final Memory: 171M/318M
[INFO] ------------------------------------------------------------------------
mavenExecutionResult exceptions not empty
message : Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile (default-compile) on project me.activity.impl: Compilation failure
cause : Compilation failure
Stack trace : 
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile (default-compile) on project me.activity.impl: Compilation failure
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
    at org.jvnet.hudson.maven3.launcher.Maven3Launcher.main(Maven3Launcher.java:79)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)
    at org.jvnet.hudson.maven3.agent.Maven3Main.launch(Maven3Main.java:158)
    at hudson.maven.Maven3Builder.call(Maven3Builder.java:104)
    at hudson.maven.Maven3Builder.call(Maven3Builder.java:70)
    at hudson.remoting.UserRequest.perform(UserRequest.java:118)
    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
    at hudson.remoting.Request$2.run(Request.java:287)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
    at java.util.concurrent.FutureTask.run(FutureTask.java:123)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
    at java.lang.Thread.run(Thread.java:595)
Caused by: org.apache.maven.plugin.CompilationFailureException: Compilation failure
    at org.apache.maven.plugin.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:516)
    at org.apache.maven.plugin.CompilerMojo.execute(CompilerMojo.java:114)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
    ... 27 more
Maven schlug mit Fehlern fehl.
An attempt to send an e-mail to empty list of recipients, ignored.

When building from the command-line we get the normal build errors:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5:24.906s
[INFO] Finished at: Fri Feb 10 08:17:31 EST 2012
[INFO] Final Memory: 173M/328M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile) on project me.activity.impl: Compilation failure: Compilation failure:
[ERROR] \p4views\VM\Base\main\modules\activity\impl\src\main\java\com\sap\me\activity\impl\ExtensionConfigurationService.java:[1018,12] cannot find symbol
[ERROR] symbol  : class ActivityRuntimeType
[ERROR] location: class com.sap.me.activity.impl.ExtensionConfigurationService
[ERROR] \p4views\VM\Base\main\modules\activity\impl\src\main\java\com\sap\me\activity\impl\ExtensionConfigurationService.java:[200,16] cannot find symbol
[ERROR] symbol  : class ActivityRuntimeType
[ERROR] location: class com.sap.me.activity.impl.ExtensionConfigurationService
[ERROR] \p4views\VM\Base\main\modules\activity\impl\src\main\java\com\sap\me\activity\impl\ExtensionConfigurationService.java:[234,16] cannot find symbol
[ERROR] symbol  : class ActivityRuntimeType
[ERROR] location: class com.sap.me.activity.impl.ExtensionConfigurationService
[ERROR] \p4views\VM\Base\main\modules\activity\impl\src\main\java\com\sap\me\activity\impl\ExtensionConfigurationService.java:[263,16] cannot find symbol
[ERROR] symbol  : class ActivityRuntimeType
[ERROR] location: class com.sap.me.activity.impl.ExtensionConfigurationService
[ERROR] \p4views\VM\Base\main\modules\activity\impl\src\main\java\com\sap\me\activity\impl\ExtensionConfigurationService.java:[294,20] cannot find symbol
[ERROR] symbol  : class ActivityRuntimeType
[ERROR] location: class com.sap.me.activity.impl.ExtensionConfigurationService
[ERROR] \p4views\VM\Base\main\modules\activity\impl\src\main\java\com\sap\me\activity\impl\ExtensionConfigurationService.java:[311,16] cannot find symbol
[ERROR] symbol  : class ActivityRuntimeType
[ERROR] location: class com.sap.me.activity.impl.ExtensionConfigurationService
[ERROR] \p4views\VM\Base\main\modules\activity\impl\src\main\java\com\sap\me\activity\impl\ExtensionConfigurationService.java:[1023,47] cannot find symbol
[ERROR] symbol  : variable ActivityRuntimeType
[ERROR] location: class com.sap.me.activity.impl.ExtensionConfigurationService
[ERROR] \p4views\VM\Base\main\modules\activity\impl\src\main\java\com\sap\me\activity\impl\ExtensionConfigurationService.java:[1025,67] cannot find symbol
[ERROR] symbol  : variable ActivityRuntimeType
[ERROR] location: class com.sap.me.activity.impl.ExtensionConfigurationService
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :me.activity.impl

We're using maven-3.0.4 both for local builds and on Jenkins. Jenkins version was 1.3ish, but I upgraded to 1.450 to see if the issue would go away - it didn't. This happened about the time we moved from maven-2.2.1 to maven-3.0.4, but I could swear (although I don't have evidence) that I was getting normal build errors on Jenkins just after the maven upgrade, so I don't think that's it, but it's the only change I can think of that would cause this. The compiler plugin version is 2.0.2.

I saw a similar posting here but his issue had to do with eclipse builds, not Jenkins.

Solution

Got an answer, which may not work for you as it seems rather dependent on many things and circumstances, BUT I suspect it is indeed relevant for most people here.

---

Troubleshooting

(Skip if you don't care about how I figured this out for my setting, but it may help for yours. The Solution 1 bit is what you want if you're impatient.)

How it Started...

Current working-directory of an actively developed project started to spit out logs showing this behavior today, when there had been no errors for a while. I was using Windows 7, Maven 3.0.4, compiler plugin 2.4, and java(c) 1.6.0_31.

When reaching a specific sub-module, maven would spit out some error messages and fail the build, except the error messages were unrelated (reporting use of proprietary com.sun.* APIs) and there would be no sign of an actual compilation error.

Things get Weird...

At this point I find this thread, and I also try to re-build with different java versions and maven-compiler-plugin versions. Here's the fun part:

• with compiler-plugin 2.5, I'd get a similar issue, and no valid error report;
• with compiler plugin 2.3, I'd get a similar issue with a nonsensical error report complaining about a missing package (that is present and on classpath, though!);
• with compiler plugin 2.3.2, I'd get a similar issue with a nonsensical error report complaining about a missing package (that is present and on classpath, though!) BUT ** at a different point **in the test-classes' compilation of the module (previously it happened during the normal classes' compilation).

From Weird to Weirder

Getting suspicious, I switched to another directory with a clean checkout of the project and re-built everything.

Everything worked FINE. Puzzlement and despair here.

So I realized I was working on a rather huge and complex class in the other workspace, in the faulty module. And I do mean rather huge, complex, and fairly poorly written. The type of beast that makes you tell your boss "please don't make me touch this" and that makes your DELETE key tickle you every time you have the misfortune of running across it in your IDE. In fact, this thing looks so much like the offspring of a dark margic experiment and weirdly unethical bioresearch that you wonder if it didn't come right out of RTC Wolfenstein or Doom, if the antagonists were bits of code. The class you don't want to be left alone to code on at night, and for which Sonar quietly reports close to 1000 code quality violations, including complexity indexes that make you wonder if PMD, JDepend and other tools are not actually all having a stroke at the same time).

But I digress...

I copied it over to this clean and seemingly working workspace (luckily for me, it didn't impact other files, so it was rather simple to do: just copy over).

Then I rebuild with Maven and it gets all cranky when it reaches the module containing this file again, except this time it prints out a load of errors a stacktrace that nevers ends. So I want to try to re-run the same build, this time saving the output to a log file to have a closer look and (here's the new odd thing, because so far that wasn't bizarre enough) ... it again doesn't show any error.

So, I'm figuring something is obviously very nasty, for javac to go belly up with what looks like a stack overflow, and look for the bit of stacktrace I could see in my log the first time around. Here's a chunk of it (as I suspect some may come across this and find this answer useful):

[ERROR] at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
[ERROR] at com.sun.tools.javac.comp.Attr.attribExpr(Attr.java:377)
[ERROR] at com.sun.tools.javac.comp.Attr.visitApply(Attr.java:1241)
[ERROR] at com.sun.tools.javac.tree.JCTree$JCMethodInvocation.accept(JCTree.java:1210)
[ERROR] at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
[ERROR] at com.sun.tools.javac.comp.Attr.visitSelect(Attr.java:1799)
[ERROR] at com.sun.tools.javac.tree.JCTree$JCFieldAccess.accept(JCTree.java:1522)
[ERROR] at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
[ERROR] at com.sun.tools.javac.comp.Attr.attribExpr(Attr.java:377)
[ERROR] at com.sun.tools.javac.comp.Attr.visitApply(Attr.java:1241)
[ERROR] at com.sun.tools.javac.tree.JCTree$JCMethodInvocation.accept(JCTree.java:1210)
[ERROR] at com.sun.tools.javac.comp.Attr.attribTree(Attr.java:360)
[ERROR] at com.sun.tools.javac.comp.Attr.visitSelect(Attr.java:1799)
[ERROR] at com.sun.tools.javac.tree.JCTree$JCFieldAccess.accept(JCTree.java:1522)

And Google nicely points me to... StackOverflow, of course :)

Maven compilation: Failure executing javac

Where I can see an easy thing to try (see below).

Of course, this would have been way easier to troubleshoot, had the stacktrace appeared in all situations and in all cases, and shown the obvious StackOverflowException. Especially considering I have otherwise already seen this error and know what it means, but it's a hell of a lot harder to figure out when you **can't see* the error.

Not sure why Maven swallowed that one, but maybe javac simply crashed without notice and there's a screw-up in the log management and it doesn't get flushed out properly.

That would also explain why this bug happens relatively randomly: it would be impact by your hardware settings, and slight changes to the classes javac is given to process.

So, now what??? Well, if there's a crazy stack overflow exception or an OutOfMemoryException when java tries to process your code, obviously you need to do something akin to what's explained below...

---

Solutions

Solution 1 (aka "short-term / quick-n-dirty / please-make-it-work")

Upgrade your memory settings. Simplest way is with something like (for a bash-like shell):

export MAVEN_OPTS="-Xss1024k -Xms512m -Xmx1024m"

BEWARE: These settings will depend on your platform's hardware, obviously (and on your shell).

In my case, I actually used to have:

export MAVEN_OPTS="-Xss128k -Xms384m -Xmx384m"

... because I have a pretty heavy project and a not so memory-capable workstation, so I need to squeeze every bit of RAM I can out of it to have multiple Eclipse instances running, multiple application servers, etc... So my JAVA_OPTS, ANT_OPTS and MAVEN_OPTS are set with a bunch of options, including these.

This is not necessarily what you want!! The default xss for a 64 bits JVM on Windows is actually 1024, so I use something considerably smaller. I'm just saying it as it may help others in the same situation. Try to raise it accordingly and sensibly for you configuration.

So, eventually, to fix this particular issue in my project, I had to change the above to:

export MAVEN_OPTS="-Xss256k -Xms384m -Xmx384m"

And now everything works peachy.

Maybe some other things are different for you. Let me know.

 

Solution 2 (aka "long-term / zen")

You know what's better than fiddling with memory settings that others won't know to fiddle with and may not have the chance to, and that make your builds less portable? [audience screams here]

You refactor the hell out of that butt-ugly class that makes javac cry for its mommy.

It's that simple. No thousands and thousands of lines long classes, with crazy static initializers, super long methods and stuff like these. If your code looks complicated to you, it sure does for poor javac as well.

Save a javac process today: refactor your code!!

Comments

[tdrury]

To the astute observer: the lower error shows I'm using compiler version 2.3.2 instead of 2.0.2. This error message was clipped from my laptop where I tried using 2.3.2 to address this issue. The output from compiler-plugin 2.0.2 is the same. i.e. both are correct when building locally but these compiler errors don't show up when building on Jenkins.

[tdrury]

Also, there's a issue here on Jenkins' Jira page, but it's over 6 months old with no action taken.

[tdrury]

I ran the Jenkins build with -e -X also, but there was no additional information in that output - it didn't include the compiler's syntax errors.

[Michael]

try entering hudson.maven.MavenBuild.debug=true; in the scripting console of jenkins, to get more info about the maven module

[Artem Oboturov]

and what about this ActivityRuntimeType class?

[andyb]

I have exactly this problem just on the command line. Using the latest maven-compiler-plugin (2.4), maven (3.0.4) and Java (1.7.0_04). I get no symbol/location information when there is a compilation error. The only way to fix it for me was to downgrade to Java 6.

[haylem]

@andyb: have a similar one on command line as well, with Win7, mvn 3.0.4, compiler-plugin 2.4 and java 1.6.0_32. That being said, it doesn't seem to happen all the time or for all errors. I haven't had this issue until today, and I've had the same settings (and been working on the same build) for more than 2 weeks now. A bit odd...

[haylem]

@andyb (and tdrury): Give a shot to what I suggest in my answer. Please let us know how that works out for you.

[haylem]

@tdrury: I think we have identified that the issue is not Jenkins-related, and in fact (if I'm right), maybe not even mostly Maven-related but javac-related (though I'm still curious why maven wouldn't report the error consitently). You might want to update the question title and tags if this works for you, and maybe just leave a note about it failing in your CI/Jenkins environment so others can find this thread when it happens in a similar configuration for them.

[andyb]

@haylem I believe I found out why Java 7 does not output errors whilst Java 6 does. The output format changed and the plexus-compiler that the maven-compiler-plugin uses under the hood cannot parse the output. I found the already raised MCOMPILER-158 but raised PLXCOMP-204 myself after looking at the code.

[tdrury]

Sorry, but my results are not conclusive and I basically gave up because I couldn't reproduce the issue on demand. However, since this issue we've updated Jenkins (1.514), maven (3.0.5), and the compiler plugin (2.3.2) and have not seen the problem again.

[tdrury]

The error doesn't occur all the time. In fact, I haven't seen it in a while.

[tdrury]

The code base was the same for both runs. When building from the command line, we got the errors we expected. When building via Jenkins, it still failed but it did not show the compiler error output. The question wasn't what was wrong with ActivityRuntimeType, but why did Jenkins fail to show the compiler output.

[Faustas]

I don't think you can get any more information from compiler here. It just fails to find that class/symbol and there is nothing more that can be said about it.

[tdrury]

Re-read my original post - there are two Maven outputs shown. The first is the output from Maven from Jenkins - it shows a plugin error instead of compiler errors. The second is running Maven from the command line and shows the expected compiler errors. I'm not asking for help debugging the code - I'm wondering when Maven doesn't show the compiler erros when running in Jenkins.

[andyb]

+1 nice investigation. I will see what impact this has on my system. I'm still using 32-bit JVM at the moment. My problem isn't a randomly failing build per se but more if there are genuine compilation errors, Jenkins and the command line do not print them. This is due to Java 7's output being different to Java 6's and the plexus compiler is not capturing the output correctly.

[haylem]

"My problem isn't a randomly failing build per se but more if there are genuine compilation errors, Jenkins and the command line do not print them" In my case, it would fail with a "compilation failure" where it wasn't one. I have been using Java 7 with Maven 3.0.4 and compiler-plugin 2.3.2 fine, but I haven't tried since the 2.4 and 2.5 compiler plugins.

[Usman Ismail]

@haylem See my answer, try defining the compiler plugin in the pom of the actual module(s) rather than parent.

[haylem]

@UsmanIsmail: I don't know if that would help, I don't have the classes to reproduce the previous error case. For me, it was really a problem of the javac compiler actually crashing because of large class files. So while your solution could have an impact, I don't think it's the root cause.

[tdrury]

Usman, it's an interesting theory, but I can't easily test this since I have 400+ sub-modules and adding the compilation plugin to each one is not realistic.