Bad Gateway 502 error with Apache mod_proxy and Tomcat

Question

Hello I am trying to wrap an javaFX-jar with launch4j maven plugin.

But the execution fails:

Exception in thread "main" java.lang.NoClassDefFoundError: javafx/applicatio
   at java.lang.ClassLoader.defineClass1(Native Method)
   at java.lang.ClassLoader.defineClass(Unknown Source)
   at java.security.SecureClassLoader.defineClass(Unknown Source)
   at java.net.URLClassLoader.defineClass(Unknown Source)
   at java.net.URLClassLoader.access$100(Unknown Source)
   at java.net.URLClassLoader$1.run(Unknown Source)
   at java.net.URLClassLoader$1.run(Unknown Source)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(Unknown Source)
   at java.lang.ClassLoader.loadClass(Unknown Source)
   at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
   at java.lang.ClassLoader.loadClass(Unknown Source)
   at sun.launcher.LauncherHelper.checkAndLoadMain(Unknown Source)
Caused by: java.lang.ClassNotFoundException: javafx.application.Application
   at java.net.URLClassLoader$1.run(Unknown Source)
   at java.net.URLClassLoader$1.run(Unknown Source)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(Unknown Source)
   at java.lang.ClassLoader.loadClass(Unknown Source)
   at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
   at java.lang.ClassLoader.loadClass(Unknown Source)
    ... 13 more

It looks like the exe cannot find the jfxruntime. If I start the program with "java -jar ..." it works fine.

Here is partial of my pom.xml

        <plugin>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.0</version>
            <configuration>
                <source>1.7</source>
                <target>1.7</target>
                <encoding></encoding>
            </configuration>
        </plugin>

        <plugin>
            <groupId>com.zenjava</groupId>
            <artifactId>javafx-maven-plugin</artifactId>
            <version>1.5</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>build-jar</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <mainClass>org.myprogram.Main</mainClass>
                <bundleType>ALL</bundleType>
                <vendor>me</vendor>
            </configuration>
        </plugin>

        <plugin>
            <groupId>org.bluestemsoftware.open.maven.plugin</groupId>
            <artifactId>launch4j-plugin</artifactId>
            <version>1.5.0.0</version>
            <executions>
                <execution>
                    <id>l4j-gui</id>
                    <phase>package</phase>
                    <goals>
                        <goal>launch4j</goal>
                    </goals>
                    <configuration>
                        <headerType>gui</headerType>
                        <outfile>target/MyProgram.exe/outfile>
                        <jar>target/${project.artifactId}-${project.version}-jfx.jar</jar>
                        <errTitle>App Err</errTitle>
                        <classPath>
                            <mainClass>org.myprogram.Main</mainClass>
                        </classPath>                            
                        <jre>
                            <minVersion>1.7.0</minVersion>                              
                            <initialHeapSize>128</initialHeapSize>
                            <maxHeapSize>1024</maxHeapSize>
                        </jre>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>

Thank you for your help.

Solution 1

So, answering my own question here. We ultimately determined that we were seeing 502 and 503 errors in the load balancer due to Tomcat threads timing out. In the short term we increased the timeout. In the longer term, we fixed the app problems that were causing the timeouts in the first place. Why Tomcat timeouts were being perceived as 502 and 503 errors at the load balancer is still a bit of a mystery.

Solution 2

Just to add some specific settings, I had a similar setup (with Apache 2.0.63 reverse proxying onto Tomcat 5.0.27).

For certain URLs the Tomcat server could take perhaps 20 minutes to return a page.

I ended up modifying the following settings in the Apache configuration file to prevent it from timing out with its proxy operation (with a large over-spill factor in case Tomcat took longer to return a page):

Timeout 5400
ProxyTimeout 5400

---

Some backgound

ProxyTimeout alone wasn't enough. Looking at the documentation for Timeout I'm guessing (I'm not sure) that this is because while Apache is waiting for a response from Tomcat, there is no traffic flowing between Apache and the Browser (or whatever http client) - and so Apache closes down the connection to the browser.

I found that if I left the Timeout setting at its default (300 seconds), then if the proxied request to Tomcat took longer than 300 seconds to get a response the browser would display a "502 Proxy Error" page. I believe this message is generated by Apache, in the knowledge that it's acting as a reverse proxy, before it closes down the connection to the browser (this is my current understanding - it may be flawed).

The proxy error page says:

Proxy Error

The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET.

Reason: Error reading from remote server

...which suggests that it's the ProxyTimeout setting that's too short, while investigation shows that Apache's Timeout setting (timeout between Apache and the client) that also influences this.

Solution 3

You can use proxy-initial-not-pooled

See http://httpd.apache.org/docs/2.2/mod/mod_proxy_http.html :

If this variable is set no pooled connection will be reused if the client connection is an initial connection. This avoids the "proxy: error reading status line from remote server" error message caused by the race condition that the backend server closed the pooled connection after the connection check by the proxy and before data sent by the proxy reached the backend. It has to be kept in mind that setting this variable downgrades performance, especially with HTTP/1.0 clients.

We had this problem, too. We fixed it by adding

SetEnv proxy-nokeepalive 1
SetEnv proxy-initial-not-pooled 1

and turning keepAlive on all servers off.

mod_proxy_http is fine in most scenarios but we are running it with heavy load and we still got some timeout problems we do not understand.

But see if the above directive fits your needs.

Solution 4

Sample from apache conf:

#Default value is 2 minutes
**Timeout 600**
ProxyRequests off
ProxyPass /app balancer://MyApp stickysession=JSESSIONID lbmethod=bytraffic nofailover=On
ProxyPassReverse /app balancer://MyApp
ProxyTimeout 600
<Proxy balancer://MyApp>
    BalancerMember http://node1:8080/ route=node1 retry=1 max=25 timeout=600
    .........
</Proxy>

Solution 5

I'm guessing your using mod_proxy_http (or proxy balancer).

Look in your tomcat logs (localhost.log, or catalina.log) I suspect your seeing an exception in your web stack bubbling up and closing the socket that the tomcat worker is connected to.

View more solutions

Comments

[Uday A. Navapara]

I am facing this problem for some time during my running application

[Dave Cheney]

Update the issue with anything you find in the log. Also check the apache error logs for, which should give you a clue who closed the socket (apache or tomcat). By default apache has a timeout of 300 seconds for proxied responses

[Tobias]

Do you have any idea yet, why the load balancer throws bad gateways if the tomcat times out?

[worcin]

I have specified the min version with 1.7.0_17 an deinstalled all other java version. Still the same problem have you used launch4j for a jfx2 project?

[wobblycogs]

One of our products uses JavaFX 2 and an exe is built with launch4j. I use a different build of the plugin to you but the configuration is the same (see answer edit). I seem to remember I got most of my settings from this page: 9stmaryrd.com/tools/launch4j-maven-plugin

[worcin]

Thank you very much, I used your config and it worked well. I Think the problem is the other wrapper.

[worcin]

Futher investigation showed, that the sampe problem occours when I add an explicit classpath.

[Ben Thurley]

This worked for me. I had inconsistent keep-alive settings which I corrected with SetEnv proxy-nokeepalive 1 and SetEnv force-proxy-request-1.0 1.

[TechnoCrat]

What is the syntax for timeout setting? TimeOut 5400 OR Timeout 5400? (small o vs capital O)

[sherb]

Configuration directives are case insensitive (but arguments may be case sensitive): httpd.apache.org/docs/2.4/configuring.html

[Uday A. Navapara]

Hello, I am fresher for this and don't know where to and how to set above property please explain me in detail. where I Can find this config file and where I have to write this property in that config file.

[JRichardsz]

In my case, console dev of browser show me ......... 502 (Proxy Error)...... I added Timeout 5400 ProxyTimeout 5400 inside <virtualHost> in file httpd.conf and WORKS!! Thanks!

[MattBianco]

In the documentation quoted in Janning's answer (stackoverflow.com/a/1287662/385571) the explanation is that there is a race condition between when the backend WAS closes the connection, and the mod_proxy_http checks it. If a new client connects without proxy-initial-not-pooled being set, it could be sent to this defunct thread.

[themenace]

Adding the ProxyTimeout Directive would not be needed in most cases as it takes the Timeout Directives value as a default. See httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxytimeout

[Joshua Achoka]

Thanks for sharing helped me fix a similar issue.

[yas]

works for me too. So now it makes me more confuse, is it a syntax issue [1] or timeout issue? [1]:ProxyPass / 192.168.1.1:3001 [1]:ProxyPassReverse / 192.168.1.1:3001