How do I fix a 'Failed to retrieve RMIServer stub' JMX error?
Solution 1
The problem is that even if JMX is configured to listen on port 8084, after the initial connection is made, the JMX host will dynamically pick a different port for the rest of the "conversation". My firewall allowed traffic through 8084, but was blocking the subsequent traffic.
There are two options.
- Allow inbound traffic on all ephemeral ports (yikes!).
- Develop a JMX wrapper that uses a single port for communication.
Solution 2
You can specify same port for both JMX and RMI conversation:
-Dcom.sun.management.jmxremote.port=8084
-Dcom.sun.management.jmxremote.rmi.port=8084
Solution 3
I struggled with this same issue trying to connect to our new Tomcat8 VMs over JMX (using VisualVM). Our Linux team had to open a ticket with Redhat to help us troubleshoot. But I stumbled on this article before they could resolve our issue.
The solution as mentioned above was to add
-Dcom.sun.management.jmxremote.rmi.port=X
and X
should match whatever your setting is for
-Dcom.sun.management.jmxremote.port
Complete List of Args:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8091
-Dcom.sun.management.jmxremote.rmi.port=8091
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=MYHOST
Hope this helps someone else having the same issue. Good luck!
Solution 4
Searching for a solution to an identical problem (after solving many many others, ready to pull my hair), I found this thread, and I can share that the solution that worked for me was https://serverfault.com/a/872756/579378 by @MaxRelax -- I would have upvoted his answer but I don't have enough reputation.
Oh, one more thing: Since -Dcom.sun.management.jmxremote.ssl=false
be sure to check the 'Do not require SSL connection' checkbox in the Java Visual VM dialog for adding a JMX connection.
GregB
Updated on September 18, 2022Comments
-
GregB over 1 year
I just completed a migration from Windows server 2008 R2 to Ubuntu 10.04. I manage a java application (Java 6, Tomcat) that is having some performance issues. I'd like to use JMX to try and troubleshoot, but I can't seem to get jvisualvm to connect.
If I do a ps -ef | grep "java", I see the following parameters.
-Dcom.sun.management.jmxremote.port=8084 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
Netstat shows that port 8084 is listening on 0.0.0.0. In my config, JMX is setup to bind to the FQDN of the server (we use a private DNS server). My firewall (IPTABLES/UFW) is setup to allow all outgoing traffic, and to allow incoming traffic on port 8084.
The server itself is virtual with two NICs, a public and a private. The public NIC's gateway is disabled so that connections can only come in on the private side.
When I try to connect jvisualvm to my app server using JMX, I get the following error in jvisualvm.
Cannot connect to [FQDN OMITTED]:8084 using server:jmx:rmi:///jndi/rmi://[FQDN OMITTED]:8084/jmxrmi
If I look in the jvisualvm log, I see the following trace.
NFO [com.sun.tools.visualvm.jmx.impl.JmxModelImpl]: connect(service:jmx:rmi:///jndi/rmi://[FQDN OMITTED]:8084/jmxrmi) java.io.EOFException: SSL peer shut down incorrectly at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789) Caused: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:808) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1120) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:623) at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at java.io.DataOutputStream.flush(DataOutputStream.java:106) at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:211) Caused: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:286) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184) at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322) at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97) Caused: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake] at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101) at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185) at javax.naming.InitialContext.lookup(InitialContext.java:392) at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886) at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856) at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257) Caused: java.io.IOException: Failed to retrieve RMIServer stub at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338) at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.tryConnect(JmxModelImpl.java:451) [catch] at com.sun.tools.visualvm.jmx.impl.JmxModelImpl$ProxyClient.connect(JmxModelImpl.java:395) at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.connect(JmxModelImpl.java:216) at com.sun.tools.visualvm.jmx.impl.JmxModelImpl.<init>(JmxModelImpl.java:205) at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:61) at com.sun.tools.visualvm.jmx.impl.JmxModelProvider.createModelFor(JmxModelProvider.java:42) at com.sun.tools.visualvm.core.model.ModelFactory.getModel(ModelFactory.java:111) at com.sun.tools.visualvm.tools.jmx.JmxModelFactory.getJmxModelFor(JmxModelFactory.java:69) at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.addJmxApplication(JmxApplicationProvider.java:267) at com.sun.tools.visualvm.jmx.impl.JmxApplicationProvider.createJmxApplication(JmxApplicationProvider.java:185) at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationImpl(JmxApplicationsSupport.java:283) at com.sun.tools.visualvm.jmx.JmxApplicationsSupport.createJmxApplicationInteractive(JmxApplicationsSupport.java:261) at com.sun.tools.visualvm.jmx.impl.AddJMXConnectionAction$1.run(AddJMXConnectionAction.java:80) at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:577) at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1030)
Does anyone have any ideas?
-
davey over 10 yearsIf you are on Java 7 update 4 or later there is hope with the flag `` -Dcom.sun.management.jmxremote.rmi.port=7091`` See this blog post: hirt.se/blog/?p=289