How to connect to Java instances running on EC2 using JMX
Solution 1
We are having problem connecting to our Java applications running in Amazon's EC2 cluster.
It turns out that the problem was a combination of two missing settings. The first forces the JRE to prefer ipv4 and not v6. This was necessary (I guess) since we are trying to connect to it via a v4 address:
-Djava.net.preferIPv4Stack=true
The real blocker was the fact that JMX works by first contacting the RMI port which responds with the hostname and port for the JMX client to connect. With no additional settings it will use the local IP of the box which is a 10.X.X.X
virtual address which a remote client cannot route to. We needed to add the following setting which is the external hostname or IP of the server -- in this case it is the elastic hostname of the server.
-Djava.rmi.server.hostname=ec2-107-X-X-X.compute-1.amazonaws.com
The trick, if you are trying to automate your EC2 instances (and why the hell would you not), is how to find this address at runtime. To do that you need to put something like the following in our application boot script:
# get our _external_ hostname
RMI_HOST=`wget -q -O - http://169.254.169.254/latest/meta-data/public-hostname`
...
java -server \
-Djava.net.preferIPv4Stack=true -Djava.rmi.server.hostname=$RMI_HOST \
-jar foo.jar other parameters here > java.log 2>&1
The mysterious 169.254.169.254
IP in the wget
command above provides information that the EC2 instance can request about itself. I'm disappointed that this does not include tags which are only available in an authenticated call.
I initially was using the extern ipv4 address but it looks like the JDK tries to make a connection to the server-port when it starts up. If it uses the external IP then this was slowing our application boot time until that timed out. The public-hostname resolves locally to the 10-net address and to the public-ipv4 externally. So the application now is starting fast and JMX clients still work. Woo hoo!
Hope this helps someone else. Cost me 3 hours today.
To force your JMX server to start the server and the RMI registry on designated ports so you can block them in the EC2 Security Groups, see this answer:
Edit:
We just had this problem re-occur. It seems that the Java JMX code is doing some hostname lookups on the hostname of the box and using them to try to connect and verify the JMX connection.
The issue seems to be a requirement that the local hostname of the box should resolve to the local-ip of the box. For example, if your /etc/sysconfig/network
has HOSTNAME=server1.foobar.com
then if you do a DNS lookup on server1.foobar.com
, you should get to the 10-NET virtual address. We were generating our own /etc/hosts
file and the hostname of the local host was missing from the file. This caused our applications to either pause on startup or not startup at all.
Lastly
One way to simplify your JMX creation is to use my SimpleJMX package.
Solution 2
Per the second answer Why does JMX connection to Amazon EC2 fail?, the difficulty here is that by default the RMI port is selected at random, and clients need access to both the JMX and RMI ports. If you're running jdk7u4 or later, the RMI port can be specified via an app property. Starting my server with the following JMX settings worked for me:
Without authentication:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.rmi.port=9998
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.rmi.server.hostname=<public EC2 hostname>
With authentication:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.rmi.port=9998
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.password.file=/path/to/jmxremote.password
-Djava.rmi.server.hostname=<public EC2 hostname>
I also opened ports 9998-9999 in the EC2 security group for my instance.
Solution 3
A bit different approach by using ssh tunnels
-
(On the Remote machine) Pass the following flags to the JVM
-Dcom.sun.management.jmxremote.port=1099 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=127.0.0.1
-
(On the Remote machine) Check which ports java started to use
$ netstat -tulpn | grep java tcp 0 0 0.0.0.0:37484 0.0.0.0:* LISTEN 2904/java tcp 0 0 0.0.0.0:1099 0.0.0.0:* LISTEN 2904/java tcp 0 0 0.0.0.0:45828 0.0.0.0:* LISTEN 2904/java
-
(On the local machine) Make ssh tunnels for all the ports
ssh -N -L 1099:127.0.0.1:1099 ubuntu@<ec2_ip> ssh -N -L 37484:127.0.0.1:37484 ubuntu@<ec2_ip> ssh -N -L 45828:127.0.0.1:45828 ubuntu@<ec2_ip>`
(On the local machine) Connect by Java Mission Control to
localhost:1099
Solution 4
The answer given by Gray worked for me, however I find that I have to open TCP ports 0 to 65535 or I don't get in. I think that you can connect on the main JMX port, and then get another one assigned. I got that from this blog post that has always worked well for me.
Solution 5
We are using AWS Elastic Container Service for running our spring boot services. The below config allowed us to connect to our docker containers.
Without Authentication:
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=9090 \
-Dcom.sun.management.jmxremote.rmi.port=9090 \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Djava.rmi.server.hostname=$(/usr/bin/curl -s --connect-timeout 2 \
http://169.254.169.254/latest/meta-data/public-ipv4)
I found it crisp and also doesn't require any other servicer side init script.
Gray
Java hacker. In a previous life a C hacker. Ton of distributed architecture experience, lot of networking/multithreading, some ORM knowledge. ORMLite primary author.
Updated on May 12, 2020Comments
-
Gray about 4 years
We are having problem connecting to our Java applications running in Amazon's EC2 cluster. We definitely have allowed both the "JMX port" (which is usually the RMI registry port) and the server port (which does most of the work) to the security-group for the instances in question. Jconsole connects but seems to hang and never show any information.
We are running our java with something like the following:
java -server -jar foo.jar other parameters here > java.log 2>&1
We have tried:
- Telnets to the ports connect but no information is displayed.
- We can run
jconsole
on the instance itself using remote-X11 over ssh and it connects and shows information. So the JRE is exporting it locally. - Opening all ports in the security group. Weeee.
- Using
tcpdump
to make sure the traffic is not going to other ports. - Simulating it locally. We can always connect to our local JREs or those running elsewhere on our network using the same application parameters.
java -version
outputs:OpenJDK Runtime Environment (IcedTea6 1.11.5) (amazon-53.1.11.5.47.amzn1-x86_64) OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
As an aside, we are using my Simple JMX package which allows us to set both the RMI registry and server ports which are typically semi-randomly chosen by the RMI registry. You can also force this with something like the following JMX URI:
service:jmx:rmi://localhost:" + serverPort + "/jndi/rmi://:" + registryPort + "/jmxrmi"
These days we use the same port for both the server and the registry. In the past we have used
X
as the registry-port andX+1
for the server-port to make the security-group rules easy. You connect to the registry-port injconsole
or whatever JMX client you are using. -
willglynn over 11 yearsYou could issue a DescribeInstances call to determine what tags apply to your instance if you really needed to (e.g. via the EC2 command-line utilities), but it's better practice to communicate configuration information to an instance via user data.
-
Gray over 11 yearsThanks @willglynn. I was hoping to get the tags without having to add my access/secret keys to my instance. Yeah, we now use the UserData but I'd rather it be
key=value
type stuff so that ops won't make a typo. -
willglynn over 11 yearsThere's a mechanism for automatically generating and delivering AWS credentials over the same instance metadata channel: see IAM roles for EC2 instances. You could create a role restricted to EC2 DescribeInstances access, letting you automate everything without without going mad from credentials management.
-
Gray over 11 yearsWe do not have to do that @Eric. We just have the RMI port and the JMX server port opened. You do need to specify both ports when you create your JMX server. As an aside, simpleJmx package does that pretty easily for you: 256.com/sources/simplejmx
-
Gray over 11 yearsSee this answer: stackoverflow.com/questions/8386001/…
-
Eric Pugh over 11 yearsGray, that is interesting... However, I am using Coda Hale Metrics library, which does it's own magic in setting up JMX.... I wonder if it can let me do what simplejmx does?
-
Rui Gonçalves almost 11 yearsI'm having the same problem as you, and I already set the hostname and preferIPv4 flags with no success. Can you please explain better your recent edit? Thank you!
-
Gray almost 11 yearsI've added some more details @RuiGonçalves. Let me know if it makes more sense now.
-
Rui Gonçalves almost 11 yearsYes, it makes sense now, thank you. Unfortunately, I don't think it applies to my problem. I'm starting
java
with-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12345 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=ec2-X-X-X-X.compute-1.amazonaws.com -Djava.net.preferIPv4Stack=true
and I still can't connect to my remote process. I simplified the scenario to a simple application that prints "Hello" every 60 seconds in a default Ubuntu AMI with OpenJDK. Any idea of the problem? -
Gray almost 11 yearsSecurity groups @RuiGonçalves? You can't just open port 12345. You also need to open the 2nd allocated port, right? See: stackoverflow.com/questions/8386001/…
-
Rui Gonçalves almost 11 years@Gray You're right, through
netstat -lp
in my instance I could see that two extra ports were opened by java. I wasn't aware that I needed one more, I guess when I was testing before I was testing with all ports opened for me... dumb me. There isn't a non-programmatic way (i.e. by the way of system properties) to define that second port, is it? If not, I guess I'll give your SimpleJmx package a try, then! -
Matt about 9 yearsIf your instance doesn't have a public EC2 hostname (hiding it behind an ELB for example) and you just want jmx to work internally, then set up the jmx ports and security group as Mark describes above, and put the instance hostname in for the java.rmi.server.hostname. You'll know you instance is set up this way if you call the /meta-data/public-hostname/ endpoint listed in @Gray's answer and you get an empty reponse back.
-
russellhoff over 7 yearsMy guess is that you could use your 10.x.x.x IP address, or the domain you own (subdomain). I think that the core is to force to use ipv4. Thank you so much!
-
Harshit about 6 yearsthis method is useful when you are automating EC2 instances because you don't know in which instance your service will run and hence you don't know IP in advance.
-
Avinash Anand over 5 yearsPlease update the link. The given link seems to be giving 404
-
devesh-ahuja almost 5 yearsAdding
-Dcom.sun.management.jmxremote.rmi.port
resolved the issue for me. Thanks. -
Eric Pugh over 4 yearsI fixed the link!
-
Gray about 4 yearsYou can set both ports to the same number I think @mmindenhall.
-
Gray about 4 yearsThe problem is that you don't know before you attach what ports you need to forward. You could however set both the port and the
com.sun.management.jmxremote.rmi.port
to the same port to get it to work. -
Gray about 4 yearsSimpleJMX has moved to 256stuff.com/sources/simplejmx
-
Timofey about 4 years@Gray--SOstopbeingevil this concern is addressed by the step 2, no?
-
Gray about 4 yearsMy point is that you have to connect to the box via ssh, run the netstat command, and then connect back again with the ports forwarded. If you set both ports then all you would need is to forward 1099.
-
Timofey about 4 yearsThis is definitely easier, need to test it though