VisualVM over Putty SSH-tunnel

Question

I'm trying to profile remote java app, actually it is a gameserver. It works normally on my local machine (windows XP x64 with JDK1.7.0_02 x64), but behaves very wierd on the production server (CentOS with JDK1.7.0_03 i586).

I've done a lot of searching and found out that I should use VisualVM for this task. So VisualVM works great on local machine, but there is no hangs on local machine, i need profiling in production environment with real payload. I started jstatd on the remote machine with arguments

jstatd -J-Djava.security.policy=jstatd.all.policy -J-Djava.rmi.server.logCalls=false &

with the policy file

grant codebase "file:/usr/java/jdk1.7.0_02/lib/tools.jar" {
    permission java.security.AllPermission;
};

then I started my java application like this

java -server -Dcom.sun.management.jmxremote\
 -Dcom.sun.management.jmxremote.port=4000\
 -Dcom.sun.management.jmxremote.ssl=false\
 -Dcom.sun.management.jmxremote.authenticate=false\
 -jar /home/pinballSocketServer/pinballSocketServer.jar

Both application and jstatd are launched with root priveledges.

and VisualVM didn't manage to connect to remote host. But on the remote host i see the following log, while VisualVM is running and remote host added:

Feb 16, 2012 7:11:52 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(3)-217.16.27.195: [217.16.27.195: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Feb 16, 2012 7:11:56 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(3)-217.16.27.195: [217.16.27.195: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Feb 16, 2012 7:12:00 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(3)-217.16.27.195: [217.16.27.195: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]
Feb 16, 2012 7:12:04 PM sun.rmi.server.UnicastServerRef logCall
FINER: RMI TCP Connection(3)-217.16.27.195: [217.16.27.195: sun.rmi.registry.RegistryImpl[0:0:0, 0]: java.rmi.Remote lookup(java.lang.String)]

After further googling, I found out that I need to use ssh tunneling. I configured putty in the following way

http://www.advancedigital.ru/show/putty_config.jpg

and VisualVM as this

http://www.advancedigital.ru/show/visualvm_config.jpg

Adter munipulations above VisualVM connects to remote host, but I can only see the threads summary chart and profiler is inactive.

I've seen some recommendations that jvms on both machines should be similar and have the same platform (x86 or x64) but i've already tried profiling from another machine (windows 7 x86 with JDK1.7.0_03 x86), and have the same result.

I've also tried this, but get the same result again. VisualVM over ssh

How can I get this profiling to work?

Solution 1

Note that if the server has X11 libraries installed you can run JVisualVM remotely and just have Putty forward the X11 connection to a X11-server running on your local machine.

If you do not have an X11-server available, Xming - http://www.straightrunning.com/XmingNotes/ - works fine for this. If your computer is secure, you can run without access control making it much easier to get up and running.

Note that X11 communication is quite verbose. Use "blowfish"-cypher and ask for compression.

Solution 2

Even though this has been answered, I managed to do it so I will add my approach:

I used the following source:

https://bowerstudios.com/node/731

I did this in windows using git bash that has ssh command. You can also do using cygwin or pure minggw.

1) Run ssh tunnel command in command prompt (I do this in git bash/MINGGW32).

ssh -D 9010 -p 22 root@IP -v

2) Run your application on server with JMX options (one liner)

 java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 
-Dcom.sun.management.jmxremote.local.only=false 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false -jar application.jar

3) Run visualVM through socks proxy to connect (one liner)

visualvm -J-Dnetbeans.system_socks_proxy=localhost:9010 
-J-Djava.net.useSystemProxies=true

4) Actually add your JMX remote connection in visualVM

Also available: http://maythesource.com/2013/12/04/connecting-to-jmx-from-visualvm-using-ssh-tunnel/ (will be updated with more info over time).