Tomcat dies suddenly
Solution 1
Sorry I had to remove the green check from @erickson. I finally figured out what was killing Tomcat.
It looks like a profiler plugin is not configured correctly with VisualVM and attempting to run a profile on the Tomcat process killed it.
Investigating why right now, and will update this answer once I know more.
Solution 2
If the JVM crashes, you should have a hs_err_pidNNN.log file; you don't have to do anything to enable this. Its location depends on your OS and how you are running Tomcat. On Windows, they can show up on your desktop, unless you are running as a service. Otherwise, they should be in the current working directory of the crashed process.
Your operating system probably provides additional tools for process monitoring; you could describe your environment more, or perhaps ask at serverfault.com.
It's also possible that jvisualvm
is actually causing the crash.
I'd try reproducing the problem, and progressively simplify the scenario to help isolate the cause.
Solution 3
Another possibility is that the OS is running out of memory and the OOM Killer is killing your process. In this case, the JVM wouldn't get an opportunity to write a heap dump, or an hs_err_pid file.
Solution 4
You can use the option java -XX:+HeapDumpOnOutOfMemoryError to create a heap dump for jvm crash due to out of memory error.
More details here Using HeapDumpOnOutOfMemoryError parameter for heap dump for JBoss.
Related videos on Youtube
IAmYourFaja
my father is a principal at burgoyne intnl and got me this job programming lisp and development. I aspire to unittesting with a concentration in mobile platforms.
Updated on July 09, 2022Comments
-
IAmYourFaja almost 2 years
Trying to diagnose some bizarre Tomcat (
7.0.21
) and/or JVM errors on a 64-bit linux (CentOS) machine.I'm load testing our server application and tried hitting it with 100K messages. Launched
jvisualvm
and kept my eye on the heap the whole time. Everything was looking great* (see below) until I got to about 93K processed messages and then Tomcat just died. Ran aps
on Tomcat's PID number to confirm it was dead.Up until this crash:
- Load test had been running for about 90 minutes; should have finished shortly thereafter since we were at 93K/100K)
- CPU was holding strong around 45%
- Used heap was around 2GB (plus or minus a bunch after GCs) but heap size grew from 4GB to
MAX_HEAP
after about 30 minutes - Class loading/unloading was cycling normally
- Thread dumps were normal
Nowhere in the server code are any calls to
System.exit()
- so we can rule that right out (and yes I've double-checked!!!).I'm not sure if this is Tomcat crashing or the JVM (how do I tell?). And even if I did know, I can't seem to find any indication of what went wrong:
- All of the server app's logs just stop without any ERROR messages (even though we have logging universally set to DEBUG and higher)
- Tomcat's
catalina.out
and respectlocalhost_access_*
files just stop without any info
I've heard it is possible to have Tomcat log a coredump when it does but not sure how to do that and online examples aren't helping much.
How would SO go about diagnosing this? What steps should I take to start ruling out all of the possible factors?
Thanks in advance!
-
brady over 12 yearsIs it Schrödinger's tomcat?
-
josh.trow over 12 yearsDid you check more logs? stackoverflow.com/questions/1575730/debugging-tomcat-crash
-
Tom over 10 yearsFYI in my experience you can tell if tomcat is killed by the OOM Killer as the word 'Killed' will appear somewhere close to the end of catalina.out.
-
Chloe almost 5 yearsTry to add a swapfile to your system. It doesn't have to be big. superuser.com/questions/1445510/…
-
IAmYourFaja over 12 yearsThanks erickson, where would I find this log file? I assume in my JDK directory or thereabouts?
-
IAmYourFaja over 12 yearsahhh nm - looks like it will either be the cwd or /tmp
-
brady over 12 yearsIt depends on your OS and how you are running Tomcat. On Windows, they can show up on your desktop, unless you are running as a service. Otherwise, they should be in the current working directory of the crashed process.
-
IAmYourFaja over 12 yearsOne last question - how could jvisualvm be causing the crash? By eating up too much memory? How could I diagnose this?
-
brady over 12 yearsI don't have a specific mechanism in mind for
jvisualvm
, it's just one more variable that you can toggle. -
brady about 12 yearsYou mean, "it's also possible that
jvisualvm
is causing the crash?" ;) -
Dilip Rajkumar about 12 yearsI am having the same issue. My tomcat is crashing with no logs.. I am using CentOs 6.2. I Checked hs_err_pid, I dont have any logs on it. Also there is no JVM crash found in my logs. I have also looked for OOM killer logs also. can you please guide me to solve this issue.
-
kodmanyagha about 7 yearsSorry for downvote. I clicked it by mistake :( I can't get back
-
Chloe almost 5 yearsWhat if it runs as a system process with
systemctl start tomcat
? -
Chloe almost 5 yearsShouldn't you only accept the answer when you know all the details?