Weblogic process just killed suddenly on redhat linux

Question

One of the process on my WAS killed at dawn.

6 process run on this server. But only one process just killed.

I don't know how to fix it. This is Weblogic last log that before process killed.

/app/orarecp/bea/domains/HRRECP/bin/startWeblogic.sh: line 180: 27066 killed               ${JAVA_HOME}/bin/java ${JAVA_VM} {MEM_ARGS} -Dweblogic.Name=${SERVER_NAME} -Djava-security.policy=${WL_HOME}/server/lib/weblogic.policy ${JAVA_OPTION} ${PROXY_SETTINGS} ${SERVER_CLASS}

And This is startWeblogic shell script. startWeblogic Shell Script: 180 line

And This is start process shell script parts.

USER_MEM_ARGS="-Xms3072m -Xmx3072m -Xverbose:gc -Xverboselog:${LOG_HOME}/gclogs/gc.${SERVER_NAME}_`date + '%Y%m%d_%H%M%S'`.out"

JAVA_OPTION="${JAVA_OPTIONS} -Xbootclasspath/p:/app/orarecp/jennifer/agent/lwst.jdk.jar:/app/orarecp/jennifer/agent/lwst.boot.jar:/app/orarecp/jennifer/agent/jennifer.boot.jar -Djennifer.config=/app/orarecp/jennifer/agent/R11.conf -javaagent:/app/orarecp/jennifer/agent/lwst.javaagent.jar -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom"

server specification is...

• CPU: 8 core, 2.4GHz // The time that CPUs utilization was under 15%
• Memory: 32Gb
  • Mem: 32922024k total, 32817492k used, 104580k free, 147464k buffers
  • Swap: 33551744k total, 148k used, 33551596k free, 15231864k cached

server stack is...

• weblogic 11g
• jrockit 1.6
• redhat linux 5.8 (64bit)

I wonder why the process is killed. What am I suppose to do?

Please answer if you need additional information.

All the answers are going to be helpful to me.

Thanks.

---

Additinal Information.

1. before that Last log But this log is occurs frequently.

<[ServletContext@228988411[app:Recruit-Front module:WebContent path:/recruit spec-version:2.5]] Servlet failed with Exception

java.lang.NullPointerException at cj.recruit.syscommon.filter.SessionLocaleFilter.doFilter(SessionLocaleFilter.java:41) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at frameone.core.common.web.filter.EnvironmentSetupFilter.doFilter(EnvironmentSetupFilter.java:88) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at cj.recruit.syscommon.filter.CrossScriptingFilter.doFilter(CrossScriptingFilter.java:22) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

<[ServletContext@228988411[app:Recruit-Front module:WebContent path:/recruit spec-version:2.5]] Servlet failed with Exception java.lang.NullPointerException

at cj.recruit.syscommon.filter.SessionLocaleFilter.doFilter(SessionLocaleFilter.java:41) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at frameone.core.common.web.filter.EnvironmentSetupFilter.doFilter(EnvironmentSetupFilter.java:88) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at cj.recruit.syscommon.filter.CrossScriptingFilter.doFilter(CrossScriptingFilter.java:22) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

2. another JAVA process (ps -ef | grep java)

ps -ef | grep java

3. another Server Java Process

   ps -ef | grep java

oraehrp 818 786 0 19:34 pts/1 00:00:00 grep java

root 9689 1 1 2015 ? 1-01:05:58 /usr/java/jdk1.6.0_23/bin/java -Djava.awt.headless=true -jar /app/CJ_agent//cjagent.jar /app/CJ_agent//conf/cjagent.conf

oraehrp 10128 10074 2 2015 ? 1-10:06:08 /usr/java/jrockit1.6/bin/java -jrockit -Xms1024m -Xmx1024m -Dweblogic.Name=EHRREP1 -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/oraehrp/bea/wlserver_10.3/server/lib/cacerts -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://52.2.205.138:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server

oraehrp 14795 14740 2 2015 ? 1-10:31:48 /usr/java/jrockit1.6/bin/java -jrockit -DAdminServer -Xms1024m -Xmx1024m -verbosegc -Dweblogic.Name=AdminServer -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server

oraehrp 23456 23395 67 18:25 ? 00:46:23 /usr/java/jrockit1.6/bin/java -jrockit -Xms2048m -Xmx2048m -Xns:256m -Xverbose:gc -Xverboselog:/svcroot/logs/was/domains/HREHRP/EHRPRD11/gclogs/gc.EHRPRD11_20160201_182544.out -Dweblogic.Name=EHRPRD11 -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/oraehrp/bea/wlserver_10.3/server/lib/cacerts -Dweblogic.threadpool.MinPoolSize=100 -Xbootclasspath/p:/app/oraehrp/jennifer/agent/lwst.jdk.jar:/app/oraehrp/jennifer/agent/lwst.boot.jar:/app/oraehrp/jennifer/agent/jennifer.boot.jar -Djennifer.config=/app/oraehrp/jennifer/agent/E11.conf -javaagent:/app/oraehrp/jennifer/agent/lwst.javaagent.jar -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://52.2.205.138:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server

oraehrp 24233 24179 49 18:27 ? 00:33:03 /usr/java/jrockit1.6/bin/java -jrockit -Xms2048m -Xmx2048m -Xns:256m -Xverbose:gc -Xverboselog:/svcroot/logs/was/domains/HREHRP/EHRPRD12/gclogs/gc.EHRPRD12_20160201_182747.out -Dweblogic.Name=EHRPRD12 -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/oraehrp/bea/wlserver_10.3/server/lib/cacerts -Dweblogic.threadpool.MinPoolSize=100 -Xbootclasspath/p:/app/oraehrp/jennifer/agent/lwst.jdk.jar:/app/oraehrp/jennifer/agent/lwst.boot.jar:/app/oraehrp/jennifer/agent/jennifer.boot.jar -Djennifer.config=/app/oraehrp/jennifer/agent/E12.conf -javaagent:/app/oraehrp/jennifer/agent/lwst.javaagent.jar -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://52.2.205.138:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server

oraehrp 25006 24944 53 18:29 ? 00:34:35 /usr/java/jrockit1.6/bin/java -jrockit -Xms2048m -Xmx2048m -Xns:256m -Xverbose:gc -Xverboselog:/svcroot/logs/was/domains/HREHRP/EHRPRD13/gclogs/gc.EHRPRD13_20160201_182933.out -Dweblogic.Name=EHRPRD13 -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/oraehrp/bea/wlserver_10.3/server/lib/cacerts -Dweblogic.threadpool.MinPoolSize=100 -Xbootclasspath/p:/app/oraehRP/jennifer/agent/lwst.jdk.jar:/app/oraehrp/jennifer/agent/lwst.boot.jar:/app/oraehrp/jennifer/agent/jennifer.boot.jar -Djennifer.config=/app/oraehrp/jennifer/agent/E13.conf -javaagent:/app/oraehrp/jennifer/agent/lwst.javaagent.jar -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://52.2.205.138:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server

Solution

This is due to lack of memory. You have slightly less than 300MB RAM free (free+buffers), and each new weblogic Java instance will be eating 3GB.

There is a functionality of the Linux kernel called the OOM killer that kills a random process whenever there is insufficient memory.

How to Configure the Linux Out-of-Memory Killer

Have a look at that JAVA arguments, and at runaway Java Scripts. You are using a lot of RAM.

Evaluate whether you need to fine tune the Java Xms/Xmx or add more RAM to the server.

I have had the opportunity to check WebLogic requirements now. As Customizing the WebLogic JVM heap size states:

The default JVM heap size for WebLogic is 3GB. ... USER_MEM_ARGS="-Xms128m -Xmx3072m ...

Pay attention to the 128m, the recommended configuration is giving it 128m when starting the process, and it growing as needed to 3GB.

Xms is the minimum memory allocated, Xmx the maximum memory. If you change Xms to 128m, the threads will only use up the memory they need when starting, and not 3GB at once by default. That will probably use less RAM over time.

From What are the Xms and Xmx parameters when starting JVMs?

The flag Xmx specifies the maximum memory allocation pool for a Java Virtual Machine (JVM), while Xms specifies the initial memory allocation pool. The memory flag can also be specified in multiple sizes, such as kilobytes, megabytes, and so on.

I would also evaluate after this change whether you need more RAM, as that the value you have now is not enough for a proper operation (e.g. too low).

Comments

[Jeff Spaulding]

Just somewhere to look: if you can't figure this out right away, watch your memory usage. I've had issues running large java servers and running out of memory (mostly due to FreeBSD's insane default settings for ZFS that eats RAM like candy). In my case it left little in the way of useful error messages, and just printed "killed" to the console.

[Rui F Ribeiro]

I have updated my answer.

[Chanho Jung]

Another my WAS is same occurred. But that WAS set memory is 1024m. It is a mystery.

[Rui F Ribeiro]

I edited the answer slightly...read it. It does not seem a big mystery to me.

[Chanho Jung]

I edited the additional information 3 on my question. That server same case. yellow column was killed process. and no log.

[Chanho Jung]

usr/java/jrockit1.6/bin/java -jrockit -Xms2048m -Xmx2048m -Xns:256m -Xverbose:gc -Xverboselog:/svcroot/logs/was/domains/HREHRP/EHRPRD12/gclog‌​s/gc.EHRPRD12_201602‌​01_182747.out -Dweblogic.Name=EHRPRD12

[Rui F Ribeiro]

I updated my answer, finally had the time to investigate properly Xms and Xmx

[Rui F Ribeiro]

Thanks for the points. updated the answer with a Xms and Xmx Java parameters explanation.

[Chanho Jung]

I have a one question. My WAS memory is 32G. so that server is not enough memory? 3G * process 3 + 1G * process 4

[Rui F Ribeiro]

Often tasks launch extra Java processes, and then there are threads too. You math says 3*3+1*4=13GB, however your free command show the server is using 31.5GB of RAM and 150MB of swap. (...) As I said, change the Xms, it makes a lot of difference, restart the services, or even better, the server, to begin with a clean slate, and then reevaluate the situation again.

[Chanho Jung]

according this article(linuxatemyram.com), my WAS memory is fine. am I right? This is "FREE" command result. Mem: 32922024(total) 32818796(used) 103228(free) cached(15138616) -/+ buffers/cache: 17520572(used) 15401452(free)