java.lang.OutOfMemoryError: Java heap space-How to resolve this error?

Question

I am developing a website in JSP(JDK 1.6) and hosting it using apache tomcat7. There is a webpage which takes user input,builds a query and sends it to Lucene Database(20GB in size) and displays the result in a downloadable textfile.However this functionality is not working properly and I get the following exception after I submit the user Input.

HTTP Status 500 -

type Exception report

message

description The server encountered an internal error () that prevented it from fulfilling this request.

exception

org.apache.jasper.JasperException: javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space
    org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:549)
    org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:455)
    org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
    org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

root cause

javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space
    org.apache.jasper.runtime.PageContextImpl.doHandlePageException(PageContextImpl.java:911)
    org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:840)
    org.apache.jsp.Getinfo_jsp._jspService(Getinfo_jsp.java:305)
    org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
    org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
    org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
    org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

root cause

java.lang.OutOfMemoryError: Java heap space
    org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
    org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
    org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
    org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
    org.apache.lucene.search.similarities.TFIDFSimilarity.exactSimScorer(TFIDFSimilarity.java:760)
    org.apache.lucene.search.PhraseQuery$PhraseWeight.scorer(PhraseQuery.java:285)
    org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:323)
    org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:323)
    org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:603)
    org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:482)
    org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:438)
    org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
    org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
    com.cde.sentiment.DataCollection.GetTweets(DataCollection.java:76)
    com.cde.sentiment.SentimentCalculation.GenerateOutputFile(SentimentCalculation.java:102)
    org.apache.jsp.Getinfo_jsp._jspService(Getinfo_jsp.java:154)
    org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
    org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
    org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
    org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

note The full stack trace of the root cause is available in the Apache Tomcat/7.0.27 logs.
Apache Tomcat/7.0.27

Here's the stacktrace of above exception

24 Jun, 2013 3:04:35 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [jsp] in context with path [/SentimentAnalysisModule] threw exception [javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space] with root cause
java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.loadNumeric(Lucene42DocValuesProducer.java:180)
        at org.apache.lucene.codecs.lucene42.Lucene42DocValuesProducer.getNumeric(Lucene42DocValuesProducer.java:146)
        at org.apache.lucene.index.SegmentCoreReaders.getNormValues(SegmentCoreReaders.java:301)
        at org.apache.lucene.index.SegmentReader.getNormValues(SegmentReader.java:259)
        at org.apache.lucene.search.similarities.TFIDFSimilarity.exactSimScorer(TFIDFSimilarity.java:760)
        at org.apache.lucene.search.PhraseQuery$PhraseWeight.scorer(PhraseQuery.java:285)
        at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:323)
        at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:323)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:603)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:482)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:438)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
        at com.cde.sentiment.DataCollection.GetTweets(DataCollection.java:76)
        at com.cde.sentiment.SentimentCalculation.GenerateOutputFile(SentimentCalculation.java:102)
        at org.apache.jsp.Getinfo_jsp._jspService(Getinfo_jsp.java:154)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)

Solution

Your code with Lucene consumes more memory than declared for the heap. It may not be an error (leak) but only a configuration issue (not enough memory).

1. Try using -Xmx to increase the heap size. For Apache Tomcat use i.e. export CATALINA_OPTS="-Xmx1024M"

2. Find how memory is used using some profiling tool i.e. VisualVM which is bundled with Oracle JDK.