Solr stopped with Error opening new searcher at org.apache.solr.core

Question

Yesterday suddenly our solr server got stopped with following exception

org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1438) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1550) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1319) at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:574) at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95) at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64) at org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1147) at org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157) at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:266) at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173) at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)

Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:889) at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:283) at org.apache.lucene.store.MMapDirectory$MMapIndexInput.

But at that time we have lot of memory left in that server. Any guess?

Solution 1

Assuming its a linux system you can try ulimit -v unlimited

Check the below links for more information.

https://support.lucidworks.com/hc/en-us/articles/206127457-Caused-by-java-io-IOException-Map-failed

https://stackoverflow.com/a/8893684/689625

You will get answers much faster if you include more information, like the version of your software (in this case solr version), OS Name and version, Java version etc.

Solution 2

The flag -fix didn't work in my later version, instead I used exorcise

java -cp lucene-core-5.5.0.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex PATH_TO_INDEX -exorcise

I hope this could help.

Solution 3

I got this error:

Error creating core [mm_shard1_replica1]: Error opening new searcher

and tried fiddling with ulimit and max_map_count

In the end it turned out to be a bad error message because the problem turned out to be a corrupt index.

Fixed it by shutting down solr and doing this:

cd /var/lib/solr/example/solr-webapp/webapp/WEB-INF/lib
java -cp lucene-core-4.10.4.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /path/to/index -fix

To just check wether the index is broken remove the -fix at the end of above command. After that I started up solr and it recreated its index from the rest of the solrcloud cluster.

Comments

[Progs]

there is any considerations before doing this?