Solr MultiCore Search

Question

I am using Apache Solr for search. I use this to provide personal user-based search. i.e. each user has a separate physical Lucene Index. So for 10 users, I have 10 separate physical indexes on disk.

To support searches on these indexes, I am planning to use Solr MultiCore Feature. With the various articles I have been reading regarding this, it looks like this would work.

Where I am actually not sure is, when a solr searcher gets a query, instead of sending the query to all the multiple-cores, how do I funnel the query to that core which has that particular user's index connected to? Is this a config change or do I need to do code level changes?

i.e. I want to send the query to only one solr-core (based on userid). Is this even possible?

UPDATE: So according to one of the solutons I can add multi-cores in the solrconfig.xml i.e. at the time of starting solr I'll need to mention the cores (or in my case the users). So now, if I want to add a new user's index, I'll probably need to stop solr, edit its config, add that users core & start solr again. Is there any way to dynamically add cores to a running solr instance?

Solution 1

Solr cores are essentially multiple indices run in the same context on an application server. You can think of it as installing 1 war-file for each user. Each core is separated by a name, hence you must yourself keep track of which url is valid for which user.

E.g.,

http://host.com/solr/usercore1/select?q=test http://host.com/solr/usercore2/select?q=test

Which is based on the config solr.xml:

<solr persistent="true" sharedLib="lib">
 <cores adminPath="/admin/cores">
  <core name="usercore1" instanceDir="usercore1" />
  <core name="usercore2" instanceDir="usercore1" />
 </cores>
</solr>

...instead of sending the query to all the multiple-cores...

This approach is called sharding and is based on distributed searching, which is a completely separate feature which focuses on splitting one users index over multiple solr instances.

[EDIT] One approach to creating new cores is by solrj which provides a routine CoreAdmin.createCore(..). You could also do this using a manual HTTP request: /cores?action=CREATE&name=usercore3...

Solr can also reload its config dynamically, if you had a script in place which edited the cores config these changes should be picked up too.

Solution 2

You can combine multicore with sharding via this following URL :

http://localhost:8983/solr/core0/select?shards=localhost:8983/solr/core0,localhost:8983/solr/core1&q=*:*

Solution 3

Im using solrj.

First creating cores. I found 2 ways.

first way:

SolrCore solrCore = coreContainer.create(new CoreDescriptor(
        coreContainer,
        coreName,
        "."));
coreContainer.register(solrCore, true);

second way:

SolrQuery solrQuery = new SolrQuery();
solrQuery.setParam(CommonParams.QT, "/admin/cores");
solrQuery.setParam(
        CoreAdminParams.ACTION,
        CoreAdminParams.CoreAdminAction.CREATE.name());
solrQuery.setParam(
        CoreAdminParams.NAME,
        name);
solrQuery.setParam(
        CoreAdminParams.INSTANCE_DIR,
        "./" + name);
solrQuery.setParam(
        CoreAdminParams.CONFIG,
        solrHomeRelativePath + solrConfigHomeRelativePath);
solrQuery.setParam(
        CoreAdminParams.SCHEMA,
        solrHomeRelativePath + solrSchemaHomeRelativePath);
solrQuery.setParam(
        CoreAdminParams.DATA_DIR,
        ".");
solrServer.query(solrQuery);

to query a particular core I just do :

SolrServer solrServer = new EmbeddedSolrServer(coreContainer, coreName);

and then perform my queries the way I usually do using solrj.

So in your case, you would simply get the corename associated with the user doing a search request. The coreContainer instance would be shared but not the SolrServer instance.

By the way Im doing something similar to you!

See you.

Solution 4

When you use mmulticore, you create separate conf folder that contain separate query and schema and and the way the results are being fetched.

And when you hit the below url

http://{your localhost}:8983/solr

The you will see the list of cores that you have created. and for each core you will have to create index like this

http://{your localhost}:8983/solr/{your_core_name1}/dataimport?command=full-import
http://{your localhost}:8983/solr/{your_core_name2}/dataimport?command=full-import
http://{your localhost}:8983/solr/{your_core_name3}/dataimport?command=full-import

and after creating index, you will have to refer to the core when performing search like this,

http://{your localhost}:8983/solr/{your_core_name3}/select/?q=*:*&start=0&rows=100

View more solutions