Elastic search, multiple indexes vs one index and types for different data sets?

Question

I have an application developed using the MVC pattern and I would like to index now multiple models of it, this means each model has a different data structure.

• Is it better to use mutliple indexes, one for each model or have a type within the same index for each model? Both ways would also require a different search query I think. I just started on this.

• Are there differences performancewise between both concepts if the data set is small or huge?

I would test the 2nd question myself if somebody could recommend me some good sample data for that purpose.

Solution 1

There are different implications to both approaches.

Assuming you are using Elasticsearch's default settings, having 1 index for each model will significantly increase the number of your shards as 1 index will use 5 shards, 5 data models will use 25 shards; while having 5 object types in 1 index is still going to use 5 shards.

Implications for having each data model as index:

• Efficient and fast to search within index, as amount of data should be smaller in each shard since it is distributed to different indices.
• Searching a combination of data models from 2 or more indices is going to generate overhead, because the query will have to be sent to more shards across indices, compiled and sent back to the user.
• Not recommended if your data set is small since you will incur more storage with each additional shard being created and the performance gain is marginal.
• Recommended if your data set is big and your queries are taking a long time to process, since dedicated shards are storing your specific data and it will be easier for Elasticsearch to process.

Implications for having each data model as an object type within an index:

• More data will be stored within the 5 shards of an index, which means there is lesser overhead issues when you query across different data models but your shard size will be significantly bigger.
• More data within the shards is going to take a longer time for Elasticsearch to search through since there are more documents to filter.
• Not recommended if you know you are going through 1 terabytes of data and you are not distributing your data across different indices or multiple shards in your Elasticsearch mapping.
• Recommended for small data sets, because you will not waste storage space for marginal performance gain since each shard take up space in your hardware.

If you are asking what is too much data vs small data? Typically it depends on the processor speed and the RAM of your hardware, the amount of data you store within each variable in your mapping for Elasticsearch and your query requirements; using many facets in your queries is going to slow down your response time significantly. There is no straightforward answer to this and you will have to benchmark according to your needs.

Solution 2

Although Jonathan's answer was correct at the time, the world has moved on and it now seems that the people behind ElasticSearch have a long term plan to drop support for multiple types:

Where we want to get to: We want to remove the concept of types from Elasticsearch, while still supporting parent/child.

So for new projects, using only a single type per index will make the eventual upgrade to ElasticSearch 6.x be easier.

Solution 3

Jonathan's answer is great. I would just add few other points to consider:

• number of shards can be customized per solution you select. You may have one index with 15 primary shards, or split it to 3 indexes for 5 shards - performance perspective won't change (assuming data are distributed equally)
• think about data usage. Ie. if you use kibana to visualize, it's easier to include/exclude particular index(es), but types has to be filtered in dashboard
• data retention: for application log/metric data, use different indexes if you require different retention period

Solution 4

Both the above answers are great!

I am adding an example of several types in an index. Suppose you are developing an app to search for books in a library. There are few questions to ask to the Library owner,

Questions:

1. How many books are you planning to store?

2. What kind of books are you going to store in the library?

3. How are you going to search for books?

Answers:

1. I am planning to store 50 k – to 70 k books (approximately)

2. I will have 15 k -20 k technology related books (computer science, mechanical engineering, chemical engineering and so on), 15 k of historical books, 10 k of medical science books. 10 k of language related books (English, Spanish and so on)

3. Search by authors first name, author last name, year of publish, name of the publisher. (This gives you the idea about what information you should store in the index)

From the above answers we can say the schema in our index should look somewhat like this.

//This is not the exact mapping, just for the example

            "yearOfPublish":{
                "type": "integer"
            },
            "author":{
                "type": "object",
                "properties": {
                    "firstName":{
                        "type": "string"
                    },
                    "lastName":{
                        "type": "string"
                    }
                }
            },
            "publisherName":{
                "type": "string"
            }
        }

In order to achieve the above we can create one index called Books and can have various types.

Index: Book

Types: Science, Arts

(Or you can create many types such as Technology, Medical Science, History, Language, if you have lot more books)

Important thing to note here is the schema is similar but the data is not identical. And the other important thing is the total data you are storing.

Hope the above helps when to go for different types in an Index, if you have different schema you should consider different index. Small index for less data . big index for big data :-)

View more solutions

Comments

[AndreKR]

This answer is not complete without the info from elasticsearch.org/guide/en/elasticsearch/guide/current/…

[Kshitiz Sharma]

What is meant by retention period? Are you referring to the time to live field? That is set on a per document basis.

[Marcel Matus]

No, here retention period is meant as document/index retention - how long to store those data. Based on data quality, size, importance - I use to specify different retention policy. Some data/indexes are deleted after 7 days, others after 6w, and some after 10years...

[oblivion]

To add to the excellent answer, I quote from ES 5.2 doc that explains why maintaining large number of shards is not recommended: "By default elasticsearch rejects search requests that would query more than 1000 shards. The reason is that such large numbers of shards make the job of the coordinating node very CPU and memory intensive. It is usually a better idea to organize data in such a way that there are fewer larger shards. In case you would like to bypass this limit, which is discouraged, you can update the action.search.shard_count.limit cluster setting to a greater value."