Elasticsearch and .NET

Question

We're considering to switch from Solr/Solr.net to Elasticsearch. We started with NEST. We have only 4 documents in search index.

private static void Main(string[] args)
{
    var node = new Uri("http://localhost:9200");
    var settings = new ConnectionSettings(
        node, "my-application");

    var client = new ElasticClient(settings);

    var stopwatch = Stopwatch.StartNew();
    var sr = client.Get<Movie>(1);

    Console.WriteLine(stopwatch.ElapsedMilliseconds);
}

The code above takes approx. 250ms, while the same code with HttpClient and JsonSerializer takes 30-45ms. 250ms is too much time for just 4 documents.

Can NEST be used on high-traffic news website, or do you recommend HttpClient + JsonSerializer combo? The search page was the most visited page on our website in 2013.

Thanks in advance.

Solution 1

There are two things that have to happen in order for NEST to make the first request.

1. The Json Serializer (Json.net) in this case has to cache the type so that it knows how to serialize and deserialize the object you are sending back and forth.

2. Nest has its own fluent language for queries that must be translated from the intial types that represent the fluent query language and the delivered as JSON to elastic search. These document types also must be learned by the Json Serializer.

3. The HTTP client has to be spun up to make the request.

I currently have over 4M documents in a single index that I use with NEST and my searches from server all the way to client over the internet are taking 50-70 ms using NEST. Like you, however, after a cold start the first request is slow.

Solution 2

I suggest you to use https://github.com/ServiceStack/ServiceStack.Text, the fastest Json serializer for C#.

For the driver, use the low-level one, http://nest.azurewebsites.net/elasticsearch-net/quick-start.html

below the code I started to write to log my apps in detail and analyze them. Still have to work, but can be a good beginning.

using System;
using System.Configuration;
using Elasticsearch.Net;
using Elasticsearch;
using Elasticsearch.Net.Connection;
using Elasticsearch.Net.ConnectionPool;

namespace Common {

    /// <summary>
    /// Elastic search. Singletone, open connection and thread safe to be open for all the time
    /// the app is running, so we send ours logs to ealsticsearch to be analyzed, assychronly
    /// See the fourth version;
    /// http://csharpindepth.com/articles/general/singleton.aspx
    /// </summary>
    public sealed class ElasticSearch {
        // our instance of ourself as a singleton
        private static readonly ElasticSearch instance = new ElasticSearch();

        ElasticsearchClient client;
        string connectionString = ConfigurationManager.ConnectionStrings["Elasticsearch"].ConnectionString;

        /// <summary>
        /// Initializes a new instance of the <see cref="Common.ElasticSearch"/> class.
        /// Follow this: http://nest.azurewebsites.net/elasticsearch-net/connecting.html
        /// We use a ConnectionPool to make the connection fail-over, that means, if the 
        /// connection breaks, it reconnects automatically
        /// </summary>
        private ElasticSearch() {
            var node = new Uri(connectionString);
            var connectionPool = new SniffingConnectionPool(new[] { node });
            var config = new ConnectionConfiguration(connectionPool);
            client = new ElasticsearchClient(config);   // exposed in this class
        }

        static ElasticSearch() {
        }

        /// <summary>
        /// Gets the instance of our singleton class
        /// </summary>
        /// <value>The instance.</value>
        public static ElasticSearch Instance {
            get {
                return instance;
            }
        }

        /// <summary>
        /// Log the specified module, id and json.
        /// </summary>
        /// <param name="type">Here the entity you want to save your log,
        /// let's use it based on classes and StateMachines</param>
        /// <param name="id">Identifier. alwayes the next</param>
        /// <param name="json">Json.</param>
        public void Log(string type, string id, string json) {
            client.Index("mta_log", type, id, json);
        }

    }
}

Comments

[Basic]

Just to add a data point, I have over 100 million in a single sharded across multiple servers and NEST adds little overhead to the raw elastic response times