Index a dynamic object using NEST

Question

I am building an API application that essentially allows a user to build a document, which can be structured however they want, that will be stored in Elasticsearch. Essentially, I'm providing a simple interface for users to access our Elasticsearch instance. I'm trying to keep the implementation as simple as possible. Here's what I'm dealing with so far.

The object for the expected body:

public class DocumentModel
{
    public string Index { get; set; }
    public string Type { get; set; }
    public string Id { get; set; }
    [ElasticProperty(Type = FieldType.Nested)]
    public dynamic Document { get; set; }
}

Simple implementation:

[HttpPost]
[Route("")]
public IHttpActionResult Post(DocumentModel document)
{
    Uri nodeLocation = new Uri("http://localhost:9200");
    IConnectionPool connectionPool = new SniffingConnectionPool(new List<Uri> { nodeLocation });
    ConnectionSettings settings = new ConnectionSettings(connectionPool);
    ElasticClient esClient = new ElasticClient(settings);

    IIndexResponse result = esClient.Index(document, i => i
        .Index(document.Index)
        .Type(document.Type)
        .Id(document.Id));

    return Ok(result.IsValid);
}

This works fine, but it includes the Index, Type, and Id in the source. What I'd really like to do is simply provide those three pieces of information when indexing, but actually just index document.Document, which is of a dynamic type. But, that seems to disagree with Nest, as it throws an error in the IDE and at compile time:

"An anonymous function or method group cannot be used as a constituent value of a dynamically bound operation"

"Cannot use a lambda expression as an argument to a dynamically dispatched operation without first casting it to a delegate or expression tree type".

How can I index just document.Document? Is there a better way to handle an incoming JSON document of unknown structure than using a dynamic type?

Solution

There's a couple ways to do this.

Trying to index the document as type dynamic won't work, but you can index it as an object through the IndexRequest object.

dynamic dynamicDoc = new { /*fill in document format here*/ };
ElasticClient esClient = new ElasticClient(esSettings);

IndexRequest<object> request = new IndexRequest<object>(dynamicDoc)
{
    Index = "someindex",
    Type = "SomeType",
    Id = "someid"
};

esClient.Index<object>(request);

Or if dealing with documents in bulk

List<dynamic> Documents = new List<dynamic>();
//Populate Documents

BulkDescriptor descriptor = new BulkDescriptor();
foreach(var doc in Documents)
{
    descriptor.Index<object>(i => i
        .Index("someindex")
        .Type("SomeType")
        .Id("someid")
        .Document(doc));
}

esClient.Bulk(descriptor);

NEST (or more accurately, Elasticsearch.Net) also has a .Raw method variant attached to the ElasticClient class, which can index raw JSON. Using Raw.Index() let's us do things like this:

string documentJson = JsonConvert.SerializeObject(document.Document);

ElasticsearchResponse<string> result = esClient.Raw.Index(document.Index, document.Type, document.Id, documentJson);

The type descriptor for the response is the type you'll expect the response to be in (string means you'll have a serialized json response which you can deserialize and do something with). This allows us to sidestep the entire object type issue and NEST indexes the document into Elasticsearch exactly as expected.

Comments

[Adrian]

I wish Raw.Index kept parent-child relationships from the old index in the new index

[nothingisnecessary]

As alternative to dynamic I have used Dictionary<string, object> or inherit from the class. Warning: if you inherit from Dictionary NEST will not automap the other properties on your document (put them in the dictionary instead). This also worked well for variable attributes: common attributes went into POCO properties, variable attributes went into a Data property (with type Dictionary<string,object>). This bulk method is easy to use. Don't forget to get result of call to Bulk to check for .Errors, etc!

[Abdul Saboor]

dynamic type converted by dynamic document = JsonConvert.DeserializeObject<dynamic>(jsonString); will not work directly. Here is the reason and NEST.JsonNetSerializer is the solution. This is how to implement the solution