ElasticSearch Spring-Data Date format always is long

Question

When using spring-data to insert Elasticsearch document with Date type, I can't get right date format, the date format always is Long.

here is the java code: Entity.java

import java.util.Date;

import org.springframework.data.annotation.Id;
import org.springframework.data.elasticsearch.annotations.DateFormat;
import org.springframework.data.elasticsearch.annotations.Document;
import org.springframework.data.elasticsearch.annotations.Field;
import org.springframework.data.elasticsearch.annotations.FieldIndex;
import org.springframework.data.elasticsearch.annotations.FieldType;

import com.fasterxml.jackson.annotation.JsonProperty;

@Document(indexName = "entity-index", type = "entity-type")
public class Entity {
    @Id
    private String id;

    @Field(type = FieldType.Date, index = FieldIndex.not_analyzed, store = true, 
            format = DateFormat.custom, pattern = "yyyy-MM-dd'T'hh:mm:ss.SSS'Z'")
    private Date createDate;

    private String system;
    private double score;

    @Field(type = FieldType.Date, format = DateFormat.date_optional_time)
    @JsonProperty(value = "@timestamp")
    private Date updateDate;
    // omit setter and getter 
}

Here is the Test

public class EntityDAOTest {
    @Autowired
    private ElasticsearchTemplate template;

    @Before
    public void init() {
        template.createIndex(Entity.class);
        template.putMapping(Entity.class);
    }


    @Test
    public void testCreate() {
        Entity entity = new Entity();
        entity.setId("5");
        entity.setCreateDate(new DateTime(2015,05,27,0,0).toDate());
        entity.setUpdateDate(new DateTime(2015,05,27,0,0).toDate());
        entity.setSystem("systemC");
        entity.setScore(5.7);
        IndexQuery query = new IndexQueryBuilder().withObject(entity).withId(entity.getId()).build();
        template.index(query);
    }

I can get the mapping of the created entity:

{
   "entity-index": {
      "mappings": {
         "entity-type": {
            "properties": {
               "@timestamp": {
                  "type": "long"
               },
               "createDate": {
                  "type": "date",
                  "store": true,
                  "format": "yyyy-MM-dd'T'hh:mm:ss.SSS'Z'"
               },
               "id": {
                  "type": "string"
               },
               "score": {
                  "type": "double"
               },
               "system": {
                  "type": "string"
               },
               "updateDate": {
                  "type": "date",
                  "format": "date_optional_time"
               }
            }
         }
      }
   }
}

However, when I search it curl -X GET /entity-index/_search, I get the following document:

 {
               "id": "5",
               "createDate": 1432656000000,
               "system": "systemC",
               "score": 5.7,
               "@timestamp": 1432656000000
 }

and the Date Fields are all Long type, how can I get the date format : '2015-08-17T12:00:00.000'?

Solution 1

Your mapping is created correctly. The problem is more likely to come from the Jackson JSON serializer. You should try adding this annotation to your date fields: @JsonFormat (shape = JsonFormat.Shape.STRING, pattern ="yyyy-MM-dd'T'HH:mm:ss.SSSZZ").

There are also some alternative solutions that might better suit your case (i.e. creating a CustomDateSerializer, etc).

Solution 2

I would suggest to give a try to existing mappings available if they suffice the need. With es version: 7.14.1, following works quite well:

@Field(type = FieldType.Date, format = DateFormat.basic_date_time)
private Date creationTime;

Good thing is one doesn't have to worry about mappings, as it's automatically created. It might be supported in older versions, which I haven't checked.

Solution 3

Starting from Elasticsearch 7 you should't use yyyy but uuuu. e.g:

@Field(type = FieldType.Date, format = DateFormat.custom, pattern = "uuuu-MM-dd'T'HH:mm:ss.SSSZZ")
private Date lastModifiedDate;

You don't need @JsonProperty because now Spring Data Elasticsearch doesn't use Jackson but instead a MappingElasticsearchConverter. With this annotation, a converter is automatically created for this property and used.

Solution 4

Something to keep in mind:

JSON doesn’t have a date data type, so dates in Elasticsearch can either be:

• Strings containing formatted dates, e.g. "2015-01-01" or "2015/01/01. 12:10:30".
• A long number representing milliseconds-since-the-epoch.
• An integer representing seconds-since-the-epoch.

Also from the same doc:

Dates will always be rendered as strings, even if they were initially supplied as a long in the JSON document

This implies that querying the data type for a date field will always be a string, a long or an integer. There is no special "Date" field in elastic search.

Read more here: https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html

Solution 5

It worked for me with below settings. Note: delete your index before to test this change. Make sure that the patterns are same at all places.

@Field(type = FieldType.Date, store = true, format = DateFormat.custom, pattern = "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'")
@JsonFormat (shape = JsonFormat.Shape.STRING, pattern ="yyyy-MM-dd'T'HH:mm:ss.SSS'Z'")
private Date date;

View more solutions

Comments

[Petr Kostroun]

I use @JsonFormat (shape = JsonFormat.Shape.STRING, pattern ="yyyy-MM-dd'T'HH:mm:ss.SSSZZ") on date field, but it still fails on Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse [creationDate]- Caused by: java.lang.NumberFormatException: For input string: "2018-04-04T14:19:49". Any Option?

[Val]

@PetrKostroun you're missing the timezone, remove ZZ from the pattern.

[Shu]

it's really weird that we have to duplicate annotation values both in Field and JsonFormat.

[evandongen]

For me, only adding the @JsonFormat annotation was enough

[Dhwanil Patel]

@Val I'm using spring boot with elasticsearch jpa developed using Jhipster.

[Dhwanil Patel]

@Column(name = "created_date", columnDefinition = "Instant.now()") private Instant createdDate = Instant.now(); public Document createdDate(Instant createdDate) { this.createdDate = createdDate; return this; }

[Dhwanil Patel]

Above code generate, index column with type long. So when i add data it store value like : 1588938113466. But at time of retrieve it gives exception like, failed to parse field [createdDate] of type [long]

[Dhwanil Patel]

Index and column configuration default generated using above code which generated using Jhipster. So i bit configure how to manage that. Can you please guide me?

[Val]

@DhwanilPatel Please create a new question to describe your needs, this one is closed already.

[DavidR]

Hopefully people read down this far as this is the much better approach and should probably be the accepted answer.

[DavidR]

I would also add that DateFormat.date_time gives a more human readable format than DateFormat.basic_date_time.