datetime in Elasticsearch - How to handle timezone

Question

I try to index a field containing a date.

How can I index a date from a different timezone ?

I've set my elasticsearch field like so : 'requested_dt': {"type": "date", "format": "date_time_no_millis"} 'local_dt': {"type": "date", "format": "date_time_no_millis"}

I've tried to index these values (local_dt) : (requested_dt is the current time in France)

IT 2016-10-27T23:46:17Z
GB 2016-10-27T22:46:19Z

I don't get the expected result through Kibana :

[local_dt]
IT October 28th 2016, 01:46:17.000
GB October 28th 2016, 00:46:19.000

[requested_dt]
IT October 27th 2016, 23:46:17.000
GB October 27th 2016, 23:46:19.000

So, for requested_dt, I get what I expect.

For local_dt, I don't get what I want.

I've tried to replace the Z value with the UTC offset but I'm not able to get the correct output.

Is there someone able to explain to me how to get the correct output for each timezone I want ?

Solution 1

As I read(and my own experience), Kibana will index the @timestamp assuming they come in UTC, and only in format like 2018-04-23T10:45:13.899Z. Note that we only have milliseconds and the T as separator and Z indicating UTC.

https://discuss.elastic.co/t/kibana-timezone/29270/5

So, if you have a local datetime object, try to convert to UTC time, and format it like above.

• In case of now(), use timezone.now() of django.utils.timezone; or, without django, you have datetime.datetime.utcnow()
• If you have an datetime() object already, you can do:

this:

import pytz, datetime
local = pytz.timezone ("Europe/Paris")
local_dt = local.localize(your_datetime_object, is_dst=None)
utc_dt = local_dt.astimezone(pytz.utc)

(thanks to this answer)

When you have the object, format like:

timeStr = datetime.strftime(your_utc_time_object, "%Y-%m-%dT%H:%M:%S.%f") # %f: microseconds, 1/10^6 seconds.
timeStr = timeStr[:-3] # delete the trailing 3 digits, convert to milliseconds
toInsert["@timestamp"] = nowStr + "Z" # add 'Z' at last

Solution 2

For the benefit of anyone coming here via Google search like me, you cannot append a military time zone letter to the end of the timestamp and have Elasticsearch recognize it.

I thought this would be the case since it recognizes and outputs the "Z" at the end of a UTC timestamp, so I appended "R" to my own timestamps to signify they came from UTC-5. Here's what they would look like next to each other:

"2020-04-09T07:35:15.100Z"  # parsed as UTC
"2020-04-09T07:35:15.100R"  # illegal argument exception

However, none of the built-in formats would recognize this additional letter; you must either specify the time offset like so

"2020-04-09T07:35:15.100-0500"
"2020-04-09T07:35:15.100-05:00"

or specify the timezone in your pipeline processor

{
  "pipeline": {
    "processors": [
      {
        "date": {
          "field": "raw_date",
          "formats": ["ISO8601"],
          "timezone": "America/New_York"
        }
      }]
  }
}