Converting between datetime, Timestamp and datetime64

Question

How do I convert a numpy.datetime64 object to a datetime.datetime (or Timestamp)?

In the following code, I create a datetime, timestamp and datetime64 objects.

import datetime
import numpy as np
import pandas as pd
dt = datetime.datetime(2012, 5, 1)
# A strange way to extract a Timestamp object, there's surely a better way?
ts = pd.DatetimeIndex([dt])[0]
dt64 = np.datetime64(dt)

In [7]: dt
Out[7]: datetime.datetime(2012, 5, 1, 0, 0)

In [8]: ts
Out[8]: <Timestamp: 2012-05-01 00:00:00>

In [9]: dt64
Out[9]: numpy.datetime64('2012-05-01T01:00:00.000000+0100')

Note: it's easy to get the datetime from the Timestamp:

In [10]: ts.to_datetime()
Out[10]: datetime.datetime(2012, 5, 1, 0, 0)

But how do we extract the datetime or Timestamp from a numpy.datetime64 (dt64)?

.

Update: a somewhat nasty example in my dataset (perhaps the motivating example) seems to be:

dt64 = numpy.datetime64('2002-06-28T01:00:00.000000000+0100')

which should be datetime.datetime(2002, 6, 28, 1, 0), and not a long (!) (1025222400000000000L)...

Solution 1

To convert numpy.datetime64 to datetime object that represents time in UTC on numpy-1.8:

>>> from datetime import datetime
>>> import numpy as np
>>> dt = datetime.utcnow()
>>> dt
datetime.datetime(2012, 12, 4, 19, 51, 25, 362455)
>>> dt64 = np.datetime64(dt)
>>> ts = (dt64 - np.datetime64('1970-01-01T00:00:00Z')) / np.timedelta64(1, 's')
>>> ts
1354650685.3624549
>>> datetime.utcfromtimestamp(ts)
datetime.datetime(2012, 12, 4, 19, 51, 25, 362455)
>>> np.__version__
'1.8.0.dev-7b75899'

The above example assumes that a naive datetime object is interpreted by np.datetime64 as time in UTC.

---

To convert datetime to np.datetime64 and back (numpy-1.6):

>>> np.datetime64(datetime.utcnow()).astype(datetime)
datetime.datetime(2012, 12, 4, 13, 34, 52, 827542)

It works both on a single np.datetime64 object and a numpy array of np.datetime64.

Think of np.datetime64 the same way you would about np.int8, np.int16, etc and apply the same methods to convert between Python objects such as int, datetime and corresponding numpy objects.

Your "nasty example" works correctly:

>>> from datetime import datetime
>>> import numpy 
>>> numpy.datetime64('2002-06-28T01:00:00.000000000+0100').astype(datetime)
datetime.datetime(2002, 6, 28, 0, 0)
>>> numpy.__version__
'1.6.2' # current version available via pip install numpy

I can reproduce the long value on numpy-1.8.0 installed as:

pip install git+https://github.com/numpy/numpy.git#egg=numpy-dev

The same example:

>>> from datetime import datetime
>>> import numpy
>>> numpy.datetime64('2002-06-28T01:00:00.000000000+0100').astype(datetime)
1025222400000000000L
>>> numpy.__version__
'1.8.0.dev-7b75899'

It returns long because for numpy.datetime64 type .astype(datetime) is equivalent to .astype(object) that returns Python integer (long) on numpy-1.8.

To get datetime object you could:

>>> dt64.dtype
dtype('<M8[ns]')
>>> ns = 1e-9 # number of seconds in a nanosecond
>>> datetime.utcfromtimestamp(dt64.astype(int) * ns)
datetime.datetime(2002, 6, 28, 0, 0)

To get datetime64 that uses seconds directly:

>>> dt64 = numpy.datetime64('2002-06-28T01:00:00.000000000+0100', 's')
>>> dt64.dtype
dtype('<M8[s]')
>>> datetime.utcfromtimestamp(dt64.astype(int))
datetime.datetime(2002, 6, 28, 0, 0)

The numpy docs say that the datetime API is experimental and may change in future numpy versions.

Solution 2

You can just use the pd.Timestamp constructor. The following diagram may be useful for this and related questions.

Solution 3

Welcome to hell.

You can just pass a datetime64 object to pandas.Timestamp:

In [16]: Timestamp(numpy.datetime64('2012-05-01T01:00:00.000000'))
Out[16]: <Timestamp: 2012-05-01 01:00:00>

I noticed that this doesn't work right though in NumPy 1.6.1:

numpy.datetime64('2012-05-01T01:00:00.000000+0100')

Also, pandas.to_datetime can be used (this is off of the dev version, haven't checked v0.9.1):

In [24]: pandas.to_datetime('2012-05-01T01:00:00.000000+0100')
Out[24]: datetime.datetime(2012, 5, 1, 1, 0, tzinfo=tzoffset(None, 3600))

Solution 4

I think there could be a more consolidated effort in an answer to better explain the relationship between Python's datetime module, numpy's datetime64/timedelta64 and pandas' Timestamp/Timedelta objects.

The datetime standard library of Python

The datetime standard library has four main objects

• time - only time, measured in hours, minutes, seconds and microseconds
• date - only year, month and day
• datetime - All components of time and date
• timedelta - An amount of time with maximum unit of days

Create these four objects

>>> import datetime
>>> datetime.time(hour=4, minute=3, second=10, microsecond=7199)
datetime.time(4, 3, 10, 7199)

>>> datetime.date(year=2017, month=10, day=24)
datetime.date(2017, 10, 24)

>>> datetime.datetime(year=2017, month=10, day=24, hour=4, minute=3, second=10, microsecond=7199)
datetime.datetime(2017, 10, 24, 4, 3, 10, 7199)

>>> datetime.timedelta(days=3, minutes = 55)
datetime.timedelta(3, 3300)

>>> # add timedelta to datetime
>>> datetime.timedelta(days=3, minutes = 55) + \
    datetime.datetime(year=2017, month=10, day=24, hour=4, minute=3, second=10, microsecond=7199)
datetime.datetime(2017, 10, 27, 4, 58, 10, 7199)

NumPy's datetime64 and timedelta64 objects

NumPy has no separate date and time objects, just a single datetime64 object to represent a single moment in time. The datetime module's datetime object has microsecond precision (one-millionth of a second). NumPy's datetime64 object allows you to set its precision from hours all the way to attoseconds (10 ^ -18). It's constructor is more flexible and can take a variety of inputs.

Construct NumPy's datetime64 and timedelta64 objects

Pass an integer with a string for the units. See all units here. It gets converted to that many units after the UNIX epoch: Jan 1, 1970

>>> np.datetime64(5, 'ns') 
numpy.datetime64('1970-01-01T00:00:00.000000005')

>>> np.datetime64(1508887504, 's')
numpy.datetime64('2017-10-24T23:25:04')

You can also use strings as long as they are in ISO 8601 format.

>>> np.datetime64('2017-10-24')
numpy.datetime64('2017-10-24')

Timedeltas have a single unit

>>> np.timedelta64(5, 'D') # 5 days
>>> np.timedelta64(10, 'h') 10 hours

Can also create them by subtracting two datetime64 objects

>>> np.datetime64('2017-10-24T05:30:45.67') - np.datetime64('2017-10-22T12:35:40.123')
numpy.timedelta64(147305547,'ms')

Pandas Timestamp and Timedelta build much more functionality on top of NumPy

A pandas Timestamp is a moment in time very similar to a datetime but with much more functionality. You can construct them with either pd.Timestamp or pd.to_datetime.

>>> pd.Timestamp(1239.1238934) #defaults to nanoseconds
Timestamp('1970-01-01 00:00:00.000001239')

>>> pd.Timestamp(1239.1238934, unit='D') # change units
Timestamp('1973-05-24 02:58:24.355200')

>>> pd.Timestamp('2017-10-24 05') # partial strings work
Timestamp('2017-10-24 05:00:00')

pd.to_datetime works very similarly (with a few more options) and can convert a list of strings into Timestamps.

>>> pd.to_datetime('2017-10-24 05')
Timestamp('2017-10-24 05:00:00')

>>> pd.to_datetime(['2017-1-1', '2017-1-2'])
DatetimeIndex(['2017-01-01', '2017-01-02'], dtype='datetime64[ns]', freq=None)

Converting Python datetime to datetime64 and Timestamp

>>> dt = datetime.datetime(year=2017, month=10, day=24, hour=4, 
                   minute=3, second=10, microsecond=7199)
>>> np.datetime64(dt)
numpy.datetime64('2017-10-24T04:03:10.007199')

>>> pd.Timestamp(dt) # or pd.to_datetime(dt)
Timestamp('2017-10-24 04:03:10.007199')

Converting numpy datetime64 to datetime and Timestamp

>>> dt64 = np.datetime64('2017-10-24 05:34:20.123456')
>>> unix_epoch = np.datetime64(0, 's')
>>> one_second = np.timedelta64(1, 's')
>>> seconds_since_epoch = (dt64 - unix_epoch) / one_second
>>> seconds_since_epoch
1508823260.123456

>>> datetime.datetime.utcfromtimestamp(seconds_since_epoch)
>>> datetime.datetime(2017, 10, 24, 5, 34, 20, 123456)

Convert to Timestamp

>>> pd.Timestamp(dt64)
Timestamp('2017-10-24 05:34:20.123456')

Convert from Timestamp to datetime and datetime64

This is quite easy as pandas timestamps are very powerful

>>> ts = pd.Timestamp('2017-10-24 04:24:33.654321')

>>> ts.to_pydatetime()   # Python's datetime
datetime.datetime(2017, 10, 24, 4, 24, 33, 654321)

>>> ts.to_datetime64()
numpy.datetime64('2017-10-24T04:24:33.654321000')

Solution 5

>>> dt64.tolist()
datetime.datetime(2012, 5, 1, 0, 0)

For DatetimeIndex, the tolist returns a list of datetime objects. For a single datetime64 object it returns a single datetime object.

View more solutions

Comments

[seberg]

@hayden if you know that its a scalar/0-d array I would rather use .item() which is far more explicit (and nobody can come around and start arguing that it should return a list).

[Andy Hayden]

I'm afraid this doesn't seem to always work: e.g. dt64 = numpy.datetime64('2002-06-28T01:00:00.000000000+0100'), which gives a long (1025222400000000000L) (!)

[Andy Hayden]

I'm afraid this doesn't seem to always work: e.g. dt64 = numpy.datetime64('2002-06-28T01:00:00.000000000+0100'), which gives a long (1025222400000000000L) (!)

[jfs]

@hayden: try type(dt64). dt64.astype(datetime) == datetime.utcfromtimestamp(dt64.astype(int)*1e-6)

[Andy Hayden]

@JFSebastian type(dt64) is numpy.datetime64 and dt64.astype(datetime) is the same long int... :s

[jfs]

@hayden: What is your numpy version? Mine: numpy.__version__ -> '1.6.1'

[Andy Hayden]

Version 1.8.0 (in python 2.7.3), if it works for you it does suggest it is a bug on my system!

[jfs]

@hayden: the type that is returned by .item() (suggested by @seberg), .tolist() depends on what units datetime64 uses e.g., D produces datetime.date(), us (microseconds) produce datetime.datetime(), ns (nanoseconds) produce long. And the units change depending on input values e.g., numpy.datetime64('2012-05-01') uses 'D', numpy.datetime64('2012-05-01T00:00:00.000') uses ms, numpy.datetime64('2012-05-01T00:00:00.000000000') uses ns. You could open an issue if you find it confusing.

[ely]

None of these examples work for me. If I try datetime.datetime.utcfromtimestamp, I see two errors: (1) TypeError: don't know how to convert scalar number to float when I try passing in the unaltered np.datetime64 and then (2) ValueError: year is out of range when I try first using .astype(int). The test date is for 2012: xx = numpy.datetime64(datetime.date(2012,1,31)) so the year is perfectly valid. I am using NumPy 1.6.1.

[jfs]

@EMS: run this (on success you should see nothing). If you see any errors; provide a link to the code that can be run.

[Amelio Vazquez-Reina]

Thanks Andy for sharing this tip. For some reason I am unable to make it work, as I discuss here: stackoverflow.com/questions/22825349/…

[Andy Hayden]

@user815423426 this was never a very robust solution, I guess you can pass a format to the datetime constructor to work more generally. Not very pandastic though!

[jfs]

You should mention that issubclass(pd.Timestamp, datetime) is True. And Timestamp class itself has to_datetime() method.

[Anton Protopopov]

pd.to_datetime('2012-05-01T01:00:00.000000+0100') returns Timestamp('2012-05-01 00:00:00') at least in pandas 0.17.1.

[wlad]

I got an error saying replace() got an unexpected keyword argument 'tzinfo'

[Crystal]

which pandas version do you use?I have Version: 0.18.1 (pip show pandas)

[wlad]

same as you. . .

[Crystal]

I dont know then but it works for me like charm. pix.toile-libre.org/upload/original/1475645621.png

[demented hedgehog]

Just looking at this diagram tells me there's something fundamentally wrong with all this time stuff.

[Mr.WorshipMe]

It's very confusing that pd.to_datetime would produce a TimeStamp if given the number of ms or ns, but would produce a datetime.datetime if given a datetime.datetime or a np.datetime64 if given a np.datetime64... Why would anyone think this is reasonable?

[zthomas.nc]

For those skimming: pd.Timestamp() changes np.datetime to pd.Timestamp

[hpaulj]

docs.scipy.org/doc/numpy/reference/… for changes in timezone handling.

[Ted Petrou]

@Mr.WorshipMe This diagram needs to be updated. pd.to_datetime converts everything to pd.Timestamp. A pd.Timestamp object has the method to_pydatetime to revert back to a datetime.datetime object and a to_datetime64 method to convert to np.datetime64.

[Mr.WorshipMe]

@TedPetrou This is much more reasonable - Do you know since what version this is the case?

[Ted Petrou]

You could just do ts.to_pydatetime()

[Andy Hayden]

It's crazy how numpy to datetime is still hard/hacky... is there really no better way? This is a good answer, I am thinking about accepting to move it to the top-level I have to read the others more deeply once by a computer.

[Ted Petrou]

What's so quirky about it? Pandas Timestamps work well and are fairly simple.

[Andy Hayden]

Numpy to datetime.

[user3226167]

How can I get a higher resolution of this pic?

[Dave X]

This chart needs the string -> mappings on it. Try these: x = pd.to_datetime('2012-05-01T01:00:00.000000+0100'); print(type(x)); print(type(x.to_datetime()); -- the first is a class 'pandas._libs.tslib.Timestamp' and the second is a class 'datetime.datetime'. (And you get a warning that to_datetime() is deprecated for to_pydatetime() ) (In Pandas 0.22.0 and Python 3.5.2)

[N M]

@AndyHayden You could also just add an extra argument, 'us' or 'ms' to ensure the same format is applied resulting in the same datetime element being produced in tolist()

[Sean McCarthy]

I think this is the best answer I've ever seen. Coming from Excel, VBA, SAS, or SQL, Python seems weird because there's not just "one way" to work with dates/times. As with many things in Python or R, it seems one must choose a favourite method/module/class and stick with it.

[SherylHohman]

Please edit to conform with proper: code formatting, quote formatting, and text formatting. Also, please adhere to proper capitalization, grammar, and check for typos, as per SO guidelines - see: How to Post, and Code Samples

[Santiago]

Yes, great answer. I finally understand this much better. Apparently there is also matplotlib.dates, why???

[cknoll]

@Quant it would be interesting to share the source file of this diagram, such that it could be updated as suggested by some other comments. With which program was it created?

[rubebop]

What is that function from np.datetime64[ns] to datetime.datetime (dotted arrow)? I find it weird.

[Xaser]

For an array of timestamps, a, you can do a.astype('datetime64[s]').

[Tom]

@TedPetrou thanks for the great answer, I think there is something that should be modified in your answer on datetime64 : from numpy.org/doc/stable/reference/arrays.datetime.html the unit can be much higher than hours, it can be up to years

[starriet]

After fixing a bug with time stuff, I feel like the theory of relativity would be simpler than these things. They ended up being so error-prone.

[Greg Graham]

"Just looking at this diagram tells me there's something fundamentally wrong with all this time stuff." And it's not just Python/Pandas. Dates and times are used both as categories and as measures depending on the scenario. Implementations that try to cover all the use cases invariably come out with a complexity that would make Mandlebrot wince.

[John Mark]

The simple way to convert a datetime64 column to a Timestamp is my_datetime64_column.apply(lambda x: pd.Timestamp(x))