string to datetime with fractional seconds, on Google App Engine

Question

I need to convert a string to a datetime object, along with the fractional seconds. I'm running into various problems.

Normally, i would do:

>>> datetime.datetime.strptime(val, "%Y-%m-%dT%H:%M:%S.%f")

But errors and old docs showed me that python2.5's strptime does not have %f...

Investigating further, it seems that the App Engine's data store does not like fractional seconds. Upon editing a datastore entity, trying to add .5 to the datetime field gave me the following error:

ValueError: unconverted data remains: .5

I doubt that fractional seconds are not supported... so this is just on the datastore viewer, right?

Has anyone circumvented this issue? I want to use the native datetime objects... I rather not store UNIX timestamps...

Thanks!

---

EDIT: Thanks to Jacob Oscarson for the .replace(...) tip!

One thing to keep in mind is to check the length of nofrag before feeding it in. Different sources use different precision for seconds.

Here's a quick function for those looking for something similar:

def strptime(val):
    if '.' not in val:
        return datetime.datetime.strptime(val, "%Y-%m-%dT%H:%M:%S")

    nofrag, frag = val.split(".")
    date = datetime.datetime.strptime(nofrag, "%Y-%m-%dT%H:%M:%S")

    frag = frag[:6]  # truncate to microseconds
    frag += (6 - len(frag)) * '0'  # add 0s
    return date.replace(microsecond=int(frag))

Solution 1

Parsing

Without the %f format support for datetime.datetime.strptime() you can still sufficiently easy enter it into a datetime.datetime object (randomly picking a value for your val here) using datetime.datetime.replace()), tested on 2.5.5:

>>> val = '2010-08-06T10:00:14.143896'
>>> nofrag, frag = val.split('.')
>>> nofrag_dt = datetime.datetime.strptime(nofrag, "%Y-%m-%dT%H:%M:%S")
>>> dt = nofrag_dt.replace(microsecond=int(frag))
>>> dt
datetime.datetime(2010, 8, 6, 10, 0, 14, 143896)

Now you have your datetime.datetime object.

Storing

Reading further into http://code.google.com/appengine/docs/python/datastore/typesandpropertyclasses.html#datetime

I can see no mentioning that fractions isn't supported, so yes, it's probably only the datastore viewer. The docs points directly to Python 2.5.2's module docs for datetime, and it does support fractions, just not the %f parsing directive for strptime. Querying for fractions might be trickier, though..

Solution 2

All ancient history by now, but in these modern times you can also conveniently use dateutil

from dateutil import parser as DUp

funky_time_str = "1/1/2011 12:51:00.0123 AM"

foo = DUp.parse(funky_time_str)

print foo.timetuple()
# time.struct_time(tm_year=2011, tm_mon=1, tm_mday=1, tm_hour=0, tm_min=51, tm_sec=0, tm_wday=5, tm_yday=1, tm_isdst=-1)

print foo.microsecond
# 12300

print foo
# 2011-01-01 00:51:00.012300

dateutil supports a surprising variety of possible input formats, which it parses without pattern strings.

Comments

[jbenet]

Thanks for the Answer! Actually, querying with fractions seems to work fine. I did a couple of quick tests and it definitely tells the difference between milliseconds.

[jfs]

"in these modern times", datetime.strptime(val, "%Y-%m-%dT%H:%M:%S.%f") works as is.

[Tim Richardson]

alas, dateutil parsing is S... L...O...W