Write and read Datetime to binary format in Python

Question

I want to store a list of datetimes in a binary file in Python.

EDIT: by "binary" I mean the best digital representation for each datatype. The application for this is to save GPS trackpoints composed by (unix-timestamp, latitude, longitude, elevation), so the whole structure is little-endian "Long, float, float, float", with four bytes to each value.

NOTE: I don't use "unix-timestamp" due to any affection to the Unix platform, but only as an unequivocal way to represent the value of a datetime.

Currently, I am doing like the code below, but besides some timezone confusion that I'm still working out (my timezone is -3), I believe converting to int and back might not be the right way, since datetime and datetime64 are native types in python/numpy, if I'm not mistaken. Thus, a datetime64 would need eight bytes instead of the four I am using for the (long)unix-timestamp.

import datetime
import calendar
import struct

now = datetime.datetime.now()
print now

stamp = calendar.timegm(now.utctimetuple())
print stamp

binarydatetime = struct.pack('<L', stamp)
recoverstamp = struct.unpack('<L', binarydatetime)[0]
print recoverstamp

recovernow = datetime.datetime.fromtimestamp(recoverstamp)
print recovernow

So the main question is: "is this the pythonic way to converting naive datetime to binary and back?"

And the aditional question is: "if everything in this code is supposed to be naive, why do I have a timezone offset?"

Thanks for reading!

Solution 1

I have found a way using the Unix timestamp and storing it as an integer. This works for me because I don't need a subsecond resolution, but I think long integers would allow for microsecond resolution with some modifications of the code.

The changes from my original consist in replacing calendar.timegm by time.mktime and also utctimetuple by timetuple, to keep everything naive.

This:

import datetime
import struct
import time

now = datetime.datetime.now()
print now

stamp = time.mktime(now.timetuple())
print stamp

recoverstamp = datetime.datetime.fromtimestamp(stamp)
print recoverstamp

binarydatetime = struct.pack('<L', stamp)
recoverbinstamp = struct.unpack('<L', binarydatetime)[0]
print recoverbinstamp

recovernow = datetime.datetime.fromtimestamp(recoverbinstamp)
print recovernow

Returns this:

2013-09-02 11:06:28.064000
1378130788.0
2013-09-02 11:06:28
1378130788
2013-09-02 11:06:28

From this, I can easily write the packed binarydatetime to file, and read it back later.

Solution 2

By far the most simple solution is to use Temporenc: http://temporenc.readthedocs.org/

It takes care of all the encoding/decoding and allows you to write a Python datetime object to a file:

now = datetime.datetime.now()
temporenc.pack(fp, now)

To read it back, this suffices:

dt = temporenc.unpack(fp)
print(dt)

Comments

[heltonbiker]

Actually I want a language- and platform- independent way to store the datetime data type. See my edit.

[Wjars]

Ah, I had the thought pickle won't do. See, I cannot figure out another way to do this, so for now it's surely good enough. ''Pythonic'' is a very abstract notion ( as is PEP 20 ). I only

[Wjars]

You can use str.encode('encodage') to have hexadecimal, utf-8 or whatever representation of your string, but it's a bit off topic. (the 5 minutes edit session is way too short) Cannot help you.

[heltonbiker]

Actually I am not storing a string representing a date and time, but actually a "native" datetime.datetime object: docs.python.org/2/library/datetime.html#datetime.datetime

[le_lemon]

For me microseconds are important what do you mean by using long integers ?

[heltonbiker]

@le_lemon the unix timestamp is too large to be stored on a 32 bit signed integer, so in order to properly store it with microsecond resolution you would need a larger number format, such as uint, long, or even better ulong (unsigned 64 bit integer). Please check all this, perhaps what I just said is not totally correct, but these are the principles.

[Andrey]

@heltonbiker looks like stamp needs to be converted to integer before packing: binarydatetime = struct.pack('<L', int(stamp))