How to specify metadata for dask.dataframe

Question

The docs provide good examples, how metadata can be provided. However I still feel unsure, when it comes to picking the right dtypes for my dataframe.

• Could I do something like meta={'x': int 'y': float, 'z': float} instead of meta={'x': 'i8', 'y': 'f8', 'z': 'f8'}?
• Could somebody hint me to a list of possible values like 'i8'? What dtypes exist?
• How can I specify a column, that contains arbitrary objects? How can I specify a column, that contains only instances of one class?

Solution 1

The available basic data types are the ones which are offered through numpy. Have a look at the documentation for a list.

Not included in this set are datetime-formats (e.g. datetime64), for which additional information can be found in the pandas and numpy documentation.

The meta-argument for dask dataframes usually expects an empty pandas dataframe holding definitions for columns, indices and dtypes.

One way to construct such a DataFrame is:

import pandas as pd
import numpy as np
meta = pd.DataFrame(columns=['a', 'b', 'c'])
meta.a = meta.a.astype(np.int64)
meta.b = meta.b.astype(np.datetime64)

There is also a way to provide a dtype to the constructor of the pandas dataframe, however, I am not sure how to provide them for individual columns each. As you can see, it is possible to provide not only the "name" for datatypes, but also the actual numpy dtype.

Regarding your last question, the datatype you are looking for is "object". For example:

import pandas as pd

class Foo:
    def __init__(self, foo):
        self.bar = foo

df = pd.DataFrame(data=[Foo(1), Foo(2)], columns=['a'], dtype='object')
df.a
# 0    <__main__.Foo object at 0x00000000058AC550>
# 1    <__main__.Foo object at 0x00000000058AC358>

Solution 2

Both Dask.dataframe and Pandas use NumPy dtypes. In particular, anything within that you can pass to np.dtype. This includes the following:

1. NumPy dtype objects, like np.float64
2. Python type objects, like float
3. NumPy dtype strings, like 'f8'

Here is a more extensive list taken from the NumPy docs: http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html#specifying-and-constructing-data-types