Updating h5py Datasets

Question

Does any one have an idea for updating hdf5 datasets from h5py? Assuming we create a dataset like:

import h5py
import numpy
f = h5py.File('myfile.hdf5')
dset = f.create_dataset('mydataset', data=numpy.ones((2,2),"=i4"))
new_dset_value=numpy.zeros((3,3),"=i4")

Is it possible to extend the dset to a 3x3 numpy array?

Solution

You need to create the dataset with the "extendable" property. It's not possible to change this after the initial creation of the dataset. To do this, you need to use the "maxshape" keyword. A value of None in the maxshape tuple means that that dimension can be of unlimited size. So, if f is an HDF5 file:

dset = f.create_dataset('mydataset', (2,2), maxshape=(None,3))

creates a dataset of size (2,2), which may be extended indefinitely along the first dimension and to 3 along the second. Now, you can extend the dataset with resize:

dset.resize((3,3))
dset[:,:] = np.zeros((3,3),"=i4")

The first dimension can be increased as much as you like:

dset.resize((10,3))

Comments

[Joseph Sheedy]

resize also accepts an axis argument so you only have to specify the new size for the axis you're extending, rather than all of them: dset.resize(10, axis=0)

[Guillem Cucurull]

That's very useful @velotron, it could be added to the accepted answer because for multidimensional arrays it is easier to do so.

[user798719]

does this example overwrite what was previously in the file? or does it just append the new contents?

[Aquiles Carattino]

You 'append' to the current data while you increase or decrease the size. The indexes of the data do not change when you resize a dataset. docs.h5py.org/en/latest/faq.html

[Arsham Arya]

As I was searching for it, If you want to resize a 1-dimentional array, you have to pass (None, ) to maxshape, don't forget to write , to make it a tuple. Also do it in resize method or use axie=0 as Joseph mentioned.