Saving numpy array to csv produces TypeError Mismatch

Question

I have a numpy array with numeric data of the form:

example = numpy.array([[[i for i in range(0, 5)],[0 for j in range(0, 5)]] for k in range(0, 10)])

So it's array of 10 groups, where each group consists of 2 lists of equal length and contains only numbers. Running the following save code gives me the error below:

numpy.savetxt('exampleData.csv', test, delimiter=',')
TypeError: Mismatch between array dtype ('int32') and format specifier ('%.18e %.18e')

I'm guessing this could be fixed with something in the fmt='xyz' argument, but the documentation isn't particularly clear. Any help would be appreciated.

(In my actual data, the i and j lists are lists of long floats, e.g.'0.0047322940571' etc.)

Solution

Your example is a 3d array

In [82]: example=np.array([[[i for i in range(0, 5)],[0 for j in range(0, 5)]] for  k in range(0, 3)])  # chg 10 to 3 for display

In [83]: example.shape
Out[83]: (3L, 2L, 5L)

In [84]: example
Out[84]: 
array([[[0, 1, 2, 3, 4],
        [0, 0, 0, 0, 0]],

       [[0, 1, 2, 3, 4],
        [0, 0, 0, 0, 0]],

       [[0, 1, 2, 3, 4],
        [0, 0, 0, 0, 0]]])

trying to save the whole thing results in an error (different message due to different version):

In [87]: np.savetxt('test.csv',example, delimiter=',')
....
TypeError: float argument required, not numpy.ndarray 

but saving one 'row' is ok

In [88]: np.savetxt('test.csv',example[1,...], delimiter=',')

Save with integer format makes a prettier output

In [94]: np.savetxt('test.csv',example[1,...], delimiter=',',fmt='%d')

In [95]: with open('test.csv') as f:print f.read()
0,1,2,3,4
0,0,0,0,0

So how do you want the 3d array to be saved? Keep in mind how you will use it/read it. Multiple files? Multiple blocks within one file?

https://stackoverflow.com/a/3685339/901925 is a 6 yr old SO answer on how to save a 3d array. The simple answer is to open a file, and perform multiple savetxt for slices of the array. This saves the data in blocks. But loading those blocks is another SO question (which has come up before).

In [100]: with open('test.csv','w') as f:
     ...:     for row in example:
     ...:         np.savetxt(f,row,delimiter=',',fmt='%d',footer='====')
     ...:         

In [101]: with open('test.csv') as f:print f.read()
0,1,2,3,4
0,0,0,0,0
# ====
0,1,2,3,4
0,0,0,0,0
# ====
0,1,2,3,4
0,0,0,0,0
# ====

In response to your comment, this works

example=np.ones((4,2,100))
np.savetxt('test.csv',example[1,...], delimiter=',',fmt='%.18e')

Another way to save a 3d array is to reshape it to 2d. You reshape it back to 3d after loading, possibly using information that you stored in a comment line

np.savetxt('test.csv',example.reshape(-1,example.shape[-1]), delimiter=',',fmt='%.18e')

Comments

[Pingk]

I tried fmt='%1.10E, %1.10E, &04d' for i, j and k respectively but I get a SyntaxError. The problem seems to be iterating through the k list?

[Pingk]

Thanks, but now I get a different error using your In[100] line and 'fmt=%.18e'. My actual array has the shape (4L, 2L, 100L), and I get the error TypeError: Mismatch between array dtype ('float64') and format specifier ('%.18e %.18e...[x100])

[hpaulj]

savetxt iterates on the 1st dimension of your array, and for each row, writes format % tuple(row). format is derived from your fmt parameter and the .shape[1] of your input.