How do I save a 3D Python/NumPy array as a text file?

Question

I have to launch a great number of calculations, and I have to save a 2D file text each time, so I would like to store results in "real-time" as a 3D text file with each slice corresponding to one calculation result.

The first calculation is OK, but when I do the second calculation, during the "np.loadtxt" step, the array dimensions become 2D... So I can't reach my aim... and I can't do a reshape when I begin to dimensions (... , ... , 1)

#MY FIRST RESULTS
test1 = open("C:/test.txt", "r")
test_r = np.genfromtxt(test, skip_header=1)
test_r = np.expand_dims(test_r, axis=2) #I create a new axis to save in 3D
test1.close()

#I test if the "Store" file to keep all my results is created.
try:
    Store= np.loadtxt('C:/Store.txt')
except:
    test=1

#If "Store" is not created, I do it or I concatenate in my file.
if test ==1:
    Store= test_r
    np.savetxt('C:/Store.txt', Store)
    test=2
else:
    Store = np.concatenate((Store,test_r), axis=2)
    np.savetxt('C:/Store.txt', Store)


#MY SECOND RESULTS
test2 = open("C:/test.txt", "r")
test_r = np.genfromtxt(test, skip_header=1)
test_r = np.expand_dims(test_r, axis=2)
test2.close()

#I launch the procedure again to "save" the results BUT DURING THE LOADTXT STEP THE ARRAY DIMENSIONS CHANGE TO BECOME A 2D ARRAY...
try:
    Store= np.loadtxt('C:/Store.txt')
except:
    test=1


if test ==1:
    Store= test_r
    np.savetxt('C:/Store.txt', Store)
    test=2
else:
    Store = np.concatenate((Store,test_r), axis=2)
    np.savetxt('C:/Store.txt', Store)

Solution 1

Here's an example of cPickle:

import cPickle

# write to cPickle
cPickle.dump( thing_to_save, open( "filename.pkl", "wb" ) )

# read from cPickle
thing_to_save = cPickle.load( open( "filename.pkl", "rb" ) )

The "wb" and "rb" parameters to the open() function are important. CPickle writes objects in a binary format, so using just "w" and "r" won't work.

Solution 2

If the save file needs to be a 'csv' style text, you could use multiple savetxt and loadtxt. The key is knowning that both of these can take an open file as input.

Writing example:

In [31]: A=np.arange(3*2*4).reshape(3,2,4)    
In [32]: A    # normal display as 3 blocks of 2d array
Out[32]: 
array([[[ 0,  1,  2,  3],
        [ 4,  5,  6,  7]],

       [[ 8,  9, 10, 11],
        [12, 13, 14, 15]],

       [[16, 17, 18, 19],
        [20, 21, 22, 23]]])

In [36]: for a in A:print a, '\n'   # or iterate on the 1st dimension
[[0 1 2 3]
 [4 5 6 7]] 

[[ 8  9 10 11]
 [12 13 14 15]] 

[[16 17 18 19]
 [20 21 22 23]] 

Following that example, I can iterate on the file, using savetxt for each subarray:

In [37]: with open('3dcsv.txt','wb') as f:
    for a in A:
        np.savetxt(f, a, fmt='%10d')
        f.write('\n')
   ....:         

Confirm the file write with a system cat (via ipython):

In [38]: cat 3dcsv.txt
         0          1          2          3
         4          5          6          7

         8          9         10         11
        12         13         14         15

        16         17         18         19
        20         21         22         23

For simple read, loadtxt apparently ignores the empty lines, returning a 6 x 4 array. So I know that it is supposed to be (2,3,4) I can easily reshape the result.

In [39]: np.loadtxt('3dcsv.txt')
Out[39]: 
array([[  0.,   1.,   2.,   3.],
       [  4.,   5.,   6.,   7.],
       [  8.,   9.,  10.,  11.],
       [ 12.,  13.,  14.,  15.],
       [ 16.,  17.,  18.,  19.],
       [ 20.,  21.,  22.,  23.]])

After a bit of debugging I got this multiple loadtxt to work. loadtxt (and genfromtxt) works with a list of lines.

In [53]: A1=[]     # list to collect blocks

In [54]: with open('3dcsv.txt') as f:
    lines = []     # list to collect lines
    while 1:
        aline = f.readline()
        if aline.strip():
            lines.append(aline)     # nonempty line
        else:              # empty line
            if len(lines)==0: break
            A1.append(np.loadtxt(lines, dtype=int))
            lines = []
   ....:             

In [55]: A1 = np.array(A1)

In [56]: A1
Out[56]: 
array([[[ 0,  1,  2,  3],
        [ 4,  5,  6,  7]],

       [[ 8,  9, 10, 11],
        [12, 13, 14, 15]],

       [[16, 17, 18, 19],
        [20, 21, 22, 23]]])

This may not be the most robust pairing of of save/load but it gives a framework for building something better.

But if it doesn't need to be text then pickle is fine, as is the native numpy 'save/load'

In [57]: np.save('3dsave.npy',A)

In [58]: np.load('3dsave.npy')
Out[58]: 
array([[[ 0,  1,  2,  3],
        [ 4,  5,  6,  7]],

       [[ 8,  9, 10, 11],
        [12, 13, 14, 15]],

       [[16, 17, 18, 19],
        [20, 21, 22, 23]]])

Comments

[Thomas Baruchel]

Maybe you could rather be interested by using the Pickle module which can easely save/load any Python object?

[chill_turner]

Depending on your use case you may be able to get away with something similar to what I did recently at work. Take your numpy array, convert to normal python list and stuff that into into a JSON file. That JSON is highly portable and you can read your arrays from there. As Baruchel mentioned there are ways of storing your numpy data in binary form (such as pickle). Numpy has built in functions for this too (first two modules on top of list docs.scipy.org/doc/numpy-1.10.0/reference/routines.io.html)