Python struct.error: unpack requires a string argument of length 2

Question

I have written some data using C++ in byte format. I am now trying to read that data again using Python, but I run into an error;

Traceback (most recent call last):
  File "binary-reader.py", line 61, in <module>
    interaction_types.append(struct.unpack('<H',fp.read(2))[0]);
struct.error: unpack requires a string argument of length 2

I don't really understand since it looks like I am giving a string of length 2, right? Furthermore, I do the same thing at line 32

There is another question like mine but it is without an answer is targeted for Python 3.

Here is my code

import sys
import struct
import os

print "Arguments : "
print str(sys.argv)

#N = #isects

#  2      2        3*4      2          3*4*N          4N            4N      3*4N        2N          2N
#imageX,imageY,throughput,#isects,isect_positions,primitive_ids,shape_ids,spectra,interaction_types,light_ids
file_path = str(sys.argv[1]);

byte_count = 0;
line_number = 1;

fp = open(file_path, "rb");
output = open('output.txt',"w");
file_size = os.path.getsize(file_path)

print "(input) file size = " + str(file_size);

while byte_count < file_size:
    print "Line number = " + str(line_number)
    print "Current byte count = " + str(byte_count)
    # Do stuff with byte.
    x = struct.unpack('<H', fp.read(2))[0]
    y = struct.unpack('<H', fp.read(2))[0]
    throughputOne = struct.unpack('<f', fp.read(4))[0]
    throughputTwo = struct.unpack('<f', fp.read(4))[0]
    throughputThree = struct.unpack('<f', fp.read(4))[0]
    nrIsects = struct.unpack('<H',fp.read(2))[0]

    # print "x = " + str(x)
    # print "y = " + str(y)
    # print "throughputOne = " + str(throughputOne)
    # print "throughputTwo = " + str(throughputTwo)
    # print "throughputThree = " + str(throughputThree)
    print "nrIsects = " + str(nrIsects)

    isect_positions = []
    for i in range(nrIsects*3):
        value = struct.unpack('<f',fp.read(4))[0]
        isect_positions.append(value);

    primitive_ids = []
    for i in range(nrIsects):
        value = struct.unpack('<I',fp.read(4))[0]
        primitive_ids.append(value);

    shape_ids = []
    for i in range(nrIsects):
        shape_ids.append(struct.unpack('<I',fp.read(4))[0]);

    spectra = []
    for i in range(nrIsects*3):
        spectra.append(struct.unpack('<f',fp.read(4))[0]);

    interaction_types = []
    for i in range(nrIsects):
        interaction_types.append(struct.unpack('<H',fp.read(2))[0]);

    light_ids = []
    for i in range(nrIsects):
        light_ids.append(struct.unpack('<H',fp.read(2))[0]);

    output_vars = [x,y,throughputOne,throughputTwo,throughputThree,nrIsects]

    line_string = ""

    for i in range(len(output_vars)):
        output.write(str(output_vars[i]))
        line_string += str(output_vars[i])
        if i is not len(output_vars) - 1:
            output.write(',')   
            line_string += ','

    print line_string


    #Update counters
    byte_count += 18 + 36*nrIsects
    line_number+=1

    # raw_input('Press any key to continue.');

    # print byte

And here is a link to a input file to use. You can run the code by passing a commandline argument specifying the path of the binary file. I have also written the code in ASCII, which reads

0,0,[0.127076,0.127076,0.127076],1,{[0.144978,-0.294863,2.991749]},{3917},{3916},{[1.375603,1.375603,1.375603]},{5},{0}

https://www.dropbox.com/s/tu1anqo5k0ygtd6/writetest.bin

EDIT: The layout of my file can be found as a comment in the code

Solution

50 bytes have already been read before the fp.read(2) that raises the error. Thus, fp.read(2) returns an empty string, and struct.unpack raises an exception:

In [83]: 2+2+4+4+4+2+12+4+4+12
Out[83]: 50

---

x = struct.unpack('<H', fp.read(2))[0]                  # 2 bytes read
y = struct.unpack('<H', fp.read(2))[0]                  # 2 bytes
throughputOne = struct.unpack('<f', fp.read(4))[0]      # 4 bytes
throughputTwo = struct.unpack('<f', fp.read(4))[0]      # 4 bytes
throughputThree = struct.unpack('<f', fp.read(4))[0]    # 4 bytes
nrIsects = struct.unpack('<H',fp.read(2))[0]            # 2 bytes

print "nrIsects = " + str(nrIsects)

isect_positions = []
for i in range(nrIsects*3):
    value = struct.unpack('<f',fp.read(4))[0]           # 12 bytes
    isect_positions.append(value)

primitive_ids = []
for i in range(nrIsects):
    value = struct.unpack('<I',fp.read(4))[0]           # 4 bytes
    primitive_ids.append(value)

shape_ids = []
for i in range(nrIsects):
    shape_ids.append(struct.unpack('<I',fp.read(4))[0]) # 4 bytes

spectra = []
for i in range(nrIsects*3):
    spectra.append(struct.unpack('<f',fp.read(4))[0])   # 12 bytes

interaction_types = []
for i in range(nrIsects):
    interaction_types.append(struct.unpack('<H', fp.read(2))[0])   # error!

---

Comments

[Gerard]

You are absolutely right! I feel so dumb, I did not count the bytes correctly, and apparently I forgot to write out those last bytes in the C++ program. I had also expected to get a sort of error when trying to read more bytes than available but I guess the whole binary reading is too low level to notice any of this. Thanks again for your help!