Python: How to pack different types of data into a string buffer using struct.pack_into

Question

I'm trying to pack some unsigned int data into a string buffer created using ctypes.create_string_buffer.

Here is the following code segment, and a running example showing the error on codepad:

import struct
import ctypes
import binascii

buf = ctypes.create_string_buffer(16)
struct.pack_into("=I=I=I", buf, 0, 1, 2, 3)
print binascii.hexlify(buf)

This yields the following error:

...
struct.error: bad char in struct format

The documentation doesn't allude to whether you can pack data of different types if the underlying buffer is of a specific C type. In this case, trying to pack unsigned int data into a string buffer with an underlying c_char type. Anyone know of a solution to do this, or is there a specific way to create a buffer that can pack any type of data?

Solution 1

You're not supposed to prefix every output specifier with the '=' code. Just say it once:

struct.pack_into("=III", buf, 0, 1, 2, 3)

This yields:

01000000020000000300000000000000

Solution 2

Sorry for resurrecting old topic, but I get the point of "snap" - being hit by probably similar background habit.

"the first character of the format string can be used to indicate the byte order, size and alignment of the packed data" I agree. However:

• Python docs deliberately (?) omit even a single example of use of order formatters (All examples assume a native byte order, size, and alignment with a big-endian machine.)
• One may assume (as I and probably snap did), that "III" consists of three format strings and we may format every one of them at will. Hence =I=I=I. I shot myself in the foot after getting used to Ruby's array.pack, where one may freely change ordering along the expression (Ruby's equivalent is I_I_I_ in this case, as order selector comes after type).

Hence I guess it might be good to add a few lines to struct.pack/unpack docs, giving examples of order & padding use (meh, padding hit me even harder... I could live with native order, but padding ruined my protocol).

Solution 3

Standard operating procedure: Read the error message.

"bad char in struct format" means what it says.

Standard operating procedure: Check the docs. Here it says "the first [my emphasis] character of the format string can be used to indicate the byte order, size and alignment of the packed data" and goes on to list = as a possibility. The next section (Format Characters) lists many letters including I.

Conclusion: your format string should be "=III".

Note: The problem has nothing to do with the destination buffer at all, let alone its underlying C type:

>>> import struct
>>> struct.pack("=I=I=I", 1, 2, 3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
struct.error: bad char in struct format
>>> struct.pack("=III", 1, 2, 3)
'\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00'
>>> 

Comments

[snap]

Thanks. Somehow I missed that critical piece of reading in the module docco :).

[Jack]

If you have a NEW question, please ask it by clicking the Ask Question button. If you have sufficient reputation, you may upvote the question. Alternatively, "star" it as a favorite and you will be notified of any new answers.

[Pawel Kraszewski]

It is not a NEW question. After seeing this post I resolved my problem. It is just a note, that Python docs is unclear - for example for people from Ruby world - at this topic.

[Jack]

I understand, but Stackoverflow is not actually a forum with topics/threads, but rather a Q & A site...

[littlebroccoli]

It is fine to answer old questions with new and/or better information. Stack Overflow is not a forum, so you are not reviving an old "topic."

[Kyle Strand]

I actually don't think that's a very good error message. It doesn't tell you which character is "bad", and in fact the "bad" character is actually a permissible character (it just happens to only be valid at the beginning of the string).