pack/unpack functions for C++

Question

NOTE: I know that this has been asked many times before, but none of the questions have had a link to a concrete, portable, maintained library for this.

I need a C or C++ library that implements Python/Ruby/Perl like pack/unpack functions. Does such a library exist?

EDIT: Because the data I am sending is simple, I have decided to just use memcpy, pointers, and the hton* functions. Do I need to manipulate a char in any way to send it over the network in a platform agnostic manner? (the char is only used as a byte, not as a character).

Solution 1

Why not boost serialization or protocol buffers?

Solution 2

In C/C++ usually you would just write a struct with the various members in the correct order (correct packing may require compiler-specific pragmas) and dump/read it to/from file with a raw fwrite/fread (or read/write when dealing with C++ streams). Actually, pack and unpack were born to read stuff generated with this method.

If you instead need the result in a buffer instead of a file it's even easier, just copy the structure to your buffer with a memcpy.

If the representation must be portable, your main concerns are is byte ordering and fields packing; the first problem can be solved with the various hton* functions, while the second one with compiler-specific directives.

In particular, many compilers support the #pragma pack directive (see here for VC++, here for gcc), that allows you to manage the (unwanted) padding that the compiler may insert in the struct to have its fields aligned on convenient boundaries.

Keep in mind, however, that on some architectures it's not allowed to access fields of particular types if they are not aligned on their natural boundaries, so in these cases you would probably need to do some manual memcpys to copy the raw bytes to variables that are properly aligned.

Solution 3

Yes: Use std::copy from <algorithm> to operate on the byte representation of a variable. Every variable T x; can be accessed as a byte array via char * p = reinterpret_cast<char*>(&x); and p can be treated like a pointer to the first element of a an array char[sizeof(T)]. For example:

char buf[100];
double q = get_value();

char const * const p = reinterpret_cast<char const *>(&q);
std::copy(p, p + sizeof(double), buf);

// more stuff like that

some_stream.write(buf) //... etc.

And to go back:

double r;

std::copy(data, data + sizeof(double), reinterpret_cast<char *>(&r));

In short, you don't need a dedicated pack/unpack in C++, because the language already allows you access to its variables' binary representation as a standard part of the language.

Comments

[user541686]

Could you explain what pack/unpack do in those languages?

[Linuxios]

I know. I want a pack/unpack for portable representation, not local byte manipulation.

[bames53]

You're guaranteed to get a valid object back only for types that are trivially copyable. I'm sure you already know this, but you didn't say it.

[Correa]

This is not portable at all. You are not considering endianess for starters, and of course it only works for POD (plain old data) types.

[Kerrek SB]

@Linux_iOS.rb.cpp.c.lisp.m.sh: Well, what's a "portable" way of storing floats? What I'm saying is that you can use the byte-access to implement any sort of serialization that you like. For example, you can read/write unsigned integers with the usual algebraic manipulations (buf[0] = n % 256; buf [1] = (n / 256) % 256;, etc. (Though you need unsigned chars in that case.) And yes, this is only for fundamental types (not even PODs).