Are packed structs portable?

Question

I have some code on a Cortex-M4 microcontroller and'd like to communicate with a PC using a binary protocol. Currently, I'm using packed structs using the GCC-specific packed attribute.

Here is a rough outline:

struct Sensor1Telemetry {
    int16_t temperature;
    uint32_t timestamp;
    uint16_t voltageMv;
    // etc...
} __attribute__((__packed__));

struct TelemetryPacket {
    Sensor1Telemetry tele1;
    Sensor2Telemetry tele2;
    // etc...
} __attribute__((__packed__));

My question is:

• Assuming that I use the exact same definition for the TelemetryPacket struct on the MCU and the client app, will the above code be portable accross multiple platforms? (I'm interested in x86 and x86_64, and need it to run on Windows, Linux and OS X.)
• Do other compilers support packed structs with the same memory layout? With what syntax?

EDIT:

• Yes, I know packed structs are non-standard, but they seem useful enough to consider using them.
• I'm interested in both C and C++, although I don't think GCC would handle them differently.
• These structs are not inherited and don't inherit anything.
• These structs only contain fixed-size integer fields, and other similar packed structs. (I've been burned by floats before...)

Solution 1

You should never use structs across compile domains, against memory (hardware registers, picking apart items read from a file or passing data between processors or the same processor different software (between an app and a kernel driver)). You are asking for trouble as the compiler has somewhat free will to choose alignment and then the user on top of that can make it worse by using modifiers.

No there is no reason to assume you can do this safely across platforms, even if you use the same gcc compiler version for example against different targets (different builds of the compiler as well as the target differences).

To reduce your odds of failure start with the largest items first (64 bit then 32 bit the 16 bit then lastly any 8 bit items) Ideally align on 32 minimum perhaps 64 which one would hope arm and x86 do, but that can always change as well as the default can be modified by whomever builds the compiler from sources.

Now if this is a job security thing, sure go ahead, you can do regular maintenance on this code, likely going to need a definition of each structure for each target (so one copy of the source code for the structure definition for ARM and another for x86, or will need this eventually if not immediately). And then every or every few product releases you get to be called in to do work on the code...Nice little maintenance time bombs that go off...

If you want to safely communicate between compile domains or processors the same or different architectures, use an array of some size, a stream of bytes a stream of halfwords or a stream of words. Significantly reduces your risk of failure and maintenance down the road. Do not use structures to pick apart those items that just restores the risk and failure.

The reason why folks seem to think this is okay because of using the same compiler or family against the same target or family (or compilers derived from other compilers choices), as you understand the rules of the language and where the implementation defined areas are you will eventually run across a difference, sometimes it takes decades in your career, sometimes it takes weeks...Its the "works on my machine" problem...

Solution 2

Considering the mentioned platforms, yes, packed structs are completely fine to use. x86 and x86_64 always supported unaligned access, and contrary to the common belief, unaligned access on these platforms has (almost) the same speed as aligned access for a long time (there's no such thing that unaligned access is much slower). The only drawback is that the access may not be atomic, but I don't think it matters in this case. And there is an agreement between compilers, packed structs will use the same layout.

GCC/clang supports packed structs with the syntax you mentioned. MSVC has #pragma pack, which can be used like this:

#pragma pack(push, 1)
struct Sensor1Telemetry {
    int16_t temperature;
    uint32_t timestamp;
    uint16_t voltageMv;
    // etc...
};
#pragma pack(pop)

Two issues can arise:

1. Endianness must be the same across platforms (your MCU must be using little-endian)
2. If you assign a pointer to a packed struct member, and you're on an architecture which doesn't support unaligned access (or use instructions which have alignment requirements, like movaps or ldrd), then you may get a crash using that pointer (gcc doesn't warn you about this, but clang does).

---

Here's the doc from GCC:

The packed attribute specifies that a variable or structure field should have the smallest possible alignment—one byte for a variable

So GCC guarantees that no padding will be used.

MSVC:

To pack a class is to place its members directly after each other in memory

So MSVC guarantees that no padding will be used.

The only "dangerous" area I've found, is the usage of bitfields. Then the layout may differ between GCC and MSVC. But, there's an option in GCC, which makes them compatible: -mms-bitfields

---

Tip: even, if this solution works now, and it is highly unlikely that it will stop working, I recommend you keep dependency of your code on this solution low.

Note: I've considered only GCC, clang and MSVC in this answer. There are compilers maybe, for which these things are not true.

Solution 3

If

• endianness is not an issue
• both compilers handle packing correctly
• the type definitions on both C implementations are accurate (Standard compliant).

then yes, "packed structures" are portable.

For my taste too many "if"s, do not do this. It's not worth the hassle to arise.

Solution 4

You could do that, or use a more reliable alternative.

For the hard core amongst the serialisation fanatics out there, there's CapnProto. This gives you a native structure to deal with, and undertakes to ensure that when it's transferred across a network and lightly worked on, it'll still make sense the other end. To call it a serialisation is nearly inaccurate; it aims to do a little as possible to the in-memmory representation of a structure. Might be amenable to porting to an M4

There's Google Protocol Buffers, that's binary. More bloaty, but pretty good. There's the accompanying nanopb (more suited to microcontrollers), but it doesn't do the whole of GPB (I don't think it does oneof). Many people use it successfully though.

Some of the C asn1 runtimes are small enough for use on micro controllers. I know this one fits on M0.

Solution 5

If you want something maximally portable, you can declare a buffer of uint8_t[TELEM1_SIZE] and memcpy() to and from offsets within it, performing endianness conversions such as htons() and htonl() (or little-endian equivalents such as the ones in glib). You could wrap this in a class with getter/setter methods in C++, or a struct with getter-setter functions in C.

View more solutions

Comments

[Tom Kuschel]

possibly find some similar answers at stackoverflow.com/questions/8568432/… ; +1 b/c of my interest too. I think, it is always a good idea to put largest type, here uint32t at the first place, smaller data types at the last place of the struct or union.

[xaxxon]

Is this a question about C or C++? They are not the same language. Please specify and remove the tag for the other in your question.

[Michaël Roy]

packed is always your best bet for any communication protocol.

[M.M]

one obvious possible problem is endianness of the integers

[M.M]

EABI v5 doesn't support passing packed structs by value either

[LyingOnTheSky]

@xaxxon just curiosity, would the answer differ if it's supposed to be answered to C++ or C?

[user14959355]

"Yes, I know packed structs are non-standard, but they seem useful enough to consider using them." I think people creating C standard do not care about networking. endian.h is also non standard.

[ad absurdum]

Shouldn't we be able to assume implementation correctness? Same (maybe to a lesser degree) for compiler correctness?

[alk]

@DavidBowling: "assume implementation correctness" As far as I remember at least "packing" isn't part of the C Standard, so well, what would be "correct" though.

[ad absurdum]

@alk-- I meant implementation correctness with respect to "the type definitions on both C implementations are accurate," and compiler correctness with respect to "compilers handle packing correctly", in the sense that if a compiler supports packing, it should get it right (but may not).

[chqrlie]

I would add a couple more ifs:1) if all members have a fixed size, no int, short, long, and of course no pointers. 2) no bit-fields.

[alk]

@chqrlie: Well yes, true. Still my answer refers to the particular case the OP is showing, which uses fixed sizes already.

[user694733]

Strict aliasing should't be an issue in this case because of exception to allow aliasing through character types (that is; view structure bytes through char pointer).

[chqrlie]

@alk: which uses fixed sizes already. The OP only shows 3 members, we can only guess what lies behind // etc...

[harold]

Unaligned accesses can even be atomic, if they do not cross a cache line boundary - but there is little control over that

[geza]

@harold: thanks, I've modified my answer a little bit.

[geza]

"because you will have to look at the byte representation of your memory". What do you mean by this?

[geza]

I kinda think that using these libraries to transfer some little data from a sensor is overkill.

[geza]

Actually, using packed structs work. If the endianness is the same, they work like a charm, there's no hassle at all.

[Andrew Henle]

@geza Actually, using packed structs work. If the endianness is the same, they work like a charm, there's no hassle at all. Is that because they've worked for you so far? Not having run into any issues yet does not mean they "work like a charm" in general.

[vpalmu]

@AndrewHenle: Always apply the compiler specific declaration to say pack this struct. If a certain compiler doesn't have one, don't use that one.

[Andrew Henle]

@Joshua You're also assuming whatever choice you make in compiler won't change its implementation in the future and that the deployment platform won't change. I'd use a more portable, stable way to pass the data that's not based on "do it this way with this compiler on just this platform". Because in general, it doesn't work. Fixed-width sizes such as int16_t are in fact optional and are not required by the C standard.

[geza]

@AndrewHenle: Yes. And I think that I can depend on it, it is very unlikely that it will ever change. It is a very simple concept, why would it change? Even if it is not standardized. Who cares? All major compilers have this option. In the viewpoint of standard, it is not portable. In the real world, it is. Besides, standards can change too. You usually don't fear that your code's behavior changes when a new standard comes out. But, sometimes standards change, and your code will behave differently. It is the same scenario.

[alk]

@geza: You name it. And that's exactly why I recommend to rule out such uncertainties by concept: Simply use another, a well defined approach: Use (hardware/implementation independent) serialisation. :-)

[geza]

People are different :) In this case, I happily go with packed structs. They worked in the last 20 years, and in my opinion they will work 20 years later. Some other people will choose some other (in this case, more complex) way, just because, there is a possibility that packed struct won't work in the future. Or maybe they have to port the program to a compiler, which doesn't support it. I've no problems with that, as I said, people are different, and choose different solutions for a problem :)

[geza]

@AndrewHenle: about int16_t, I've just asked this question: stackoverflow.com/questions/45119928/c11-c14-on-exotic-hardw‌​are, hopefully it won't get closed.

[bazza]

@geza, they're not if one wants that little sensor to reliably talk to software running on Windows, Linux and OSX, as the original poster asked, without having to do some boring, error prone structure layout verification work. Using a reputable library designed for the job helps avoid having to do all that work oneself. Look at all the comments (including your own) for alk's excellent post; endianness, layout, integer width; boring, pointless discussion in my opinion; it's a solved problem.

[geza]

if you use packed structs with intXX_t, then I'm absolutely sure that you don't have to do anything like "layout verification". It will work out of the box, on any of the platforms mentioned. Put a static_assert after the struct to assert that the sizeof is OK (to catch possible future errors), and that's all.

[bazza]

I echo alk's post. Too many if's for it to be worthwhile. Why not just use a lightweight serialisation library and never, ever have to worry about it ever again?

[Margaret Bloom]

I believe unaligned accesses still make a difference: the DIMMs have a 64-bit data channel, doing some math on the number of signals in the datasheets it seems that memory is addressed in multiple of 64-bit. So reading a DWORD at address 3 reads 64-bit from address 0 (a single access) but reading a DWORD at address 6 reads 64-bit from address 0 and 64-bit from address 8 (double access). All this without considering the cache subsystem.

[harold]

@MargaretBloom will that affect cached reads then? Maybe give them a slightly higher latency (the first time that is) since critical-word-first doesn't work?

[vpalmu]

@AndrewHenle: I'm talking about packed, not integer types. Indeed I would use the intX types declaring a portable struct.

[Andrew Henle]

@Joshua Indeed I would use the intX types declaring a portable struct. But fixed-width intX types such as int16_t are optional in C. Per 7.20.1.1 Exact-width integer types of the C Standard: "The typedef ame intN_t designates a signed integer type with width N, no padding bits, and a two’s complement representation. ... These types are optional. ..."

[Andrew Henle]

(cont) And C++ merely follows the C <stdint.h> header with the C++ equivalent <cstdint>. Per cplusplus.com/reference/cstdint: "Notice that some types are optional (and thus, with no portability guarantees)." The fixed-width integer types are again optional. Strictly speaking, int32_t makes for a non-portable structure for strictly-conforming code.

[Cody Gray]

Unaligned access is still slower on x86 than aligned access. The difference compared to other architectures is that x86 forgives attempts to access unaligned data, silently fixing up the misalignment. (Unless you're using aligned SIMD instructions, then it faults. Use the unaligned ones. The performance difference is minimal on the latest architectures, although that hasn't been true for "a long time", unless your definition of "long" is quite different from mine.)

[geza]

@CodyGray: That's why I written "(almost)". I've written a little benchmark (which is already very far from how a typical program uses memory - it summed 64-bit numbers from 8 streams), and the difference is 20%. If I use 32-bit numbers instead, I get 9% difference. Yes, one can argue about that 20% is not "almost", and I kinda agree. Maybe I'll edit my answer. About "long". I cannot remember when, about 10 years ago, I did a similar benchmark on my current computer, and I saw similar numbers. Unfortunately I don't have a 10-year old machine to do the test. (I did the test on a Haswell CPU)

[geza]

@CodyGray: and maybe my little test is flawed, maybe one can create a test which shows even bigger differences. But my point is, that for a typical program, aligned/unaligned access is not a huge factor of speed.

[geza]

@CodyGray: here's my test: stackoverflow.com/questions/45128763/…. As it turned out, for 128-bit access, when data is in the cache, the difference grows to 40%

[Alexander Oh]

@geza you want to write this to a socket or a file right? Thus you need to cast your struct to char * or void * if you do this wrong you might see it serializing stale data.

[geza]

@Alex: can you give an example of this stale data? I don't see why -fno-strict-aliasing is needed in this case. Looking at any data with char * doesn't violate strict aliasing rules. It is allowed. But why would I need to do that? I simply call write(fd, &myStruct, sizeof(myStruct)), and it will be written perfectly. No stale data.

[Alexander Oh]

if you call write it's fine.

[BeeOnRope]

I think the summary is for that for recent Intel x86, misaligned access is free or nearly so, for accesses that don't span a 64B cache line. There is hardware that still lets you do 2 reads even if both are misaligned. Your throughput is cut in half for line spanning loads. Larger loads (like 128-bit) are more likely to span a line (indeed, nearly half of 256-bit loads will cross a boundary).

[Venemo]

Thanks for the link! It looks like something similar to, although more complicated than what I had in mind with my tuple-like stuff.

[Antti Haapala -- Слава Україні]

No it is not really fine on x86 either. There are instructions for SSE that require aligned access, otherwise a fault occurs. If you ever pass a pointer to a double that is misaligned somewhere, you never know what will happen. There is no good reason to use packed, like, ever; there are only some where you need to judge the alternatives.

[geza]

@AnttiHaapala: if packed struct is used, then the compiler won't use these instructions. The only problem if the compiler doesn't know about misalignment. That's what I described in 2.

[Antti Haapala -- Слава Україні]

@geza while it might be true, however you can never safely reference a pointer to a member to a packed struct if the type of member has fundamental alignment requirements

[geza]

@AnttiHaapala: it not might be true. It is true. If not, then it's a bug in the compiler. One typical use case for packed structs is data transfer, where pointer to member is not typical. But I've already described this problem in my answer.

[Antti Haapala -- Слава Україні]

Your answer states that ~"x86_64 always supports aligned access" and therefore it would be safe to always use a pointer to a member; that is not true.

[geza]

@AnttiHaapala: It is almost safe to use on x86_64. The only exception I know is SSE 16 byte move. Which is highly unlikely that is used in this scenario. But even, there's a warning for this in my answer. But I'll edit my answer to contain this information.

[Qix - MONICA WAS MISTREATED]

@bazza because it's 1) another dependency, 2) more development and maintenance overhead, 3) I've yet to get Cap'nProto working correctly cross-platform, 4) I don't wish to support Google by using Protobufs, 5) other serialization libraries are generally too high level and do too much processing at the cost of performance. People have reasons for things, sometimes.

[ZeZNiQ]

"picking apart items read from a file" - I disagree. You can pack structs to make sure there is no padding and then use that struct to represent items read from a file (given fixed offsets of data in file, ofcourse).

[ManuelAtWork]

uint8_t is only defined on platforms supporting it. You may want to used unsigned char instead for better portability.

[Davislor]

@ManuelAtWork Or uint_least8_t has a stronger guarantee that it’s the smallest type at least 8 bits wide. You probably can’t pack data using the same logic on your oddball 9-bit or 12-bit computer from 1970 that has a C99 compiler anyway.

[ManuelAtWork]

Does the standard make any alignment guarantees for uint_least8_t as it does for unsigned char (alignment of 1)? Or can there be padding between array elements on my oddball computer?

[Davislor]

@ManuelAtWork There can be padding bits between uint_least8_t, but not uint8_t if it exists.

[Davislor]

@ManuelAtWork As you know. :)

[U. W.]

I disagree with the disagreement. I JUST NOW have to deal with a compiler, that chooses to ignore the pack pragma, because it is free to do so. It is a PITA, First Class!!

[U. W.]

Addendum: Used the __attribute__ mentioned below instead of #pragma, and that is working as intended. God bless stackoverflow!!!