SSE, intrinsics, and alignment

Question

I've written a 3D vector class using a lot of SSE compiler intrinsics. Everything worked fine until I started to instatiate classes having the 3D vector as a member with new. I experienced odd crashes in release mode but not in debug mode and the other way around.

So I read some articles and figured I need to align the classes owning an instance of the 3D vector class to 16 bytes too. So I just added _MM_ALIGN16 (__declspec(align(16)) in front of the classes like so:

_MM_ALIGN16 struct Sphere
{
    // ....

    Vector3 point;
    float radius
};

That seemed to solve the issue at first. But after changing some code my program started to crash in odd ways again. I searched the web some more and found a blog article. I tried what the author, Ernst Hot, did to solve the problem and it works for me too. I added new and delete operators to my classes like this:

_MM_ALIGN16 struct Sphere
{
    // ....

    void *operator new (unsigned int size)
     { return _mm_malloc(size, 16); }

    void operator delete (void *p)
     { _mm_free(p); }

    Vector3 point;
    float radius
};

Ernst mentions that this aproach could be problematic as well, but he just links to a forum which does not exist anymore without explaining why it could be problematic.

So my questions are:

1. What's the problem with defining the operators?

2. Why isn't adding _MM_ALIGN16 to the class definition enough?

3. What's the best way to handle the alignment issues coming with SSE intrinsics?

Solution 1

First of all you have to care for two types of memory allocation:

• Static allocation. For automatic variables to be properly aligned, your type needs a proper alignment specification (e.g. __declspec(align(16)), __attribute__((aligned(16))), or your _MM_ALIGN16). But fortunately you only need this if the alignment requirements given by the type's members (if any) are not sufficient. So you don't need this for you Sphere, given that your Vector3 is already aligned properly. And if your Vector3 contains an __m128 member (which is pretty likely, otherwise I would suggest to do so), then you don't even need it for Vector3. So you usually don't have to mess with compiler specific alignment attributes.

• Dynamic allocation. So much for the easy part. The problem is, that C++ uses, on the lowest level, a rather type-agnostic memory allocation function for allocating any dynamic memory. This only guarantees proper alignment for all standard types, which might happen to be 16 bytes but isn't guaranteed to.

  For this to compensate you have to overload the builtin operator new/delete to implement your own memory allocation and use an aligned allocation function under the hood instead of good old malloc. Overloading operator new/delete is a topic on its own, but isn't that difficult as it might seem at first (though your example is not enough) and you can read about it in this excellent FAQ question.

  Unfortunately you have to do this for each type that has any member needing non-standard alignment, in your case both Sphere and Vector3. But what you can do to make it a bit easier is just make an empty base class with proper overloads for those operators and then just derive all neccessary classes from this base class.

  What most people sometimes tend to forget is that the standard allocator std::alocator uses the global operator new for all memory allocation, so your types won't work with standard containers (and a std::vector<Vector3> isn't that rare a use case). What you need to do is make your own standard conformant allocator and use this. But for convenience and safety it is actually better to just specialize std::allocator for your type (maybe just deriving it form your custom allocator) so that it is always used and you don't need to care for using the proper allocator each time you use a std::vector. Unfortunately in this case you have to again specialize it for each aligned type, but a small evil macro helps with that.

  Additionally you have to look out for other things using the global operator new/delete instead of your custom one, like std::get_temporary_buffer and std::return_temporary_buffer, and care for those if neccessary.

Unfortunately there isn't yet a much better approach to those problems, I think, unless you are on a platform that natively aligns to 16 and know about this. Or you might just overload the global operator new/delete to always align each memory block to 16 bytes and be free of caring for the alignment of each and every single class containing an SSE member, but I don't know about the implications of this approach. In the worst case it should just result in wasting memory, but then again you usually don't allocate small objects dynamically in C++ (though std::list and std::map might think differently about this).

So to sum up:

• Care for proper alignment of static memory using things like __declspec(align(16)), but only if it is not already cared for by any member, which is usually the case.

• Overload operator new/delete for each and every type having a member with non-standard alignment requirements.

• Make a cunstom standard-conformant allocator to use in standard containers of aligned types, or better yet, specialize std::allocator for each and every aligned type.

---

Finally some general advice. Often you only profit form SSE in computation-heavy blocks when performing many vector operations. To simplify all this alignment problems, especially the problems of caring for the alignment of each and every type containing a Vector3, it might be a good aproach to make a special SSE vector type and only use this inside of lengthy computations, using a normal non-SSE vector for storage and member variables.

Solution 2

Basically, you need to make sure that your vectors are properly aligned because SIMD vector types normally have larger alignment requirements than any of the built-in types.

That requires doing the following things:

1. Make sure that Vector3 is properly aligned when it is on the stack or a member of a structure. This is done by applying __attribute__((aligned(32))) to Vector3 class (or whatever attribute is supported by your compiler). Note, that you don't need to apply the attribute to structures containing Vector3, that is not necessary and not enough (i.e. no need to apply it to Sphere).

2. Make sure that Vector3 or its enclosing structure is properly aligned when using heap allocation. This is done by using posix_memalign() (or a similar function for your platform) instead of using plain malloc() or operator new() because the latter two align memory for built-in types (normally 8 or 16 bytes) which is not guaranteed to be enough for SIMD types.

Solution 3

1. The problem with the operators is that by themselves they're not sufficient. They don't affect stack allocations, for which you still need __declspec(align(16)).

2. __declspec(align(16)) affects how the compiler places objects in memory, if and only if it has the choice. For new'ed objects, the compiler has no choice but to use the memory returned by operator new.

3. Ideally, use a compiler which handles them natively. There is no theoretical reason why they need to be treated differently from double. Else, read the compiler documentation for workarounds. Each handicapped compiler will have its own set of issues and therefore its own set of workarounds.

Comments

[Thomas]

In the first case, are you allocating your structs on the stack or the heap? I am not sure malloc returns aligned memory by default, whereas _mm_malloc certainly would - what do you mean by "after a while my program started to crash again"? Do you mean after leaving it running for a bit (and what was it doing)?

[Jan Deinhard]

The problems began when I started allocating the structs on the heap. With the "after a while" sentence I mean that it started crashing after I changed code. I guess the alignment was right by accident and then I destroyed it. I think malloc does not return memory 16 byte aligned which is the problem I guess. My question is really what the problem with operator approach is and what is the best way to manage code using SSE intrinsics.

[Christian Rau]

In fact you don't need to specify the alignment of Sphere (using this _MM_ALIGN16 thing), since the compiler is smart enough to see that Sphere has a 16-aligned member and automatically adjusts Sphere's alignment requirements (given that Vector3 is properly aligned). That's the reason why you don't have to explicitly align Vector3 either if it already has an __m128 member. It's only the dynamic allocation that's a problem and this can be overcome by overloading operator new/delete, like written in the blog (and usually additional things, like specializing std::allocator).

[Jan Deinhard]

Thanks! From Christian Rau's comment I take that the __declspec(align(16)) is obsolete. I guess that part depends on the compiler? I'm not sure whether I understand the 3. part of your answer. What do you mean by 'handle them natively'. I use the compiler that comes with Visual Studio 2012 Express.

[MSalters]

@FairDinkumThinkum: What I mean by "a compiler that handles them natively" is a compiler that can align SSE types just like aligns FP types, i.e. without help from the programmer. You don't need a __declspec(align(8)) for those. I don't have VS2012 so I can't say for certain whether it's that smart already.

[Alexandre C.]

You most likely have to implement a custom allocator too.

[Christian Rau]

@FairDinkumThinkum The __declspec(align(16)) (or whatever your compiler uses) is not obsolete, it is only obsolete if a type already has a properly aligned member, which is the case for Sphere. The thing is just that it is most of the times also obsolete for your Vector3 class, because it will most probably contain a member of type __m128, which is guaranteed to be 16-aligned.

[Christian Rau]

@AlexandreC. +1 - People often forget that the standard allocator uses the global operator new. But I would rather specialize std::allocator instead of making your own, because there is virtually never a case where you need non-aligned memory for an aligned type and this way you don't have to care for using the proper allocator each time you instantiate a std::vector.

[andrewmu]

Would the std::aligned_storage from C++11 enable all this without the need for specialist call conventions?

[Christian Rau]

@graham.reeds Rather the alignas keyword. std::aligned_storage isn't really needed, given that __m128 is already properly aligned and you'd rather want an __m128 member instead of a std::aligned_storage member. But sure, alignas is the new platform independent way of saying __declspec(align()) (or whatever gcc likes), even if none of them are usually needed at all. But all this only helps for static memory alignment anyway.