Marshal.PtrToStructure (and back again) and generic solution for endianness swapping

Question

I have a system where a remote agent sends serialized structures (from an embedded C system) for me to read and store via IP/UDP. In some cases I need to send back the same structure types. I thought I had a nice setup using Marshal.PtrToStructure (receive) and Marshal.StructureToPtr (send). However, a small gotcha is that the network big endian integers need to be converted to my x86 little endian format to be used locally. When I'm sending them off again, big endian is the way to go.

Here are the functions in question:

    private static T BytesToStruct<T>(ref byte[] rawData) where T: struct
    {
        T result = default(T);
        GCHandle handle = GCHandle.Alloc(rawData, GCHandleType.Pinned);
        try
        {
            IntPtr rawDataPtr = handle.AddrOfPinnedObject();
            result = (T)Marshal.PtrToStructure(rawDataPtr, typeof(T));
        }
        finally
        {
            handle.Free();
        }
        return result;
    }

    private static byte[] StructToBytes<T>(T data) where T: struct
    {
        byte[] rawData = new byte[Marshal.SizeOf(data)];
        GCHandle handle = GCHandle.Alloc(rawData, GCHandleType.Pinned);
        try
        {
            IntPtr rawDataPtr = handle.AddrOfPinnedObject();
            Marshal.StructureToPtr(data, rawDataPtr, false);
        }
        finally
        {
            handle.Free();
        }
        return rawData;
    }

And a quick example structure that might be used like this:

byte[] data = this.sock.Receive(ref this.ipep);
Request request = BytesToStruct<Request>(ref data);

Where the structure in question looks like:

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack = 1)]
private struct Request
{
    public byte type;
    public short sequence;
    [MarshalAs(UnmanagedType.ByValArray, SizeConst = 5)]
    public byte[] address;
}

What (generic) way can I swap the endianness when marshalling the structures? My need is such that the locally stored 'request.sequence' in this example should be little-endian for displaying to the user. I don't want to have to swap the endianness in a structure-specific way since it's a generic problem.

My first thought was to use Reflection, but I'm not very familiar with that feature. Also, I hoped that there would be a better solution out there that somebody could point me towards. Thanks in advance :)

Solution 1

Reflection does seem like the only real way to accomplish what you're after.

I've put together some code below. It creates an attribute called EndianAttribute that can be applied at the field level on a struct. I've included the definition for this attribute and it's associated enum, as well as the modifications to your code necessary to use it.

As a side note, you did not need to define rawData as a ref parameter.

Note that this does require the use of C# 3.0/.NET 3.5, since I'm using LINQ and anonymous types in the function doing the work. It would not be difficult to rewrite the function without these features, though.

[AttributeUsage(AttributeTargets.Field)]
public class EndianAttribute : Attribute
{
    public Endianness Endianness { get; private set; }

    public EndianAttribute(Endianness endianness)
    {
        this.Endianness = endianness;
    }
}

public enum Endianness
{
    BigEndian,
    LittleEndian
}

private static void RespectEndianness(Type type, byte[] data)
{
    var fields = type.GetFields().Where(f => f.IsDefined(typeof(EndianAttribute), false))
        .Select(f => new
        {
            Field = f,
            Attribute = (EndianAttribute)f.GetCustomAttributes(typeof(EndianAttribute), false)[0],
            Offset = Marshal.OffsetOf(type, f.Name).ToInt32()
        }).ToList();

    foreach (var field in fields)
    {
        if ((field.Attribute.Endianness == Endianness.BigEndian && BitConverter.IsLittleEndian) ||
            (field.Attribute.Endianness == Endianness.LittleEndian && !BitConverter.IsLittleEndian))
        {
            Array.Reverse(data, field.Offset, Marshal.SizeOf(field.Field.FieldType));
        }
    }
}

private static T BytesToStruct<T>(byte[] rawData) where T : struct
{
    T result = default(T);

    RespectEndianness(typeof(T), rawData);     

    GCHandle handle = GCHandle.Alloc(rawData, GCHandleType.Pinned);

    try
    {
        IntPtr rawDataPtr = handle.AddrOfPinnedObject();
        result = (T)Marshal.PtrToStructure(rawDataPtr, typeof(T));
    }
    finally
    {
        handle.Free();
    }        

    return result;
}

private static byte[] StructToBytes<T>(T data) where T : struct
{
    byte[] rawData = new byte[Marshal.SizeOf(data)];
    GCHandle handle = GCHandle.Alloc(rawData, GCHandleType.Pinned);
    try
    {
        IntPtr rawDataPtr = handle.AddrOfPinnedObject();
        Marshal.StructureToPtr(data, rawDataPtr, false);
    }
    finally
    {
        handle.Free();
    }

    RespectEndianness(typeof(T), rawData);     

    return rawData;
}

Solution 2

For those of us without Linq, a replacement RespectEndianness():

private static void RespectEndianness(Type type, byte[] data) {
    foreach (FieldInfo f in type.GetFields()) {
        if (f.IsDefined(typeof(EndianAttribute), false)) {
            EndianAttribute att = (EndianAttribute)f.GetCustomAttributes(typeof(EndianAttribute), false)[0];
            int offset = Marshal.OffsetOf(type, f.Name).ToInt32();
            if ((att.Endianness == Endianness.BigEndian && BitConverter.IsLittleEndian) ||
                (att.Endianness == Endianness.LittleEndian && !BitConverter.IsLittleEndian)) {
                Array.Reverse(data, offset, Marshal.SizeOf(f.FieldType));
            }
        }
    }
}

Solution 3

Here's my variation - it handles nested structs and arrays, with the assumption that arrays are of a fixed size, eg marked with a [MarshalAs(UnmanagedType.ByValArray, SizeConst = N)] attribute.

public static class Serializer
{
    public static byte[] GetBytes<T>(T structure, bool respectEndianness = true) where T : struct
    {
        var size = Marshal.SizeOf(structure); //or Marshal.SizeOf<T>(); in .net 4.5.1
        var bytes = new byte[size];
        var ptr = Marshal.AllocHGlobal(size);

        Marshal.StructureToPtr(structure, ptr, true);
        Marshal.Copy(ptr, bytes, 0, size);
        Marshal.FreeHGlobal(ptr);

        if (respectEndianness) RespectEndianness(typeof(T), bytes);  

        return bytes;
    }

    public static T FromBytes<T>(byte[] bytes, bool respectEndianness = true) where T : struct
    {
        var structure = new T();

        if (respectEndianness) RespectEndianness(typeof(T), bytes);    

        int size = Marshal.SizeOf(structure);  //or Marshal.SizeOf<T>(); in .net 4.5.1
        IntPtr ptr = Marshal.AllocHGlobal(size);

        Marshal.Copy(bytes, 0, ptr, size);

        structure = (T)Marshal.PtrToStructure(ptr, structure.GetType());
        Marshal.FreeHGlobal(ptr);

        return structure;
    }

    private static void RespectEndianness(Type type, byte[] data, int offSet = 0)
    {
        var fields = type.GetFields()
            .Select(f => new
            {
                Field = f,
                Offset = Marshal.OffsetOf(type, f.Name).ToInt32(),
            }).ToList();

        foreach (var field in fields)
        {
            if (field.Field.FieldType.IsArray)
            {
                //handle arrays, assuming fixed length
                var attr = field.Field.GetCustomAttributes(typeof(MarshalAsAttribute), false).FirstOrDefault();
                var marshalAsAttribute = attr as MarshalAsAttribute;
                if (marshalAsAttribute == null || marshalAsAttribute.SizeConst == 0)
                    throw new NotSupportedException(
                        "Array fields must be decorated with a MarshalAsAttribute with SizeConst specified.");

                var arrayLength = marshalAsAttribute.SizeConst;
                var elementType = field.Field.FieldType.GetElementType();
                var elementSize = Marshal.SizeOf(elementType);
                var arrayOffset = field.Offset + offSet;

                for (int i = arrayOffset; i < arrayOffset + elementSize * arrayLength; i += elementSize)                    {
                    RespectEndianness(elementType, data, i);
                }
            }
            else if (!field.Field.FieldType.IsPrimitive) //or !field.Field.FiledType.GetFields().Length == 0
            {
                //handle nested structs
                RespectEndianness(field.Field.FieldType, data, field.Offset);
            }
            else
            {
                //handle primitive types
                Array.Reverse(data, offSet + field.Offset, Marshal.SizeOf(field.Field.FieldType));
            }
        }
    }
}

Comments

[cgyDeveloper]

Just the slick kind of setup I was looking for. I'll give this a shot.

[cgyDeveloper]

Works so far without any other modifications... I like this solution a lot.

[rkagerer]

Great solution! If you don't like the performance hit of walking through the reflection every time you convert, then you could use ILGenerator to dyamically generate (and cache) a "converter" class for each of your structs. It may sound complicated, but there's a nice little example here: varjabedian.net/archive/2008/07/29/…

[Spooles]

Nice solution. Still some typos in the enum and the attribute ctor.

[Nathan Moinvaziri]

This works really well when you make BytesToStruct and StructToBytes extensions of a stream.

[Jeppe Stig Nielsen]

Since .NET 4.5.1 you can use var size = Marshal.SizeOf<T>(); instead of var size = Marshal.SizeOf(structure);.

[A.J.Bauer]

I just came across your code and I think it's pretty good, except for a little error in your recursion where you handle arrays. I added "var arrayOffset = field.Offset + offSet;" and "for (int i = arrayOffset; i < arrayOffset + elementSize * arrayLength; i += elementSize)". Thanks for posting.