C# unsafe value type array to byte array conversions

Question

I have a .ckpt file that contains the weights of a model based on tensorflow v1 resnet_v1_50. I tried to inspect the checkpoint file using the function tensorflow.python.tools.inspect_checkpoint.print_tensors_in_checkpoint_file and the output is like:

video_model/resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta (DT_FLOAT) [64]
video_model/resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta/Adam (DT_FLOAT) [64]
video_model/resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/beta/Adam_1 (DT_FLOAT) [64]
video_model/resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/gamma (DT_FLOAT) [64]
video_model/resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/gamma/Adam (DT_FLOAT) [64]
video_model/resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/gamma/Adam_1 (DT_FLOAT) [64]
video_model/resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean (DT_FLOAT) [64]
video_model/resnet_v1_50/block1/unit_1/bottleneck_v1/conv1/BatchNorm/moving_variance (DT_FLOAT) [64]
[...]

What I want to do is to load this variable (or a subset of them) into tf.keras ResNet50 model as all the other layers in my model are written in keras; The structure of the two architectures should be the same, but if I inspect a compiled version of keras ResNet50 here's what I obtain:

[...]
layer_with_weights-0/bias/.ATTRIBUTES/VARIABLE_VALUE (DT_FLOAT) [64]
layer_with_weights-0/kernel/.ATTRIBUTES/VARIABLE_VALUE (DT_FLOAT) [7,7,3,64]
layer_with_weights-1/beta/.ATTRIBUTES/VARIABLE_VALUE (DT_FLOAT) [64]
layer_with_weights-1/gamma/.ATTRIBUTES/VARIABLE_VALUE (DT_FLOAT) [64]
layer_with_weights-1/moving_mean/.ATTRIBUTES/VARIABLE_VALUE (DT_FLOAT) [64]
layer_with_weights-1/moving_variance/.ATTRIBUTES/VARIABLE_VALUE (DT_FLOAT) [64]
[...]

I think that it's just a re-naming of the layers. Is there any chance I can do that and reuse the pretrained checkpoint?

Solution 1

Yes, the type information and data is in the same memory block, so that is impossible unless you overwrite the type information in a float array to fool the system that it's byte array. That would be a really ugly hack, and could easily blow up...

Here's how you can convert the floats without unsafe code if you like:

public static byte[] ToByteArray(this float[] floatArray) {
    int len = floatArray.Length * 4;
    byte[] byteArray = new byte[len];
    int pos = 0;
    foreach (float f in floatArray) {
        byte[] data = BitConverter.GetBytes(f);
        Array.Copy(data, 0, byteArray, pos, 4);
        pos += 4;
    }
    return byteArray;
}

Solution 2

You can use a really ugly hack to temporary change your array to byte[] using memory manipulation.

This is really fast and efficient as it doesn't require cloning the data and iterating on it.

I tested this hack in both 32 & 64 bit OS, so it should be portable.

The source + sample usage is maintained at https://gist.github.com/1050703 , but for your convenience I'll paste it here as well:

public static unsafe class FastArraySerializer
{
    [StructLayout(LayoutKind.Explicit)]
    private struct Union
    {
        [FieldOffset(0)] public byte[] bytes;
        [FieldOffset(0)] public float[] floats;
    }

    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    private struct ArrayHeader
    {
        public UIntPtr type;
        public UIntPtr length;
    }

    private static readonly UIntPtr BYTE_ARRAY_TYPE;
    private static readonly UIntPtr FLOAT_ARRAY_TYPE;

    static FastArraySerializer()
    {
        fixed (void* pBytes = new byte[1])
        fixed (void* pFloats = new float[1])
        {
            BYTE_ARRAY_TYPE = getHeader(pBytes)->type;
            FLOAT_ARRAY_TYPE = getHeader(pFloats)->type;
        }
    }

    public static void AsByteArray(this float[] floats, Action<byte[]> action)
    {
        if (floats.handleNullOrEmptyArray(action)) 
            return;

        var union = new Union {floats = floats};
        union.floats.toByteArray();
        try
        {
            action(union.bytes);
        }
        finally
        {
            union.bytes.toFloatArray();
        }
    }

    public static void AsFloatArray(this byte[] bytes, Action<float[]> action)
    {
        if (bytes.handleNullOrEmptyArray(action)) 
            return;

        var union = new Union {bytes = bytes};
        union.bytes.toFloatArray();
        try
        {
            action(union.floats);
        }
        finally
        {
            union.floats.toByteArray();
        }
    }

    public static bool handleNullOrEmptyArray<TSrc,TDst>(this TSrc[] array, Action<TDst[]> action)
    {
        if (array == null)
        {
            action(null);
            return true;
        }

        if (array.Length == 0)
        {
            action(new TDst[0]);
            return true;
        }

        return false;
    }

    private static ArrayHeader* getHeader(void* pBytes)
    {
        return (ArrayHeader*)pBytes - 1;
    }

    private static void toFloatArray(this byte[] bytes)
    {
        fixed (void* pArray = bytes)
        {
            var pHeader = getHeader(pArray);

            pHeader->type = FLOAT_ARRAY_TYPE;
            pHeader->length = (UIntPtr)(bytes.Length / sizeof(float));
        }
    }

    private static void toByteArray(this float[] floats)
    {
        fixed(void* pArray = floats)
        {
            var pHeader = getHeader(pArray);

            pHeader->type = BYTE_ARRAY_TYPE;
            pHeader->length = (UIntPtr)(floats.Length * sizeof(float));
        }
    }
}

And the usage is:

var floats = new float[] {0, 1, 0, 1};
floats.AsByteArray(bytes =>
{
    foreach (var b in bytes)
    {
        Console.WriteLine(b);
    }
});

Solution 3

This question is the reverse of What is the fastest way to convert a float[] to a byte[]?.

I've answered with a union kind of hack to skip the whole copying of the data. You could easily reverse this (length = length *sizeof(Double).

Solution 4

I've written something similar for quick conversion between arrays. It's basically an ugly proof-of-concept more than a handsome solution. ;)

public static TDest[] ConvertArray<TSource, TDest>(TSource[] source)
    where TSource : struct
    where TDest : struct {

    if (source == null)
        throw new ArgumentNullException("source");

        var sourceType = typeof(TSource);
        var destType = typeof(TDest);

        if (sourceType == typeof(char) || destType == typeof(char))
            throw new NotSupportedException(
                "Can not convert from/to a char array. Char is special " +
                "in a somewhat unknown way (like enums can't be based on " +
                "char either), and Marshal.SizeOf returns 1 even when the " +
                "values held by a char can be above 255."
            );

        var sourceByteSize = Buffer.ByteLength(source);
        var destTypeSize = Marshal.SizeOf(destType);
        if (sourceByteSize % destTypeSize != 0)
            throw new Exception(
                "The source array is " + sourceByteSize + " bytes, which can " +
                "not be transfered to chunks of " + destTypeSize + ", the size " +
                "of type " + typeof(TDest).Name + ". Change destination type or " +
                "pad the source array with additional values."
            );

        var destCount = sourceByteSize / destTypeSize;
        var destArray = new TDest[destCount];

        Buffer.BlockCopy(source, 0, destArray, 0, sourceByteSize);

        return destArray;
    }
}

Solution 5

You should check my answer to a similar question: What is the fastest way to convert a float[] to a byte[]?.

In it you'll find portable code (32/64 bit compatible) to let you view a float array as a byte array or vice-versa, without copying the data. It's the fastest way that I know of to do such thing.

If you're just interested in the code, it's maintained at https://gist.github.com/1050703 .

View more solutions

Comments

[Omer Mor]

And if you're interested in the suggested hack, check out the implementation in my answer below: stackoverflow.com/questions/621493/…

[Cristian Diaconescu]

There is a newer, more portable "one year later" version of this answer here: stackoverflow.com/a/3577253/11545 . @Omer - maybe update this answer too?

[Cristian Diaconescu]

Thanks! :) +1 Btw you should add a license to that code. CodingHorror explains why

[Omer Mor]

You're right. I added a FreeBSD license to the gist. The code in SO already has a license. Check the footer: "user contributions licensed under cc-wiki with attribution required"

[Jan Kotas]

These hacks are corrupting the internal garbage collector data structures. It will cause intermittent crashes, data corruptions, and security bugs of the same class as use-after-free in C++. Hacking internal garbage collector data structures like this is absolutely not supported by the .NET runtime. github.com/HelloKitty/Reinterpret.Net/issues/1 has a long discussion about the crashes that this hack will lead to

[Syroot]

It would probably be faster to use Buffer.BlockCopy(floatArray, 0, data, 0, data.Length) rather than using foreachwith BitConverter.