Passing byte array between C++ and C# ByRef raises AccessViolationException

Question

I am trying to create a Win32 DLL exposes some functions which are called in C# as follows

__declspec(dllexport) int GetData(unsigned char* *data, int* size)
{
    try
    {
        int tlen = 3;
        unsigned char* tchr = new unsigned char[5];
        tchr[0] = 'a';
        tchr[1] = 'b';
        tchr[2] = 'c';

        *size = tlen;
        *data = tchr;

        return 1;
    }
    catch (char *p)
    {
        return 0;
    }
}

And on C# side

[DllImport("MyDll.dll")]
static extern int GetData(ref byte[] data, ref int size);

static void Main()
{
    try
    {
        int hr = 0;
        byte[] gData = null;
        int gSize = 0;
        hr = GetData(ref gData, ref gSize);
        Console.WriteLine(gSize);
        for (int i = 0; i < gSize; i++)
            Console.WriteLine((char)gData[i]);
    }
    catch (Exception p)
    {
        Console.WriteLine(p.ToString());
    }
}

When I run C# code, AccessViolationException happens on GetData function which is a sign of exception in C++ code however, following C++ code snippet works fine without any error.

int _tmain(int argc, _TCHAR* argv[])
{
    unsigned char* data = NULL;
    int size = NULL;
    GetData(&data, &size);
    printf("%d", size);
    for (int i = 0; i < size; i++)
        printf("%c,", data[i]);
    return 0;
}

If you compare C# main function and C++ _tmain, they are almost analoguous so where I may make a mistake?

Solution

You are returning an array allocated by a call to C++ new and hoping that the marshaler will turn it into a C# byte[]. That won't happen.

You'll need to pass a pointer by reference and then marshal it by hand. Your p/invoke should look like this:

[DllImport("MyDll.dll")]
static extern int GetData(out IntPtr data, out int size);

When the function returns data will point to the array and you can read the contents using the Marshal class. I guess you would copy it to a new byte array.

var arr = new byte[size];
Marshal.Copy(data, arr, 0, size);

Some other points:

1. The calling conventions do not match. The native side is cdecl and the managed is stdcall.
2. You'll need to export a deallocator to delete the memory returned by the native function. Consider a re-design where the caller allocates the buffer.

Comments

[Петър Петров]

...or allocate with CoTaskMemAlloc / g_malloc in NAtive and use Marshal.FreeCoTaskMem on returned IntPtr in managed