Protobuf-net serialization/deserialization

Question

I checked but seem to be unable to see how to directly serialize a class to a byte array and subsequently deserialize from a byte array using Marc Gravell's protobuf-net implementation.

Edit: I changed the question and provided code because the original question of how to serialize into byte[] without having to go through stream was admittedly trivial. My apologies.

Updated Question: Is there any way to not have to deal with generics and instead infer the type of the property "MessageBody" through reflection when it is passed through the constructor? I assume I cannot serialize object type, correct? The current solution looks very cumbersome in that I need to pass in the type of the MessageBody each time I instantiate a new Message. Is there a sleeker solution to this?

I came up with the following:

class Program
{
    static void Main(string[] args)
    {
        Message<string> msg = new Message<string>("Producer", "Consumer", "Test Message");

        byte[] byteArray = msg.Serialize();
        Message<string> message = Message<string>.Deserialize(byteArray);

        Console.WriteLine("Output");
        Console.WriteLine(message.From);
        Console.WriteLine(message.To);
        Console.WriteLine(message.MessageBody);

        Console.ReadLine();

    }
}

[ProtoContract]
public class Message<T>
{
    [ProtoMember(1)]
    public string From { get; private set; }
    [ProtoMember(2)]
    public string To { get; private set; }
    [ProtoMember(3)]
    public T MessageBody { get; private set; }

    public Message()
    {

    }

    public Message(string from, string to, T messageBody)
    {
        this.From = from;
        this.To = to;
        this.MessageBody = messageBody;
    }

    public byte[] Serialize()
    {
        byte[] msgOut;

        using (var stream = new MemoryStream())
        {
            Serializer.Serialize(stream, this);
            msgOut = stream.GetBuffer();
        }

        return msgOut;
    }

    public static Message<T> Deserialize(byte[] message)
    {
        Message<T> msgOut;

        using (var stream = new MemoryStream(message))
        {
            msgOut = Serializer.Deserialize<Message<T>>(stream);
        }

        return msgOut;
    }   
}

What I like to get to is something such as:

Message newMsg = new Message("Producer", "Consumer", Foo); byte[] byteArray = newMsg.Serialize();

and Message msg = Message.Deserialize(byteArray);

(where Deserialize is a static method and it always deserializes into an object of type Message and only needs to know what type to deserialize the message body into).

Solution 1

there's a few different questions here, so I'll answer what I can see: if I've missed anything just let me know.

Firstly, as noted, a MemoryStream is the most common way of getting to a byte[]. This is consistent with most serializers - for example, XmlSerializer, BinaryFormatter and DataContractSerializer also don't have an "as a byte[] overload", but will accept MemoryStream.

Generics: you don't need to use generics; v1 has Serializer.NonGeneric, which wraps this away from you. In v2, the "core" is non-generic, and can be accessed via RuntimeTypeModel.Default; of course Serializer and Serializer.NonGeneric continue to work.

For the issue of having to include the type: yes, the protobuf spec assumes the receiver knows what type of data they are being given. A simple option here is to use a simple wrapper object as the "root" object, with multiple typed properties for the data (only one of which is non-null). Another option might spring from the inbuilt inheritance support via ProtoInclude (note: as an implementation detail, these two approaches are identical).

In your specific example, perhaps consider:

[ProtoContract]
[ProtoInclude(1, typeof(Message<Foo>))]
.... More as needed
[ProtoInclude(8, typeof(Message<Bar>))]
public abstract class Message
{   }
[ProtoContract]
public class Message<T> : Message
{
    ...
}

Then just serialize with <Message> - the API will create the right type automatically.

With recent builds, there is also a DynamicType option that includes type data for you, for example:

[ProtoContract]
public class MyRoot {
    [ProtoMember(1, DynamicType=true)]
    public object Value { get; set; }
}

This will work for any Value that holds a contract-type instance (but not for primitives, and ideally not involving inheritance).

Solution 2

The code the OP posted wouldn't quite work for me, the following is a slight adaptation taking onboard a little more of Marc Gravell's suggestions. Inheriting from Message was needed to prevent "Cyclic inheritance is not allowed", and as noted in the code comments below GetBuffer wasn't working out either.

Hoping it helps someone else, took me a good few hours to get it all to work...

      [ProtoContract]
      public abstract class Message
      {
        public byte[] Serialize()
        {
          byte[] result;
          using (var stream = new MemoryStream())
          {
            Serializer.Serialize(stream, this);
            result = stream.ToArray(); //GetBuffer was giving me a Protobuf.ProtoException of "Invalid field in source data: 0" when deserializing
          }
          return result;
        }
      }

      [ProtoContract]
      public class Message : Message
      {
        [ProtoMember(1)]
        public string From { get; private set; }
        [ProtoMember(2)]
        public string To { get; private set; }
        [ProtoMember(3)]
        public T MessageBody { get; private set; }

        public Message()
        { }

        public Message(string from, string to, T messageBody)
        {
          this.From = from;
          this.To = to;
          this.MessageBody = messageBody;
        }

        public static Message Deserialize(byte[] message)
        {
          Message result;
          using (var stream = new MemoryStream(message))
          {
            result = Serializer.Deserialize>(stream);
          }
          return result;
        }
      }

Comments

[Matt]

Marc, thanks for the comments. The last of your suggestions looks like exactly what I was looking for. Let me play with it a bit and get back to you.

[Marc Gravell]

@Freddy I would have advised the ProtoInclude one personally, but: whatever works...

[Matt]

Looking at both...thanks a thousand for your advise. Protobuf-net is an awesome library by the way

[Matt]

Question though: if I use protoinclude won't I limit myself to having to specify the T types at compile time? I look for a solution that essentially lets me serialize any kind of objects without having to specify which components the object consists of until runtime.

[Matt]

Marc, I run performance tests and I think the DynamicType solution works very well. I use Protobuf-net to serialize/deserialize control messages that are not latency critical. I also run another high throughput/low latency message stream that processes over 10 million messages per second but convert to raw byte arrays based on my own custom serialization routines of primitive types because it proved to be faster. Your solution was exactly what I was looking for, thanks a lot for your help especially given its a weekend.

[Marc Gravell]

@Freddy protobuf-net is designed, like XmlSerializer and DataContractSerializer, primarily for the "I know what I'm expecting" scenario. If the other works then fine, but...

[Matt]

Marc, I guess I miss-communicated: My process also knows what types to expect when deserializing, I was simply looking for a workaround in order to a) not having to specify types at serialization time (precisely because I know the type at deserialization time), and b) not having to list/specify all possible object types as part of the protocontract base object. The dynamic type allows me to do such.

[Marc Gravell]

@Freddy at serialization, you don't have to tell it the type... The system can get that itself from GetType. See the NonGeneric API or RuntimeTypeModel.Default

[João Antunes]

DynamicType works, but only with one level, how can we handle multiple nested dynamics? stackoverflow.com/questions/22512610/…

[9swampy]

The error's a markup issue: Serializer.Deserialize<Message<T>>(stream);