Performance Tests of Serializations used by WCF Bindings

c# .net wcf performance wcf-binding

14,704

Solution 1

OK; I'll bite... here's some raw serializer metrics (emph: you may need to consider base-64/MTOM to get overall bandwidth requirements, plus whatever fixed overheads (both space and CPU) that WCF adds), however; results first:

BinaryFormatter
Length: 1314
Serialize: 6746
Deserialize: 6268

XmlSerializer
Length: 1049
Serialize: 3282
Deserialize: 5132

DataContractSerializer
Length: 911
Serialize: 1411
Deserialize: 4380

NetDataContractSerializer
Length: 1139
Serialize: 2014
Deserialize: 5645

JavaScriptSerializer
Length: 528
Serialize: 12050
Deserialize: 30558

(protobuf-net v2)
Length: 112
Serialize: 217
Deserialize: 250

(so I conclude protobuf-net v2 the winner...)

Numbers updated with .NET 4.5 and current library builds, on a newer machine:

BinaryFormatter
Length: 1313
Serialize: 2786
Deserialize: 2407

XmlSerializer
Length: 1049
Serialize: 1265
Deserialize: 2165

DataContractSerializer
Length: 911
Serialize: 574
Deserialize: 2011

NetDataContractSerializer
Length: 1138
Serialize: 850
Deserialize: 2535

JavaScriptSerializer
Length: 528
Serialize: 8660
Deserialize: 8468

(protobuf-net v2)
Length: 112
Serialize: 78
Deserialize: 134

with test rig (compiled with optimizations, run at command line):

(and note I had to invent the Player class and some sample data):

using System;
using System.Diagnostics;
using System.IO;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;
using System.Text;
using System.Web.Script.Serialization;
using System.Xml.Serialization;
using ProtoBuf.Meta;


static class Program
{
    static void Main()
    {
        var orig = new Game {
             Finished = true, GameGUID = Guid.NewGuid(), GameID = 12345, GameSetup = false, MaximumCardsInDeck = 20,
             Player = new Player { Name = "Fred"}, Player1 = new Player { Name = "Barney"}, Player1Connected = true,
             Player1EnvironmentSetup = true, Player1ID = 12345, Player1Won = 3, Player2Connected = true, Player2EnvironmentSetup = true,
             Player2ID = 23456, Player2Won = 0, Round = 4, RoundsToWin = 5, Started = true, StateXML = "not really xml",
             TimeEnded = null, TimeLimitPerTurn = 500, TimeStamp = new byte[] {1,2,3,4,5,6}, TimeStarted = DateTime.Today};
        const int LOOP = 50000;

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        using (var ms = new MemoryStream())
        {
            var ser = new BinaryFormatter();
            Console.WriteLine();
            Console.WriteLine(ser.GetType().Name);
            ser.Serialize(ms, orig);
            Console.WriteLine("Length: " + ms.Length);
            ms.Position = 0;
            ser.Deserialize(ms);

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ms.SetLength(0);
                ser.Serialize(ms, orig);
            }
            watch.Stop();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ser.Deserialize(ms);
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        using (var ms = new MemoryStream())
        {
            var ser = new XmlSerializer(typeof(Game));
            Console.WriteLine();
            Console.WriteLine(ser.GetType().Name);
            ser.Serialize(ms, orig);
            Console.WriteLine("Length: " + ms.Length);
            ms.Position = 0;
            ser.Deserialize(ms);

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ms.SetLength(0);
                ser.Serialize(ms, orig);
            }
            watch.Stop();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ser.Deserialize(ms);
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        using (var ms = new MemoryStream())
        {
            var ser = new DataContractSerializer(typeof(Game));
            Console.WriteLine();
            Console.WriteLine(ser.GetType().Name);
            ser.WriteObject(ms, orig);
            Console.WriteLine("Length: " + ms.Length);
            ms.Position = 0;
            ser.ReadObject(ms);

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ms.SetLength(0);
                ser.WriteObject(ms, orig);
            }
            watch.Stop();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ser.ReadObject(ms);
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        using (var ms = new MemoryStream())
        {
            var ser = new NetDataContractSerializer();
            Console.WriteLine();
            Console.WriteLine(ser.GetType().Name);
            ser.Serialize(ms, orig);
            Console.WriteLine("Length: " + ms.Length);
            ms.Position = 0;
            ser.Deserialize(ms);

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ms.SetLength(0);
                ser.Serialize(ms, orig);
            }
            watch.Stop();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ser.Deserialize(ms);
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        {
            var sb = new StringBuilder();
            var ser = new JavaScriptSerializer();
            Console.WriteLine();
            Console.WriteLine(ser.GetType().Name);
            ser.Serialize(orig, sb);
            Console.WriteLine("Length: " + sb.Length);
            ser.Deserialize(sb.ToString(), typeof(Game));

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                sb.Length = 0;
                ser.Serialize(orig, sb);
            }
            watch.Stop();
            string s = sb.ToString();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ser.Deserialize(s, typeof(Game));
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        using (var ms = new MemoryStream())
        {
            var ser = CreateProto();
            Console.WriteLine();
            Console.WriteLine("(protobuf-net v2)");
            ser.Serialize(ms, orig);
            Console.WriteLine("Length: " + ms.Length);
            ms.Position = 0;
            ser.Deserialize(ms, null, typeof(Game));

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ms.SetLength(0);
                ser.Serialize(ms, orig);
            }
            watch.Stop();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ser.Deserialize(ms, null, typeof(Game));
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        Console.WriteLine();
        Console.WriteLine("All done; any key to exit");
        Console.ReadKey();
    }
    static TypeModel CreateProto()
    {
        var meta = TypeModel.Create();
        meta.Add(typeof(Game), false).Add(Array.ConvertAll(typeof(Game).GetProperties(),prop=>prop.Name));
        meta.Add(typeof(Player), false).Add(Array.ConvertAll(typeof(Player).GetProperties(),prop=>prop.Name));
        return meta.Compile();
    }
}

[Serializable, DataContract]
public partial class Game
{
    [DataMember]
    public bool Finished { get; set; }
    [DataMember]
    public Guid GameGUID { get; set; }
    [DataMember]
    public long GameID { get; set; }
    [DataMember]
    public bool GameSetup { get; set; }
    [DataMember]
    public Nullable<int> MaximumCardsInDeck { get; set; }
    [DataMember]
    public Player Player { get; set; }
    [DataMember]
    public Player Player1 { get; set; }
    [DataMember]
    public bool Player1Connected { get; set; }
    [DataMember]
    public bool Player1EnvironmentSetup { get; set; }
    [DataMember]
    public long Player1ID { get; set; }
    [DataMember]
    public int Player1Won { get; set; }
    [DataMember]
    public bool Player2Connected { get; set; }
    [DataMember]
    public bool Player2EnvironmentSetup { get; set; }
    [DataMember]
    public long Player2ID { get; set; }
    [DataMember]
    public int Player2Won { get; set; }
    [DataMember]
    public int Round { get; set; }
    [DataMember]
    public Nullable<int> RoundsToWin { get; set; }
    [DataMember]
    public bool Started { get; set; }
    [DataMember]
    public string StateXML { get; set; }
    [DataMember]
    public Nullable<DateTime> TimeEnded { get; set; }
    [DataMember]
    public Nullable<int> TimeLimitPerTurn { get; set; }
    [DataMember]
    public byte[] TimeStamp { get; set; }
    [DataMember]
    public Nullable<DateTime> TimeStarted { get; set; }
}
[Serializable, DataContract]
public class Player
{
    [DataMember]
    public string Name { get; set; }
}

Solution 2

I also have benchmarks graphs for different serializers in .NET that show @Marc Gravell's binary protobuf-net serializer as the clear winner. Although I maintain the fastest text serializers .NET which come closest to matching it and are also much faster than all the serializers that come in BCL in .NET.

These benchmarks are based on the Nortwind sample Database from Microsoft and show how much slower each serializer is compared to Protobuf-net's.

ProtoBuf.net(v1)                      1x
ServiceStack TypeSerializer           2.23x
ServiceStack JsonSerializer           2.58x
Microsoft DataContractSerializer      6.93x
NewtonSoft.Json                       7.83x
Microsoft BinaryFormatter             9.21x
Microsoft JsonDataContractSerializer  9.31x

The full benchmarks are available here

So if you prefer/need to use a fast text-serializer here are links to Service Stack's open source text serializers:

By the way the Microsoft's JavaScriptSerializer showed the worst performance and at times were 40x-100x slower than protobuf-nets. Took it out because they were slowing down my benchmarks :)

Solution 3

I've modified @Marc's benchmark source code and added results for ServiceStack's JSV and JSON Serializers Here are the results on my 3yo iMac:

BinaryFormatter
Length: 1313
Serialize: 3959
Deserialize: 3395

XmlSerializer
Length: 1049
Serialize: 1710
Deserialize: 2716

DataContractSerializer
Length: 911
Serialize: 712
Deserialize: 2117

NetDataContractSerializer
Length: 1138
Serialize: 1093
Deserialize: 4825

TypeSerializer
Length: 431
Serialize: 496
Deserialize: 887

JsonSerializer
Length: 507
Serialize: 558
Deserialize: 1213

Here is the source code that I added to @Marc's benchmark above.

GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
GC.WaitForPendingFinalizers();
var sbJsv = new StringBuilder(4096);
using (var sw = new StringWriter(sbJsv))
{
    Console.WriteLine();
    Console.WriteLine(typeof(TypeSerializer).Name);
    TypeSerializer.SerializeToWriter(orig, sw);
    var jsv = sbJsv.ToString();
    Console.WriteLine("Length: " + sbJsv.Length);
    TypeSerializer.DeserializeFromString<Game>(jsv);

    var watch = Stopwatch.StartNew();
    for (int i = 0; i < LOOP; i++)
    {
        sbJsv.Length = 0;
        TypeSerializer.SerializeToWriter(orig, sw);
    }
    watch.Stop();
    Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);

    watch = Stopwatch.StartNew();
    for (int i = 0; i < LOOP; i++)
    {
        TypeSerializer.DeserializeFromString<Game>(jsv);
    }
    watch.Stop();
    Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
}

GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
GC.WaitForPendingFinalizers();
var sbJson = new StringBuilder(4096);
using (var sw = new StringWriter(sbJson))
{
    Console.WriteLine();
    Console.WriteLine(typeof(JsonSerializer).Name);
    JsonSerializer.SerializeToWriter(orig, sw);
    var json = sbJson.ToString();
    Console.WriteLine("Length: " + sbJson.Length);
    JsonSerializer.DeserializeFromString<Game>(json);

    var watch = Stopwatch.StartNew();
    for (int i = 0; i < LOOP; i++)
    {
        sbJson.Length = 0;
        JsonSerializer.SerializeToWriter(orig, sw);
    }
    watch.Stop();
    Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);

    watch = Stopwatch.StartNew();
    for (int i = 0; i < LOOP; i++)
    {
        JsonSerializer.DeserializeFromString<Game>(json);
    }
    watch.Stop();
    Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
}

Note: I couldn't get a hold of @Marc's protobuf-net v2 r352 dlls he used for this so I've had to comment protobuf-net benchmarks out.

Solution 4

At the simplest level; simply serialize a shed-load of data, time it, and measure the bandwidth. And shed-load should include both large and small (but lots-of) payloads.

You should also consider with/without MTOM. And although I am perhaps biased I suggest you should include alternative WCF serializers such as protobuf-net (let me know if you need help hooking that in). From lots of work in the area it usually defeats all the ones you've mentioned by a decent margin on every measure.

Much of what is involved here can be investigated at the serializer level without even touching WCF, however that bypases base-64/MTOM so isn't a 100 percent image.

We can't, however, define your measures for you; only you can decide what is key. I do have a number of measure though - it is usually simply:

serialize once to MemorySteam (and deserialize); this gets you the size and primes the JIT
now keep that memory-stream (as a handy buffer) and (inside stopwatch) serialize many thousands if times. Divide. Rewind each time so you are overwriting (not extending).
repeat but deserializibg many thousands of times. Divide.

Solution 5

Use an object of constant size; getting "size" information from a Type is messy and won't gain you much in terms of figuring out which is "best". Any object decorated as a DataContract can be serialized to binary (DataContract inherits Serializable), basic XML (any object with a default constructor can be serialized into XML) or DataContract XML (this requires the most markup to start with, but it's pretty simple).

For a running test, create a method that will take an Object and a Serializer. It should create a MemoryStream and start a StopWatch, then serialize the object into the MemoryStream (make sure to Flush()). Then it stops the stopwatch and returns you the results as a TimeSpan, and the Stream's length. Then reset and start the Stopwatch and deserialize the Stream, and remember that time. You can set the return results up as a simple struct.

Run this with the same object for each serializer you want to test. Output each of the results to the console or debug output, and may the best serializer win.

Generally speaking, I think you'll find:

BinarySerializer will be fastest and smallest, as it has the least overhead of bytes to write while serializing. However, .NET's binary serializations are platform-specific; if you want to talk to anything but another .NET assembly that knows about your exact type, forget it.
XMLSerializer, SoapSerializer and DataContractSerializer all output various forms of XML. DataContract is actually the simplest format (the XML is extremely basic because the handshake and other protocol/communication information is seperate) and will probably be rather fast. SOAP has a lot of bloat in the serialized file because of transport and metadata information, but is easy to generate as it's a pretty strict format. Basic XML serialization, because it's so flexible, has a lot of overhead, but can generate a very simple or very complex schema.

View more solutions

14,704

Author by

Ben

http://www.linkedin.com/in/benbachhuber@benbachhuber

Updated on June 15, 2022

Comments

Ben almost 2 years

I have the following object:

public partial class Game
{
    public bool Finished { get; set; }

    public Guid GameGUID { get; set; }

    public long GameID { get; set; }

    public bool GameSetup { get; set; }

    public Nullable<int> MaximumCardsInDeck { get; set; }

    public Player Player { get; set; }

    public Player Player1 { get; set; }

    public bool Player1Connected { get; set; }

    public bool Player1EnvironmentSetup { get; set; }

    public long Player1ID { get; set; }

    public int Player1Won { get; set; }

    public bool Player2Connected { get; set; }

    public bool Player2EnvironmentSetup { get; set; }

    public long Player2ID { get; set; }

    public int Player2Won { get; set; }

    public int Round { get; set; }

    public Nullable<int> RoundsToWin { get; set; }

    public bool Started { get; set; }

    public string StateXML { get; set; }

    public Nullable<DateTime> TimeEnded { get; set; }

    public Nullable<int> TimeLimitPerTurn { get; set; }

    public byte[] TimeStamp { get; set; }

    public Nullable<DateTime> TimeStarted { get; set; }    
}

This class gonna be filled with some test data.

I need the to compare the Performance of different Serializers used by the different forms of bindings for WCF Services:

basicHttpBinding => SoapFormatter (TextFormatter?)
binaryBinding => BinaryFormatter
XMLFormatter

What i need to do in detail is:

Get to now the size of the Object being serialized
Get to now the size after serizlization
Time to serialize
Time to deserialize

I already tried some stuff, but i am struggling a bit. Maybe there is already some simple code for this kind of measurement.

Marc Gravell over 13 years

Note; you will usually need a large number of cycles to give any kind of comparison. And I'm not sure I agree with some of the points; BinaryFormatter can be overweight depending on the data, and XmlSerializer is often simpler than DataContractSerializer. It all depends on the data, of course.
Marc Gravell over 13 years

(note: protobuf-net v2 based on "r352")
Ben over 13 years

Thank you very much. Seems like Binary Formatter isn't that good for my case
Marc Gravell over 13 years

@Ben - I suspect WCF binary (inbuilt) is NetDataContractSerializer, actually. BinaryFormatter is the default in remoting though.
Marc Gravell over 13 years

ditto; I have a different iteration count for the slowest serializers ;p I must find time to try your benchmarks against v2 at some point. Although I think I'll prioritise finishing it first ;p
mythz over 13 years

@marcgravell Definitely, Been hanging out for v2 to be complete as well :) I want to include it as another ServiceStack endpoint.
Marc Gravell over 13 years

Actually, your test inflates the timing (I.e. Slower than it needs to show), as it seems to extend rather than overwrite the buffer. Up to you, bit it might be even faster with a more comparable test
Ben over 13 years

Am i right that you are using a regular console application project for your tests?
mythz over 13 years

Yeah you're right, I was looking to reset the Position of the StringWriter when I was refactoring but couldn't see it so I left it as-is - didn't even think about resetting the StringBuilder.Length :)
Richard Hauer over 9 years

Old thread I know, but FWIW I tested this against a custom binary serializer I created which a) only works for 2 known types and b) has full knowledge of the format of the types in advance. This is the leanest, fastest I could make it. Using the above test method I can only get about 35% faster and smaller than protoBuf - so there's not much to gain over protobuf in a real-world application.
Isaac over 6 years

Based on these results the protobuf the is the clear winner. Any insights on this? Also, does it have any drawbacks/limitations?
Iúri dos Anjos over 6 years

@Isaac Protobuff does now support all types of objects. Don't know exactly the full list, but "Marc Gravell" already answered to me in an e-mail listing those: - interfaces (well, there's some limited support) - delegates (i guess its okay to no support this one) - nested arrays / lists (fine if there's an intermediate object in the middle) - inheritance of unexpected types (expected types are fine)