Kryo serialization library: is it used in production?

Question

Kryo is a very new and interesting Java serialization library, and one of the fastest in the thrift-protobuf benchmark. If you've used Kryo, has it already reached enough maturity to try it out in production code?

Update (10/27/2010): We're using Kryo, though not yet in production. See my answer below for details.

Update (3/9/2011): Updating to the latest Jackson and Kryo libraries shows that Jackson's binary Smile serialization is pretty competitive.

Solution 1

There is a bug report and a discussion thread. The DateSerializer that comes with Kryo is slightly more efficient size-wise than the SimpleSerializer implementation posted on SO because it uses LongSerializer optimized for positive values.

Edit: I forgot to answer the original question. I believe Kryo is used in at least a few production systems. There is mention of it in this article, Jive SBS cache redesign: Part 3. In the Destroy All Humans project, Kryo is used to communicate with an Android phone that serves as a robot brain (video here).

Not a direct answer, but you might browse the Kryo source and/or javadocs. Check out the read* and write* methods on the Kryo class, then look at the Serializer class. This is really the core of the library.

Solution 2

I'll try to answer my own question (Kyro is still very new!).

We have a set of about 120 different web services implemented using the Restlet framework. These are consumed by web service clients generally built on top of a Restlet-based client library. The representations sent back and forth between server and client include XML (using the XStream serialization library), JSON (Using Jackson), XHTML, Java Object Serialization, and as of yesterday, Kryo. So we're in a position to do some solid side-by-side comparisons.

Kryo 1.0.1 seems reasonably stable. Once I actually read up on how to use the API, the only real problem I found was that the default java.util.Date serializer seemed to warp dates a few months into the past. I just had to provide my own override:

kryo.register(Date.class, 
  new SimpleSerializer<Date>() {
   @Override public void write (ByteBuffer b, Date d) { b.putLong(d.getTime()); }
   @Override public Date read (ByteBuffer b) { return new Date(b.getLong()); }
  });

But that was the only possible issue I've found so far. We have a set of JavaBeans that have String, Float, Integer, Long, Date, Boolean and List fields.

Here are some rough benchmarks. First, I did 100,000 serializations and deserializations of an object hierarchy that describes one TV program (ie, made 100,000 deep copies of it). The speeds were:

XStream XML:                 360/sec
Java Object Serialization: 1,570/sec
Jackson JSON:              5,000/sec
Kryo:                      8,100/sec

Next, I also serialized a catalog of 2,000 TV program descriptions and counted bytes:

XStream XML:         6,837,851 bytes
Jackson JSON:        3,656,654 bytes
Kryo:                1,124,048 bytes

I also found that registering serializers was very important:

kryo.register(List.class);
kryo.register(ArrayList.class);
// ...
kryo.register(Program.class);
kryo.register(Catalog.class);
// ...

If I didn't do that, the serializations were almost double the size, and the speed was maybe 40% slower.

We also ran complete end-to-end tests of several web services using each of these four serialization methods, and they also showed that Kryo was running faster than the others.

So in summary, Kryo seems reasonably robust. I'm going to keep support for it in our code base and as we gain experience with it I hope to use it in more places. Kudos to the Kryo team!

Update (3/9/2011): I finally got around to @StaxMan's suggestion to try Jackson 1.6's binary "Smile" serializer. Using Jackson 1.6 and Kryo 1.04, I did 100,000 deep copies (serialization/deserialiations) of a somewhat different TV program object hierarchy:

XStream XML:     429/sec    5,189 bytes
Jackson JSON:  4,474/sec    2,657 bytes
Kryo:          4,539/sec    1,066 bytes  
Jackson Smile: 5,040/sec    1,689 bytes

This test didn't mesh with a macro-level test, where I tried different serializers in a REST web service that delivers many of these objects. There the overall system throughput supports @StaxMan's intuition about performance:

Jackson JSON:     92 requests/sec
Jackson Smile     97 requests/sec
Kryo:            108 requests/sec

Solution 3

Kryo is part of Yahoo's S4 (Simple Scalable Streaming System) project. S4 isn't production yet as far as I know.

Solution 4

With the help of Jim Ferrans responses and comments above I found a more detailed explanation about Date Serialization Issue with Kryo on this page: http://groups.google.com/group/kryo-users/browse_thread/thread/91969c6f48a45bdf/ and also a how to use DateSerializer() of Kryo:

kryo.register(Date.class, new DateSerializer());

I hope this could help others.

Solution 5

Kryo 2.x is also used by Mule ESB, and so widely used in production.

View more solutions

Comments

[Petr Gladkikh]

Did you file a bug for date serialization problem? I have not found relevant issues in kryo issue tracker.

[Jim Ferrans]

@Petr, I've been meaning to create an isolated test case, and verify that it's actually a Kryo bug, though I haven't had a chance yet. :-(

[Jim Ferrans]

+1: And upvoted. Thanks Nate for doing Kryo, it's really great!

[StaxMan]

Good stuff, thank you for sharing this! One thing that might be interesting and relatively easy to test is to see how "Smile" format (binary JSON-compatible variant, see wiki.fasterxml.com/JacksonBinaryFormat) would fare. Should be easy to see because it is (a) bundled with Jackson and (b) has API identical to Jackson's JSON handling. Performance and size-wise it should be faster than Jackson/JSON, but how close to Kryo it'd be is hard to say -- probably half-way between JSON and Kryo, maybe closer to Kryo.

[Jim Ferrans]

@StaxMan: Thanks! I'll see if I can squeeze in that experiment and report on it.

[Jim Ferrans]

@StaxMan: Your intuition is right. Smile is very competitive.

[StaxMan]

Excellent, thank you for sharing this. More data points, better the picture. Kryo is very fast, nicely optimized, glad to know Smile fares well too -- the two have bit different goals, but sometimes either can be used.

[Jim Ferrans]

Yes, I'm sure they'd love to see your fixes.

[NateS]

Kryo v2 no longer attempts to be thread safe. Apps are expected to use a Kryo instance per thread.

[NateS]

Just an FYI, Kryo has been updated to v2 and is faster, smaller, and has a cleaner API than before.

[Jim Ferrans]

@NateS: Thanks, it looks very promising! I'm on a different project now, but told my colleagues the good news. Sounds like they'll be switching.

[Jose Martinez]

Thanks for posting your question and answer. How exactly can you use kyro in a RESTful service? Is there an example of kro being sent/read from an HTTP connection? Thanks in advance.