Proto2 vs. Proto3 in C#

Question

I have to send messages to another team using the proto2 version of Google Protocol Buffers. They are using Java and C++ on Linux. I'm using C# on Windows.

Jon Skeet's protobuf-csharp-port (https://github.com/jskeet/protobuf-csharp-port) supports proto2. If I understand correctly, Google has taken this code and folded an updated version of it into the main protobuf project (https://github.com/google/protobuf/tree/master/csharp). But it no longer supports proto2 for C#, only proto3.

I'm not sure which project I should use. It seems like the new one will be better supported (performance, support for proto3 if the other team ever upgrades). But I would have to convert the .proto file that I was given from proto2 to proto3 and risk any issues that come with that.

I've read that for the most part, the messages for proto2 and proto3 are compatible. I have no experience with Protocol Buffers, but the .proto file I'm working with looks pretty vanilla, no default values or oneof or nested anything. So it seems like I could just delete their "required" and "optional" keywords and use the new library, treating this as a proto3 file.

In your opinion, is it worth the hassle to use the newer library? Is there a list of proto features that would make the proto2 and proto3 messages incompatible?

Solution

If the other team has any required fields and you send messages to them without specifying those fields (or even explicitly specifying the default value, for primitives) then the other end will fail to receive the messages - they won't validate.

There are various differences between proto2 and proto3 - some are listed on the releases page:

The following are the main new features in language version 3:

• Removal of field presence logic for primitive value fields, removal of required fields, and removal of default values. This makes proto3 significantly easier to implement with open struct representations, as in languages like Android Java, Objective C, or Go.
• Removal of unknown fields.
• Removal of extensions, which are instead replaced by a new standard type called Any.
• Fix semantics for unknown enum values.
• Addition of maps.
• Addition of a small set of standard types for representation of time, dynamic data, etc.
• A well-defined encoding in JSON as an alternative to binary proto encoding.

The removal of unknown fields could be a significant issue to you - if the other team expects to be able to send you a message with some fields your code is unaware of, and you be able to return a message to them maintaining those fields, proto3 could pose problems for you.

If you can use proto3, I'd suggest using proto3 version, partly as it will have proper support whereas the proto2 version is basically in maintenance mode. There are significant differences between the two, primarily in terms of mutability - the generated message classes in the proto3 codebase are mutable, which is great for immediate usability, but can pose challenges in other areas.

Comments

[Meirion Hughes]

still can't believe they got rid of defaults :(

[Manoj]

The language guide for proto2 lists maps as being supported. Also I tried using proto compiler 3 with proto3 and it seems to ignore fields set with default values (int with 0 was removed from binary and json representation).

[Jon Skeet]

@Manoj: I think they were supported to some extent, but not by all languages, and quite possibly with slightly different semantics. Not sure what you mean by "it seems to ignore fields set with default values". If you just mean "the value 0 isn't serialized" then you're right.

[Manoj]

@Jon : Thank you for the quick reply! Yes, I meant it was not getting serialized. I think it is easy to get confused with proto compiler and proto language versions. Proto compiler 3 seems to support both proto language 2 and 3. With proto language 2 - map does not work when using proto compiler 2 but works fine with proto compiler 3 (tried only with Python). If default values are still supported can you please clarify what you meant by removal of default values.

[Jon Skeet]

@Manoj: Explicitly set default values are not supported in proto3 syntax, no. And note that the C# code only supports proto3 syntax.