Google Protocol Buffers and HTTP

Question

I'm refactoring legacy C++ system to SOA using gSoap. We have some performance issues (very big XMLs) so my lead asked me to take a look at protocol buffers. I did, and it looks very cool (We need C++ and Java support). However protocol buffers are solution just for serialization and now I need to send it to Java front-end. What should I use from C++ and Java perspective to send those serialized stuff over HTTP (just internal network)?

PS. Another guy tries to speed-up our gSoap solution, I'm interested in protocol buffers only.

Solution 1

You can certainly send even a binary payload with an HTTP request, or in an HTTP response. Just write the bytes of the protocol buffer directly into the request/response, and make sure to set the content type to "application/octet-stream". The client, and server, should be able to take care of the rest easily. I don't think you need anything more special than that on either end.

Solution 2

ProtoBuf is a binary protocol. It doesn't mix well with SOAP. I suggest you either stick with gSOAP or convert to ProtoBuf entirely.

With protoBuf, you define your protocol in a special format like this,

message Product {
  required string id = 1;
  required string description = 2;
  required int32 quantity = 3;
  optional bool discontinued = 4;
}

The protoc tool can generate code in C++/Java/Python so you can serialize it on one end and deserialize on another.

As you can see, ProtoBuf is designed to serialize individual object. It doesn't provide all the facilities provided by SOAP, like headers. To get around this issue, we use ProtoBuf inside ProtoBuf. We define an Envelope like this,

message Envelope {
  enum Type { 
    SEARCH = 1;
    SEARCH_RESPONSE = 2;
    RETRIEVE = 3;
    RETRIEVE_RESPONSE = 4; 
  }
  required Type type = 1;

  required bytes encodedMessage = 2;

  message Header {
    required string key = 1;
    required bytes value = 2;
  }    
  repeated Header headers = 3;
}

The encodedMessage is another serialized ProtoBuf message. All the stuff in SOAP header now goes to headers.

Solution 3

Google frontends prefer application/protobuf.

The ProtocolBufferModel of the Google API client uses application/x-protobuf.

Solution 4

You can serialize/de-serialize protobuf encoded data to/from strings. Send the serialized string as the body of an HTTP POST to Java and de-serialize it. That is one approach. Another way is to make use of the protobuf Service interface. Protobuf allows you to define a service interface in a .proto file and the protocol buffer compiler will generate service interface code and stubs in your chosen language. You only need to implement the protobuf::RpcChannel and protobuf::RpcController classes to get a complete RPC framework. Probably you can write an HTTP wrapper for these classes. See the following links for more information:

http://code.google.com/apis/protocolbuffers/docs/proto.html#services http://code.google.com/apis/protocolbuffers/docs/reference/cpp-generated.html#service http://code.google.com/apis/protocolbuffers/docs/reference/cpp/google.protobuf.service.html

Solution 5

To my knowledge protocol buffers support is available in both C++ and Java, you should be able to exchange protocol buffer serialized data between both systems.

That said, it seems your real question is "How do I send stuff over HTTP between a C++ backend and Java client"

It sound like you need to learn how to use gSOAP, read the docs.

Alternatively you could host a RESTful web server from your C++ app: Look at this: https://stackoverflow.com/questions/298113/how-can-i-implement-a-restful-webservice-using-c++

Next you would need to access the data hosted on your new C++ RESTful server: Look at this: Rest clients for Java?

View more solutions

Comments

[Tim Sylvester]

We're using application/x-protobuf as described on slide 21 here: slideshare.net/mokeefe/…

[Sean Owen]

Ah even better. Didn't know anyone had suggested a MIME type for it.

[n0rd]

MIME types having a name starting with "x-" are nowhere near standard and usually treated as application/octet-stream.

[Will]

gSOAP is massively overcomplicating the issue. Protobufs work fine over HTTP in their native binary format.

[Tommy Stanton]

Looks like there was a commit in Apr 2012 in the ProtoRPC project that states that application/octet-stream is the preferred content-type nowadays.

[Dr. Alex RE]

Overcomplicated? Sure, if you want to transfer binary content then just use Protobufs and REST. But for XML it made our work a lot easier. The complicated parts of WSDL, SOAP, and XML are all taken care of. Also, we got over 5,000 service invocations per second with gsoap. The main overhead is with the TCP/IP, not the gsoap stack.

[Alex Bitek]

@znlyj If you have your own client and server, yes, otherwise it is not guaranteed.