Accessing field of Protobuf message of unknown type in Python

17,038

I know this is an old thread, but I'll respond anyway for posterity:

Firstly, as you know, it's not possible to determine the type of a protocol buffer message purely from its serialized form. The only information in the serialized form you have access to is the field numbers, and their serialized values.

Secondly, the "right" way to do this would be to have a proto that contains both, like

message Parent {
   required int32  FormatVersion             = 1;
   optional bool   SomeFlag [default = true] = 2;
   optional Common CommonSettings            = 3;

   oneof letters_of_alphabet {
      A a_specific = 4;
      B b_specific = 5;
   }
}

This way, there's no ambiguity: you just parse the same proto (Parent) every time.


Anyway, if it's too late to change that, what I recommend you do is define a new message with only the shared fields, like

message Shared {
   required int32  FormatVersion             = 1;
   optional bool   SomeFlag [default = true] = 2;
   optional Common CommonSettings            = 3;
}

You should then be able to pretend that the message (either A or B) is in fact a Shared, and parse it accordingly. The unknown fields will be irrelevant.

Share:
17,038
djf
Author by

djf

Updated on June 05, 2022

Comments

  • djf
    djf over 1 year

    Let's say I have 2 Protobuf-Messages, A and B. Their overall structure is similar, but not identical. So we moved the shared stuff out into a separate message we called Common. This works beautifully.

    However, I'm now facing the following problem: A special case exists where I have to process a serialized message, but I don't know whether it's a message of type A or type B. I have a working solution in C++ (shown below), but I failed to find a way to do the same thing in Python.

    Example:

    // file: Common.proto
    // contains some kind of shared struct that is used by all messages:
    message Common {
     ...
    }
    
    // file: A.proto
    import "Common.proto";
    
    message A {
       required int32  FormatVersion             = 1;
       optional bool   SomeFlag [default = true] = 2;
       optional Common CommonSettings            = 3;
    
       ... A-specific Fields ...
    }
    
    // file: B.proto
    import "Common.proto";
    
    message B {
       required int32  FormatVersion             = 1;
       optional bool   SomeFlag [default = true] = 2;
       optional Common CommonSettings            = 3;
    
       ... B-specific Fields ...
    }
    

    Working Solution in C++

    In C++ I'm using the reflection API to get access to the CommonSettings field like this:

    namespace gp = google::protobuf;
    ...
    Common* getCommonBlock(gp::Message* paMessage)
    {
       gp::Message* paMessage = new gp::Message();
       gp::FieldDescriptor* paFieldDescriptor = paMessage->GetDescriptor()->FindFieldByNumber(3);
       gp::Reflection* paReflection = paMessage->GetReflection();
       return dynamic_cast<Common&>(paReflection->GetMessage(*paMessage,paFieldDescriptor));
    }
    

    The method 'getCommonBlock' uses FindFieldByNumber() to get hold of the descriptor of the field I'm trying to get. Then it uses reflection to fetch the actual data. getCommonBlock can process messages of type A, B or any future type as long as the Common field remains located at index 3.

    My Question is: Is there a way to do a similar thing Python? I've been looking at the Protobuf documentation, but couldn't figure out a way to do it.

  • djf
    djf over 10 years
    That would work, if I had an instance of obj. Maybe I should clarify my problem a little: I receive the message - in typical protobuf fashion - as a serialized blob (binary memory stream). How do I instantiate obj from that without knowing the underlying message type?
  • deddebme
    deddebme almost 9 years
    I see "Type" within message A and B. Don't you suppose to put in "Base" in those places?
  • kraptor
    kraptor almost 9 years
    If you imported correctly, TYPE is an enum defined in the protobuf file so you are able to call it directly.
  • Kartik Ayyar
    Kartik Ayyar almost 4 years
    As noted in another answer, use oneof, as that natively at the protobuf level does something similar for you. You can query the type of the oneof field using APIs in different languages.