Unable to decode network response from AWS

Question

I'm trying to setup a websocket for AWS transcribe, but I've been stuck for over a day trying to decode the byte data / Uint8ArrayView that I receive as a response. Any help is very much appreciated, I've tried tons of decoding, ByteBuffers, etc, and can't figure it out.

Here is a little document on how the data should be coming in: https://docs.aws.amazon.com/transcribe/latest/dg/event-stream.html

Here is how somebody easily does this with the same response in Javascript: Using marshaller (Full code on GitHub)

let messageWrapper = eventStreamMarshaller.unmarshall(Buffer(message.data));
let messageBody = JSON.parse(String.fromCharCode.apply(String, messageWrapper.body));

And here is the data that I receive as a response:

[0, 0, 3, 212, 0, 0, 0, 103, 76, 21, 128, 165, 15, 58, 101, 120, 99, 101, 112, 116, 105, 111, 110, 45, 116, 121, 112, 101, 7, 0, 25, 73, 110, 118, 97, 108, 105, 100, 83, 105, 103, 110, 97, 116, 117, 114, 101, 69, 120, 99, 101, 112, 116, 105, 111, 110, 13, 58, 99, 111, 110, 116, 101, 110, 116, 45, 116, 121, 112, 101, 7, 0, 16, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 13, 58, 109, 101, 115, 115, 97, 103, 101, 45, 116, 121, 112, 101, 7, 0, 9, 101, 120, 99, 101, 112, 116, 105, 111, 110, 123, 34, 77, 101, 115, 115, 97, 103, 101, 34, 58, 34, 84, 104, 101, 32, 114, 101, 113, 117, 101, 115, 116, 32, 115, 105, 103, 110, 97, 116, 117, 114, 101, 32, 119, 101, 32, 99, 97, 108, 99, 117, 108, 97, 116, 101, 100, 32, 100, 111, 101, 115, 32, 110, 111, 116, 32, 109, 97, 116, 99, 104, 32, 116, 104, 101, 32, 115, 105, 103, 110, 97, 116, 117, 114, 101, 32, 121, 111, 117, 32, 112, 114, 111, 118, 105, 100, 101, 100, 46, 32, 67, 104, 101, 99, 107, 32, 121, 111, 117, 114, 32, 65, 87, 83, 32, 83, 101, 99, 114, 101, 116, 32, 65, 99, 99, 101, 115, 115, 32, 75, 101, 121, 32, 97, 110, 100, 32, 115, 105, 103, 110, 105, 110, 103, 32, 109, 101, 116, 104, 111, 100, 46, 32, 67, 111, 110, 115, 117, 108, 116, 32, 116, 104, 101, 32, 115, 101, 114, 118, 105, 99, 101, 32, 100, 111, 99, 117, 109, 101, 110, 116, 97, 116, 105, 111, 110, 32, 102, 111, 114, 32, 100, 101, 116, 97, 105, 108, 115, 46, 92, 110, 92, 110, 84, 104, 101, 32, 67, 97, 110, 111, 110, 105, 99, 97, 108, 32, 83, 116, 114, 105, 110, 103, 32, 102, 111, 114, 32, 116, 104, 105, 115, 32, 114, 101, 113, 117, 101, 115, 116, 32, 115, 104, 111, 117, 108, 100, 32, 104, 97, 118, 101, 32, 98, 101, 101, 110, 92, 110, 39, 71, 69, 84, 92, 110, 47, 115, 116, 114, 101, 97, 109, 45, 116, 114, 97, 110, 115, 99, 114, 105, 112, 116, 105, 111, 110, 45, 119, 101, 98, 115, 111, 99, 107, 101, 116, 92, 110, 88, 45, 65, 109, 122, 45, 65, 108, 103, 111, 114, 105, 116, 104, 109, 61, 65, 87, 83, 52, 45, 72, 77, 65, 67, 45, 83, 72, 65, 50, 53, 54, 38, 88, 45, 65, 109, 122, 45, 67, 114, 101, 100, 101, 110, 116, 105, 97, 108, 61, 65, 75, 73, 65, 84, 52, 55, 65, 90, 72, 82, 81, 79, 82, 88, 72, 77, 85, 77, 79, 37, 50, 70, 50, 48, 50, 49, 48, 52, 49, 48, 37, 50, 70, 117, 115, 45, 101, 97, 115, 116, 45, 50, 37, 50, 70, 116, 114, 97, 110, 115, 99, 114, 105, 98, 101, 37, 50, 70, 97, 119, 115, 52, 95, 114, 101, 113, 117, 101, 115, 116, 38, 88, 45, 65, 109, 122, 45, 68, 97, 116, 101, 61, 50, 48, 50, 49, 48, 52, 49, 48, 84, 49, 56, 48, 50, 52, 57, 90, 38, 88, 45, 65, 109, 122, 45, 69, 120, 112, 105, 114, 101, 115, 61, 56, 54, 52, 48, 48, 38, 88, 45, 65, 109, 122, 45, 83, 105, 103, 110, 101, 100, 72, 101, 97, 100, 101, 114, 115, 61, 104, 111, 115, 116, 38, 108, 97, 110, 103, 117, 97, 103, 101, 45, 99, 111, 100, 101, 61, 101, 110, 45, 85, 83, 38, 109, 101, 100, 105, 97, 45, 101, 110, 99, 111, 100, 105, 110, 103, 61, 112, 99, 109, 38, 115, 97, 109, 112, 108, 101, 45, 114, 97, 116, 101, 61, 49, 54, 48, 48, 48, 92, 110, 104, 111, 115, 116, 58, 116, 114, 97, 110, 115, 99, 114, 105, 98, 101, 115, 116, 114, 101, 97, 109, 105, 110, 103, 46, 117, 115, 45, 101, 97, 115, 116, 45, 50, 46, 97, 109, 97, 122, 111, 110, 97, 119, 115, 46, 99, 111, 109, 58, 56, 52, 52, 51, 92, 110, 92, 110, 104, 111, 115, 116, 92, 110, 101, 51, 98, 48, 99, 52, 52, 50, 57, 56, 102, 99, 49, 99, 49, 52, 57, 97, 102, 98, 102, 52, 99, 56, 57, 57, 54, 102, 98, 57, 50, 52, 50, 55, 97, 101, 52, 49, 101, 52, 54, 52, 57, 98, 57, 51, 52, 99, 97, 52, 57, 53, 57, 57, 49, 98, 55, 56, 53, 50, 98, 56, 53, 53, 39, 92, 110, 92, 110, 84, 104, 101, 32, 83, 116, 114, 105, 110, 103, 45, 116, 111, 45, 83, 105, 103, 110, 32, 115, 104, 111, 117, 108, 100, 32, 104, 97, 118, 101, 32, 98, 101, 101, 110, 92, 110, 39, 65, 87, 83, 52, 45, 72, 77, 65, 67, 45, 83, 72, 65, 50, 53, 54, 92, 110, 50, 48, 50, 49, 48, 52, 49, 48, 84, 49, 56, 48, 50, 52, 57, 90, 92, 110, 50, 48, 50, 49, 48, 52, 49, 48, 47, 117, 115, 45, 101, 97, 115, 116, 45, 50, 47, 116, 114, 97, 110, 115, 99, 114, 105, 98, 101, 47, 97, 119, 115, 52, 95, 114, 101, 113, 117, 101, 115, 116, 92, 110, 52, 57, 54, 98, 50, 57, 49, 57, 54, 100, 100, 101, 98, 100, 57, 55, 101, 53, 100, 56, 54, 56, 54, 98, 54, 54, 100, 97, 101, 102, 48, 97, 100, 98, 102, 56, 53, 49, 48, 54, 53, 54, 99, 53, 52, 102, 100, 99, 98, 98, 101, 101, 100, 98, 49, 50, 54, 101, 55, 100, 54, 51, 57, 99, 39, 92, 110, 34, 125, 25, 123, 100, 219]

Solution

Huge thanks to @Anon-Coward I converted his python code into flutter code which decodes the message and makes the headers and body readable. It's a bit sloppy, but here it is: (testVal is the response from Amazon which is in my question).

                        Uint8List sample = Uint8List.fromList(testVal);
                        ByteData bytes = sample.buffer.asByteData();

                        var totalLength = bytes.lengthInBytes; //980
                        var headersLength = bytes.getInt8(7); //103

                        var data = sample.sublist(12);
                        
                        var left = headersLength;

                        while(left > 0){
                          //get the header name
                          var nameLength = data[0];
                          data = data.sublist(1);
                          var name = utf8.decode(data.sublist(0, nameLength));
                          data = data.sublist(nameLength);
                          //get the value type
                          var valueType = data[0];
                          data = data.sublist(1);
                          //get the value
                          var tempLenList = data.sublist(0, 2);
                          var tempLenBytes = tempLenList.buffer.asByteData();
                          var valueLength = tempLenBytes.getUint16(0, Endian.big);
                          data = data.sublist(2);
                          var value = data.sublist(0, valueLength);
                          data = data.sublist(valueLength);
                          var valueFinal;
                          if(valueType == 7){
                            valueFinal = utf8.decode(value);
                          }

                          left -= (4 + nameLength + valueLength);
                          print(name+ '   ' + valueFinal);
                        }

                        var remainingData = utf8.decode(data.sublist(0, (data.lengthInBytes - 4)));
                        print(remainingData);

Comments

[Anon Coward]

The first link you provided described the format of the message you posted. You'll need to decode the values yourself looking at the header and reading the messages in the body. That said, the header decodes as a InvalidSignatureException, which suggests some code you didn't post is broken.

[August Kimo]

Thanks for the input, can you tell me how you were able to decode the header? I am working on GZIP decoding now, and the code for the request is likely wrong somewhere, but my main goal is to decode the response in hopes in helps me out. Currently I'm working on a GZIP error FormatException: Invalid GZip Signature

[Anon Coward]

I wrote a small python script to decode the header. That said, in this case, simply putting this data in a hex viewer would have shown you the error

[Anon Coward]

The access key ID was in the error message. You can of course delete it and create a new one if you'd like, but it's not the end of the world if it gets out.