Unknown magic byte with kafka-avro-console-consumer
Solution 1
The Confluent Schema Registry serialiser/deserializer uses a wire format which includes information about the schema ID etc in the initial bytes of the message.
If your message has not been serialized using the Schema Registry serializer, then you won't be able to deserialize it with it, and will get the Unknown magic byte!
error.
So you'll need to write a consumer that pulls the messages, does the deserialization using your Avro avsc schemas, and then assuming you want to preserve the data, re-serialize it using the Schema Registry serializer
Edit: I wrote an article recently that explains this whole thing in more depth: https://www.confluent.io/blog/kafka-connect-deep-dive-converters-serialization-explained
Solution 2
kafka-console-consumer
has no knowledge about key.schema
or value.schema
, only the Avro producer does. Source code here
The regular console consumer doesn't care about the format of the data - it'll just print UTF8 encoded bytes
The property that kafka-avro-console-consumer
accepts is only schema.registry.url
. So, to answer the question, yes, it needs to be serialized using the Confluent serializers.
Related videos on Youtube
Martin Macak
Updated on June 04, 2022Comments
-
Martin Macak almost 2 years
I have been trying to connect with kafka-avro-console-consumer from Confluent to our legacy Kafka cluster, which was deployed without Confluent Schema Registry. I provided schema explicitly using properties like:
kafka-console-consumer --bootstrap-server kafka02.internal:9092 \ --topic test \ --from-beginning \ --property key.schema='{"type":"long"}' \ --property value.schema='{"type":"long"}'
but I am getting 'Unknown magic byte!' error with
org.apache.kafka.common.errors.SerializationException
Is it possible to consume Avro messages from Kafka using Confluent kafka-avro-console-consumer that were not serialized with AvroSerializer from Confluent and with Schema Registry?