Error "Unknown magic byte" occurred while deserializing Avro message

[ZHU Hua B]

Hi,


We tried to produce and consume a AVRO message (zookeeper, broker and schema registry have been launched), error "Unknown magic byte" occurred while deserializing Avro message, if I missed anything? Thanks!


From producer:
# bin/kafka-avro-console-producer --broker-list localhost:9092 --topic test --property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"f1","type":"string"}]}'
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/confluent/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/confluent/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/confluent/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

{"f1": "value1"}


From consumer:
# bin/kafka-avro-console-consumer --topic test --zookeeper localhost:2181 --from-beginning
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/confluent/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/confluent/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/confluent/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

[2016-11-23 16:15:11,108] ERROR Unknown error when running consumer: (kafka.tools.ConsoleConsumer$:103)
org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!
[2016-11-23 16:15:11,108] ERROR Unknown error when running consumer: (kafka.tools.ConsoleConsumer$:103)
org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!


Best Regards

Johnny

[Dayong]

As I remember this is to complain the fist byte of msg is not x00. I think console producer does not support json since it uses string schema.

Thanks,
Dayong

Post by ZHU Hua B
Hi,
We tried to produce and consume a AVRO message (zookeeper, broker and schema registry have been launched), error "Unknown magic byte" occurred while deserializing Avro message, if I missed anything? Thanks!
# bin/kafka-avro-console-producer --broker-list localhost:9092 --topic test --property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"f1","type":"string"}]}'
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/confluent/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/confluent/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/confluent/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{"f1": "value1"}
# bin/kafka-avro-console-consumer --topic test --zookeeper localhost:2181 --from-beginning
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/confluent/share/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/confluent/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/confluent/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2016-11-23 16:15:11,108] ERROR Unknown error when running consumer: (kafka.tools.ConsoleConsumer$:103)
org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!
[2016-11-23 16:15:11,108] ERROR Unknown error when running consumer: (kafka.tools.ConsoleConsumer$:103)
org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!
Best Regards
Johnny

[Ewen Cheslack-Postava]

Are you sure you have not produced any other data into that topic, e.g.
perhaps you were testing the regular kafka-console-producer before? This
would cause it to fail on the non-Avro messages (as Dayong says, because
the initial magic byte mismatches).

Can you try starting the consumer first without --from-beginning and then
running the producer to add more data to the topic? If that works, you can
dump the raw bytes using kafka-console-consumer to try to figure out where
the bad data came from, but I'd guess since it looks like you're just
testing you might also just want to delete the topic to get it back to a
clean state.

-Ewen

Post by Dayong
As I remember this is to complain the fist byte of msg is not x00. I think
console producer does not support json since it uses string schema.
Thanks,
Dayong

Post by ZHU Hua B
Hi,
We tried to produce and consume a AVRO message (zookeeper, broker and

schema registry have been launched), error "Unknown magic byte" occurred
while deserializing Avro message, if I missed anything? Thanks!

Post by ZHU Hua B
# bin/kafka-avro-console-producer --broker-list localhost:9092 --topic

test --property value.schema='{"type":"record"
,"name":"myrecord","fields":[{"name":"f1","type":"string"}]}'

Post by ZHU Hua B
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/confluent/shar

e/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/
impl/StaticLoggerBinder.class]

Post by ZHU Hua B
SLF4J: Found binding in [jar:file:/root/confluent/shar

e/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/
impl/StaticLoggerBinder.class]

Post by ZHU Hua B
SLF4J: Found binding in [jar:file:/root/confluent/shar

e/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/
impl/StaticLoggerBinder.class]

Post by ZHU Hua B
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an

explanation.

Post by ZHU Hua B
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{"f1": "value1"}
# bin/kafka-avro-console-consumer --topic test --zookeeper

localhost:2181 --from-beginning

Post by ZHU Hua B
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/confluent/shar

e/java/kafka-serde-tools/slf4j-log4j12-1.7.6.jar!/org/slf4j/
impl/StaticLoggerBinder.class]

Post by ZHU Hua B
SLF4J: Found binding in [jar:file:/root/confluent/shar

e/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/
impl/StaticLoggerBinder.class]

Post by ZHU Hua B
SLF4J: Found binding in [jar:file:/root/confluent/shar

e/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/
impl/StaticLoggerBinder.class]

Post by ZHU Hua B
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an

explanation.

Post by ZHU Hua B
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

(kafka.tools.ConsoleConsumer$:103)

Post by ZHU Hua B
org.apache.kafka.common.errors.SerializationException: Error

deserializing Avro message for id -1
Unknown magic byte!
(kafka.tools.ConsoleConsumer$:103)

Post by ZHU Hua B
org.apache.kafka.common.errors.SerializationException: Error

deserializing Avro message for id -1
Unknown magic byte!

Post by ZHU Hua B
Best Regards
Johnny