Mirrormaker producing to only one partition in a topic

[Ryan En]

Hi, I'm using Kafka version 0.10.2.0 and trying to use Mirrormaker to the messages from one Kafka cluster to another.


The source and target Kafka cluster are pretty much set up the same... replication factor is 3, number of partitions is 3, auto.create.topics.enable is true.


I am finding that in the target Kafka cluster, all the messages produced by Mirrormaker are going into one partition only.


Here are my configs:


mirrormaker producer


security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="xxxxxxxx" password="xxxxxxxxxx";
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=xxxxxxxxxxx
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=xxxxxxxxx
ssl.key.password=xxxxxxxx
retries=5000000
max.in.flight.requests.per.connection=1
client.id=xxxxxxxx
compression.type=lz4
bootstrap.servers=<aggregate-broker>:9292
acks=all


mirrormaker consumer


security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="xxxxxxxx" password="xxxxxxxxxx";
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=xxxxxxxxxx
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=xxxxxxxxxx
ssl.key.password=xxxxxxxxx
client.id=xxxxxxxxxxxx
partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
auto.commit.interval.ms=1000
auto.offset.reset=earliest
bootstrap.servers=<local-broker>:9192
session.timeout.ms=30000
group.id=xxxxxxxxxxxxxx
enable.auto.commit=false


Any ideas why this may happen?


Thanks in advance

[Stephen Powis]

Hey Ryan,

I ran into a similar issue and it was how the RoundRobinAssignor/Partitioner
was hashing the keys in my messages. You may want to look at how thats
implemented and see if its causing all of your messages to end up in the
same partition.

For what its worth, this ticket has the implementation I put together to
deal with it: https://issues.apache.org/jira/browse/KAFKA-3333

Stephen

Post by Ryan En
Hi, I'm using Kafka version 0.10.2.0 and trying to use Mirrormaker to the
messages from one Kafka cluster to another.
The source and target Kafka cluster are pretty much set up the same...
replication factor is 3, number of partitions is 3,
auto.create.topics.enable is true.
I am finding that in the target Kafka cluster, all the messages produced
by Mirrormaker are going into one partition only.
mirrormaker producer
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
required username="xxxxxxxx" password="xxxxxxxxxx";
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=xxxxxxxxxxx
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=xxxxxxxxx
ssl.key.password=xxxxxxxx
retries=5000000
max.in.flight.requests.per.connection=1
client.id=xxxxxxxx
compression.type=lz4
bootstrap.servers=<aggregate-broker>:9292
acks=all
mirrormaker consumer
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
required username="xxxxxxxx" password="xxxxxxxxxx";
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=xxxxxxxxxx
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=xxxxxxxxxx
ssl.key.password=xxxxxxxxx
client.id=xxxxxxxxxxxx
partition.assignment.strategy=org.apache.kafka.clients.
consumer.RoundRobinAssignor
auto.commit.interval.ms=1000
auto.offset.reset=earliest
bootstrap.servers=<local-broker>:9192
session.timeout.ms=30000
group.id=xxxxxxxxxxxxxx
enable.auto.commit=false
Any ideas why this may happen?
Thanks in advance

[b***@binance.com]

Post by Ryan En
Hi, I'm using Kafka version 0.10.2.0 and trying to use Mirrormaker to the messages from one Kafka cluster to another.
The source and target Kafka cluster are pretty much set up the same... replication factor is 3, number of partitions is 3, auto.create.topics.enable is true.
I am finding that in the target Kafka cluster, all the messages produced by Mirrormaker are going into one partition only.
mirrormaker producer
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="xxxxxxxx" password="xxxxxxxxxx";
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=xxxxxxxxxxx
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=xxxxxxxxx
ssl.key.password=xxxxxxxx
retries=5000000
max.in.flight.requests.per.connection=1
client.id=xxxxxxxx
compression.type=lz4
bootstrap.servers=<aggregate-broker>:9292
acks=all
mirrormaker consumer
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="xxxxxxxx" password="xxxxxxxxxx";
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=xxxxxxxxxx
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=xxxxxxxxxx
ssl.key.password=xxxxxxxxx
client.id=xxxxxxxxxxxx
partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
auto.commit.interval.ms=1000
auto.offset.reset=earliest
bootstrap.servers=<local-broker>:9192
session.timeout.ms=30000
group.id=xxxxxxxxxxxxxx
enable.auto.commit=false
Any ideas why this may happen?
Thanks in advance
i met the same question as yours. have you solved it by adding some configs. very thank for your share

[b***@binance.com]

have you solved this problem ? how do you solved... tanks reply

Post by Ryan En
Hi, I'm using Kafka version 0.10.2.0 and trying to use Mirrormaker to the messages from one Kafka cluster to another.
The source and target Kafka cluster are pretty much set up the same... replication factor is 3, number of partitions is 3, auto.create.topics.enable is true.
I am finding that in the target Kafka cluster, all the messages produced by Mirrormaker are going into one partition only.
mirrormaker producer
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="xxxxxxxx" password="xxxxxxxxxx";
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=xxxxxxxxxxx
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=xxxxxxxxx
ssl.key.password=xxxxxxxx
retries=5000000
max.in.flight.requests.per.connection=1
client.id=xxxxxxxx
compression.type=lz4
bootstrap.servers=<aggregate-broker>:9292
acks=all
mirrormaker consumer
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="xxxxxxxx" password="xxxxxxxxxx";
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=xxxxxxxxxx
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=xxxxxxxxxx
ssl.key.password=xxxxxxxxx
client.id=xxxxxxxxxxxx
partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
auto.commit.interval.ms=1000
auto.offset.reset=earliest
bootstrap.servers=<local-broker>:9192
session.timeout.ms=30000
group.id=xxxxxxxxxxxxxx
enable.auto.commit=false
Any ideas why this may happen?
Thanks in advance