How to manipulate client settings through AdminAPI?

[Will Weber]

Hey all,

Working on a tool for Kafka lifecycle management and one of the features
we're keen to try to wrap is user quotas.

This capability definitely exists through sets of scripts included with
Kafka, case and point: https://kafka.apache.org/documentation/#quotas

bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config

'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200'
--entity-type users --entity-name user1 --entity-type clients --entity-name
clientA

Updated config for entity: user-principal 'user1', client-id 'clientA'.

Which would set limits on a given client.

Based on this command, it leads me to believe that the
AdminClient#alterConfigs
<https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/admin/AdminClient.html#alterConfigs-java.util.Map->
method
is probably the way to go.

However, I noticed that in the ConfigResource.Type
<https://kafka.apache.org/20/javadoc/org/apache/kafka/common/config/ConfigResource.Type.html>
enum, there are only Broker, Unknown and Topic options. Naively, I imagine
there to be a corresponding Enum to match the entity-type flag options as
well, but no such luck.

Anyone have recommendations as to how I could go about enforcing quotas
through the admin api?

Mostly just looking for a nudge in the right direction, any help is
appreciated!

Best,

-Will

[SJTU胡昊坤]

Hi Will,
Unfortunately, you can't use AdminClient to do that. Because Kafka use
zookeeper to keep the user quotas. And there is no API for this.
You can implement this feature by writing raw data to zookeeper. The
protocol is simple:
first, write data to:
znode: ${kafka_rootdir}/config/users/${user_id}/clients/${client_id}
value: {"version": 1, "config": {"producer_byte_rate": "1024",
"consumer_byte_rate": "1024"}}
second, write config change with sequence to:
znode: ${kafka_rootdir}/config/changes/config_change_${sequence} value:
{"version": 2, "entity_path": "users/${user_id}/clients/${client_id}"}

You can find more impelment detail in these source files:
src/main/scala/kafka/admin/ConfigCommand.scala
src/main/scala/kafka/zk/AdminZkClient.scala
src/main/scala/kafka/server/DynamicConfigManager.scala ( The comments
helps!)

from DynamicConfigManager.scala:

Config is stored under the path: /config/entityType/entityName
* E.g. /config/topics/<topic_name> and /config/clients/<clientId>
* This znode stores the overrides for this entity in properties format
with defaults stored using entityName "<default>".
* Multiple entity names may be specified (eg. <user, client-id>
quotas) using a hierarchical path:
* E.g. /config/users/<user>/clients/<clientId>
*
* To avoid watching all topics for changes instead we have a notification path
* /config/changes
* The DynamicConfigManager has a child watch on this path.
*
* To update a config we first update the config properties. Then we
create a new sequential
* znode under the change path which contains the name of the
entityType and entityName that was updated, say
* /config/changes/config_change_13321
* The sequential znode contains data in this format: {"version" : 1,
"entity_type":"topic/client", "entity_name" : "topic_name/client_id"}
* This is just a notification--the actual config change is stored only
once under the /config/entityType/entityName path.
* Version 2 of notifications has the format: {"version" : 2,
"entity_path":"entity_type/entity_name"}
* Multiple entities may be specified as a hierarchical path (eg.
users/<user>/clients/<clientId>).
*


However, I suggest not relying on this implement for a long time. Because
it may change in new version of Kafka. It's not part of public API.

Hope this will help :)

Post by Will Weber
Hey all,
Working on a tool for Kafka lifecycle management and one of the features
we're keen to try to wrap is user quotas.
This capability definitely exists through sets of scripts included with
Kafka, case and point: https://kafka.apache.org/documentation/#quotas

bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config

'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200'
--entity-type users --entity-name user1 --entity-type clients --entity-name
clientA
Updated config for entity: user-principal 'user1', client-id 'clientA'.
Which would set limits on a given client.
Based on this command, it leads me to believe that the
AdminClient#alterConfigs
<
https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/admin/AdminClient.html#alterConfigs-java.util.Map-
method
is probably the way to go.
However, I noticed that in the ConfigResource.Type
<
https://kafka.apache.org/20/javadoc/org/apache/kafka/common/config/ConfigResource.Type.html
enum, there are only Broker, Unknown and Topic options. Naively, I imagine
there to be a corresponding Enum to match the entity-type flag options as
well, but no such luck.
Anyone have recommendations as to how I could go about enforcing quotas
through the admin api?
Mostly just looking for a nudge in the right direction, any help is
appreciated!
Best,
-Will