Discussion:
Kafka ACL issue - Operation denied despite having full access to the topic
Bala
2018-09-28 14:12:40 UTC
Permalink
I have a kafka with kerberos security and trying to use the ACL and am not able to make it work.

Here is the error I am seeing in the server log.[2018-09-28 14:06:54,152] INFO Principal = User:storm-mytestcluster is Denied Operation = Describe from host = <ip address of host> on resource = Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,312] INFO Principal = User:storm-mytestcluster is Denied Operation = Describe from host = <ip address of host> on resource = Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,472] INFO Principal = User:storm-mytestcluster is Denied Operation = Describe from host = <ip address of host> on resource = Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,631] INFO Principal = User:storm-mytestcluster is Denied Operation = Describe from host = <ip address of host> on resource = Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,793] INFO Principal = User:storm-mytestcluster is Denied Operation = Describe from host = <ip address of host> on resource = Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,953] INFO Principal = User:storm-mytestcluster is Denied Operation = Describe from host = <ip address of host> on resource = Topic:icd_alpha (kafka.authorizer.logger)



But the user has full access to the topic: Here is the output of `list ` command

Current ACLs for resource `Topic:icd_alpha`:
     user:storm-mytestcluster has Allow permission for operations: All from hosts: *

Please help me, as I am kind of blocked and don't know how to proceed further.
ThanksBala
Vahid Hashemian
2018-09-28 15:56:08 UTC
Permalink
Hi Bala,

What operation/command are you trying that gives you this error?

--Vahid
Post by Bala
I have a kafka with kerberos security and trying to use the ACL and am not
able to make it work.
Here is the error I am seeing in the server log.[2018-09-28 14:06:54,152]
INFO Principal = User:storm-mytestcluster is Denied Operation = Describe
from host = <ip address of host> on resource = Topic:icd_alpha
(kafka.authorizer.logger)
[2018-09-28 14:06:54,312] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource =
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,472] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource =
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,631] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource =
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,793] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource =
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,953] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource =
Topic:icd_alpha (kafka.authorizer.logger)
But the user has full access to the topic: Here is the output of `list ` command
user:storm-mytestcluster has Allow permission for operations: All
from hosts: *
Please help me, as I am kind of blocked and don't know how to proceed further.
ThanksBala
Bala
2018-09-28 16:13:29 UTC
Permalink
Producer using the Java API. I did configure the jaas config as per docs. It looks like is working and the authentication is succeeded but the authorization is not honoring the ACL
On Friday, September 28, 2018, 11:56:24 AM EDT, Vahid Hashemian <***@gmail.com> wrote:

Hi Bala,

What operation/command are you trying that gives you this error?

--Vahid
Post by Bala
I have a kafka with kerberos security and trying to use the ACL and am not
able to make it work.
Here is the error I am seeing in the server log.[2018-09-28 14:06:54,152]
INFO Principal = User:storm-mytestcluster is Denied Operation = Describe
from host = <ip address of host> on resource = Topic:icd_alpha
(kafka.authorizer.logger)
[2018-09-28 14:06:54,312] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource =
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,472] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource =
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,631] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource =
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,793] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource =
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,953] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource =
Topic:icd_alpha (kafka.authorizer.logger)
But the user has full access to the topic: Here is the output of `list ` command
      user:storm-mytestcluster has Allow permission for operations: All
from hosts: *
Please help me, as I am kind of blocked and don't know how to proceed further.
ThanksBala
Vahid Hashemian
2018-09-29 00:14:06 UTC
Permalink
Your produce needs to have Write access to the topic. But as you mentioned
All should cover Write. Which version of Kafka are you using?
FYI, more authn/authz information can be found here for some of the common
client operations:
https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/

--Vahid
Post by Bala
Producer using the Java API. I did configure the jaas config as per docs.
It looks like is working and the authentication is succeeded but the
authorization is not honoring the ACL
On Friday, September 28, 2018, 11:56:24 AM EDT, Vahid Hashemian <
Hi Bala,
What operation/command are you trying that gives you this error?
--Vahid
Post by Bala
I have a kafka with kerberos security and trying to use the ACL and am
not
Post by Bala
able to make it work.
Here is the error I am seeing in the server log.[2018-09-28 14:06:54,152]
INFO Principal = User:storm-mytestcluster is Denied Operation = Describe
from host = <ip address of host> on resource = Topic:icd_alpha
(kafka.authorizer.logger)
[2018-09-28 14:06:54,312] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,472] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,631] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,793] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,953] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
But the user has full access to the topic: Here is the output of `list ` command
user:storm-mytestcluster has Allow permission for operations: All
from hosts: *
Please help me, as I am kind of blocked and don't know how to proceed further.
ThanksBala
Bala
2018-09-29 15:34:28 UTC
Permalink
Yes, I followed all the docs and thing is it's connecting as expected and sending the right user to the server. The problem is that when I list ACL's I am seeing that the user has all the access. But when the producer trying to connect to the server, the server is logging message saying that the user is denied. Something is not right and I want to know how to debug this further to find out why the server thinks the user has no access.

I even tried adding the user to the super users and still not able to access the topic.
I am using Kafka 1.0.0.
On Friday, September 28, 2018, 8:14:30 PM EDT, Vahid Hashemian <***@gmail.com> wrote:

Your produce needs to have Write access to the topic. But as you mentioned
All should cover Write. Which version of Kafka are you using?
FYI, more authn/authz information can be found here for some of the common
client operations:
https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/

--Vahid
Post by Bala
Producer using the Java API. I did configure the jaas config as per docs.
It looks like is working and the authentication is succeeded but the
authorization is not honoring the ACL
    On Friday, September 28, 2018, 11:56:24 AM EDT, Vahid Hashemian <
  Hi Bala,
What operation/command are you trying that gives you this error?
--Vahid
Post by Bala
I have a kafka with kerberos security and trying to use the ACL and am
not
Post by Bala
able to make it work.
Here is the error I am seeing in the server log.[2018-09-28 14:06:54,152]
INFO Principal = User:storm-mytestcluster is Denied Operation = Describe
from host = <ip address of host> on resource = Topic:icd_alpha
(kafka.authorizer.logger)
[2018-09-28 14:06:54,312] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,472] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,631] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,793] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,953] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on resource
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
But the user has full access to the topic: Here is the output of `list ` command
      user:storm-mytestcluster has Allow permission for operations: All
from hosts: *
Please help me, as I am kind of blocked and don't know how to proceed further.
ThanksBala
Manikumar
2018-09-29 15:42:49 UTC
Permalink
PrincipalType string "User" is case sensitive. Try creating acls for
"User:storm-mytestcluster" principal.
Post by Bala
Yes, I followed all the docs and thing is it's connecting as expected and
sending the right user to the server. The problem is that when I list ACL's
I am seeing that the user has all the access. But when the producer trying
to connect to the server, the server is logging message saying that the
user is denied. Something is not right and I want to know how to debug this
further to find out why the server thinks the user has no access.
I even tried adding the user to the super users and still not able to access the topic.
I am using Kafka 1.0.0.
On Friday, September 28, 2018, 8:14:30 PM EDT, Vahid Hashemian <
Your produce needs to have Write access to the topic. But as you mentioned
All should cover Write. Which version of Kafka are you using?
FYI, more authn/authz information can be found here for some of the common
https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/
--Vahid
Post by Bala
Producer using the Java API. I did configure the jaas config as per docs.
It looks like is working and the authentication is succeeded but the
authorization is not honoring the ACL
On Friday, September 28, 2018, 11:56:24 AM EDT, Vahid Hashemian <
Hi Bala,
What operation/command are you trying that gives you this error?
--Vahid
Post by Bala
I have a kafka with kerberos security and trying to use the ACL and am
not
Post by Bala
able to make it work.
Here is the error I am seeing in the server log.[2018-09-28
14:06:54,152]
Post by Bala
Post by Bala
INFO Principal = User:storm-mytestcluster is Denied Operation =
Describe
Post by Bala
Post by Bala
from host = <ip address of host> on resource = Topic:icd_alpha
(kafka.authorizer.logger)
[2018-09-28 14:06:54,312] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on
resource
Post by Bala
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,472] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on
resource
Post by Bala
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,631] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on
resource
Post by Bala
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,793] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on
resource
Post by Bala
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,953] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on
resource
Post by Bala
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
But the user has full access to the topic: Here is the output of `list
`
Post by Bala
Post by Bala
command
user:storm-mytestcluster has Allow permission for operations: All
from hosts: *
Please help me, as I am kind of blocked and don't know how to proceed further.
ThanksBala
Bala
2018-09-29 16:06:06 UTC
Permalink
Wow, good catch. I am using the HDP distribution and when I looked at the config for the kafka, the super user is set as `user:kafka`. I think this deserves a explicit mention in the docs about using the upper case "User".
Thanks for the help. Now I am unblocked on this issue, am getting a different error "Error while fetching metadata with correlation id 2 : {mytopic=LEADER_NOT_AVAILABLE}". Will look into that
ThanksBala
On Saturday, September 29, 2018, 11:43:13 AM EDT, Manikumar <***@gmail.com> wrote:

PrincipalType  string "User" is case sensitive. Try creating acls for
"User:storm-mytestcluster" principal.
  Yes, I followed all the docs and thing is it's connecting as expected and
sending the right user to the server. The problem is that when I list ACL's
I am seeing that the user has all the access. But when the producer trying
to connect to the server, the server is logging message saying that the
user is denied. Something is not right and I want to know how to debug this
further to find out why the server thinks the user has no access.
I even tried adding the user to the super users and still not able to access the topic.
I am using Kafka 1.0.0.
    On Friday, September 28, 2018, 8:14:30 PM EDT, Vahid Hashemian <
  Your produce needs to have Write access to the topic. But as you mentioned
All should cover Write. Which version of Kafka are you using?
FYI, more authn/authz information can be found here for some of the common
https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/
--Vahid
Post by Bala
Producer using the Java API. I did configure the jaas config as per docs.
It looks like is working and the authentication is succeeded but the
authorization is not honoring the ACL
    On Friday, September 28, 2018, 11:56:24 AM EDT, Vahid Hashemian <
  Hi Bala,
What operation/command are you trying that gives you this error?
--Vahid
Post by Bala
I have a kafka with kerberos security and trying to use the ACL and am
not
Post by Bala
able to make it work.
Here is the error I am seeing in the server log.[2018-09-28
14:06:54,152]
Post by Bala
Post by Bala
INFO Principal = User:storm-mytestcluster is Denied Operation =
Describe
Post by Bala
Post by Bala
from host = <ip address of host> on resource = Topic:icd_alpha
(kafka.authorizer.logger)
[2018-09-28 14:06:54,312] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on
resource
Post by Bala
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,472] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on
resource
Post by Bala
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,631] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on
resource
Post by Bala
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,793] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on
resource
Post by Bala
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
[2018-09-28 14:06:54,953] INFO Principal = User:storm-mytestcluster is
Denied Operation = Describe from host = <ip address of host> on
resource
Post by Bala
=
Post by Bala
Topic:icd_alpha (kafka.authorizer.logger)
But the user has full access to the topic: Here is the output of `list
`
Post by Bala
Post by Bala
command
      user:storm-mytestcluster has Allow permission for operations: All
from hosts: *
Please help me, as I am kind of blocked and don't know how to proceed further.
ThanksBala
Loading...