Kafka with ACL fails to connect zk and stops

kafkazookeeper

I have a kafka installation (with ssl listener and ssl client authentication). It works fine until I try to add ACL.
this is the error (full error in the bottom):

[edit-1]
the first error on kafka startup is this:

[2018-07-25 14:15:10,498] ERROR Exception while trying to create SASL client: java.lang.ArrayIndexOutOfBoundsException: 0 (org.apache.zookeeper.client.ZooKeeperSaslClient)

below are the configuration and some more info from logs:

[2018-07-25 12:22:27,156] ERROR SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: saslClient failed to initialize properly: it's null. (org.apache.zookeeper.ClientCnxn)

In kafka/conf/server.properties I've added the following:

listeners=SASL_PLAINTEXT://:9092,SSL://:9093

security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN

ssl.keystore.location=/some-path/server.keystore.jks
ssl.keystore.password=password
ssl.key.password=password
ssl.truststore.location=/some-path/server.truststore.jks
ssl.truststore.password=password
ssl.client.auth=required
# without the next 2 lines kafka works fine but with no ACL
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:CN=bob,OU=bob,O=bob,L=bob,ST=bob,C=il
allow.everyone.if.no.acl.found=false
sasl.enabled.mechanisms=PLAIN

zookeeper/conf/zoo.cfg:

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
#requireClientAuthScheme=sasl
quorum.auth.enableSasl=true
quorum.auth.learnerRequireSasl=true
quorum.auth.serverRequireSasl=true
#quorum.auth.learner.loginContext=QuorumLearner
#quorum.auth.server.loginContext=QuorumServer
#quorum.auth.kerberos.servicePrincipal=servicename/_HOST
quorum.cnxn.threads.size=20

There are also jaas files
kafka jaas:

KafkaServer {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="kafkabroker"
   password="kafkabroker-secret";
};

Client {
    org.apache.zookeeper.server.auth.DigestLoginModule required
    username="admin"
    password="admin";
};

zookeeper jaas:

Server {
    org.apache.zookeeper.server.auth.DigestLoginModule required
    username="admin"
    password="admin";
};

full error from kafka log on startup:

[2018-07-25 12:22:27,120] ERROR Exception while trying to create SASL client: java.lang.ArrayIndexOutOfBoundsException: 0 (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2018-07-25 12:22:27,121] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2018-07-25 12:22:27,132] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-07-25 12:22:27,154] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100007b700f0003, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2018-07-25 12:22:27,156] ERROR SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: saslClient failed to initialize properly: it's null. (org.apache.zookeeper.ClientCnxn)
[2018-07-25 12:22:27,161] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientAuthFailedException: Auth failed either before or while waiting for connection
    at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:231)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:250)
    at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:221)
    at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:95)
    at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1548)
    at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:348)
    at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:202)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
    at kafka.Kafka$.main(Kafka.scala:75)
    at kafka.Kafka.main(Kafka.scala)
[2018-07-25 12:22:27,163] INFO shutting down (kafka.server.KafkaServer)
[2018-07-25 12:22:27,165] WARN  (kafka.utils.CoreUtils$)
java.lang.NullPointerException
    at kafka.server.KafkaServer.$anonfun$shutdown$6(KafkaServer.scala:572)
    at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:85)
    at kafka.server.KafkaServer.shutdown(KafkaServer.scala:572)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:329)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
    at kafka.Kafka$.main(Kafka.scala:75)
    at kafka.Kafka.main(Kafka.scala)
[2018-07-25 12:22:27,173] INFO shut down completed (kafka.server.KafkaServer)
[2018-07-25 12:22:27,174] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2018-07-25 12:22:27,177] INFO shutting down (kafka.server.KafkaServer)

Best Answer

jaas file format is not completely clear to me, anyway what eventually worked for me was to use all user/passoword formats for both Client and Server jaas files

Client { org.apache.zookeeper.server.auth.DigestLoginModule required username="admin" password="admin-pass" user_admin="admin"; };

Related Topic