Skip to Main Content

Kafka SASL Authentication: Usage & Best Practices

Discover the robust security mechanisms provided by SASL authentication for Apache Kafka clusters in this comprehensive guide. Learn about the different SASL mechanisms supported, their configuration, implementation, and best practices to ensure secure communication between Kafka clients and brokers.

Kafka SASL Authentication: Usage & Best Practices

Overview

SASL (Simple Authentication and Security Layer) authentication provides robust security mechanisms for Apache Kafka clusters, enabling secure communication between clients and brokers. This comprehensive guide explores SASL authentication in Kafka, including its mechanisms, implementation details, configuration options, and best practices for production environments.

Understanding SASL Authentication in Kafka

SASL is a framework that provides authentication and data security in network protocols. In Kafka, SASL is used to authenticate clients (producers and consumers) and brokers using various mechanisms. Each mechanism offers different security features and complexity levels, allowing organizations to choose the one that best fits their requirements.

Key Concepts

KafkaPrincipal represents the identity of a user or service interacting with the Kafka cluster. When clients attempt to connect, they present their KafkaPrincipal, which Kafka verifies before allowing access to resources[16]. This principal is then used for subsequent authorization checks through Access Control Lists (ACLs).

SASL vs. Other Authentication Methods

Kafka supports multiple authentication methods:

SASL Mechanisms Supported by Kafka

Kafka supports several SASL mechanisms, each with distinct characteristics:

SASL Authentication Mechanisms in Detail

SASL/PLAIN

SASL/PLAIN is a simple username/password authentication mechanism. While straightforward to implement, it transmits credentials in plaintext, making it vulnerable if not used with TLS encryption[7].

PLAIN should not be confused with PLAINTEXT, which refers to the absence of transport encryption. Configuration parameters such as sasl.enabled.mechanisms may be set to use the SASL mechanism PLAIN, whereas parameters like security.inter.broker.protocol may be configured to use SASL_PLAINTEXT (SASL authentication without encryption) or SASL_SSL (SASL authentication with TLS encryption)[7].

SASL/SCRAM

SCRAM (Salted Challenge Response Authentication Mechanism) addresses security concerns with traditional mechanisms like PLAIN by:

  1. Protecting against password sniffing on networks

  2. Preventing dictionary attacks on password files

  3. Storing authentication information in salted form to protect against database compromises[18]

Confluent Platform supports both SCRAM-SHA-256 and SCRAM-SHA-512 variants, storing credentials in KRaft or ZooKeeper[18].

SASL/GSSAPI (Kerberos)

GSSAPI with Kerberos provides ticket-based authentication, eliminating the need to transmit passwords. It requires a functioning Kerberos infrastructure and is more complex to set up but offers strong security guarantees[6].

SASL/OAUTHBEARER

OAUTHBEARER leverages OAuth tokens for authentication, allowing integration with external identity providers. Users must provide custom code to acquire and verify credentials[19].

How SASL Authentication Works in Kafka

The SASL authentication process follows these general steps:

  1. Client initiates connection to Kafka broker

  2. Broker responds with supported SASL mechanisms

  3. Client selects a mechanism and begins authentication handshake

  4. Credentials are exchanged according to the mechanism's protocol

  5. Broker verifies credentials and either allows or denies the connection

  6. If successful, the client's KafkaPrincipal is used for subsequent authorization

Configuring SASL Authentication

JAAS Configuration

Kafka uses the Java Authentication and Authorization Service (JAAS) for SASL configuration. There are two approaches to configuring JAAS:

  1. Using the sasl.jaas.config property (recommended)

  2. Passing a JAAS configuration file via the java.security.auth.login.config system property[17]

For brokers, JAAS configuration should be prefixed with the listener name and SASL mechanism:


listener.name.<listenerName>.<saslMechanism>.sasl.jaas.config

Broker Configuration

The following example shows a broker configuration for SASL/PLAIN:


# Enable SASL mechanisms
sasl.enabled.mechanisms=PLAIN

# Configure security protocol
listeners=SASL_SSL://hostname:9093
advertised.listeners=SASL_SSL://hostname:9093
security.inter.broker.protocol=SASL_SSL

# Set mechanism for inter-broker communication
sasl.mechanism.inter.broker.protocol=PLAIN

# JAAS configuration for the listener
listener.name.sasl_ssl.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="admin" \
password="admin-secret" \
user_admin="admin-secret" \
user_alice="alice-secret";


Client Configuration

For clients, you can embed JAAS configuration directly in the properties:


bootstrap.servers=hostname:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="alice" \
password="alice-secret";


Enabling Multiple SASL Mechanisms

Kafka brokers can support multiple SASL mechanisms simultaneously, while each client must choose one. Configure each mechanism with its own JAAS configuration[17]:


sasl.enabled.mechanisms=SCRAM-SHA-512,GSSAPI

listener.name.sasl_ssl.gssapi.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required \
useKeyTab=true \
storeKey=true \
keyTab="/var/lib/secret/kafka.key" \
principal="kafka/kafka.host@REALM";

listener.name.sasl_ssl.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
username="admin" \
password="admin-secret";


SASL with KRaft Mode

KRaft mode allows running Kafka without ZooKeeper. While SASL authentication can be used with KRaft, there are some considerations:

  1. KRaft-backed clusters cannot use SCRAM for controller-to-controller authentication, though Confluent Server brokers can use SCRAM to authenticate to controllers and other brokers[18]

  2. SASL credentials should be created before brokers are running

  3. For KRaft with SASL/PLAIN, you need the configuration property sasl.mechanism.controller.protocol=PLAIN [1]

Best Practices for SASL Authentication

Security Recommendations

  1. Always use TLS with SASL to encrypt credentials in transit

  2. For production environments, prefer SASL/SCRAM or SASL/GSSAPI over SASL/PLAIN[19]

  3. Implement proper credential management and rotation procedures

  4. Separate quorum members from brokers in KRaft mode for better fault tolerance[4]

  5. Configure ACLs to restrict access to sensitive topics and operations[2]

Mechanism Selection

Choose your SASL mechanism based on your existing infrastructure and security requirements:

Avoiding Common Issues

  1. Always use TLS with SASL/PLAIN to prevent credential exposure

  2. Ensure the correct JAAS configuration for each listener and mechanism

  3. When using KRaft mode, ensure you've set super.users correctly to allow broker-to-controller communication[2]

  4. Verify that client configurations match broker configurations for the selected mechanism

Common Issues and Troubleshooting

SaslAuthenticationException

This error indicates authentication failure, typically due to incorrect credentials or misconfiguration[12]. Check:

  • Username and password correctness

  • SASL mechanism configuration

  • JAAS configuration

SSL handshake failed

This occurs when TLS is misconfigured. Ensure:

  • Correct TLS certificates are in place

  • Client and broker truststores/keystores are properly configured

  • The client is connecting to the correct port[14]

Could not find KafkaServer entry in JAAS configuration

In KRaft mode, this indicates JAAS configuration issues. Ensure:

  • Proper JAAS configuration for controllers

  • Setting sasl.mechanism.controller.protocol=PLAIN for SASL/PLAIN[1]

Unexpected Kafka request of type metadata during sasl handshake

This error suggests the client is not configured for SASL authentication while the server expects it[5]. Verify client configuration matches server expectations.

Conclusion

SASL authentication provides flexible security options for Kafka deployments, from simple username/password authentication to more sophisticated mechanisms like SCRAM and Kerberos. By following the configuration guidelines and best practices outlined in this guide, you can secure your Kafka cluster while meeting your organization's specific security requirements.

Remember that authentication is just one aspect of a comprehensive security strategy for Kafka. Consider combining SASL authentication with TLS encryption, authorization through ACLs, and proper network security measures to create a robust security posture for your Kafka deployment.

If you find this content helpful, you might also be interested in our product AutoMQ. AutoMQ is a cloud-native alternative to Kafka by decoupling durability to S3 and EBS. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. AutoMQ now is source code available on github. Big Companies Worldwide are Using AutoMQ. Check the following case studies to learn more:

References:

  1. Kafka KRaft SASL Plaintext

  2. How can I start Kafka with user anonymous?

  3. Kafka topics not showing up with the command but they exist

  4. Kafka cluster without Zookeeper

  5. I have did the setup Kafka SASL/Kerberos on a cluster

  6. Configuring SASL authentications in Apache Kafka

  7. SASL/PLAIN Authentication

  8. Kafka KRaft Authentication

  9. Configure Authentication for Kafka

  10. Gain visibility into your Amazon MSK cluster by deploying the Conduktor Platform

  11. Authentication in Redpanda

  12. Kafka Topic SaslAuthenticationException

  13. Kafka Connection Configuration

  14. Getting SSL handshake failed when creating Kafka producer and consumer

  15. Apache Kafka SASL Authentication Sequence

  16. Kafka Authentication Guide

  17. Authentication Using SASL

  18. SASL/SCRAM Authentication

  19. Secure Kafka Deployment Best Practices

  20. Issue with Azure Function (Java) and Kafka Trigger

  21. Kafkatopical: The Kafka UI for Engineers and Admins

  22. Yozefu: A TUI for exploring data of a Kafka cluster

  23. What are the most frustrating parts of Kafka?

  24. MSK Topic Level Security

  25. Apache Kafka SASL SSL Configure Failure

  26. What tools do you use for visualizing Kafka?

  27. Read/Write Kafka with SQL and Proton

  28. Integrating Kafdrop with Kafka

  29. Unable to connect to broker using TLS librdkafka

  30. Confluent local Kafka start doesn't work

  31. Beat Connection Error: Error while reading from

  32. Kafka-acls CLI Error with Confluent Cloud Instance

  33. Help Please! Not able to connect to Kafka on AWS

  34. Authentication Using SASL

  35. Azure Kafka OAuth Bearer

  36. Authentication in Kubernetes

  37. Issue with SASL mechanism in Kafka

  38. Kafka Authentication

  39. SASL Authentication Overview

  40. Connecting to a Secure Kafka

  41. Redpanda Issue #8095

  42. SASL/PLAIN for client connection configuration problems

  43. Essential Kafka Security Best Practices for 2024

  44. Kafka Security Tutorial

  45. Kafka Security Implementation

  46. Cloud Authentication in Redpanda

  47. Spring Native Issue #1416

  48. Reddit User Golden-Trash_Number

  49. AWS MSK: Understanding Kafka Clusters Behavior

  50. Connecting to AWS MSK from Rails Application

  51. Is anyone exposing Kafka publicly?

  52. Authentication and Authorization in Microservices

  53. ServiceNow vs QRadar and Apache Kafka

  54. Go Zookeeper Client

  55. How do you identify producers writing to Kafka?

  56. Kafka client metrics through JMX configuration

  57. Kafka SASL Plaintext Authentication Setup

  58. Apache Kafka Authentication using SASL

  59. Plain SASL set up is not working

  60. Securing Kafka

  61. SASL Authentication in Kafka

  62. Kafka SASL SCRAM Failed Authentication

  63. Authentication Basics

  64. Kafka Deployment Guide

  65. librdkafka Issue #2959

  66. Best Practices for Securing Kafka Real-Time Data

  67. Introduction to Kafka Topics

  68. Kafka Security

  69. Best Practices for Gateway Cases

  70. Kafka Authentication

  71. What is the best way to download and install?

  72. Kafka Authentication Issue

  73. Kowl: Open Source WebUI for Kafka

  74. Apache Kafka Documentation

  75. Apache Kafka 2.4 Documentation

  76. Apache Kafka Troubleshooting Guide

  77. Unable to connect to Kafka with SASL SSL SCRAM

  78. MSK SASL SCRAM Issues

  79. Getting Started Apache Kafka .NET

  80. How can I configure Kafka SSL with NestJS?

  81. Best Practices for Kafka Security

  82. Apache Kafka Security Best Practices

  83. Unexpected Kafka request of type METADATA during SASL handshake

  84. New release of FastKafka improves testability

  85. Apache Kafka Concepts: SASL Authentication Sequence Explained

  86. Kafka Authentication and Authorization

  87. Exploring Apache Kafka Internals and Codebase

  88. Recommended fairly new courses for Kafka

  89. Apache Kafka 3.9 Documentation

  90. Configuring Connection to Apache Kafka with SASL

  91. KafkaJS Configuration Documentation

  92. Red Hat Streams for Apache Kafka: Configuration Properties

  93. Using AMQ Streams on RHEL: Configuring Kafka

  94. Kafka Exporter: SASL Authentication Issue

  95. Apache Kafka 0.10.0.0 Release Notes

  96. librdkafka: SASL PLAIN Authentication Issue

  97. StreamSets: HTTP to Kafka Origin

  98. How to Set Up SASL Authentication with Kafka

  99. Kafka Producer - SASL Authentication

  100. SpiceAI Community

  101. MSK Tutorial Troubleshooting

  102. Creating Users and ACLs in MSK

  103. Oracle: Configurations for SASL Authentication

  104. Kafka SASL Configuration for Clients Only

  105. Multiplatform Kafka Desktop Client Release

  106. Kafka CLI Usage Best Practices

  107. Kafka Clients Usage Best Practices

  108. Setting Up a Connection to Kafka in Conduktor

  109. Kafka Security Manager

  110. MirrorMaker2 Consumer Groups Replication Issue

  111. AutoMQ SASL Security Authentication Guide

  112. Kafka Broker SASL Connection Issues with Zookeeper

  113. AWS Lambda Kafka Troubleshooting Guide

  114. How to Backup Kafka Clusters

  115. Identifying Kafka Clients with Fetch Leaks

  116. Best Practices for Kafka Data Security

  117. Apache Kafka Security Best Practices

  118. Unexpected Metadata Request During SASL Handshake

  119. Kafka Authentication: SSL and SASL/SSL

  120. Apache Kafka Protocol Guide

  121. Aiven Kafka SASL Authentication Guide

  122. How to Enable SASL Mechanism Locally

  123. Troubleshooting Topic Message Visibility

  124. Red Hat Kafka Connect Configuration Properties

  125. Double Cloud: Connecting to Kafka Cluster

  126. Understanding Kafka SASL Authentication

  127. Troubleshooting Slow SASL Handshake