Skip to Main Content

Kafka Docker: Setup Guide & Best Practices

Overview

Deploying Apache Kafka using Docker offers a streamlined way to manage and scale Kafka clusters. This guide covers the setup process, best practices, and common configurations for running Kafka on Docker using both Bitnami and Apache Kafka Docker images.

Prerequisites

Before setting up Kafka on Docker, ensure you have:

  • Docker Engine installed on your system.

  • Docker Compose for managing multiple containers.

  • A basic understanding of Docker concepts (containers, images, volumes).

Setting Up Kafka on Docker

Using Bitnami Kafka Image

In this example, we will create an Apache Kafka client instance that will connect to the server instance that is running on the same docker network as the client.

Step 1: Create a network


docker network create app-tier --driver bridge

Step 2: Launch the Apache Kafka server instance

Use the --network app-tier argument to the docker run command to attach the Apache Kafka container to the app-tier network.


docker run -d --name kafka-server --hostname kafka-server \
--network app-tier \
-e KAFKA_CFG_NODE_ID=0 \
-e KAFKA_CFG_PROCESS_ROLES=controller,broker \
-e KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \
-e KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \
-e KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka-server:9093 \
-e KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER \
bitnami/kafka:latest

Step 3: Launch your Apache Kafka client instance

Finally we create a new container instance to launch the Apache Kafka client and connect to the server created in the previous step:


docker run -it --rm \
--network app-tier \
bitnami/kafka:latest kafka-topics.sh --list --bootstrap-server kafka-server:9092

Using Apache Kafka Image

Apache Kafka provides official Docker images for Kafka and ZooKeeper. Here's how to set up a Kafka cluster using these images.

Start a Kafka broker:


docker run -d --name broker apache/kafka:latest

Open a shell in the broker container:


docker exec --workdir /opt/kafka/bin/ -it broker sh

A topic is a logical grouping of events in Kafka. From inside the container, create a topic called test-topic :


./kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test-topic

Write two string events into the test-topic topic using the console producer that ships with Kafka:


./kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test-topic

This command will wait for input at a > prompt. Enter hello , press Enter , then world , and press Enter again. Enter Ctrl\+C to exit the console producer.

Now read the events in the test-topic topic from the beginning of the log:


./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning

You will see the two strings that you previously produced:


hello
world

The consumer will continue to run until you exit out of it by entering Ctrl\+C .

When you are finished, stop and remove the container by running the following command on your host machine:


docker rm -f broker

Best Practices for Kafka on Docker

  1. Resource Management

Ensure your host machine has sufficient resources. For production environments, consider at least 16GB RAM and 4 CPU cores .

  1. Persistence Configuration

Use Docker volumes for persistent storage to avoid data loss when containers restart:


volumes:
- ./kafka-data:/var/lib/kafka/data


  1. Network Configuration

Configure listeners correctly to enable external access:


KAFKA_ADVERTISED_HOST_NAME: your-host-ip
KAFKA_ADVERTISED_PORT: 9092


  1. Security

Implement security measures such as SSL/TLS encryption and SASL authentication:


KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,SSL:SSL


  1. Monitoring and Logging

volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml


  1. Scalability

For multi-broker setups, ensure unique hostnames and ports:


kafka1:
...
hostname: kafka1
ports:
- "9092:9092"

kafka2:
...
hostname: kafka2
ports:
- "9093:9093"


Conclusion

Deploying Kafka on Docker provides a flexible and scalable way to manage messaging systems. By following best practices for resource management, persistence, network configuration, security, and monitoring, you can ensure a robust Kafka setup. For complex environments, consider using Redpanda or Confluent's Kubernetes operator for enhanced performance and automation. Always monitor your Kafka cluster's performance and adjust configurations as needed to maintain optimal throughput and latency.

If you find this content helpful, you might also be interested in our product AutoMQ. AutoMQ is a cloud-native alternative to Kafka by decoupling durability to S3 and EBS. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. AutoMQ now is source code available on github. Big Companies Worldwide are Using AutoMQ. Check the following case studies to learn more:

References

  1. Bitnami Kafka Docker Container Guide

  2. Kafka Docker Setup Guide - Baeldung

  3. Apache Kafka Docker Examples

  4. Bitnami Kafka Docker Setup Guide - UnoGeeks

  5. How to Deploy Kafka on Docker - Hostinger Tutorial

  6. Top 10 Security Practices for Confluent Kafka with AWS ECS

  7. Kafka Monitoring Guide - OpenObserve

  8. Confluent Platform Docker Quick Start Guide

  9. Getting Started with Conduktor Platform on Docker

  10. Redpanda Quick Start Guide

  11. Confluent Platform Docker Development Guide

  12. Simplifying Kafka Development with Redpanda and Testcontainers

  13. Setting Up Kafka & Zookeeper Locally Using Docker

  14. Kafka 101: Getting Started with Confluent Kafka

  15. Conduktor Kafka Stack Docker Compose Guide

  16. Running Bitnami Kafka in Docker

  17. Official Docker Guide for Kafka

  18. Bitnami Kafka Cluster Docker Compose Example

  19. Apache Kafka Official Docker Image

  20. Bitnami Kafka Container Solutions

  21. Apache Kafka Quick Start Guide

  22. Docker部署Kafka教程

  23. Conduktor Kafka Stack Docker Compose Repository

  24. Bitnami Kafka Docker Image

  25. How to Start Kafka Using Docker - Conduktor Guide

  26. Docker Compose CLI Reference

  27. Creating Kafka Topics with Docker Compose - Baeldung

  28. Confluent Platform Docker Security Guide

  29. Confluent Platform Docker Monitoring Guide

  30. Best Practices for Running Kafka on Docker Containers

  31. Apache Kafka Security Best Practices

  32. Confluent Platform 7.0 Docker Monitoring Guide

  33. Kafka Rack Configuration in Bitnami Docker Image

  34. Kafka Docker Security ACLs Examples

  35. Complete Guide to Kafka Monitoring

  36. Running Apache Kafka on Containers

  37. Docker Container Kafka Local Connection Guide

  38. Redpanda and Tyk Integration Guide

  39. Confluent Kafka Docker Parameters Guide

  40. When to Choose Redpanda Instead of Apache Kafka

  41. Getting Confluent Kafka Working with Docker Desktop on Linux

  42. Apache Kafka Docker Setup Tutorial

  43. AutoMQ Kafka UI Integration with Redpanda Console

  44. Docker Compose Kafka Setup with Confluent Cloud

  45. Kafka 安装部署指南

  46. Confluent Developer Tutorial: Kafka on Docker

  47. Kafka Docker Compose Setup Guide

  48. How to Use Kafka with Docker Compose

  49. Confluent Platform 7.3 Docker Security Guide

  50. Monitoring Kafka in Docker with Prometheus

  51. QuestDB Integration with Redpanda