Skip to Main Content

Deploy Multi-Nodes Test Cluster on Docker

This document explains how to deploy a three-node AutoMQ cluster in a local development environment (by deploying MinIO to provide object storage services). In this setup, users can test cluster-related features such as partition reassignment and data auto-balancing.

Besides the Docker deployment option, users can reference the following documentation for alternative deployment methods to verify cluster-related features:

Deploying AutoMQ and tuning parameters for production environment loads can be complex. For necessary assistance and best practices, you can contact the AutoMQ team through this form.

Additionally, if you'd like to completely bypass the installation and deployment process, you can try the fully managed cloud service provided by the AutoMQ team through the link below. A free two-week trial is currently available in all cloud marketplaces.

Prerequisites​

  • Linux/Mac/Windows Subsystem for Linux

  • Docker

  • Docker Compose version > 2.22.0

  • At least 8GB of available memory

Install and Launch a Three-node AutoMQ Cluster​

Use the following command to download the Docker Compose configuration file for the single-node deployment plan and start the single-node AutoMQ cluster using Docker Compose.


curl -O https://raw.githubusercontent.com/AutoMQ/automq/refs/tags/1.5.0/docker/docker-compose-cluster.yaml && docker compose -f docker-compose-cluster.yaml up -d

After a successful startup, the cluster address will be exposed under the Docker network named automq_net, and the cluster's bootstrap address will be "server1:9093,server2:9093,server3:9093." You can connect to the cluster using this address subsequently.

Note:

In the above test environment, AutoMQ utilizes a Docker network named automq_net. To connect to the cluster, you need to configure the --network automq_net option to use the same Docker network for the connection.

Test Message Sending and Receiving.​

After installing and starting the AutoMQ cluster, you can test message sending, consumption, and other functionalities by running the following commands.

  1. Execute the following command to create a Topic using the Kafka CLI tool kafka-topics.sh within a Docker environment.

docker run --network automq_net automqinc/automq:1.5.0 /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server server1:9092,server2:9092,server3:9092"

  1. Execute the following command to send test messages using the Kafka CLI tool kafka-console-producer.sh within a Docker environment.

docker run -it --network automq_net automqinc/automq:1.5.0 /bin/bash -c "/opt/kafka/kafka/bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server server1:9092,server2:9092,server3:9092"

  1. Execute the following command to consume test messages using the Kafka CLI tool kafka-console-consumer.sh within a Docker environment.

docker run --network automq_net automqinc/automq:1.5.0 /bin/bash -c "/opt/kafka/kafka/bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server server1:9092,server2:9092,server3:9092"

Explore the Features of the AutoMQ Cluster.​

In addition to core message production and consumption features, AutoMQ offers advanced capabilities like second-level partition reassignment and automatic data rebalancing, which address Apache Kafka's challenges with slow reassignment and scaling. You can refer to the following documentation for testing and validation:

Stop and Remove a Three-node AutoMQ Cluster.​

After completing the functional test, please run the following command to stop and uninstall the current AutoMQ cluster.


curl -O https://raw.githubusercontent.com/AutoMQ/automq/refs/tags/1.5.0/docker/docker-compose-cluster.yaml && docker compose -f docker-compose-cluster.yaml down -v