Skip to Main Content

Deploy Multi-Nodes Test Cluster on Docker

This document explains how to deploy a three-node AutoMQ cluster in a local development environment by using MinIO for object storage services. In this setup, users can test cluster-related features such as partition reassignment and data auto-balancing.

Besides the Docker deployment option, users can check the following documents to explore other deployment methods to verify cluster-related functionalities:

Tip

Deploying and tuning AutoMQ for production workloads can be quite complex. You can reach out to the AutoMQ team through this form to receive necessary support and best practice guidelines.

Moreover, if you prefer to skip the installation and deployment hassle, you can try the fully managed cloud service offered by the AutoMQ team through the link below. Currently, all cloud marketplaces offer a free two-week trial.

Prerequisites​

  • Linux/Mac/Windows Subsystem for Linux

  • Docker

  • Docker Compose version > 2.22.0

  • More than 8GB of available memory

Install and Launch a Three-node AutoMQ Cluster​

Enter the command below to download the Docker Compose configuration file for the single-node deployment plan, and start a single-node AutoMQ cluster using Docker Compose.


curl -O https://raw.githubusercontent.com/AutoMQ/automq/refs/heads/main/docker/docker-compose-cluster.yaml && docker compose -f docker-compose-cluster.yaml up -d

Once successfully started, the cluster address will be exposed within a Docker network named automq_net, with the cluster's bootstrap address being "server1:9093,server2:9093,server3:9093." You can later connect to the cluster using this address.

Tip

Note:

In the test environment mentioned above, AutoMQ uses a Docker network named automq_net. To connect to the cluster, you need to configure the --network automq_net option and use the same Docker network to connect to the cluster.

Test Message Sending and Receiving.​

After installing and starting the AutoMQ cluster, you can test message sending, consumption, and other functionalities using the following commands.

  1. Use the Kafka CLI command kafka-topics.sh within the Docker environment to create a Topic.

docker run --network automq_net automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server server1:9092,server2:9092,server3:9092"

  1. Use the Kafka CLI command kafka-console-producer.sh within the Docker environment to send test messages.

docker run -it --network automq_net automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server server1:9092,server2:9092,server3:9092"

  1. Use the Kafka CLI command kafka-console-consumer.sh within the Docker environment to consume test messages.

docker run --network automq_net automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server server1:9092,server2:9092,server3:9092"

Explore the Functionality of the AutoMQ Cluster.​

In addition to basic message production and consumption functionalities, AutoMQ supports second-level partition reassignment, automatic data rebalancing, and other capabilities, addressing issues in Apache Kafka related to sluggish reassignment and scaling. You can refer to the following documents for testing verification:

Stop and Uninstall Single-node AutoMQ Cluster​

After completing the feature tests, please execute the following commands to stop and uninstall the current AutoMQ cluster.


curl -O https://raw.githubusercontent.com/AutoMQ/automq/refs/heads/main/docker/docker-compose-cluster.yaml && docker compose -f docker-compose-cluster.yaml down -v