Deploy Multi-Nodes Cluster on Linux

This document explains how to deploy a multi-node AutoMQ cluster on Linux hosts using object storage services provided by cloud vendors or S3-compatible storage software. In this development environment, users can validate cluster-related features such as partition reassignment and automatic data balancing.

Apart from the Linux host deployment solution, users can refer to the following documents to explore other deployment options:

• Deploy Multi-Nodes Cluster on Kubernetes▸

Deploying AutoMQ for production loads and parameter tuning is relatively complex. You can contact the AutoMQ team via this form to get necessary assistance and best practices.

Additionally, if you prefer to completely avoid the burden of installation and deployment, you can try the fully managed cloud service provided by the AutoMQ team via the link below. A free two-week trial is currently available from major cloud marketplaces.

• Experience AutoMQ Cloud for free from the AWS Marketplace.

• Experience AutoMQ Cloud from Alibaba Cloud Marketplace.

Prerequisites

This document provides an example for deploying a 5-node AutoMQ cluster, where 3 nodes run both the Controller and Broker, and the remaining 2 nodes run only the Broker.

Ensure the following prerequisites are met:

• Deploy an AutoMQ cluster using 5 Linux hosts. It is recommended to use network-optimized virtual machines with 4 cores and 16GB of memory. Ensure the system disk storage space is at least 20GB, and the data volume disk space is no less than 20GB (required only for deploying Controller nodes). Example configuration is as follows:

  Role	IP	Node ID	System Volume	Metadata Volume
  Controller + Broker	192.168.0.1	0	EBS 20GB	EBS 20GB
  Controller + Broker	192.168.0.2	1	EBS 20GB	EBS 20GB
  Controller + Broker	192.168.0.3	2	EBS 20GB	EBS 20GB
  Broker	192.168.0.4	1000	EBS 20GB	Not Applicable
  Broker	192.168.0.5	1001	EBS 20GB	Not Applicable

• Download the binary installation package for installing AutoMQ. Refer to Software Artifact▸.

• Create two custom-named object storage buckets; this documentation uses automq-data and automq-ops as examples.

• Create an IAM user and generate an Access Key ID and Access Key Secret for it. Then, ensure this IAM user has full read and write permissions on the previously created object storage bucket.

• Install jemalloc to optimize memory allocation. Refer to Getting Started.

AutoMQ relies on the Kafka KRaft component to maintain cluster metadata. Therefore, in a production environment, each cluster requires at least 3 nodes (running both Controller and Broker).

Install and Start the AutoMQ Cluster.

Step 1: Create a Cluster Deployment Project

AutoMQ provides the tool automq-cli.sh for operating and maintaining the AutoMQ cluster. The command automq-cli.sh cluster create [project] will automatically create a cluster configuration template in the current directory at clusters/[project]/topo.yaml.

bin/automq-cli.sh cluster create poc

The execution result is as follows:

Success create AutoMQ cluster project: poc
========================================================
Please follow the steps to deploy AutoMQ cluster:
1. Modify the cluster topology config clusters/poc/topo.yaml to fit your needs
2. Run ./bin/automq-cli.sh cluster deploy --dry-run clusters/poc , to deploy the AutoMQ cluster

Step 2: Edit the Cluster Configuration Template

Edit the configuration template created in Step 1 of the cluster setup. An example of the configuration template is as follows:

global:
  clusterId: ''
  # Bucket URI Pattern: 0@s3://$bucket?region=$region&endpoint=$endpoint
  # Bucket URI Example:
  # AWS : 0@s3://xxx_bucket?region=us-east-1
  # OCI: 0@s3://xxx_bucket?region=us-ashburn-1&endpoint=https://xxx_namespace.compat.objectstorage.us-ashburn-1.oraclecloud.com&pathStyle=true
  config: |
    s3.data.buckets=0@s3://xxx_bucket?region=us-east-1
    s3.ops.buckets=1@s3://xxx_bucket?region=us-east-1
    s3.wal.path=0@s3://xxx_bucket?region=us-east-1
    log.dirs=/root/kraft-logs
  envs:
    - name: KAFKA_S3_ACCESS_KEY
      value: 'xxxxx'
    - name: KAFKA_S3_SECRET_KEY
      value: 'xxxxx'
controllers:
  # The Controllers Default Are Combined Nodes Which Roles Are Controller and Broker.
  # The Default Controller Port Is 9093 and the Default Broker Port Is 9092
  - host: 192.168.0.1
    nodeId: 0
  - host: 192.168.0.2
    nodeId: 1
  - host: 192.168.0.3
    nodeId: 2
brokers:
  - host: 192.168.0.5
    nodeId: 1000
  - host: 192.168.0.6
    nodeId: 1001

• global.clusterId : A randomly generated unique ID; no modification is needed.

• global.config: Custom incremental configuration for all nodes in the cluster. Here, you must change s3.data.buckets, s3.ops.buckets, and s3.wal.path to their actual values. Additional configuration items can be added by appending lines. For details regarding object storage parameter settings for different cloud providers, refer to Object Storage Configuration▸.

• global.envs: These are the environment variables for the nodes. Ensure to replace the values for KAFKA_S3_ACCESS_KEY and KAFKA_S3_SECRET_KEY with the actual credentials.

• controllers: This section should list the Controller nodes, which must be replaced with their actual values.

• brokers: This section should list the Broker nodes, also requiring replacement with the actual values.

Step 3: Start AutoMQ

Execute the cluster pre-check command to generate the final startup command.

bin/automq-cli.sh cluster deploy --dry-run clusters/poc

This command will first verify the correctness of the S3 configuration, ensuring S3 can be accessed successfully, and will finally output the startup commands for each node. An example output is as follows:

Host: 192.168.0.1
KAFKA_S3_ACCESS_KEY=xxxx KAFKA_S3_SECRET_KEY=xxxx ./bin/kafka-server-start.sh -daemon config/kraft/server.properties --override cluster.id=JN1cUcdPSeGVnzGyNwF1Rg --override node.id=0 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override advertised.listener=192.168.0.1:9092 --override s3.data.buckets='0@s3://xxx_bucket?region=us-east-1' --override s3.ops.buckets='1@s3://xxx_bucket?region=us-east-1' --override s3.wal.path='0@s3://xxx_bucket?region=us-east-1' --override log.dirs='/root/kraft-logs'

...

To start the cluster, execute the list of commands from the previous step on the designated CONTROLLER or BROKER hosts one by one. For instance, to start the first CONTROLLER process on 192.168.0.1, run the corresponding command for that host from the generated startup command list.

KAFKA_S3_ACCESS_KEY=xxxx KAFKA_S3_SECRET_KEY=xxxx ./bin/kafka-server-start.sh -daemon config/kraft/server.properties --override cluster.id=JN1cUcdPSeGVnzGyNwF1Rg --override node.id=0 --override controller.quorum.voters=0@192.168.0.1:9093,1@192.168.0.2:9093,2@192.168.0.3:9093 --override advertised.listeners=PLAINTEXT://192.168.0.1:9092 --override s3.data.buckets='0@s3://xxx_bucket?region=us-east-1' --override s3.ops.buckets='1@s3://xxx_bucket?region=us-east-1' --override s3.wal.path='0@s3://xxx_bucket?region=us-east-1' --override log.dirs='/root/kraft-logs'

Sending and Receiving Test Messages

After installing and launching the AutoMQ cluster, you can test message sending, consumption, and other functionalities by executing the Kafka CLI commands located in the installation package's bin directory.

1. Run the following command to invoke kafka-topics.sh and create a topic.

bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server 192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092

1. Run the following command to invoke kafka-console-producer.sh to send test messages.

bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server 192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092

1. Run the following command to invoke kafka-console-consumer.sh to consume test messages.

bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server 192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092

Stopping and Uninstalling the AutoMQ Cluster

After completing the tests, you can refer to the following steps to stop and uninstall the AutoMQ cluster.

1. Execute the following command on each node to stop the process.

bin/kafka-server-stop.sh

2. You can configure object storage lifecycle rules to automatically clear the data from s3-data-bucket and s3-ops-bucket before deleting these buckets.

3. Delete the created compute instances along with their corresponding system volumes and data volumes.

4. Delete the test IAM user and its associated Access Key ID and Access Key Secret.

Table Topic is a core feature of AutoMQ designed for modern data lake architectures. Its main function is to seamlessly integrate streaming data with static data lakes, addressing challenges such as traditional stream-batch separation, complex ETL processes, and high costs through architectural innovation. For enabling the Table Topic feature in a cluster, see Overview▸.