This topic describes how to use Strimzi to deploy a multi-node AutoMQ cluster, where users can verify cluster-related features such as partition migration and automatic data balancing in this Development Environment. In addition to the Strimzi-based Kubernetes deployment solution, users can refer to the following documents to experience other deployment solutions:
Deploying AutoMQ for production environment workloads and tuning its parameters are relatively complex. You can contact the AutoMQ team via the form here to obtain the necessary assistance and best practices.
Additionally, if you wish to completely avoid the workload of installation and deployment, you can experience the full-service Cloud as a Service provided by the AutoMQ team via the following link. Currently, all cloud marketplaces offer a free 2-week trial experience.

​
Prerequisites

This document example is used to deploy a 4-node AutoMQ cluster, where 3 nodes run Controller ( Controller and Broker), and the remaining 1 node runs only Broker. The following conditions need to be checked in advance:
  • Prepare a Kubernetes cluster, ensure it has at least 4 Nodes, and it is recommended to use network-optimized virtual machines with 2 cores and 16GB of memory, on which Pod creation and other operations will be performed later.
  • Ensure that the Helm Chart version is v3.8.0 or higher. Refer to Helm Chart Quick Start.
  • Prepare 2 Object Storage Buckets, one for storing message data and one for storing system logs and Metric data.
Strimzi provides multiple types of Operators. Unless otherwise specified below, all references refer to the Cluster Operator.

​
Deploy Strimzi Operator

​
Step 1: Edit the configuration file

Create an empty strimzi-values.yaml file, which is recommended to be consistent with the example configuration strimzi-values.yaml , and then add additional parameters as needed. Strimzi supports multiple Kafka-related components, and AutoMQ is mainly adapted to the KAFKA component, and currently only supports version 3.9.0.

​
Step 2: Install the Strimzi Operator

Use the configuration file just created and install the Strimzi Operator via helm install .
  • Specify version as 0.47 to ensure stability and avoid unnecessary issues.

helm install automq-strimzi-operator oci://quay.io/strimzi-helm/strimzi-kafka-operator \
  --version 0.47.0 \
  --namespace automq \
  --create-namespace \
  --values strimzi-values.yaml
Waiting for Operator to be ready 

kubectl --namespace automq rollout status deployment strimzi-cluster-operator --watch
When the Operator is ready, it should output something similar to the following: 

deployment "strimzi-cluster-operator" successfully rolled out
View Pod List: 

kubectl get pods -n automq -w       
                                                            
NAME                                       READY   STATUS    RESTARTS   AGE
strimzi-cluster-operator-8dfd94b6d-nb9z6   1/1     Running   0          82s

​
Deploy AutoMQ Cluster

​
Step 1: Edit the configuration file

Create an empty automq-demo.yaml file, edit the file, and add specific parameters. You can refer to the recommended configuration examples for different scenarios under automq-demo.yaml and its subdirectories. For more details, refer to README.md.
  • Replace ${ops-bucket} , ${data-bucket} , ${region} , ${endpoint} with the specific values of Object Storage. For details, see Object Storage Configuration .
  • Replace ${access-key} and ${secret-key} with actual values. You can also choose other authorization methods such as IAM Role.
  • Multi-Available Zone deployment can use the topologySpreadConstraints parameter to ensure that Pods are evenly distributed across specified Available Zones. 

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              strimzi.io/pool-name: controller or broker
  • Avoid cross-Available Zone traffic. The following configuration will obtain the topology.kubernetes.io/zone of the current node label, ultimately achieving the effect of eliminating cross-Available Zone traffic costs.

spec:
  kafka:
    rack:
      topologyKey: topology.kubernetes.io/zone
  • Disable the shrinkage check. Enabling this configuration allows automq to quickly complete shrinkage, and details can be found in the volume expansion and contraction section below.

metadata:
  name: my-cluster
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
    strimzi.io/skip-broker-scaledown-check: "true"
  • It is recommended to deploy AutoMQ exclusive nodes at the production level (to avoid competing for resources such as internet bandwidth with other Pods), and it is advisable to match them through labels for node affinity and tolerations.
  • Other server level parameters can be set as needed in spec.kafka.config . For details, please refer to: Broker and Controller Configuration.

​
Step 2: Install AutoMQ

Install AutoMQ using the custom automq-demo.yaml file 

kubectl apply -f automq-demo.yaml -n automq
Waiting for AutoMQ cluster to be ready: 

kubectl --namespace automq get kafka my-cluster --watch
When the AutoMQ cluster is ready, the output should be similar to the following: 

NAME         READY   METADATA STATE   WARNINGS
my-cluster   True    KRaft            True
View Pod List: 

kubectl get pods -n automq -w   
NAME                                          READY   STATUS    RESTARTS   AGE
my-cluster-broker-1000                        1/1     Running   0          9m28s
my-cluster-controller-0                       1/1     Running   0          9m28s
my-cluster-controller-1                       1/1     Running   0          9m28s
my-cluster-controller-2                       1/1     Running   0          9m28s
my-cluster-entity-operator-6948bd66c4-4962d   2/2     Running   0          103s
strimzi-cluster-operator-678cc9fbc8-x46z4     1/1     Running   0          11m

​
Test message sending and receiving

  • When accessing the Kafka cluster within the k8s cluster, the format of its server address is:

<kafka_cluster_name>-kafka-bootstrap
For example, in our example, the access address is: my-cluster-kafka-bootstrap:9092
For details, see: Strimzi ref list of kafka cluster resources str
Next, you can perform Topic message sending and consumption tests based on kafka-console-producer.sh , kafka-console-consumer.sh .
  • Start the producer and test sending messages (the topic will be automatically created by default):

kubectl -n automq run automq-producer -ti --image=automqinc/automq:1.5.5 --rm=true --restart=Never -- /opt/kafka/kafka/bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic
If you don't see a command prompt, try pressing enter.
>Hi, this is AutoMQ!
  • Start the consumer and test receiving messages:

kubectl -n automq run kafka-consumer -ti --image=automqinc/automq:1.5.5 --rm=true --restart=Never -- /opt/kafka/kafka/bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic --from-beginning
If you don't see a command prompt, try pressing enter.
Hi, this is AutoMQ!

​
Uninstall

  • After completing the test, you can stop and uninstall the Strimzi Operator via helm uninstall .

helm uninstall automq-strimzi-operator --namespace automq
  • Stop and uninstall automq

kubectl delete -f automq-demo.yaml -n automq
  • If historical data is no longer needed, the PVC and Bucket data of the cluster must be deleted together to prevent Dirty Data from affecting the next deployment.

​
Precautions for the production environment

​
Fix the Strimzi Operator version

To avoid unexpected changes during deployment, it is recommended to fix the Strimzi Operator version to 0.47 . This helps:
  • Ensure Compatibility : Ensure that the behavior of the deployed application is consistent with that in testing and is not affected by the release of new versions.
  • Avoid accidental updates : Prevent automatic updates from introducing changes that are incompatible with current deployment or operational practices.

​
Namespace Management

The Strimzi Operator is only responsible for managing AutoMQ clusters created within the namespace where it resides (e.g., automq). The Operators of Apache Kafka and AutoMQ should be deployed in different Namespaces. To avoid conflicts, it is recommended to create a new Namespace when using AutoMQ, and completely delete this Namespace after use .

​
Docker mirroring

The Strimzi Operator uses the Kafka 3.9 version mirroring address by default as quay.io/strimzi/kafka:latest-kafka-3.9.0 , which needs to be modified to the AutoMQ custom mirroring and specific version: 

  - name: STRIMZI_KAFKA_IMAGES
    value: |
      3.9.0=automqinc/automq:1.6.0-rc0-strimzi
      3.9.1=quay.io/strimzi/kafka:0.47.0-kafka-3.9.1
      4.0.0=quay.io/strimzi/kafka:0.47.0-kafka-4.0.0

​
Storage Class

In this example, we are using AWS’s gp2 storage type. If you are using non-AWS or have different requirements, you can make adjustments based on the actual situation. 

storage:
  type: jbod
  volumes:
    - id: 0
      type: persistent-claim
      size: 20Gi
      kraftMetadata: shared
      deleteClaim: false
      class: gp2

​
Autobalancer Configuration

The autobalancer in PLAINTEXT mode must explicitly configure parameters in the following format: listener.name - port . 

spec:
  kafka:
    config:
        autobalancer.client.listener.name: PLAIN1-9092

​
Scheduling Policy

Through node affinity (nodeAffinities) and toleration (tolerations), fine-grained scheduling policies can be implemented for AutoMQ in Kubernetes. We recommend that production-level AutoMQ be exclusive and non-mixed workload. It is recommended to customize label matching rules based on node types:

​
Tolerance

It is recommended to add taints to the Kubernetes node group key: "dedicated", operator: "Equal", value: "automq", effect: "NoSchedule" . 

  tolerations:
    - key: "dedicated"
      operator: "Equal"
      value: "automq"
      effect: "NoSchedule"

​
Node Affinity

Override the default values in the controller/broker configuration to match node labels (such as node-type: r6in.large ) 

    nodeAffinity:  
      requiredDuringSchedulingIgnoredDuringExecution:  
        nodeSelectorTerms:  
        - matchExpressions:  
          - key: "node-type"  
            operator: In  
            values: ["r6in.large"]

​
Pod Anti-Affinity

Ensure that the controller component and the broker component are not scheduled to the same node, using the podAntiAffinity parameter: 

  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: strimzi.io/component-type
                operator: In
                values:
                  - kafka
          topologyKey: kubernetes.io/hostname

​
Volume Expansion and Contraction

​
Controller

The number of Controller instances is specified through the controller.replicas configuration item, and currently does not support horizontal volume expansion and contraction. It is recommended to configure 3 instances, but it can be adjusted according to actual needs.
Note: After the Clustered Deployment is completed, adjusting the Replicas of the Controller is not currently supported to avoid unexpected risks.

​
Broker

The number of instances is configured through broker.replicas , supporting horizontal volume expansion and contraction. Note that the following parameters must be configured during volume contraction: 

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  annotations:
    strimzi.io/skip-broker-scaledown-check: "true"
When there is Partition data on the Broker to be deleted, Strimzi will default to preventing volume contraction. For traditional Kafka, Cruise Control can be used to complete data migration, ensuring that the Broker has no data before performing volume contraction. Strimzi also provides the parameter skip-broker-scaledown-check to skip this check. Since AutoMQ adopts the Shared Storage architecture , this parameter can be enabled without the risk of data loss.
For details, see: Strimzi prevent broker scale down if containing partition replicas

​
Resource Allocation

Each Pod of AutoMQ is recommended to run on resources of 2Core16GB. Adjust the resource parameters through the following configuration: 

spec:
  replicas: 3
  roles:
    - controller
    - broker
  resources:
    requests:
      cpu: 1000m
      memory: 12Gi
    limits:
      cpu: 2000m
      memory: 16Gi
  template:
    kafkaContainer:
      env:
        - name: "KAFKA_HEAP_OPTS"
          value: "-Xmx6g -Xms6g -XX:MaxDirectMemorySize=6g -XX:MetaspaceSize=96m"



spec:
  replicas: 1
  roles:
    - broker
  resources:
    requests:
      cpu: 1000m
      memory: 12Gi
    limits:
      cpu: 2000m
      memory: 16Gi
  template:
    kafkaContainer:
      env:
        - name: "KAFKA_HEAP_OPTS"
          value: "-Xmx6g -Xms6g -XX:MaxDirectMemorySize=6g -XX:MetaspaceSize=96m"

​
Monitoring

It mainly involves the Prometheus-related configuration in Strimzi. For details, please refer to the Using Prometheus in Strimzi section.