Skip to Main Content

Example: Partition Reassignment in Seconds

This document outlines the process for using Kafka CLI tools to verify AutoMQ's sub-second partition reassignment capability. The Kafka CLI tool is executed via a Docker image provided by AutoMQ.

First, create a new Kafka topic with a single partition and initiate a producer to write a certain amount of data into it. Next, attempt to reassign the partition among different AutoMQ brokers using the reassign command. Observe the time it takes to complete the partition reassignment.

Prerequisites​

Before carrying out a partition reassignment test, the following conditions must be met:

Complete the installation and deployment of the AutoMQ cluster. You can refer to the following methods to install and deploy AutoMQ:

Additionally, the host running the test program must meet the following conditions:

  • Linux/Mac/Windows Subsystem for Linux

  • Docker

Info

If downloading container images is slow, please refer to Docker Hub Mirror Configuration▸.

If the previous AutoMQ cluster was deployed using Deploy Multi-Nodes Test Cluster on Docker▸, the acquired cluster Bootstrap address will resemble "server1:9092,server2:9092,server3:9092," and the AutoMQ cluster will be located in the "automq_net" Docker network.

Tip

Please update the bootstrap-server address below to match the actual address of the cluster based on the deployment configuration.

Create Topic​


docker run --network automq_net automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --create --topic reassign-topic --bootstrap-server server1:9092,server2:9092,server3:9092"

Send Messages​

Execute the following command to send a specified amount of data.


docker run --network automq_net  automqinc/automq:latest /bin/bash -c  "/opt/kafka/kafka/bin/kafka-producer-perf-test.sh --topic reassign-topic --num-records=1024000 --throughput 5120 --record-size 1024 --producer-props bootstrap.servers=server1:9092,server2:9092,server3:9092 linger.ms=100 batch.size=524288 buffer.memory=134217728 max.request.size=67108864"

Check Partition Distribution​


docker run --network automq_net  automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --topic reassign-topic --describe --bootstrap-server server1:9092,server2:9092,server3:9092"

Reassign Partitions Across Brokers​

In this step, assume that partition 0 of the "reassign-topic" is located on node 1, and reassign this partition to node 2. During the actual testing process, you can choose a different node for reassignment based on partition distribution.

  1. Create a plan for partition reassignment.

echo '{
    "partitions": [
        {
            "topic": "reassign-topic",
            "partition": 0,
            "replicas": [
                2
            ]
        }
    ],
    "version": 1
}' > move.json

  1. Implement the partition reassignment plan.

docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server  server1:9092,server2:9092,server3:9092 --reassignment-json-file /move.json --execute"


3. Verify if the Reassignment Is Complete.​


docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server server1:9092,server2:9092,server3:9092 --reassignment-json-file /move.json --verify"