Skip to Main Content

Example: Partition Reassignment in Seconds

This document outlines the process for using Kafka CLI tools to verify AutoMQ's sub-second partition reassignment capability. The Kafka CLI tool is executed via a Docker image provided by AutoMQ.

First, create a new Kafka topic with a single partition and initiate a producer to write a certain amount of data into it. Next, attempt to reassign the partition among different AutoMQ brokers using the reassign command. Observe the time it takes to complete the partition reassignment.

Prerequisites​

Before carrying out a partition reassignment test, the following conditions must be met:

Complete the installation and deployment of the AutoMQ cluster. You can refer to the following methods to install and deploy AutoMQ:

Additionally, the host running the test program must meet the following conditions:

  • Linux/Mac/Windows Subsystem for Linux

  • Docker

Experience Partition Reassignment​

If the previous AutoMQ cluster was deployed by following the instructions in Deploy Multi-Nodes Test Cluster on Docker▸, the resulting cluster Bootstrap address would resemble "server1:9092,server2:9092,server3:9092", and the AutoMQ cluster is located within the "automq_net" Docker network.

Please replace the bootstrap-server address below with the actual cluster's address based on the deployment configuration.

Create Topic​


docker run --network automq_net automqinc/automq:1.5.1 /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --create --topic reassign-topic --bootstrap-server server1:9092,server2:9092,server3:9092"

Send Messages​

Execute the following command to send a specified amount of data.


docker run --network automq_net  automqinc/automq:1.5.1 /bin/bash -c  "/opt/kafka/kafka/bin/kafka-producer-perf-test.sh --topic reassign-topic --num-records=1024000 --throughput 5120 --record-size 1024 --producer-props bootstrap.servers=server1:9092,server2:9092,server3:9092 linger.ms=100 batch.size=524288 buffer.memory=134217728 max.request.size=67108864"

View Partition Distribution​


docker run --network automq_net  automqinc/automq:1.5.1 /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --topic reassign-topic --describe --bootstrap-server server1:9092,server2:9092,server3:9092"

Reassign Partitions Between Brokers​

In this step, it's assumed that partition 0 of the test topic "reassign-topic" is currently on node 1, and the goal is to reassign this partition to node 2. Depending on the actual partition distribution during testing, you might opt to reassign to different nodes.

  1. Create the partition reassignment plan.

echo '{
    "partitions": [
        {
            "topic": "reassign-topic",
            "partition": 0,
            "replicas": [
                2
            ]
        }
    ],
    "version": 1
}' > move.json

  1. Execute the partition reassignment plan.

docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:1.5.1 /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server  server1:9092,server2:9092,server3:9092 --reassignment-json-file /move.json --execute"


3. Check if the Reassignment Is Complete.​


docker run --network automq_net -v $(pwd)/move.json:/move.json automqinc/automq:1.5.1 /bin/bash -c "/opt/kafka/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server server1:9092,server2:9092,server3:9092 --reassignment-json-file /move.json --verify"