Executing Migration

After completing the Prerequisites▸, you can proceed with the reassignment process. This article provides a detailed plan and implementation process for migrating from Apache Kafka® to AutoMQ.

Reassignment Plan

Using Kafka Linking to migrate from Apache Kafka® to AutoMQ, the following tasks should be considered:

• Message data synchronization: Kafka stores historical data of both consumed and unconsumed messages. Migrating clusters must ensure that message data is replicated to the new cluster as needed, without losing any messages.

• Producer switching: In addition to data synchronization, the reassignment process requires switching producer applications at an appropriate time so that producers connect to the target cluster to produce new messages.

• Consumer switching: In addition to data synchronization, the reassignment process requires switching consumer applications at an appropriate time so that consumers connect to the target cluster and continue consuming messages from the previous consumption progress.

Refer to the following flowchart for the overall reassignment plan:

Steps

Step 1: Create a Kafka Link and Start Synchronizing Data.

After completing the Prerequisites▸, you have identified the source cluster, destination cluster, Topic, and Consumer Group required for reassignment. Next, begin creating the reassignment task.

1. Set up Source Kafka Integration configuration. Go to the integration page to create Apache Kafka integration.

Tip

Only one integration needs to be created for each source cluster, and this integration configuration can be reused for multiple reassignment tasks subsequently.

The Apache Kafka® integration configuration also supports other Kafka distributions; you only need to fill in the corresponding endpoint parameters.

To create an Apache Kafka® integration, you need to fill in the following parameters:

• Integration Name: Used as an alias to identify the source cluster. It is customizable by the user for easier recognition later on.

• Endpoint Protocol: Choose the appropriate endpoint protocol based on the type of source cluster.

• Endpoint Address: Fill in the Bootstrap address list for the corresponding protocol.

Info

The AutoMQ reassignment tool currently supports the PLAINTEXT protocol and does not support SASL_PLAINTEXT, SSL, and SASL_SSL protocols. If you have any requirements, please contact AutoMQ technical personnel for help and support.

2. Create Kafka Link by clicking Destination Instance >> Kafka Links, and fill in the relevant parameters according to the instructions.

Parameter Name	Description
Kafka Link Id	Meaning: Used to represent a group of reassignment tasks, generally recommended to be short and recognizable. Constraint: Supports only English uppercase and lowercase letters, numbers, hyphens (-), and underscores (_).
Source Cluster	Specify the source cluster that needs reassignment. Here, select the Apache Kafka integration created in the previous step.
Target Cluster	Specify the target cluster for the reassignment. Here, select the already created AutoMQ instance.
Starting Sync Position	Specifies the initial synchronization position after the reassignment task starts. The following options are supported: From Latest Position: Kafka Linking only replicates new messages generated subsequently to the target cluster. This mode is suitable for cold read sensitivity scenarios where migration needs to catch up with both clusters from the current moment.
Sync Topic List	Select several Source Topics from the source cluster, Kafka Linking will create a one-to-one mapped Mirror Topic in the target instance. Danger If the target instance already has the current Topic, the Mirror Topic cannot be created.
Sync Consumer Group List	Select several Consumer Groups from the source cluster. Kafka Linking will create a one-to-one mapped Consumer Group on the target instance.

3. After creation, you can enter the Kafka Link details to check that the specified Topic and Group have already entered the synchronization status. Once Kafka Link is created, it also supports adding new Topics and Consumer Groups. You can add the required business resources for reassignment as needed.

Danger

If you delete Mirror Topics or Consumer Groups in Kafka Link, corresponding Topics and Consumer Groups will be deleted from the target cluster (AutoMQ instance). This operation cannot be undone; you will need to recreate them afterward.

Step 2: Switch Producers and Consumers, Execute Reassignment Process

When the Kafka Link is created, users need to update the Producer and Consumer configurations, switching the Producer and Consumer from the source cluster to the target cluster. This mainly involves the following three operations:

• Producer Reassignment: Update the producer access parameters to point to the target cluster.

• Consumer Reassignment: Update the consumer access parameters to point to the target cluster.

• Promote Mirror Topic Status: In the AutoMQ console, select the Mirror Topic for status promotion. The promotion operation essentially controls the Kafka Linking component to stop proxying Producer write traffic and replicating data from the source cluster.

Tip

Note:

Before switching the Consumer, ensure that the consumption offset of the Consumer in the source cluster is >= the starting sync offset of the Mirror Topic in the target cluster.

Specific steps for switching the Producer and Consumer stream can be found below:

Phase 1: Switching the Producer to the Target Cluster

During the reassignment process, first switch the Producer's access configuration so that the Producer connects to the target cluster (AutoMQ instance). The traffic topology during the switching process is shown in the figure below:

Operating steps:

• The producer updates the access parameters in batches and restarts the application, redirecting the production traffic to the designated AutoMQ instance.

Expected Outcome:

• Production traffic is migrated to the target instance in phases, ensuring no downtime or disruption in production traffic.

• Consumers in the source cluster continue to consume all Messages seamlessly.

• Messages in the source cluster are synchronized to AutoMQ using replication tasks.

Rollback Operation:

• Producer rollback configuration: simply switch back to the source cluster.

Stage Two: Consumer Switch to Target Cluster

The second stage involves switching the Consumer's access configuration, allowing the Consumer to connect to the target cluster (AutoMQ instance). The traffic topology during the switch is shown in the figure below:

Steps:

• Consumers modify their connection parameters in batches and restart the application to connect to the AutoMQ instance.

• Kafka Linking automatically detects when Consumer Group switchovers are completed and promotes the Consumer Group, allowing consumers to continue consumption from the new cluster.

Expected Outcome:

• Production traffic remains in the same state as the previous phase, continuing to proxy back to the source cluster and synchronize to the target cluster.

• The Consumer Group will continue to consume from the offsets of the source cluster and will not be affected.

Rollback Operation:

• For the consumer rollback configuration, simply switch back to the source cluster.

Danger

Since the consumption offsets of the source cluster will not be updated automatically, it is recommended to reset the offsets before rolling back to avoid potential duplicate consumption.

Phase Three: Promote Mirror Topic Status

Once all Producers and Consumers have successfully switched and are functioning as expected, you can elevate the status of the Mirror Topic, stop the traffic proxy, and synchronization.

Operations steps:

• After the Consumer switch is completed, it is recommended to monitor for a certain period to ensure that Producer and Consumer applications are running as expected.

• Once confirmed, in the AutoMQ console, click on "elevate Mirror Topic" and stop traffic proxy and replication.

Danger

Prerequisites for Enhancing Mirror Topics:

All producers have been reassigned.

All consumer groups have been reassigned.

Expected Outcome:

• All read and write operations for mirror topics are centralized in the target cluster, without proxying back to the source cluster.

Rollback Operation:

• If the message data in the target cluster exceeds that of the source cluster after promoting the Mirror Topic, you should be cautious about data inconsistencies during the rollback.

Step 3: Migration Complete, Delete Kafka Link

Once all Mirror Topics within Kafka Link have completed migration and promotion as described in Step 2, the migration task is complete, and Kafka Link can be deleted.