Skip to main content
Before migrating to AutoMQ, make sure to verify the following conditions and constraints to ensure that your production environment remains unaffected.

​
Function Compatibility Assessment

Due to version support differences, AutoMQ currently provides only Kafka Server services and does not offer managed services for other ecosystem components. Therefore, before migrating to AutoMQ, you need to confirm whether the following functionalities are in use and refer to the following handling guidelines:
  • Kafka Server: AutoMQ offers a more cost-effective and flexible Kafka Server, compatible with Apache Kafka versions 0.11.x ~ 3.9.x. If your current Apache Kafka Server version falls outside this supported range, please contact us for support and updates.
  • Kafka Client: AutoMQ is compatible with the original Apache Kafka Client SDK. Ensure that your existing SDK version is between 0.10.x ~ 3.9.x to maintain compatibility.
In addition to the differences in service components, before migrating to AutoMQ, you need to confirm whether the following functionalities are in use and handle them according to the guidance below.
  • Access Point Protocols: AutoMQ supports protocols such as PLAINTEXT, SASL_PLAINTEXT, SASL_SSL, and mTLS for internal network access. If your application currently uses other protocols to connect to the cluster, it is recommended to consult with AutoMQ technical staff to evaluate the risks associated with modifications.

​
Business Scope Sorting

For migrating from Apache Kafka to AutoMQ in a production environment, it is generally recommended to perform the migration in batches. This involves gradually selecting Topics and Consumer Groups from the source cluster and migrating them in batches based on business chains and business scope. Therefore, it is recommended to perform business scope sorting according to the following process:
  1. Inventory of Source Cluster Resources: It is advisable to use the Kafka Admin CLI or other tools to export all Topics and Consumer Groups from the source cluster and classify and organize them by business system. Clean up invalid or unclaimed Topics and Consumer Groups in advance.
It is recommended to check the Topics in the source cluster and clean up Topics and Consumer Groups that have had no writes or subscriptions for a long time in advance to reduce the scope of the migration.
  1. Batch Division by Business System: In a production environment, it is recommended to plan migration batches by business system. Start by migrating offline and non-core business systems, and gradually expand the migration scope to avoid unexpected risks.
  2. Coordinate with Business Personnel for Migration: During the process of migrating Apache Kafka® to AutoMQ, it is necessary for business personnel to cooperate in changing access points and restarting applications. Therefore, before executing the migration plan, it is important to coordinate with the business personnel for their cooperation.
Apache Kafka Client does not disconnect from the source cluster after establishing a server connection. Therefore, even if domain names are used to access the server, it is still necessary to restart the client application to connect to the new AutoMQ cluster.

​
Cluster Environment Preparation

​
Create an AutoMQ Instance and Ensure the Instance Capacity Meets Your Requirements.

Before executing the migration tasks, an AutoMQ instance needs to be created in advance, and it must be ensured that the AutoMQ instance is in a running state. When creating an AutoMQ instance, please pay attention to the following considerations:
  • Network Connectivity Check: When creating an AutoMQ instance, it is necessary to select a VPC and subnet. Please ensure that the VPC subnet where the AutoMQ instance is located can communicate with the current Apache Kafka. Otherwise, data migration and synchronization cannot be executed subsequently.
To utilize Kafka Linking, ensure that your AutoMQ instance version is >= 5.0. If your version is below this requirement, please upgrade the instance version first.
  • Instance Capacity Specification Check: When creating an AutoMQ instance, it is crucial to confirm the throughput scale of the source cluster. Based on the peak read/write throughput of the source cluster, select the appropriate AutoMQ instance specifications to avoid system failures due to insufficient capacity during the later stages of migration.
When using Kafka Linking, it is recommended to reserve an additional 50% capacity on top of the peak throughput specification of the source cluster for the target AutoMQ instance to account for the consumption of cluster capacity by Kafka Linking.Example:If the peak throughput of the source Kafka cluster is 100MBps, and you are migrating to AutoMQ using Kafka Linking, it is advisable to reserve 100*150% = 150MBps during migration. After the migration is completed, the capacity can be reduced to match the specifications of the source cluster.
For detailed instructions on creating an instance, please refer to Experience AutoMQ▸.

​
Retrieve Source Cluster Configuration

The principle of migrating from Apache Kafka® to AutoMQ is to use migration tools to synchronize existing data and configurations from the source cluster to the new cluster, then switch client configurations at the appropriate time to enable access to the new cluster. In the process described above, synchronizing data and configurations from the source cluster requires obtaining the source cluster’s configuration. Therefore, it is necessary to confirm in advance whether the following configuration information is prepared:
  • Source Cluster Access Points: The migration tool needs to access the source cluster, requiring the configuration of the source cluster’s Bootstrap access point list.
  • Source Cluster Access Protocol: The migration tool needs to access the source cluster, and you must confirm the access protocol used by the source cluster. Currently, Kafka Linking supports PLAINTEXT, SASL_PLAINTEXT, SASL_SSL, and mTLS protocols.
  • Source Cluster Username and Password (Optional): If the source cluster requires access using the SASL_PLAINTEXT or SASL_SSL protocol, you need to configure the username and password and ensure that the username has permission to subscribe to all topics.
  • Source Cluster Network Firewall Policy (Optional): If the source cluster is configured with network firewall or security group policies, the subnet of AutoMQ needs to be allowed.

​
Next Steps

Once the prerequisites are sorted out and met, you can proceed with the subsequent migration process. It is recommended to handle it in the following order: