Skip to Main Content

Prerequisites

Before migrating to AutoMQ, make sure to verify the following prerequisites and constraints to ensure that your production environment remains unaffected.

Function Compatibility Assessment​

Due to version support differences, AutoMQ currently provides only Kafka Server services and does not offer managed services for other ecosystem components. Before migrating to AutoMQ, please confirm that the following functionalities are in use and refer to the following handling guidelines:

  • Kafka Server: AutoMQ offers a more cost-effective and flexible Kafka Server, compatible with Apache Kafka versions 0.9.x ~ 3.9.x. If your current Apache Kafka Server version falls outside this supported range, please contact us for support and updates.

  • Kafka Client: AutoMQ is compatible with the original Apache Kafka Client SDK. Ensure that your existing SDK version is between 0.9.x ~ 3.9.x to maintain compatibility.

  • Kafka Connector: AutoMQ does not currently provide managed Kafka Connector services. If you are using Kafka Connector, it is recommended to continue using the existing service and just replace the Kafka Server endpoint in the Connector configuration with the AutoMQ instance endpoint.

  • Kafka Schema Registry: AutoMQ does not currently provide managed Schema Registry services. If you are using Kafka Registry, it is recommended to continue using the existing service and just replace the Kafka Server endpoint in the Registry configuration with the AutoMQ instance endpoint.

In addition to the differences in service components, confirm whether the following functionalities are in use before migrating to AutoMQ and handle them according to the guidance below.

  • ACL Functionality: The current version of AutoMQ does not support ACL functionality. If your application heavily relies on ACL for access control, it is essential to evaluate the risks involved in making modifications.

  • Access Point Protocols: The current version of AutoMQ supports PLAINTEXT, SASL_PLAINTEXT, SASL_SSL, and mTLS protocols for internal network access. If your current application uses other protocols to access Kafka, it is recommended to contact AutoMQ technical personnel to assist in evaluating the risk of modifications.

Business Scope Sorting​

For migrating from Apache Kafka to AutoMQ in a production environment, it is generally recommended to perform the migration in batches. This involves gradually selecting Topics and Consumer Groups from the source cluster and migrating them in batches based on business chains and business scope.

Therefore, it is recommended to perform business scope sorting according to the following process:

  1. Inventory of Source Cluster Resources: It is advisable to use the Kafka Admin CLI or other tools to export all Topics and Consumer Groups from the source cluster and classify and organize them by business system. Clean up invalid or unclaimed Topics and Consumer Groups in advance.
Tip

It is recommended to check the Topics in the source cluster and clean up Topics and Consumer Groups that have had no writes or subscriptions for a long time in advance to reduce the scope of the migration.

  1. Batch Division by Business System: In a production environment, it is recommended to plan migration batches by business system. Start by migrating offline and non-core business systems, and gradually expand the migration scope to avoid unexpected risks.

  2. Coordinate with Business Personnel for Reassignment: During the process of migrating Apache Kafka® to AutoMQ, it is necessary for business personnel to cooperate in changing access points and restarting applications. Therefore, before executing the reassignment plan, it is important to coordinate with the business personnel for their cooperation.

Info

Apache Kafka Client does not disconnect from the source cluster after establishing a server connection. Therefore, even if domain names are used to access the server, it is still necessary to restart the client application to connect to the new AutoMQ cluster.

Cluster Environment Preparation​

Create an AutoMQ Instance and Ensure the Instance Capacity Meets Your Requirements.​

Before executing the reassignment tasks, an AutoMQ instance needs to be created in advance, and it must be ensured that the AutoMQ instance is in a running state. When creating an AutoMQ instance, please pay attention to the following considerations:

  • Network Connectivity Check: When creating an AutoMQ instance, it is necessary to select a VPC and subnet. Please ensure that the VPC subnet where the AutoMQ instance is located can communicate with the current Apache Kafka. Otherwise, data reassignment and synchronization cannot be executed subsequently.
Tip

To use Kafka Linking, ensure that the AutoMQ instance version is >= 5.0. If this requirement is not met, please upgrade the instance version first.

  • Instance Capacity Specification Check: When creating an AutoMQ instance, it is crucial to confirm the throughput scale of the source cluster. Based on the peak read/write throughput of the source cluster, select the appropriate AutoMQ instance specifications to avoid system failures due to insufficient capacity during the later stages of reassignment.
Tip

When using Kafka Linking, it is recommended to reserve an additional 50% capacity on top of the peak throughput specification of the source cluster for the target AutoMQ instance to account for the consumption of cluster capacity by Kafka Linking.

Example:

If the peak throughput of the source Kafka cluster is 100MBps, and you are reassigning to AutoMQ using Kafka Linking, it is advisable to reserve 100*150% = 150MBps during reassignment. After the reassignment is completed, the capacity can be reduced to match the specifications of the source cluster.

Detailed instance creation instructions can be found at Experience AutoMQ▸.

Retrieve Source Cluster Configuration​

The principle of migrating from Apache Kafka® to AutoMQ is to use migration tools to synchronize existing data and configurations from the source cluster to the new cluster, then switch client configurations at the appropriate time to enable access to the new cluster.

In the process described above, synchronizing data and configurations from the source cluster requires obtaining the source cluster's configuration. Therefore, it is necessary to confirm in advance whether the following configuration information is prepared:

  • Source Cluster Access Points: The migration tool needs to access the source cluster, requiring the configuration of the source cluster's Bootstrap access point list.

  • Source Cluster Access Protocol: The migration tool needs to access the source cluster, requiring confirmation of the access protocol for the source cluster. Currently, PLAINTEXT and SASL_PLAINTEXT protocols are supported.

  • Source Cluster Username and Password (Optional): If the source cluster requires access using the SASL_PLAINTEXT protocol, it is necessary to configure the username and password and ensure that the username has permission to subscribe to all topics.

  • Source Cluster Network Firewall Policy (Optional): If the source cluster is configured with network firewall or security group policies, the subnet of AutoMQ needs to be allowed.

Next Steps​

Once the prerequisites are sorted out and met, you can proceed with the subsequent reassignment process. It is recommended to handle it in the following order:

  • Implement the reassignment plan: After completing the prerequisites, follow the reassignment plan step-by-step. Executing Migration ▸