Table of Contents
Table of Contents
A Kafka migration is rarely broken by copying the last topic. It breaks when a producer keeps writing to the old bootstrap endpoint, a consumer resumes from an offset that means something different on the target, or a connector arrives with credentials and network assumptions that were never recorded. The data can be present and the migration can still be unsafe.
Moving from Kafka on EC2, Kubernetes, or bare metal to Amazon MSK is therefore a compatibility exercise before it is a replication exercise. The target must preserve the contracts that applications depend on: protocol behavior, topic semantics, security, ordering, transactions, consumer positions, and the way clients discover brokers. A staged plan can keep those contracts visible while traffic moves between clusters.
1Start with the application contract, not the cluster inventory
Begin with a map of every client that touches Kafka. Include application producers and consumers, Kafka Connect workers, schema and stream-processing services, command-line jobs, and operational scripts. For each one, record the client library and version, bootstrap configuration, security protocol, authentication mechanism, TLS trust material, topic subscriptions, consumer group, partition assumptions, and whether it uses transactions or idempotent production.
This inventory is more valuable than a broker list because clients carry the migration risk. A producer using SASL_SSL with IAM credentials has a different cutover path from a Java client using mutual TLS. A consumer that subscribes to a literal topic list can move when its configuration changes; one that relies on a prefix or regex may accidentally read both source and replicated topics. A transactional producer needs a test for transaction completion, fencing, and producer identity after the endpoint changes.
Capture the topic contract as well. Record partition counts, replication factor, retention, cleanup policy, compaction settings, maximum message size, and any topic-level overrides. Replication tools can copy records and selected metadata, but they do not make an implicit application assumption explicit. Treat the resulting inventory as the acceptance baseline for the target cluster.
A client is ready when its protocol, identity, topic view, and recovery position are all defined on the target. “It can resolve the target hostname” is only one line on that checklist.
2Validate clients against a disposable MSK path
Create a small MSK test cluster with the authentication mode and network pattern you expect to use in production. The point is not to benchmark it. The point is to make each client perform the operations that matter before a migration window creates pressure.
Use a compatibility matrix with at least these tests:
| Test | What it proves | Evidence to keep |
|---|---|---|
| TLS handshake and broker metadata | Trust store, hostname verification, and security protocol are correct | Client logs and broker connection result |
| Produce and consume with the real serializer | Message format and compression work on the target | Record key, headers, and payload checksum |
| Consumer group restart | The client can commit and resume its group position | Group ID, committed offset, and resumed record |
| Transaction commit and abort | Transactional IDs and isolation settings behave as expected | Commit/abort result and read_committed check |
| ACL or IAM authorization | The identity has only the operations it needs | Allowed and denied operation results |
| Connector restart | External systems can reconnect with target credentials | Connector status, task state, and offset store |
Amazon MSK offers IAM, SCRAM/SASL, and TLS client authentication, with different credential and authorization workflows. Select the mode for the target before you copy production traffic, then test credential rotation and failure behavior. The TLS certificate rotation runbook is useful when certificate lifecycle is part of the cutover. Do not change authentication and cluster endpoints in the same unobserved step if you can separate them; otherwise a failed connection has too many possible causes.
The network path deserves the same treatment. A target cluster in a different VPC may require routing, security-group rules, DNS resolution, and private connectivity that the self-managed brokers never needed. Test from the same subnets and runtime identities used by production clients, including the connector workers. A successful test from an engineer laptop is not evidence for a private workload.
3Choose the replication path by the control you need
A migration needs a source-to-target data path that can run while the source remains authoritative. For a self-managed Kafka source, two common choices are Amazon MSK Replicator and Apache Kafka MirrorMaker 2.
Amazon MSK Replicator is an AWS-managed asynchronous service. AWS documents self-managed Kafka to MSK Provisioned replication as a supported pattern, subject to the service’s current network, version, and quota requirements. The service reduces the replication infrastructure your team operates, but the migration still depends on topic filters, security configuration, lag monitoring, and a tested cutover.
MirrorMaker 2 runs through Kafka Connect. It gives the team control over worker placement, topic and group filters, replication policy, converters, and connector settings. That control can be useful when the source is not reachable by MSK Replicator or when the migration needs custom routing. It also means the workers, internal topics, upgrades, and connector failure modes belong in the migration runbook.
The choice should follow the migration’s failure boundary:
- Choose the managed path when the target is MSK, the supported topology fits, and reducing replication infrastructure is more important than changing replication behavior.
- Choose MM2 when you need a configurable Connect deployment, custom topic policy, or a provider-neutral path that can be reused for a later exit.
- In either case, define how topic names and consumer offsets map before records start flowing. A replication job that is healthy but maps the wrong topic is still a failed migration.
4Replicate, then shadow the target
Build the target topics with enough partitions for the planned workload and with retention and cleanup policies that match the application contract. If the target uses a prefix for replicated topics, make that prefix part of the test configuration rather than hiding it in a one-off command. If you use identical topic names, document how loop prevention works and how writers are fenced; identical names reduce one kind of client change while making authority harder to see.
Start replication and measure more than connector health. Track source and target record counts for selected partitions, replication lag, errors, rejected records, and the age of the oldest record not yet visible on the target. For consumer groups, maintain a separate view of source offsets and translated target offsets. An offset number without its topic, partition, and cluster context is not useful evidence.
Shadow traffic means exercising the target without allowing it to become an unplanned second writer. Run read-only validation consumers against replicated topics. Replay representative records through a staging consumer or a canary service. Compare application-level checksums, schema IDs, key distribution, and ordering where the application requires it. Keep the target consumers in distinct group IDs until you have decided how production offsets will be mapped.
This stage is also where you catch hidden dependencies: a consumer hard-coded to the old topic prefix, a connector that writes to a local filesystem, a serializer that expects a broker-side setting, or an IAM policy that permits the source ARN but not the target. Fixing those while the source is still authoritative is much safer than discovering them after the producer switch. For a fuller traffic-shadowing runbook, see Traffic Shadowing for Kafka Migration.
5Cut over producers and consumers in a controlled order
A low-risk cutover has an explicit write owner. The source cluster remains authoritative until the target has caught up and the team has a measured handoff point. The exact order depends on whether the workload tolerates duplicates and whether the application uses transactions, but the gates should be visible:
- Announce a change window and freeze unrelated topic or ACL changes.
- Confirm target connectivity, authentication, replication lag, and capacity headroom.
- Stop, drain, or fence source producers. For transactional applications, wait for in-flight transactions to finish or abort according to the application runbook.
- Record source high-water marks and the translated target positions for each critical topic and group.
- Change producer bootstrap endpoints and credentials to the MSK values, then send a canary record with a traceable key.
- Verify the record reaches the expected target partition and downstream consumer before moving the remaining producers.
- Move consumers using the planned group and offset mapping. Watch lag, rebalances, error rates, and application checkpoints.
- Keep the source available for the agreed rollback window, but do not allow old producers to resume writing silently.
The canary should test the application path, more than a Kafka command. Include a header or key that downstream systems can identify, then verify the record at the final business boundary. A green producer metric does not prove that the consumer is reading the intended topic or that a connector has committed its target offset.
5.1Transactions and duplicate processing need a decision
Replication is asynchronous, and offset translation is not a magic preservation of application state. During a cutover, a consumer may re-read records or a producer may retry after an uncertain acknowledgment. Design the migration around the application’s actual tolerance: idempotent writes, deduplication keys, transactional sinks, or an explicit replay procedure.
Kafka transactions also have scope. A transaction committed on the source does not become one atomic transaction with a target consumer merely because the record was replicated. Validate the target consumer’s isolation.level, transactional ID behavior, and downstream side effects. If the application cannot tolerate duplicate side effects, the rollback and replay gates must be stricter than the replication health check.
6Define rollback before you change the endpoint
Rollback is a decision tree, not a promise to “switch the DNS back.” First decide which cluster owns writes after the cutover. If the target has accepted records, switching producers back without reconciling those records can create divergence or duplicate business events.
Set rollback gates before the window begins. Useful gates include a failed authentication rate above the tested baseline, target consumer lag that does not recover within the agreed interval, a missing canary at the application boundary, or a connector that cannot commit offsets. Each gate should name an owner and an action.
A safe return path usually requires stopping target producers, preserving evidence of target offsets, replaying or reconciling records written during the target period, and moving consumers with a known position. Keep replication in the direction needed for the return path until the rollback window closes. Test this path with a disposable topic before the production cutover; a rollback that exists only in a document is not a rollback plan.
7When a Kafka-compatible storage model changes the decision
The migration checklist above assumes the target’s broker and storage model are accepted as part of the move. Some teams are using the MSK project to revisit that assumption: they want Kafka protocol compatibility, but they also want storage and compute to scale independently and a deployment model that can stay inside their own cloud account.
That is where AutoMQ, a Kafka-compatible cloud-native streaming platform built around object-storage-backed durability, can enter the evaluation. The relevant question is not whether it is a drop-in replacement for every MSK deployment. It is whether the team’s client contract can stay stable while the storage layer, broker recovery model, and operating boundary change. Validate client behavior, Kafka features, networking, and migration tooling against the workload before treating compatibility as a conclusion.
For a team that has already invested in a careful inventory, the comparison becomes concrete: which platform preserves the application contract, who owns the recovery path, and what state must be proven at cutover? Those are the same questions whether the target is MSK or another Kafka-compatible platform.
8FAQ
8.1Can I migrate to Amazon MSK without stopping producers?
You can keep the source serving traffic while records replicate, but a final cutover still needs a clear write owner. Most teams pause, drain, or fence source producers briefly so they can establish a handoff point and prevent split-brain writes. The duration depends on the application and replication lag; it is not a universal zero-downtime guarantee.
8.2Does MSK Replicator preserve consumer offsets?
MSK Replicator can replicate consumer group offsets in supported configurations, but the target position is a translated position and should be validated with the application. Check topic naming, group filters, and the current AWS behavior for your source and target versions before relying on automatic translation.
8.3Do Kafka clients need code changes when moving to MSK?
Often the application code can remain unchanged, but client configuration usually changes. Bootstrap endpoints, TLS trust settings, SASL or IAM credentials, ACLs, and sometimes topic names or consumer groups must be updated and tested.
8.4What should I test first?
Test the client contract that is hardest to recover: transactions, consumer offsets, connector state, and identity. A basic produce-and-consume test is necessary, but it does not cover the failure modes that cause a migration to break after the endpoint switch.
9References
- Amazon MSK Replicator
- Configure clients for Amazon MSK
- Amazon MSK client authentication
- Apache Kafka MirrorMaker 2 documentation
When the old Kafka cluster has become a dependency nobody can describe, migration feels like a leap. When the client contract, offset evidence, and rollback gates are written down, it becomes a sequence of reversible decisions. Evaluate a Kafka-compatible operating model with AutoMQ if your next decision includes both client compatibility and a different storage boundary.
