Table of Contents
Table of Contents
Failing traffic over to a secondary Amazon MSK cluster is only half of a disaster-recovery test. The harder test begins when the primary Region is healthy again and you need to move producers and consumers back without creating a second stream of writes, skipping committed records, or quietly processing the same records twice.
Amazon MSK Replicator copies records and consumer-group metadata asynchronously. That means a consumer offset on the secondary cluster is a translated position, not a byte-for-byte coordinate in the primary cluster. AWS documents failback as a sequence of replication catch-up, producer and consumer stops, and metric checks. Treat that sequence as an evidence gate rather than a checklist you can mark complete because a console operation succeeded.
This runbook uses that idea as its organizing rule: define which cluster is authoritative, fence every writer, prove the replicated data boundary, move consumers in a controlled order, and keep the evidence that explains what happened. The result is a repeatable Amazon MSK Replicator failback validation plan that exposes duplicate and recovery behavior before an incident does.
1Define the failback boundary before touching traffic
Write down the boundary in terms of clusters, topics, consumer groups, and time. For example, “secondary is authoritative for orders and payments; primary accepts no writes; consumers in group billing-v3 must resume from the translated position.” Avoid a vague statement such as “switch back to us-east-1.” A failback plan that does not name the authoritative writer can produce a split-brain stream even while every component reports healthy.
The boundary also depends on the topic naming mode. With identical topic replication, both clusters expose the same topic names and MSK Replicator uses record headers to avoid sending a record back to its origin. With prefixed replication, the secondary contains names such as <sourceKafkaClusterAlias>.orders; consumers must deliberately read the replicated topic first or use a carefully scoped wildcard. AWS's failback procedure describes different stop and start orders for these modes.
Capture the state that makes the boundary testable:
- The Replicator direction used during the outage and the new direction used for the return path.
- The topic and consumer-group filters on both Replicators, including excluded patterns.
- The last observed
MessageLag,ReplicatorThroughput, and consumer lag for the groups you will move. - The client bootstrap configuration and the mechanism used to prevent a stale client from reconnecting to the secondary.
Keep this record with the incident or change ticket. It lets a reviewer distinguish “the cluster was reachable” from “the return path was safe.”
2Freeze or fence writers
Failback starts with a single-writer decision. Stop producers on the secondary only after the secondary-to-primary Replicator has caught up enough for your stated recovery objective. Then prevent old clients from coming back while DNS, configuration, or service discovery changes propagate.
Operational fencing and Kafka producer fencing are related but different controls. Operational fencing stops a workload from opening a new connection: revoke its MSK permission, remove its route, scale the deployment to zero, or apply the deployment's maintenance switch. Kafka's transactional producer fencing happens when a new producer instance with the same transactional.id obtains a higher producer epoch; the broker then rejects operations from the older instance. A transactional epoch does not stop a non-transactional producer, and it does not replace an account- or network-level stop.
Use at least two independent signals that the secondary is fenced. A deployment replica count of zero is one signal; an authorization-denied or connection-failure event from a canary producer is another. If the production clients share a role, do not revoke that role until you have verified that consumer credentials still have the access required for the next step. Record the timestamp of the fence and the last accepted producer sequence or application event ID.
For applications that use Kafka transactions, also record the transactional.id ownership and the producer error rate during the stop. A fenced producer should fail loudly. For non-transactional or at-least-once producers, use an application-level idempotency key so a retry after failback can be recognized downstream.
3Compare offsets with the data they describe
MSK Replicator's offset synchronization has three conceptual steps: it maps source offsets to target offsets, translates committed group offsets using those mappings, and commits the translated position to the target's __consumer_offsets topic. Because the clusters have different append histories, the translation is approximate. AWS explicitly favors at-least-once behavior, so consumers can replay a small number of records after a cluster switch.
That behavior changes what “offset validation” means. Do not compare the numeric offset for partition 3 in the two clusters and expect equality. Compare the record identity at or around the translated position. A practical worksheet includes:
| Evidence | What to capture | Why it matters |
|---|---|---|
| Replication catch-up | MessageLag and ReplicatorThroughput for the return Replicator | Shows whether secondary writes have reached primary and whether the flow is still active |
| Group translation | ConsumerGroupCount, ConsumerGroupOffsetSyncFailure, and per-group OffsetLag | Shows that the intended groups were included and translated |
| Record identity | Application event ID, key, timestamp, and partition around the handoff | Distinguishes replay from data loss when numeric offsets differ |
| Client position | Committed offset and records-lag-max after the consumer joins primary | Shows where the client actually resumed |
Run the comparison while the consumer group is stopped on the target. The Replicator will not overwrite offsets for a group that is actively consuming on the target, so starting the client too early can make the translated checkpoint look missing. Also verify that the group matches the Replicator's include and exclude filters; a healthy Replicator can legitimately report no offset for a group it was never configured to sync.
4Replay and deduplicate by design
A replay is a normal outcome of conservative offset translation. The application should make the replay harmless before you run the exercise. For commands and payments, carry an immutable event ID and enforce idempotency at the sink. For state updates, use a version or sequence field and reject an older version. For analytics, mark the test window and make the aggregation job replayable rather than trying to prove that every record is delivered exactly once across two clusters.
Do not infer loss from a gap in numeric offsets alone. Compaction, retention, and topic recreation can change the relationship between a source offset and a target offset. Instead, sample records on both sides of the handoff and reconcile counts by partition and event ID. If the workload has no stable event ID, document that limitation as part of the recovery result; it is a data-model gap, not an MSK Replicator metric problem.
The same rule applies to Kafka transactions. A transaction boundary can prevent partial visibility within a cluster, but it does not make a cross-cluster replay disappear. Consumers and downstream systems still need a policy for a record that was committed before the failover and delivered again after the translated offset.
5Redirect clients in a controlled order
Once the return Replicator has caught up and writers on the secondary are fenced, move clients in an order that matches the topic mode and the application's ordering requirements. For identical topics, start producers on the primary, then start consumers after the target group offsets are present. For prefixed topics, applications that require ordering should consume the replicated topic first, wait for its lag to reach zero, then switch to the local topic. Applications that can tolerate interleaving may consume both with a narrowly scoped pattern, but validate the pattern against the actual topic names before production.
Client recovery has more failure modes than a new bootstrap server. Check all of the following during the canary:
- The client resolves the primary bootstrap hostname and completes TLS and authentication.
- The consumer joins the expected group and receives the translated assignment.
records-lag-max, rebalance rate, and application processing latency return toward their pre-failover range.- Producers receive acknowledgments from the primary and no stale secondary producer continues to publish.
- The application sees expected event IDs, with duplicates handled by the chosen idempotency rule.
Keep the canary small enough that you can stop it without changing the recovery boundary. A successful connection test is necessary, but it does not prove that offsets, ordering, or duplicate handling are correct.
6Document evidence and the rollback gate
Close the exercise with a timeline, not a sentence that says “failback passed.” Include the outage end time, Replicator creation time, first and last non-zero MessageLag, writer-fence time, consumer start time, first primary acknowledgment, and the final lag readings. Attach the metric screenshots or exported time series and the sampled event-ID reconciliation.
Set a rollback gate before starting. If ConsumerGroupOffsetSyncFailure is non-zero, the expected group count is wrong, a producer cannot be fenced, or the sample shows an unexplained event-ID gap, keep traffic on the secondary and investigate. If duplicates exceed the application's stated tolerance, stop the canary and fix idempotency before attempting another full cutover. The gate protects the system from a well-intentioned operator who sees green infrastructure metrics and misses a data-plane failure.
For recurring drills, turn the worksheet into a versioned runbook. Record which steps are provider-specific and which belong to the application: AWS owns Replicator execution, while your team owns writer fencing, client configuration, consumer semantics, and downstream reconciliation. That ownership split is why a failback test should exercise the entire path, not only the Replicator console.
7Where AutoMQ fits in the decision
If repeated failback drills expose a deeper constraint, such as broker storage and compute having to scale together or a migration needing the same Kafka client contract across environments, AutoMQ is one architecture to evaluate. AutoMQ is a Kafka-compatible streaming system that separates broker compute from object-storage-backed data. That can change the storage and recovery assumptions in a future migration, but it does not remove the need to define writer ownership, consumer recovery, or duplicate handling.
The decision boundary is practical: keep MSK Replicator when managed cross-Region replication and AWS-native operations fit the runbook; evaluate another Kafka-compatible platform when portability, storage elasticity, or a different ownership boundary is the reason the runbook keeps expanding. The failback evidence gives you the facts for that decision.
8FAQ
8.1Does MSK Replicator guarantee exact consumer offsets during failback?
No. Offset translation maps positions between different append histories and is approximate. The documented behavior favors at-least-once processing, so a consumer can replay records after switching clusters. Validate event identity and downstream idempotency instead of requiring equal numeric offsets.
8.2Do I need two Replicators for failback?
For a one-way setup, the original Replicator does not translate consumer offsets from the target back to the source. A return Replicator is required to copy secondary writes to the primary. If consumers must move in both directions with enhanced offset synchronization, configure two opposite Replicators and identical topic names, then validate both directions before relying on the design.
8.3Should producers or consumers move first?
Fence and stop secondary producers first, then confirm the return Replicator has caught up. Start primary producers, and start consumers only after the translated offsets are present on the primary. Prefixed topics can require an additional replicated-topic drain when ordering matters.
8.4What is the most useful failback metric?
No single metric proves recovery. Use MessageLag for replication catch-up, ReplicatorThroughput to confirm the flow is active, ConsumerGroupOffsetSyncFailure and OffsetLag for offset synchronization, and application event IDs to check replay or gaps.
9Take the runbook beyond the console
The return path is safe when you can show who was allowed to write, which records crossed the boundary, where each consumer resumed, and how a duplicate was handled. Run the procedure against a disposable MSK pair, keep the evidence, and make the rollback gate part of the change request. If you are comparing Kafka-compatible architectures after that exercise, start with the AutoMQ open-source repository and test the same client and recovery assumptions against your workload.
