Amazon MSK Active-Active Architecture: Replication, Conflicts, and Client Routing

An Amazon MSK active-active design lets two AWS Regions serve reads and writes at the same time. That sounds like a simple availability upgrade until two Regions accept a change to the same business key. Replication can move both records, but it cannot decide whether shipped or cancelled should win, and it cannot create a single global order from two independent partition logs.

The useful design question is therefore narrower than “Can MSK replicate between Regions?” The question is whether the workload has an explicit ownership, ordering, and duplicate-processing contract that makes concurrent writes safe. Amazon MSK Replicator provides asynchronous data replication and consumer group offset synchronization, but the application still owns conflict resolution and client routing.

1What active-active changes in the MSK data path

Amazon MSK Replicator runs one replication direction at a time. A bidirectional design uses two Replicators, one from cluster A to cluster B and another from B to A. Each cluster can have local producers and consumers, while the Replicators copy records and selected metadata asynchronously. AWS documents cross-Region and same-Region replication, and requires the source and target clusters to exist before a Replicator is created.

This has three consequences that should be on the architecture diagram:

• The write paths are independent. A producer in Region A receives a local partition offset. A producer in Region B receives a different local offset. Neither offset is a global sequence number.
• Replication is asynchronous. Region B can be behind Region A during a network or capacity event. The MessageLag and throughput metrics are part of the failover decision, not dashboard decoration.
• The consumer path is a routing decision. A consumer can read local topics only, or it can read local and replicated topics. The choice changes ordering, duplicate handling, and the work required during a regional switch.

The default Prefixed topic name replication creates a topic such as A.topic in the target cluster for data sourced from Region A. This separation is useful in active-active systems because consumers can distinguish local records from replicated records. AWS recommends a wildcard subscription pattern so a regional consumer can read both its local topic and the replicated A.topic. If the application requires ordering, that wildcard pattern may be too broad, as the failover procedure needs the consumer to drain replicated records before switching to the local topic.

Identical topic name replication keeps the same topic names and can reduce client reconfiguration. It also adds data processing and transfer work in an active-active topology because the Replicators must prevent records from circulating forever. AWS notes that Identical mode processes each record more than once in this pattern and recommends waiting at least 30 seconds before recreating a deleted topic with the same name. Topic naming is part of the failure model, not a cosmetic setting.

2Define the conflict model before choosing topology

Kafka preserves order within a partition. It does not provide a total order across two Regions that are accepting writes independently. If both Regions can write the same key, the application must define what “latest” means and how a consumer can identify a record that was already applied.

A practical conflict review starts with the business key rather than the Kafka topic. For every active-active topic, record:

The cleanest active-active workloads partition ownership so one Region writes a given key at a time. A customer account can be assigned to a home Region; a device can follow a consistent regional owner; a tenant can be moved only through an explicit handoff. Replication then distributes the result, rather than asking two Regions to arbitrate a conflict after the fact.

Some workloads can merge concurrent updates. Counters, append-only telemetry, and event records with commutative operations can tolerate a different arrival order than a workflow state machine. That tolerance must be demonstrated at the consumer or sink. “The producer is idempotent” only addresses retries from that producer. It does not make two distinct business events with the same key equivalent.

A state transition such as pending → paid → refunded needs a stronger rule. Include a domain version or monotonic sequence in the event, reject an older transition, or make the sink apply a compare-and-set operation. If neither Region can prove which transition is newer, active-active has moved the conflict into application code where it is harder to observe and recover.

3Treat duplicate processing as a normal path

MSK Replicator synchronizes consumer group offsets by mapping source offsets to target offsets. AWS describes this translation as approximate. A translated offset can be slightly behind the equivalent source position, which favors at-least-once behavior. A failover can therefore cause consumers to read a small number of records again.

The offset pipeline has three useful stages: the Replicator records source-to-target mappings while it copies data, translates committed source offsets using those mappings, and commits translated offsets to the target __consumer_offsets topic. The target group is not overwritten when a consumer is already active there. A consumer started too early can therefore use its own reset policy instead of the translated position.

Design the downstream operation so a replay has a bounded result. Common patterns include a durable event ID with a uniqueness constraint, an idempotent upsert keyed by (entity_id, version), and a sink that records the source Region and source offset for audit. A retry of the same event should be a no-op or a safe update. If the side effect is an external payment, email, or inventory reservation, the idempotency key must cross the Kafka boundary into that service.

Measure the behavior before a production cutover. Test a consumer that is close to the tip of the stream, then repeat with intentional lag. AWS notes that lagging consumer groups can see higher offset lag on the target and may reprocess more records after failover. The test should record processed IDs, duplicates, application outcomes, and the time between the last source commit and the translated target commit. If you need a broader diagnostic method for lag, use the Kafka consumer lag guide as a companion to the MSK-specific runbook.

4Client routing is part of consistency

A regional architecture fails in a different way when the cluster is healthy but clients route to the wrong topic. Keep routing rules explicit in the client configuration and in the runbook.

With Prefixed replication, a Region B consumer might subscribe to .*topic so it receives both topic and A.topic. That is convenient for workloads where order across local and replicated records is irrelevant. For ordered workloads, the documented failover sequence is stricter: start consumers on the replicated topic, wait for replicated lag to reach zero, stop them, and then start consumers on the local topic. This creates a handoff boundary that the application can reason about.

With Identical replication, clients keep the same topic name, but the replication loop and charge model become more complex. A client can also accidentally produce into a replicated topic after failover if the producer path does not distinguish local ownership. Keep producer bootstrap servers, topic permissions, and deployment Region in the same configuration release so a traffic switch changes them together.

DNS can provide a stable application endpoint, but DNS does not move an in-flight Kafka connection. Consumers and producers need reconnect behavior, a bounded DNS cache policy, and a test that proves they establish sessions against the intended cluster. Capture the time from route change to successful produce and consume, along with any offset reset or rebalance observed by the client.

5Plan failover and failback as different tests

A planned failover is a controlled change. Stop producers and consumers in the source Region, create the reverse Replicator needed to copy new writes back, then start the target clients. Use the test to prove that the routing change, offset translation, ACLs, and data checks all work together.

An unplanned failover has a different boundary. The source Region may not be reachable, and asynchronous replication may leave records that were never copied to the target. Start target producers and consumers using the documented topic mode, then measure the gap with MessageLag and application-level event counts. Do not call the event lossless unless the workload and the measured lag justify that claim.

Failback is a third operation, not a reversal button. After the source Region recovers, replicate the target’s writes back to the source, wait for the reverse path to catch up, stop target producers, and restart source producers. With Identical topic names and an earliest starting position, MSK Replicator filters records that originated at the source, but AWS warns that the Replicator still processes data and may incur transfer charges. With Prefixed topics, the consumer sequence has to respect the same ordering boundary used during failover.

Build a recovery exercise around evidence:

1. Record the last source event ID, committed offset, MessageLag, and consumer lag.
2. Change the client route and record reconnect, rebalance, and first-success timestamps.
3. Compare event IDs at the target, including duplicates and missing IDs.
4. Verify IAM policies or Kafka ACLs on the target cluster. Replicator copies literal topic ACLs, but IAM policies must be configured separately, and write ACLs are not copied.
5. Repeat the steps in reverse for failback and retain the run output.

6When active-active is the right boundary

Active-active is a good fit when both Regions must serve traffic, key ownership can be separated or conflicts can be merged, consumers can safely reprocess events, and the team can test routing and failback. It is a poor fit when a workflow requires one total order, when side effects cannot be deduplicated, or when the client fleet cannot be made Region-aware.

Active-passive is often a better starting point for a stateful workflow. Only one Region writes at a time, the standby receives a continuous copy, and identical topic names can make the cutover less invasive. The cost is a planned handoff and a recovery window during which the standby becomes active.

A single-writer topology with remote readers can also meet regional read-latency goals without introducing two write authorities. The decision should be made per workload, not per cluster. One MSK cluster can host a telemetry topic that is active-active and an order-state topic that is active-passive, provided the client and permission boundaries remain clear.

7Where AutoMQ changes the deployment question

If the requirement is regional traffic placement, Kafka API compatibility, or a data plane that stays in the customer’s cloud account, compare the storage and deployment boundary separately from the replication semantics. AutoMQ is a Kafka-compatible cloud-native streaming platform with shared storage architecture. AutoMQ BYOC can run the data plane in a customer-controlled cloud environment, which may change network ownership, storage placement, and the way a team operates regional clusters.

That architecture does not remove the need for a conflict model. A shared storage layer can simplify broker replacement and capacity changes, but it does not decide which Region owns an order or whether a duplicate side effect is safe. If active-active semantics are the hard requirement, keep the same workload tests, event IDs, routing checks, and failback evidence when comparing platforms.

For teams that want to compare MSK Replicator’s operational boundary with a customer-controlled deployment, review the AutoMQ BYOC installation path. The AWS MSK storage cost model is useful context when replication and retained data are part of the same review. If the question is whether a regional design also needs a different storage boundary, compare Kafka compute-storage separation with Tiered Storage. The relevant question is not whether one product has a more attractive topology diagram. It is which system boundary your team can own, observe, and recover under the failure you actually need to handle.

8FAQ

8.1Does Amazon MSK active-active provide a global message order?

No. Kafka ordering is scoped to a partition, and two Regions can append to their local partitions independently. If a workflow requires a total order, assign one writer or introduce an application-level sequence and conflict rule.

8.2Is MSK Replicator bidirectional by itself?

A bidirectional design uses two unidirectional Replicators, A to B and B to A. Enhanced bidirectional offset synchronization helps consumers move in either direction, but data replication and offset translation remain asynchronous.

8.3Should active-active use Prefixed or Identical topic names?

AWS recommends Prefixed topic name replication for active-active because the local and replicated topics remain distinguishable and avoid the extra processing needed to prevent loops in Identical mode. Identical names can reduce client changes, but they add processing and transfer work and still require duplicate-safe consumers.

8.4Will consumers process duplicate records after failover?

They may. Offset translation is approximate and favors at-least-once delivery. Test the actual lag and make downstream effects idempotent before treating a failover as complete.

8.5Can MSK Replicator resolve two writes to the same key?

No. It copies records and selected metadata. The application must define key ownership, version checks, merge behavior, or rejection rules for concurrent writes.

8.6How should a team test failback?

Replicate writes from the active secondary cluster back to the primary, wait for the reverse path to catch up, switch producers, drain consumers according to the topic mode, and verify event IDs, lag, ACLs, and client reconnects. Keep the run output as recovery evidence.

9References

• Amazon MSK Replicator
• How MSK Replicator works
• Active-active replication patterns
• Active-passive replication patterns
• Topic naming: Prefixed vs Identical
• Consumer group offset synchronization
• Metadata and ACL replication
• Planned failover
• Unplanned failover
• Failback
• Apache Kafka producer idempotence
• Apache Kafka replication and ordering
• AutoMQ BYOC on AWS

The next useful step is a bounded failover exercise with your own keys, consumer groups, and side effects. Compare a customer-controlled streaming deployment when storage and operating boundaries are part of that review.