Table of Contents
Table of Contents
Cross-AZ traffic in Amazon MSK usually begins as a placement decision. A producer runs in one Availability Zone, a partition leader runs in another, and a consumer group is spread across a third. The cluster remains healthy, but the path adds latency and may create a data-transfer line on the AWS bill. During an Availability Zone failure, the same locality preference can become a recovery constraint.
The useful goal is therefore not “keep every byte in one AZ.” A production design has to keep normal reads and writes close to their clients while leaving enough replica and failover capacity across failure domains. That means separating three questions: which client paths should be local, which replica movement is required for durability, and which remote paths are acceptable during recovery.
1Locality has three different jobs
Apache Kafka traffic crosses zones for different reasons, and each reason needs a different control. Producer locality is about the route from an application to the partition leader. Consumer locality is about the route from a fetcher to a replica that can serve the read. Replica placement is about keeping enough copies in separate failure domains. Treating all three as “network traffic” hides the decision that created the bytes.
| Path | Normal objective | What can force a remote path |
|---|---|---|
| Producer → leader | Keep the write path close to the producer when practical | Partition leadership, uneven client placement, or an AZ event |
| Leader → follower | Place replicas across failure domains and keep them in sync | Multi-AZ durability and replica recovery |
| Consumer → replica | Prefer a nearby eligible replica for the read | Leader-only reads, client rack settings, lagging local replicas, or failover |
| Broker or client recovery | Restore service and catch up safely | Broker loss, reassignment, replay, or an AZ outage |
The distinction matters because a consumer optimization cannot remove replica replication, and replica placement cannot guarantee that every consumer fetch is local. The design is a set of compromises around failure boundaries.
2Map the traffic before changing the topology
Start with a graph of subnets, Availability Zones, producers, brokers, consumers, connectors, and sinks. Record where the application runs, where the MSK brokers are placed, and which topics are read by multiple groups. Add the paths used during a rebalance, replay, or broker replacement; they often have a different shape from the steady-state path.
A simple inventory should answer:
- Which workloads produce or consume from each subnet and AZ?
- Which topics have high fan-out or long replay windows?
- Which consumers can read from a follower, and which require leader ordering or a specific acknowledgement path?
- Which connectors, stream processors, and sinks run outside the MSK VPC or in another AZ?
- Which path is allowed to become remote when an AZ or broker is unavailable?
The last question prevents a common mistake: optimizing the bill by making the failure plan implicit. If the design only works while every client and preferred replica stays healthy, the locality policy is too strict for production.
Use AWS billing data as a second view of the graph. AWS publishes EC2 data-transfer pricing separately from MSK service pricing, and the applicable rate depends on the service path, Region, direction, and current pricing terms. Do not apply a universal per-GB number to an MSK estimate. Instead, join the traffic map to the line items for the account and test the paths that the workload actually uses.
3Place producers and consumers with intent
The most efficient byte is usually the one that does not leave its Availability Zone, but forcing all clients into one zone trades a network line item for a larger failure domain. A better pattern is to place clients in the same AZs as the brokers they normally use, spread the application across at least the failure domains required by its availability objective, and let the client fail over when the preferred path is unavailable.
For producers, partition leadership is the first constraint. A producer can be close to its application subnet and still cross an AZ if the partition leader is elsewhere. Partition distribution, key skew, and leader imbalance must be checked together. Changing a subnet without checking leadership usually changes only one part of the path.
Consumers have a second lever: replica selection. Apache Kafka exposes rack-aware concepts through broker rack metadata, consumer rack metadata, and a replica selector. A consumer can express its location, and a selector can prefer a replica in the same rack or zone when that replica is eligible. The exact configuration and support boundary depend on the Kafka version and the managed service controls available to the cluster, so test the behavior on the MSK version and client library you operate.
Locality should be a preference with a fallback, not a hard requirement. If the local replica is behind, unavailable, or excluded by the isolation policy, the consumer needs a defined remote path. Measure both outcomes:
| Measure | Why it matters |
|---|---|
| Local fetch ratio | Shows whether the placement policy works during normal operation |
| Cross-AZ fetch bytes | Connects locality behavior to the network bill |
| Fetch latency by AZ pair | Separates placement delay from broker or client saturation |
| Consumer lag during local-replica loss | Tests whether the fallback path preserves the recovery objective |
| Rebalance and reassignment bytes | Captures the cost of topology changes, not only steady-state traffic |
A high local fetch ratio is useful only if the consumer still catches up when the preferred replica disappears. The recovery test is the part that turns a topology diagram into an operating contract.
4Rack awareness protects the failure model
Kafka rack awareness is often described as a placement feature, but it is really a statement about correlated failure. Brokers marked with different racks or zones allow replica placement to avoid putting all copies in one failure domain. On MSK, AWS manages the service infrastructure and broker placement behavior, while your team still has to understand the cluster’s AZ layout, topic replication settings, and client routing.
Replica separation and consumer locality can pull in opposite directions. A three-copy topic may place replicas across three AZs, which is good for availability but guarantees that some replica traffic and recovery traffic are remote. A consumer that always reads the leader may add another remote path. A consumer that prefers a local follower may reduce normal read distance, while increasing the amount of lag and availability state you need to watch.
That trade-off should be visible in the design review. Use the following sequence:
- Confirm the failure domains in which replicas may be placed.
- Confirm the client AZ metadata and the selector behavior for the Kafka version in use.
- Verify that local reads remain eligible when the follower is catching up.
- Kill or isolate the preferred broker in a staging test.
- Record fetch latency, lag, bytes by AZ pair, and the time to return to the preferred path.
A locality policy that has never been exercised under a broker or AZ fault is only a cost hypothesis.
5Failover changes the meaning of “nearby”
During a failure, the nearest healthy replica may be in another AZ. That remote hop is not a defect; it is the mechanism that keeps the stream available. The design problem is deciding how much latency and network movement the application can tolerate while the preferred path is restored.
Write the fallback contract for each important consumer group. It should state the maximum acceptable catch-up lag, whether a remote replica is allowed, how long a rebalance may take, and which alert signals require an operator. Include connectors and stream processors; they often have different retry and timeout behavior from application consumers.
A useful review compares three operating states:
| State | Preferred path | Evidence to collect |
|---|---|---|
| Steady state | Same-AZ client to preferred broker or replica | Fetch latency, local ratio, cross-AZ bytes |
| Degraded state | Remote healthy replica or broker | Lag growth, retry rate, recovery throughput |
| Recovered state | Locality restored after leadership or placement changes | Rebalance duration, data movement, residual lag |
This framing avoids the false choice between low cost and high availability. The normal path can be local, while the degraded path remains explicit and measurable.
6Build a cost model from bytes, not slogans
Cost analysis should start with a traffic ledger. For each topic or workload, estimate the monthly bytes for producer writes, replica synchronization, consumer fetches, connector traffic, and recovery or reassignment. Mark the source and destination AZ for each path, then apply the AWS pricing rule that corresponds to that path.
The calculation is intentionally simple:
Monthly cross-AZ cost = billable cross-AZ GB × the applicable regional data-transfer rate.
The hard part is the numerator. A topology with three consumer groups can produce very different traffic from a topology with one group, even when the producer ingress is identical. Compression, batching, fetch behavior, follower reads, retention replay, and rebalances all change the bytes that move.
Use a worksheet with an assumption column rather than one headline estimate:
| Input | Example question | Validation source |
|---|---|---|
| Producer bytes | Which AZ hosts the producer and partition leader? | Client metrics and partition leadership |
| Replica bytes | Which replica pairs cross AZ during normal replication? | Cluster topology and broker metrics |
| Consumer bytes | Can each group fetch from a local eligible replica? | Client configuration and fetch metrics |
| Recovery bytes | What data moves after a broker or AZ failure? | Failure drill and reassignment logs |
| Rate | Which AWS data-transfer price applies to this path? | Current AWS pricing page and account bill |
This model also exposes where a tuning change can help. Rack-aware consumer reads may reduce fetch bytes, but they do not remove cross-AZ replica synchronization. Moving an application to a new subnet may improve one path while making another remote. Review the ledger after every topology or client change.
7When a different storage model changes the question
If most of the cross-AZ movement comes from broker-local replica synchronization, the limit is architectural rather than a missing client setting. The brokers are doing two jobs at once: serving Kafka requests and holding durable copies on local disks. Scaling, recovery, and retention then move data between brokers as the cluster changes.
A Shared Storage architecture separates those responsibilities. AutoMQ is a Kafka-compatible cloud-native streaming platform that uses object-storage-backed durability and stateless brokers. In supported deployment patterns, its documented zero cross-AZ traffic approach changes which bytes need to travel between brokers; compute placement and client routing still require measurement.
That is a different decision boundary, not a universal replacement for MSK. Teams still need to validate Kafka protocol features, consumer behavior, object-storage access, WAL choice, recovery objectives, and deployment ownership. AutoMQ's cross-AZ traffic documentation explains the supported routing assumptions. The right comparison is whether the workload's durable-data path, client locality, and failure contract fit the architecture.
8A locality decision matrix
Use this matrix before changing MSK client or broker settings:
| Workload condition | Starting choice | What to verify |
|---|---|---|
| Low fan-out, clients already spread by AZ | Prefer local client and replica paths | Local fetch ratio and failover lag |
| Many consumer groups or replay-heavy reads | Measure fan-out before adding brokers | Read bytes by AZ pair and downstream capacity |
| Strict single-AZ latency target | Keep a remote fallback and an AZ failure test | Recovery SLO and acceptable degraded latency |
| High replication or reassignment traffic | Review storage and replica movement assumptions | Traffic ledger and recovery drill |
| Retention growth outpaces active compute | Compare broker-local and Shared Storage models | Storage growth, compute headroom, and recovery cost |
The matrix is deliberately conservative. It makes the failure path a first-class input, so a cost optimization cannot silently reduce availability.
9FAQ
9.1Does Amazon MSK automatically keep consumer reads in the same AZ?
No. Locality depends on where clients run, where partition leaders and replicas are placed, and how the Kafka client selects replicas. Verify the behavior with client and broker metrics on the MSK configuration and Kafka version you use.
9.2Does rack awareness eliminate cross-AZ traffic?
No. Rack-aware placement and replica selection can reduce avoidable reads and improve failure-domain separation, but multi-AZ replication and recovery can still move data across zones. Model each path separately.
9.3Is cross-AZ traffic always a large MSK cost?
Not necessarily. The impact depends on throughput, replication, consumer fan-out, replay, and the applicable AWS pricing path. Use your account’s data-transfer line items and a topology-specific byte model.
9.4Should all MSK consumers run in one Availability Zone?
That can reduce normal read distance, but it concentrates application capacity in one failure domain. Keep enough multi-AZ capacity for the recovery objective, and test what happens when the preferred zone is unavailable.
9.5When should I evaluate a Shared Storage Kafka architecture?
Evaluate it when retention, active compute, replay, or replica movement grows independently and the traffic ledger shows broker-local storage as a recurring source of cross-AZ bytes. Run the same client and failure tests against the candidate architecture.
10References
- Amazon MSK Developer Guide
- Connecting to an Amazon MSK cluster
- Amazon MSK pricing
- Amazon EC2 data transfer pricing
- Apache Kafka broker and replica configuration
- Apache Kafka rack-aware replica selector
A consumer that is local during normal traffic is useful only when its remote fallback is measurable. Start with the traffic graph, run the AZ failure drill, and compare the resulting byte ledger with the AWS bill. If broker-local replication is the dominant path, review a Kafka-compatible Shared Storage operating model with AutoMQ using the same locality and recovery contract.
