Table of Contents
Table of Contents
A cloud Kafka bill can grow even when producer throughput stays flat. The culprit is often not the producer payload itself, but the copies created after the leader accepts it. In a traditional multi-AZ deployment, a partition leader sends records to follower brokers so the cluster can maintain its replication factor. When those followers sit in other Availability Zones, the same record crosses an inter-zone network boundary as part of normal durability work.
That makes the bill easy to misread. A team sees stable ingress, stable retention, and stable broker count, then finds a variable data-transfer line. The missing step is the replica path. The useful question is which bytes are required for Kafka replication and which bytes are required for clients to use the log.
A different storage architecture can change that answer by changing the boundary below the Kafka protocol. It can give durable stream data a shared owner instead of asking every broker to hold a local copy. That may remove the broker-to-broker cross-AZ replication path, but it does not make every network or storage charge disappear. The deployment and storage choices still determine the final bill.
1The network line starts with replication
Apache Kafka uses a leader and follower model for partition replication. A Producer sends a record to the partition leader, and the leader appends it to its log. Followers fetch the leader's log and maintain their own copies. The acknowledgment policy, min.insync.replicas, and the producer's delivery settings determine when a write is considered successful, but the physical work of maintaining the copies remains.
For a first-order cost model, let B be the compressed bytes written by producers during a billing period and let R be the replication factor. The bytes sent from leaders to followers are approximately:
replication bytes ~= B * (R - 1) + protocol and retry overhead
Only the portion that crosses an Availability Zone boundary belongs in the inter-zone term. A topology with followers in the same zone has a different network profile from a topology that deliberately spreads replicas across zones. Compression, batching, retries, acknowledgments, provider-specific billing direction, and whether traffic stays on a private path all affect the bill, so this formula is a workload model, not a cloud-provider invoice.
The useful accounting exercise is to classify traffic by owner:
| Traffic path | Why it exists | Cost question |
|---|---|---|
| Producer to leader | The application writes records | Is the client in another zone, region, or network? |
| Leader to followers | Kafka maintains local replicas | How many replica bytes cross a billable zone boundary? |
| Broker to Consumer | Clients fetch records | Where are consumers placed, and how often do they replay? |
| Broker or storage to object storage | A shared-storage system persists or reads data | Are requests, retrieval, and egress billed by the provider? |
This table separates two things that are often combined under "Kafka traffic." A change that removes follower replication can materially change the second row without changing the first or third. That is the narrower claim an architecture review should test.
The implication is important for FinOps. Reducing retention or compressing records may reduce the number of bytes stored or transmitted, but neither changes the fact that the traditional design creates a copy for each follower. Moving followers closer to leaders can change the rate or route, yet it also changes the failure-domain decision. If the recurring cost is structural, tuning the rate only treats the symptom.
2Evaluate the storage boundary before choosing a fix
There are several ways to respond to an inter-zone replication charge, and none should be selected from the bill line alone. Lowering replication factor changes the durability and availability objective. Concentrating brokers in one zone changes the failure model. Using a provider's managed Kafka service may simplify operations while preserving a broker-local replication path. Tiered Storage can move older records to object storage while keeping a local log for active data, but it does not automatically remove the local replica traffic for the hot path.
The neutral evaluation starts with four questions:
- Which bytes are durable at acknowledgment? Record the producer acknowledgment mode, the WAL (Write-Ahead Log) or disk boundary, and the replicas that must be in sync.
- Who owns the second copy? It may be another broker, a regional block-storage service, or a shared object-storage system. Those are different failure and billing boundaries.
- What happens on a read? Tailing reads, catch-up reads, and consumer replays exercise different caches, storage requests, and network paths.
- What happens after a node change? Capacity expansion, broker replacement, and partition reassignment can move metadata only, or they can move large amounts of durable data.
A replacement architecture is relevant when the recurring bill comes from the first copy being replicated between broker-local stores. It is less relevant when the main issue is a consumer in another region, an unbounded replay, an object-storage retrieval pattern, or an oversized retention policy. The same ledger should cover both designs: producer bytes, retention, consumer reads, WAL or disk, object requests, compute, and provider network charges.
That evaluation leads to a concrete requirement. To remove broker replication traffic without weakening the durability objective, the durable stream must have a shared owner that is reachable by the brokers, while the brokers retain enough local state to handle Kafka requests and recover safely. Storage has to be shared without turning every write into an unbounded object-storage round trip. This is where a Shared Storage architecture becomes a mechanism rather than a slogan.
3Why WAL on Shared Storage changes replication
AutoMQ is a Kafka-compatible stream storage platform built around Separation of compute and storage. Its brokers handle Kafka protocol requests, partition leadership, caching, and scheduling. S3Stream writes the stream through a configured WAL storage layer and then uploads or compacts data into S3 storage. The broker is still in the write path, but a second broker does not need to maintain a durable Kafka replica because the partition has to survive a broker failure.
The WAL is the part that makes this boundary practical. A WAL is a fixed-size, circular persistent cache used for durable writes and recovery. It absorbs writes before background upload and compaction turn them into the longer-lived object layout. The exact latency and cost profile depends on the selected WAL type. AutoMQ Open Source uses S3 WAL, while AutoMQ commercial editions can support other choices such as Regional EBS WAL or NFS WAL. Those choices have different storage, failure-domain, and performance implications, so "AutoMQ cost" is not a complete workload description by itself.
The resulting path looks like this:
Producer
-> AutoMQ Broker
-> S3Stream append
-> WAL storage
-> S3 storage upload and compaction
-> Consumer fetch through a broker
The key difference is ownership. Traditional Kafka keeps durable partition copies on multiple broker-local stores. AutoMQ Brokers are stateless with respect to long-term partition data, while shared storage holds the persistent stream. KRaft metadata and the storage layer coordinate ownership, offsets, and recovery. A broker replacement therefore does not inherently require a new broker to receive a full local replica before it can serve the partition.
This does not mean the object store is a magic replica that costs nothing. The provider manages its own durability implementation, and the customer still pays according to the object-storage service's storage, request, retrieval, and network terms. The architectural change is that customer-visible broker replication no longer has to be the mechanism that creates those durable copies across zones.
4Reads do not recreate the broker replication bill
Removing follower replication does not remove the need to serve records. A Consumer still connects to a broker, and that broker still reads from the fastest available layer for the requested offsets. A Tailing Read may be served from data caching or data near the WAL. A Catch-up Read may require prefetch from S3 storage before the broker returns historical data.
The important distinction is between broker-to-broker replication and broker-to-client serving. A consumer fetch is not a follower replica, even when it crosses a zone boundary. If the consumer is placed in another zone or region, the provider may charge for that client path according to its network rules. If the broker retrieves objects from a different region, retrieval or egress terms may apply. Those costs remain outside the claim that AutoMQ removes the broker replication line.
The deployment model matters too. In AutoMQ BYOC, the control plane and data plane run in the customer's cloud account and VPC (Virtual Private Cloud), while the customer owns the object-storage bucket and network placement. A bucket's region, endpoint type, private connectivity, broker placement, and client placement all belong in the design review. AutoMQ's Zero Cross-AZ Traffic documentation describes the architecture-level goal, but it should not be read as a blanket promise that every client or storage request is free of every inter-zone charge.
For a replay-heavy workload, measure at least these quantities separately:
- bytes produced and acknowledged;
- WAL writes and recovery reads;
- object PUT, GET, LIST, and retrieval activity as reported by the provider;
- bytes served to Consumers, including replay and catch-up traffic; and
- cross-AZ, cross-region, or external egress attributed to clients and storage.
The separation prevents a common accounting error: moving the durable copy to shared storage, then treating all object GETs or consumer egress as if they were the old replication charge. The paths are different, so the owners and mitigations are different.
5The bill changes, it does not vanish
An honest comparison uses formulas before rates. Let S be retained bytes, W the number of WAL bytes written, P and G the object PUT and GET counts, D the object bytes retrieved, C broker compute, and E client egress. A simplified monthly model for the shared-storage design is:
shared-storage cost ~= storage(S)
+ requests(P, G, ...)
+ retrieval(D)
+ WAL(W)
+ compute(C)
+ client egress(E)
+ other platform charges
The corresponding traditional model adds broker-local storage and the provider's applicable inter-zone replication term:
broker-replication cost ~= local storage(S * R)
+ compute(C)
+ cross-AZ replication(B * replica share)
+ client egress(E)
+ other platform charges
These are deliberately not price quotes. Substitute the provider's current rate card, region, storage class, and billing rules. AWS, Azure, and Google Cloud do not expose identical network products or charge every direction the same way. The AWS data transfer pricing page, AWS S3 pricing, and your provider's managed Kafka documentation are the right places to validate the local terms before making a budget decision.
This ledger makes the accounting boundary visible. The old line is the broker-to-broker copy. The new lines are object requests, retained object data, WAL storage, and any client or regional egress that the workload creates. A design wins only when the removed line is larger than the replacement terms for the same availability, retention, and read behavior.
6Where the mechanism fits, and where it does not
AutoMQ is a candidate when all of the following are true: inter-zone replication is a material recurring cost, the workload needs durable data across a broker failure domain, and the team can place shared storage and clients deliberately. Long retention, frequent broker changes, or capacity cycles that repeatedly trigger data movement strengthen the case because storage is no longer bound to the lifecycle of a broker.
The case needs a workload test when the dominant pattern is low-latency tailing, high-frequency catch-up, log compaction, large consumer fan-out, or a non-default WAL type. Measure producer acknowledgment latency, Consumer lag, replay time, object request volume, and failure recovery with the actual client mix. The architectural benefit can be real while the workload still needs cache or storage tuning.
Fix the local cause first when the bill comes from a consumer that should be co-located, a replay job that lacks a bounded window, a retention setting that exceeds the business requirement, or a cross-region client path. No storage architecture can turn an avoidable egress path into a free one. The fit test is a short ledger exercise followed by a failure and replay test, not a comparison of product adjectives.
The first line on your Kafka invoice still matters. If it represents follower copies crossing zones, AutoMQ's Shared Storage architecture removes that particular work by giving durable stream data a shared storage owner and using WAL to keep writes and recovery practical. If it represents clients, retrieval, or another boundary, the answer lies elsewhere in the ledger. Teams evaluating the change can start with the AutoMQ architecture overview, verify the WAL storage choices, and run the same traffic and failure tests against their current bill. For a BYOC evaluation, open AutoMQ BYOC with that ledger in hand.
7References
- AutoMQ architecture overview
- AutoMQ Zero Cross-AZ Traffic
- AutoMQ WAL storage
- AutoMQ compatibility with Apache Kafka
- Apache Kafka design documentation
- AWS data transfer pricing
- AWS S3 pricing
8FAQ
8.1Does AutoMQ have zero network cost?
No. The narrower claim is that its Shared Storage architecture can remove the broker-to-broker cross-AZ replication path for an applicable deployment. Client reads, object-storage requests and retrieval, WAL traffic, cross-region movement, and external egress still need to be measured.
8.2Does AutoMQ replace Kafka replication with object-storage requests?
It replaces the broker-local durable replica path with S3Stream, WAL storage, and S3 storage. Object-storage operations and their pricing remain part of the design. The exact request and retrieval pattern depends on retention, compaction, caching, tailing reads, catch-up reads, and the selected WAL type.
8.3Does the WAL have the same behavior in every AutoMQ deployment?
No. The WAL type is a deployment choice with its own latency, durability, network, and storage characteristics. AutoMQ Open Source uses S3 WAL; AutoMQ BYOC and AutoMQ Software may support other WAL types. Cost comparisons must name the actual configuration.
8.4Will Consumers in another Availability Zone avoid all network charges?
No. The removal applies to broker-to-broker replication traffic. Consumer-to-broker traffic and any object-storage retrieval or egress remain subject to the provider's placement and billing rules.
8.5Does Kafka client code need to change?
AutoMQ is designed for Kafka protocol and semantic compatibility, but a production evaluation should still test client versions, transactions, security, connectors, Consumer group behavior, monitoring, and failure recovery with the existing workload.
