Object Storage Is Cheap, but Kafka Access Patterns Decide the Real Bill

At the end of a month, a platform team sees a small object-storage line beside a much larger set of Apache Kafka charges. The storage line looks reassuring until a Saturday replay starts, three analytics consumers scan the same retention window, and a compaction job rewrites keyed records. The bill rises even though the retained data volume has barely changed.

That is the cost question behind Kafka on S3: how often does the system write, read, retrieve, rewrite, and move the bytes? A per-gigabyte storage rate answers only one part. The rest depends on access patterns, object layout, cache behavior, retention, and where the reader sits relative to the storage service.

1A low per-gigabyte rate is not the whole story

Object storage changes the capacity calculation because retained data is priced by storage class and time rather than by a pre-provisioned disk volume. That can suit a topic with long retention and uneven utilization. It does not make Kafka a passive archive. Producers, consumers, replay jobs, log cleanup, compaction, metadata operations, and network paths still create work.

The important distinction is between capacity cost and activity cost. Capacity cost grows with bytes kept and days in a class. Activity cost grows with operations and bytes moved. A 20 TiB archive can have lower activity cost than a 2 TiB workload that is repeatedly scanned, depending on storage class, request mix, and network boundaries.

Before comparing storage architectures, write down the workload shape:

• Ingested bytes per day, including peaks and producer retries.
• Retained bytes and retention duration by storage class.
• Tailing versus historical bytes read by each Consumer group.
• Compaction, backfill, export, and recovery read volumes.
• Object operations, average object size, and each reader's network path.

This turns “Kafka S3 cost” into a ledger a FinOps team can reconcile with provider usage reports. It prevents a storage rate from becoming a proxy for total cost of ownership.

2Storage, requests, retrieval, and egress are four different meters

Object-storage providers expose different names and exceptions, but a Kafka estimate usually needs four separate meters. Storage is the recurring charge for retained capacity. Requests are operations such as PUT, GET, LIST, and metadata calls. Retrieval applies to certain colder storage classes when data is read back. Egress or data transfer applies when bytes cross a provider-defined boundary, such as a region, service, account, or public network path.

The different units require a workload ledger rather than a single “price per GB” shortcut.

The formula is deliberately unexciting:

Object-storage access cost
= storage cost
+ request cost
+ retrieval cost
+ egress or transfer cost

It is not a complete Kafka TCO (Total Cost of Ownership) model. Compute, broker memory, WAL storage, observability, and operations belong in separate lines. Keeping them separate matters because moving Kafka records to object storage can lower local-disk capacity while changing the write path, cache requirement, or network pattern. A fair comparison shows which line moved and why.

Rates vary by region, storage class, request type, minimum billable duration, and transfer rules. Use the current provider page for the final number, and record the region, account layout, and estimate date.

3Which Kafka access patterns hit which meter

The same Topic can hit all four meters, but not at the same time or with the same intensity. Treat each access pattern as a distinct path through the storage system.

Tailing reads follow the newest Offset. A Consumer group that stays near the head of the log may be served mostly from data caching or a WAL-adjacent path, depending on the Kafka implementation. Its storage bill can be dominated by retained capacity and steady writes rather than historical retrieval. The check is cache hit behavior, not the consumer's average throughput alone.

Catch-up and replay reads reach older data. A backfill that reads a week of history after a schema change can turn a quiet topic into a burst of GET operations and retrieved bytes. If the replay runs across a region or through an external analytics account, the same event can add a transfer line. Record the bytes requested, bytes actually returned, and whether the source storage class applies a retrieval charge.

Log compaction rewrites the physical representation of a keyed Topic while preserving the latest value for each key. The logical result may be smaller, but the maintenance path still reads and writes objects and updates metadata. Compaction can therefore reduce long-term storage while increasing request activity during the rewrite window. Measure both outcomes instead of assuming that a smaller final dataset means a lower monthly bill.

Fan-out reads occur when several Consumer groups need overlapping history. One reader may be inexpensive when its working set is hot in cache. Ten readers with different offsets can create repeated fetches, more prefetch, and more transfer across the relevant boundary. Consumer count is not a price by itself, but it is a useful prompt to measure overlap and locality.

Producer behavior belongs in the analysis. Record size, compression, batch size, flush policy, and retries affect how many logical records fit into each uploaded object. Larger batches are not automatically better, because latency and recovery requirements set a floor on batching.

For a first estimate, use a separate row for each pattern instead of averaging them:

Monthly retrieval cost = cold bytes read by replay and recovery × retrieval rate
Monthly request cost = operations by type × request rate
Monthly transfer cost = bytes crossing each billed boundary × transfer rate

This makes a replay spike visible. It also stops a high-volume tailing workload from hiding a low-frequency operation that drives a retrieval minimum or a cross-region transfer charge.

4Workload types and their object-storage bills

Different Kafka workloads stress different meters. The categories below are not product tiers. They are a way to decide what to measure before selecting a storage architecture.

An event log often has predictable writes and mostly tailing reads. Its object-storage bill may be storage-heavy if retention is long, with request activity shaped by object aggregation. An audit stream can have a low daily read rate and still produce a material retrieval charge when an investigation opens a large historical window. CDC sits between the two: its normal path is steady, but recovery and backfill can dominate a month.

Analytics fan-out is the case most likely to invalidate a storage-only estimate. A shared object store may avoid duplicate physical copies of data, but it does not make repeated logical reads disappear. If each consumer scans a different time window, the system still has to serve those bytes, and the network path still matters.

Use an explicit scenario before pricing. Assume a Topic retains 2 TiB, receives 200 GiB per day, and has three Consumer groups. Do not infer a bill from 2 TiB alone. Add tailing reads, one planned replay, compaction rewrite volume, object operations, and the fraction crossing a billed boundary. Change one assumption at a time and note which meter moves.

That scenario also exposes topology: is the storage endpoint close to the brokers, do analytics readers use another account, and does a recovery copy leave the region? “S3 Kafka pricing” is incomplete until these boundaries are written down.

5Keeping request amplification under control

Request amplification is the ratio between storage operations or bytes fetched and the useful records delivered to an application. It is not one fixed property of object storage. It emerges from object layout, batching, cache policy, prefetch, compaction, replay concurrency, and consumer offsets.

Start with measurements tied to the four meters:

1. Group request counts by operation type and record the distribution of object sizes.
2. Separate tailing, catch-up, compaction, export, and recovery traffic in dashboards.
3. Compare bytes fetched with bytes delivered to the application, including cache hits and misses.
4. Plot retrieval and transfer activity against replay windows, consumer lag, and maintenance jobs.

Then tune the dominant path. Larger object aggregation can lower write-side request counts, but oversized objects may make catch-up reads less selective. More prefetch can improve replay throughput while fetching unused bytes. A longer cache can reduce repeated reads at the cost of memory. Compaction can lower retained capacity while creating a temporary request wave.

The target is a predictable cost and latency profile that still meets durability, recovery, and retention requirements. A request reduction that delays acknowledgments or makes recovery too slow has moved cost into reliability work.

6When shared storage changes the comparison

The cost model also exposes a topology requirement. If every broker owns durable local data, a cluster spread across Availability Zones needs broker-to-broker replication for the replication factor. That can create cross-AZ traffic as well as local disk or EBS (Elastic Block Store) capacity. Tiered Storage can place older log segments in object storage while keeping recent data on local disks, but it does not by itself make brokers stateless or remove the local retention and replication model.

If the goal is to separate Kafka compute from persistent storage, the required capabilities are clearer: Kafka protocol compatibility, a shared durable storage layer, a low-latency write-ahead path, cache-aware reads, and a way to replace or scale brokers without copying every retained byte. The cost comparison still has to include object requests, retrieval, and transfer after those capabilities are in place.

AutoMQ is a Kafka-compatible streaming platform built around a Shared Storage architecture. Its S3Stream layer moves the durable data path away from broker-local log storage, while stateless brokers handle Kafka requests, ownership, caching, and scheduling. A WAL provides the write and recovery boundary, and the exact WAL cost depends on the deployment: AutoMQ Open Source uses S3 WAL, while AutoMQ commercial editions can use EBS WAL, Regional EBS WAL, or NFS WAL where the latency and fault-domain requirements call for them.

This changes which line to inspect. Traditional In-Sync Replicas (ISR) replication between brokers is no longer the same data-movement path, so its cross-AZ replication traffic can be removed from the comparison. Network costs still remain for clients, storage endpoints, exports, control-plane traffic, and cross-region designs. Shared Storage also does not make replay free: historical reads can still create requests, retrieval, cache misses, and egress.

The S3Stream shared storage documentation describes the storage path, including WAL, data caching, and object handling. Use it as an architecture reference, then run the same workload ledger against a representative deployment. The meaningful comparison is not “local disk versus S3” in isolation. It is local disk, broker replication, and recovery behavior versus shared storage, WAL, cache, object requests, retrieval, and transfer under the same workload.

7A decision guide for Kafka on object storage

Object storage is a strong candidate when the workload has long retention, variable volume, and historical access that can be measured and controlled. It deserves more scrutiny when replay is frequent, consumers are geographically scattered, or the chosen storage class charges materially for retrieval. A simple decision sequence is:

• If storage dominates and historical reads are rare, compare retained GiB-month plus recovery objectives.
• If requests dominate, inspect object aggregation, flush cadence, metadata behavior, and cache misses before changing storage class.
• If retrieval dominates, model the replay calendar and test whether data can stay in a class that meets the recovery objective.
• If transfer dominates, place brokers, storage endpoints, and readers deliberately, then separate cross-AZ, cross-region, and public egress lines.
• If broker replication and local-disk movement dominate, evaluate Shared Storage and stateless-broker designs, with WAL type and failure domain stated explicitly.

Return to the Saturday replay that exposed the bill. The retained-data number did not explain it because the replay was an access event, not a capacity event. Put its reads, requests, retrieval, and transfer in separate rows, then compare architectures using the same rows. That is how a low per-GB storage rate becomes an engineering decision instead of a pricing slogan.

For a hands-on architecture and workload review, start with AutoMQ's evaluation path and bring the tailing, replay, compaction, and fan-out measurements with you.

8References

• Amazon S3 pricing
• Amazon S3 storage classes
• AWS data transfer pricing
• Apache Kafka design documentation
• Apache Kafka log compaction
• KIP-405: Kafka Tiered Storage
• AutoMQ S3Stream shared storage

9FAQ

9.1Is Kafka on S3 always lower cost than Kafka on EBS?

No. The result depends on retained capacity, WAL type, request count, retrieval, egress, compute, recovery objectives, and provider pricing. Compare the same workload ledger for both designs, including replay and recovery reads.

9.2Which Kafka workload fits object storage well?

Long-retention workloads with predictable tailing and controlled historical reads are useful candidates. Audit, event-log, and CDC workloads still need recovery and replay tests. Analytics fan-out deserves extra attention because repeated reads can outweigh storage.

9.3Does Kafka Tiered Storage have the same cost profile as a diskless Kafka architecture?

No. Tiered Storage can offload older segments while keeping local storage and replication for recent data. A diskless Shared Storage architecture changes durable ownership and broker replacement. Compare local storage, replication, WAL, cache, and object access rather than treating both as “Kafka on S3.”

9.4What is the most common object-storage cost mistake?

Counting retained GiB and ignoring activity. Replay, compaction, small-object writes, fan-out reads, cache misses, and cross-boundary traffic can each move a different bill line. Measure operations and bytes by access pattern.