Table of Contents
Table of Contents
It is the first cost review after a tiered-storage rollout. Broker disks are no longer close to full, so the storage line looks healthier. Then the FinOps dashboard shows object-storage requests, retrieval, and network charges climbing. The total has barely moved.
That outcome is not unusual. Moving an old Kafka segment to object storage changes where bytes live, but it does not make reads, metadata, replication, or recovery free. Tiered Storage can reduce the cost of long retention when old data is rarely touched. It can also turn a disk-capacity problem into a request and retrieval problem when "cold" data is part of a routine replay path.
The useful question is not whether object storage has a lower capacity rate. It is whether the cost avoided on broker storage is larger than every cost added by the remote path for this workload.
1The case of the bill that did not drop
Consider a platform team that retains event history for audit and replay. Producers write continuously, most consumers stay near the log end, and a policy moves closed segments after a local retention window. During normal traffic, the design looks sensible: active data stays near the broker, while older segments occupy a remote tier.
The surprise arrives during a month-end backfill. An analytics consumer reads several weeks of history, a compliance job repeats part of the scan after a validation failure, and a recovery exercise pulls data from more than one availability zone. The local-disk graph still looks better than before, but the storage report now has more lines to explain.
This is the boundary that a storage-only estimate misses. A tiered design still has a local tier, a remote tier, and a handoff between them. In the Apache Kafka tiered-storage model described by KIP-405, remote log storage extends retention beyond the broker-local log; it does not turn every broker operation into an object-storage operation with one uniform price or latency.
The local tier remains part of the operating model. Active segments, indexes, caches, replication traffic, headroom, and failure recovery still need capacity. Confirm which segments are copied, when local deletion happens, how remote reads are served, and which network path they take.
Age is only a proxy for coldness. A segment scanned every morning is economically active even when it is old. The bill follows access decisions, not the timestamp alone.
2The savings formula includes the request side
Start with a model that separates the avoided local capacity from the cost of keeping and reading the remote copy:
net monthly benefit = local storage cost avoided
- remote storage cost
- object write and read request cost
- retrieval cost
- egress and cross-zone transfer cost
- cache, WAL (Write-Ahead Log), and operational cost added by the design
For a more explicit estimate, define:
local storage avoided = cold bytes x local price per byte-month
remote storage cost = cold bytes x remote price per byte-month
request cost = PUT requests x PUT rate
+ GET requests x GET rate
retrieval cost = retrieved bytes x retrieval rate
network cost = charged bytes x applicable transfer rate
The prices must come from the provider, region, storage class, and deployment model under review. AWS, for example, publishes Amazon S3 pricing separately from Amazon EBS pricing; the same comparison pattern applies to other clouds. Do not copy a price from a different region into a business case, and do not assume that a request is free because the stored byte is inexpensive.
An illustrative calculation can expose the break-even point without pretending to be a quote. Suppose a topic has C bytes that would otherwise remain on local storage, and the remote design creates R requests plus D bytes of retrieval and transfer during the billing period. The design saves money only when:
C x (local capacity rate - remote capacity rate)
> R x request rate + D x retrieval and transfer rate + added operations
If the right-hand side changes sharply during replay, use separate rates for tailing, historical, and recovery reads. Small objects or frequent segment transitions can multiply request counts even when byte volume looks modest. Include metadata reads, lifecycle transitions, monitoring, and retries when they are material.
That gives FinOps, site reliability engineers (SREs), and platform teams one worksheet for the review.
3How to judge hot from cold in your own topics
The first measurement is the distance between the consumer offset and the log end. A narrow distance with sustained tail consumption usually means the consumer is using the hot path. A large distance is not automatically cold, though, because a backfill can make a historical range the busiest part of the system for a short period.
Measure each topic over a representative operating window and separate ordinary traffic from planned replays. At minimum, capture:
| Signal | What it tells you | Cost question it answers |
|---|---|---|
| Tail-read share | How much consumption stays near the log end | How large must the hot tier and cache remain? |
| Replay depth | How far consumers regularly move behind the log end | How many remote bytes can one incident or job touch? |
| Historical read frequency | How often older ranges are opened again | Is the remote tier actually cold between reads? |
| Read fan-out | How many consumers scan the same range | Will one retained range create repeated GET and retrieval work? |
| Segment and object size | How many storage objects represent the data | Could request count dominate byte savings? |
The measurement should be topic-aware. An audit topic may have low read frequency but high evidence requirements. A change-data-capture topic may see periodic reprocessing after an application change. A compacted topic may have a different physical read and rewrite pattern from an append-only event log. They can share a cluster while needing different tiering decisions.
Use age-based policies as a starting control, then compare them with observed access. If the policy says data is cold after one day but dashboards show daily scans through day ten, the policy is describing age, not workload behavior. That distinction is where most cost surprises begin.
4Four scenarios where tiering fools you
Tiering is a reasonable answer to long retention, but four workload shapes deserve a second calculation before rollout.
4.1"Cold" history is a scheduled product feature
Many backfills are not rare emergencies. They are part of the data platform's normal calendar: daily aggregates, periodic model training, customer exports, or compliance scans. If the same historical window is read on a schedule, the remote tier is serving an active workload even though the records are old.
Compare that repeated read path with the local capacity it replaces. Include consumer count, read window, prefetch behavior, and latency target. A scheduled job is active workload even when no user-facing request reads it directly.
4.2Replay reads create request amplification
Object storage charges can be sensitive to request count and storage-class behavior, not only retrieved bytes. A consumer replaying a broad range may cause many object lookups, range reads, metadata operations, and retries. Small segments make this effect more visible, but large segments can create a different problem if a consumer fetches a small slice from a large object and the implementation cannot reuse the rest efficiently.
Instrument the remote path during a production-shaped replay. Compare records returned, bytes read, object requests, cache hit rate, and time waiting for remote data. A small application replay can be large from the provider's perspective.
4.3Compaction or rewrites change the byte flow
Compaction and rewrite-heavy workloads need their own model. Logical retention may describe the final record set, while the physical system still writes, reads, merges, or expires intermediate data. A tiering policy that works for an append-only log may not have the same request pattern for a topic whose state is continuously rewritten.
Check which topic semantics and remote-storage features the implementation supports. Measure bytes written remotely, bytes read during compaction or recovery, object count, and how long old data remains local. The cost question is physical I/O and object operations, not the logical size of the latest state.
4.4Tiering does not address the dominant line item
If replication traffic, cross-zone transfer, broker compute, or baseline cluster capacity dominates the bill, moving old segments out of local storage may produce a small total change. Tiering can lower one line while leaving the largest multiplier intact. It also does not remove the need to size active disks, maintain replicas, or plan recovery for the local tier.
Map producer writes, replica traffic, local retention, remote upload, consumer reads, retrieval, and recovery. If the main problem is broker-local ownership of durable data, the decision may be architectural rather than a threshold-setting exercise.
5When tiering stops being an architecture decision
Tiered Storage and Shared Storage architecture solve different problems. In a tiered model, the broker-local log remains part of the active design. Older segments are copied or made available in a remote tier, but broker ownership, local hot data, replication, and the handoff between tiers still shape operations. Tiered Storage is a retention policy layered onto a broker-centered storage model.
Shared Storage architecture starts with a different ownership boundary. Durable stream data is stored outside an individual broker, while brokers handle protocol traffic, partition work, caching, and coordination. A broker can be closer to stateless compute, but "stateless" does not mean that metadata, caches, WAL, or recovery disappear. It means long-lived partition data is not tied to one broker's local volume in the same way.
The WAL boundary matters here. A WAL is a write-ahead durability and recovery path, not a synonym for the long-term object store. A Shared Storage architecture may acknowledge writes through a WAL, keep frequently accessed data in memory or local cache, and retain the durable stream in object storage. Storage, retrieval, access latency, topology, and WAL costs still belong in the same economic model.
Use Tiered Storage when the principal problem is retaining old data while most reads stay in the hot tier. Evaluate Shared Storage architecture when retained data is large, broker replacement or expansion is constrained by partition data, or compute should follow traffic without carrying the full history on each broker. Neither choice removes replay, recovery, object-request, and network tests.
6How AutoMQ fits the evaluation
Once the requirement is "Kafka-compatible access with durable stream storage separated from broker-local capacity," a Shared Storage architecture becomes a candidate. AutoMQ is a Kafka-compatible streaming system that uses Shared Storage architecture, with S3Stream coordinating stream persistence and access.
The relevant difference is the storage boundary, not a promise that every workload will have the same bill. AutoMQ uses WAL storage for write and recovery, object storage for durable stream data, and caching for frequent access. Its read path distinguishes tailing reads from catch-up reads, so a cost test should record both. The S3Stream overview describes that model; the WAL documentation explains the separate WAL boundary.
AutoMQ does not make object requests, retrieval, network transfer, cache capacity, or WAL storage vanish. It changes which component owns durable data and how brokers interact with it. That may remove the need to scale broker-local durable capacity with history, while repeated catch-up reads can still make access behavior the dominant cost.
7A decision you can defend in a review
Bring these questions to the architecture and FinOps review:
| Workload signal | First option to test | Why |
|---|---|---|
| Long retention, rare historical reads, local storage is material | Tiered storage | The avoided local capacity may outweigh remote access costs. |
| Routine backfills or repeated scans | Keep the read window hot, then model alternatives | Remote requests and retrieval may recur every billing period. |
| Large retained history plus broker replacement or scaling friction | Shared Storage architecture | The main gain may be ownership and recovery behavior, not a lower object rate. |
| Cross-zone transfer or replication dominates the bill | Fix topology and replication assumptions first | A storage tier does not automatically remove network multipliers. |
Run the comparison with the same retention window, replay workload, recovery objective, region, availability-zone layout, and data durability assumptions. Record local capacity, object storage, request count, retrieval, network, WAL, cache, and operational work as separate rows. A result that cannot explain its rows is not ready for a production decision.
The team in the opening scenario did not necessarily choose the wrong feature. It asked a storage question with a capacity-only answer. Before the next replay, follow the bytes from producer to broker, from broker to remote storage, and back to the consumer. If the avoided disk charge is larger than the access path you actually operate, tiering can save money. If the read path is the workload, the bill has not disappeared. It has moved.
To test the Shared Storage architecture branch with your own retention and replay profile, start with AutoMQ and compare the resulting storage, WAL, cache, request, and network rows against the current design.
8References
- Apache Kafka KIP-405: Kafka Tiered Storage
- Apache Kafka documentation: Tiered Storage
- Apache Kafka documentation: Topic Configs
- Amazon S3 pricing
- Amazon EBS pricing
- AutoMQ Shared Storage architecture
9FAQ
9.1Does tiered storage always reduce Kafka cost?
No. It can reduce broker-local capacity cost when old data is rarely read, but request charges, retrieval, network transfer, cache behavior, and operational work can erase the difference. Use a workload-specific break-even calculation.
9.2What makes Kafka data truly cold?
Observed access, not age alone. Track tail-read share, replay depth, historical read frequency, fan-out, and the physical segment or object pattern. A scheduled scan of old data is still an active cost driver.
9.3What does KIP-405 change?
KIP-405 describes a remote log-storage model that extends retention beyond broker-local storage. It does not define one universal implementation, service boundary, latency target, or cloud bill. Confirm the behavior and pricing of the Kafka distribution or managed service you operate.
9.4Is Tiered Storage the same as Shared Storage architecture?
No. Tiered Storage keeps a broker-centered hot path and adds remote storage for older data. Shared Storage architecture makes durable stream data external to an individual broker, while brokers handle compute, coordination, caching, and a WAL or recovery path. The two models have different scaling and failure boundaries.
9.5When should I evaluate AutoMQ instead of only tuning a tiering policy?
Evaluate it when long retention, broker-local storage ownership, or partition data movement is the larger operational constraint. Use the same retention, replay, recovery, topology, and cost assumptions for the test. AutoMQ's Kafka-compatible interface does not remove the need to validate object access and WAL behavior for your workload.
