Kafka Retention Cost Review: The Math Behind Your Longest-Lived Topics

At the quarterly FinOps review, the Kafka bill looks almost unchanged until someone groups storage by Topic. Three topics account for most of the retained bytes. Their ingress is ordinary, but one keeps 365 days of audit history, another keeps 90 days for backfills, and a third inherited a 30-day default that nobody has revisited. The cost is hiding in the calendar, not in the traffic graph.

That pattern makes Kafka retention cost easy to misdiagnose. Teams often start with broker size, instance type, or a request for a lower-cost storage class. The first question should be narrower: which topics are paying for old data, and what does that data actually do? A topic-level review turns a vague storage problem into a ranked set of decisions: expire data, move cold history, or change the storage architecture.

1The oldest topics are usually the problem

Kafka does what its retention policy asks. retention.ms and retention.bytes define the cleanup boundary, but deletion is segment-oriented and asynchronous. A topic can therefore retain data beyond an exact wall-clock boundary while eligible segments are being cleaned. That behavior matters for capacity planning, but it does not change the main cost relationship: a steady input rate multiplied by a longer window creates a larger durable working set.

The first useful ranking is not "highest ingress." It is retained bytes, followed by retained byte-months. A busy topic with a seven-day window may occupy less storage than a quiet topic with a year-long window. Byte-months also expose policies that will keep adding cost after a one-time cleanup, because the topic continues to carry the same historical window every month.

There are two boundaries to mark before changing anything:

• The data boundary: whether the Topic is append-only, compacted, or using both delete and compact cleanup policies. Log compaction keeps the latest record for a key and has its own rewrite and tombstone behavior; it is not a substitute for a retention requirement.
• The billable boundary: whether the retained bytes are charged as broker-local volume, replicated volume, remote object storage, requests, retrieval, or a combination. Availability topology, replica placement, and cross-zone paths can change this boundary.

That distinction prevents a common mistake: treating a logical retention setting as if it were a complete invoice. Retention tells you how long the platform must make data available. The storage design and access pattern determine how that availability is paid for.

2A per-topic storage bill in three numbers

Start with three measurements: ingress, retention window, and physical-bytes-per-logical-byte. Let I be the average logical input rate in GiB per hour, T be the retention window in days, and C be the observed storage multiplier after compression. The first estimate is:

logical retained GiB = I x 24 x T
physical retained GiB = logical retained GiB x C

For an illustrative Topic receiving 12 GiB per hour, retained for 30 days, with C = 0.45, the estimate is 12 x 24 x 30 x 0.45 = 3,888 GiB. That is a capacity estimate, not a price quote. Add segment and index overhead, cleanup lag, compaction rewrite headroom, and any local cache when sizing real capacity. Measure C from the workload instead of borrowing a generic compression ratio: record shape, producer compression, batch size, and compaction can all change it.

The next step is to map that physical volume to the deployment. In a broker-local design, a replication factor of three can mean roughly three durable copies before other overheads, so the illustrative 3,888 GiB becomes about 11,664 GiB of local capacity. The bill depends on the platform's storage model, usable-volume reserve, and whether replicas share a provider durability layer. Do not multiply by three when the underlying service already prices a single durable object layer, and do not omit replication when every broker carries its own copy.

The complete planning equation is closer to this:

monthly cost = storage capacity
             + compute and broker capacity
             + write/read requests
             + retrieval or egress
             + inter-zone or cross-region transfer
             + backup, replication, and operational overhead

The storage term is the starting point because it is the part retention controls most directly. It is not the whole bill. Cold archive storage may have a favorable rate but an unpleasant retrieval bill during a large replay, while a local tier may force the platform to provision for the full retained window. Keep those cases in separate columns.

The practical rule: calculate the retained bytes first, then apply the provider's pricing and topology rules. Never reverse that order by multiplying a headline storage rate by an unverified logical volume.

3The retention audit, step by step

An audit is useful only when it ends with an owner and a decision date. A spreadsheet with topic names and retention values is inventory, not governance. Use the following sequence.

1. Inventory Topic policy. Export retention.ms, retention.bytes, cleanup policy, partition count, replication factor, and the effective broker or cluster default. Record the observed oldest and newest offsets where your monitoring makes that possible. Include topics whose settings inherit defaults; inherited policy is still policy.
2. Join policy to measured flow. Use a stable period long enough to avoid a single batch or incident. Capture ingress, stored bytes, compression, segment count, consumer lag, and read-back volume. Keep peak and average ingress separate when bursts drive capacity.
3. Rank by retained bytes and byte-months. The first ranking shows what is already occupying storage. The second shows what will continue to cost money if the policy remains unchanged. Add the billable storage tier and topology so the ranking reflects the actual cost surface.
4. Confirm the oldest data's job. Ask the owning team whether historical records support replay, backfill, audit, incident investigation, model training, or a legal hold. Ask how often the oldest quartile is read and how quickly that read must complete. "We may need it" is a prompt for a recovery exercise, not a retention period.
5. Write down the service boundary. Retention is not the same as backup, disaster recovery, or a system of record. State whether the Topic is the authoritative copy, how restoration works, and which records can be regenerated elsewhere. This is where a shorter Kafka window may be safe, or where a long window needs an intentional archive.

The table below uses illustrative values to show the shape of a review. It is not a pricing comparison and should be replaced with measured values from the estate.

The ranking changes the conversation. audit.records is the largest retained object even though it has the lowest ingress. That does not mean it should be deleted. It means the owner must explain why 365 days belongs in the live Topic, whether a legal hold exists, and whether an archive or separate audit store should carry part of the obligation. The right action follows the data's use, not the row's position.

Compacted Topics deserve a separate review pass. Their logical state may be smaller than the input history, but compaction rewrites segments and can create temporary storage and I/O pressure. Measure the current logical result and the physical work required to maintain it.

4Cutting order: expire, tier, then rearchitect

The lowest-complexity architecture change is often a policy change. Use the least complex action that satisfies the requirement, and move down the list only when the earlier action fails the access or recovery test.

4.1Expire data that has no active requirement

Shortening retention removes future storage growth and may release capacity after cleanup catches up. It is the cleanest move when old records are neither read nor required for audit, but confirm downstream replay, incident response, and legal-hold processes first. A default should not be shortened globally until owners of high-impact Topics have tested their recovery path.

4.2Tier data when history is real but cold

Tiered Storage can move older log segments to remote storage while a local hot tier serves recent reads. It is a reasonable fit when consumers mostly tail the stream and historical reads are occasional. Check which data remains local, how remote reads behave, who pays request and retrieval charges, and whether a replay competes with live traffic. A tiering feature can reduce local disk pressure without removing the broker, partition placement, replication, or recovery boundaries that still shape operations.

The terms matter. Tiered Storage is an offload strategy. It keeps a local tier and a remote tier with different access behavior. A Shared Storage architecture makes object storage part of the primary durable storage model and can make Brokers more stateless. The two approaches may solve related retention problems, but they do not have the same topology or recovery consequences.

4.3Re-architect when local retention is the recurring constraint

Architecture work is justified when the business needs long retention, the oldest data is read often enough to matter, and broker-local storage keeps forcing capacity, rebalance, or recovery decisions. The evaluation must include Kafka API and client behavior, write durability, tailing reads, historical reads, compaction, failure recovery, and the cloud account and network boundaries.

That requirement points to a specific design: Kafka-compatible clients with shared durable storage, a separate write-ahead path, and broker capacity that is not tied one-to-one to every retained byte. It does not prove that a migration is right. It gives the migration a testable reason to exist.

5How AutoMQ fits the evaluation

AutoMQ is a Kafka-compatible streaming platform built around a Shared Storage architecture. Its S3Stream library uses object storage as the durable stream layer, with WAL storage and caching serving different parts of the write and read path. The relevant architectural question is not whether object storage has a lower rate in the abstract. It is whether this separation removes the broker-local retention constraint in your workload.

That distinction changes the audit rather than replacing it. Long-lived data can sit in S3 storage instead of requiring every Broker to hold the full history locally, while the WAL and cache affect write durability, recovery behavior, and hot-read latency. The exact boundary depends on the WAL implementation, object storage service, region, network path, and deployment model. An S3 WAL has a different latency and recovery profile from a block or NFS WAL, so the WAL choice belongs in the test plan and the cost model.

The read pattern still decides whether the architecture fits. A tailing Consumer may mostly use recent data and cache. A backfill job can walk much deeper into the Topic and create object-storage reads, requests, and possible egress. Compaction, Consumer group behavior, and retention cleanup still require application-level validation. Shared storage can change where the cost sits; it does not make retention, access, or recovery free.

For a deployment that must stay in the customer's cloud boundary, review the AutoMQ Cloud deployment model and S3Stream shared storage documentation. Then run the same topic worksheet against a representative hot tail, cold replay, compaction case, and broker failure scenario. The comparison is useful only when both platforms carry the same data contract and recovery target.

6Make it a quarterly ritual

A quarterly retention review can be small. Publish the ranked Topics, the change in retained byte-months since the last review, the owner, the oldest data's purpose, and the selected action. The owner confirms the decision and a date for the next check. Figure 3 turns that loop into a repeatable operating habit rather than a once-a-year capacity surprise.

Return to the opening storage graph each quarter. If the same three Topics still dominate it, the next question is no longer "which broker should we resize?" It is "which requirement is worth carrying, and which storage boundary should carry it?" Start with the worksheet, and when the evidence points to shared storage, use the same scenarios for an AutoMQ evaluation.

7References

• Apache Kafka topic configurations
• Apache Kafka log cleanup and compaction
• Apache Kafka KIP-405: Tiered Storage
• Amazon S3 pricing
• Amazon EBS pricing
• AutoMQ S3Stream shared storage

8FAQ

8.1How do I calculate Kafka retention cost?

Estimate effective ingress multiplied by 24 hours and the retention days, then apply the measured physical-bytes multiplier. Map that volume to the actual billable layer, including replicas or provider durability, requests, retrieval, egress, and network transfer. Use Topic-level inputs so a long-lived low-ingress Topic does not disappear inside a cluster average.

8.2What is a good Kafka retention policy?

The policy should match a stated replay, audit, backfill, or recovery requirement. Set retention.ms or retention.bytes with the owning team, document the oldest data that must be available, and review the result after downstream consumers or regulatory requirements change. A single default is a starting point, not proof that every Topic needs the same window.

8.3Does log compaction reduce Kafka storage cost?

It can reduce the retained logical state for key-based Topics, but it does not eliminate physical rewrite work, tombstones, segment cleanup, or the need to size for compaction behavior. Measure the post-compaction state and the temporary I/O and storage headroom separately from append-only retention.

8.4Is Tiered Storage the same as Shared Storage?

No. Tiered Storage usually keeps a local hot tier and moves older segments to remote storage. Shared Storage makes remote object storage part of the primary durable storage design and can decouple Broker capacity from retained bytes. Compare local data requirements, WAL behavior, read paths, failure recovery, and billing boundaries before treating either option as a direct substitute.

8.5Can AutoMQ make long retention free?

No. It can change the storage boundary by using shared object storage with a WAL and cache, but object-storage capacity, requests, retrieval, egress, network topology, and compute still depend on the workload. A valid evaluation uses the same retention, tailing, replay, compaction, and recovery scenarios as the current platform.