Log Compaction Cost Drivers Nobody Puts in the Estimation Spreadsheet

At 02:10, the platform team gets a disk alert for a broker that hosts a compacted topic. The topic's retained state is only 180 GB, well below the 500 GB forecast in the capacity sheet. Yet broker CPU has climbed, disk write latency is erratic, and the cleaner backlog is growing. The first instinct is to blame a producer spike. The producer rate is normal.

The missing line is rewrite work. A compacted Kafka topic is logically small only after old records have been superseded and cleaned. Until then, new values append to the log, the cleaner reads eligible segments, and replacement data is written somewhere before obsolete data can be removed. That work consumes resources even when the final retained state barely changes.

This is why a compacted topic can be cheap to store and expensive to operate. The estimate needs more than retained bytes: it needs key-update shape, cleaner work, physical write amplification, storage requests, and the latency budget that cleaning consumes.

1Small topic, busy broker

Compaction is a contract about what consumers may recover from the log. For a key, consumers eventually see the latest value that survives the cleaner, subject to the topic's timing and tombstone rules. The log remains append-only; compaction does not update records in place.

That distinction creates the common production mismatch. Suppose a state topic receives 2,000 updates per second for 50 million keys, but the application repeatedly updates the same 5 million of them. The retained state may be close to the size of one current value per key. The input path, however, has accepted every update, and the cleaner must later inspect enough segment data to decide which records are still relevant. The example is a workload shape, not a universal performance result; the point is that retained state and physical work are different measures.

Look for these signals together rather than treating any one as proof:

• Broker CPU or disk activity rises while producer bytes remain flat.
• Cleaner work is pending for longer, or eligible segments age beyond the intended window.
• Disk writes exceed the logical bytes arriving from producers by a widening margin.
• Produce or fetch latency moves with cleaning activity rather than with client traffic.

Key cardinality, updates per key, segment size, cleaner capacity, compression, and tombstone volume determine how much work is exposed by those signals. A topic configured with compact,delete has another clock: retention can remove old data even after compaction has produced a latest-value view. Treat compaction retention and deletion retention as separate boundaries in the estimate.

2Why compaction rewrites what you already wrote

Kafka's cleaner works on log segments, not individual records stored in place. A segment becomes eligible according to the topic and broker settings that determine how dirty it is, how long delete markers are retained, and which segments are old enough to process. The cleaner reads records, retains the records it needs, and writes a compacted segment. Later, the obsolete segment can be removed according to the implementation's cleanup rules.

The physical path therefore has at least three byte counts:

1. Logical producer bytes: the records accepted by the topic during a window.
2. Cleaner read bytes: the segment data inspected to build a replacement.
3. Cleaner write bytes: the output written before obsolete data is reclaimed.

Those counts are not interchangeable. Compression changes the relationship between application payload and on-disk bytes. A segment with many superseded records can require a large read while producing a much smaller replacement. A segment with mostly unique keys may produce output closer to its input. A tombstone-heavy workload can add a delayed cleanup phase instead of immediately reducing every byte.

Cleaner parallelism and throttling add a second trade-off. More cleaner capacity can reduce backlog, but it competes with produce, fetch, flush, and other broker work. More throttling protects foreground traffic, but stale segments and their storage footprint live longer. log.cleaner.io.max.bytes.per.second is not a price control by itself; it is a resource-sharing control whose cost shows up in backlog, retention, and latency.

The relevant unit is not “GB stored.” It is “GB inspected and rewritten per unit of logical ingress, under a defined latency target.” That is the unit a FinOps sheet usually lacks.

3The three hidden bills: I/O, write amplification, latency

The first hidden bill is cleaning I/O. On a broker with local or attached block storage, cleaner reads and writes share a finite bandwidth and IOPS budget with foreground requests. On a shared-storage design, some durable work may move to an object-storage path, but the system still has to read, write, index, cache, and eventually reclaim data. The owner of the I/O changes; the work does not become free.

The second bill is write amplification. A useful first metric is:

write amplification = cleaner bytes written / logical producer bytes

Measure numerator and denominator over the same interval and at the same compression boundary. If cleaner output is measured after compression but producer bytes are measured before compression, the ratio is not an economic ratio; it mixes two units. Add storage request counts and bytes read when the platform charges or throttles those dimensions separately.

The third bill is latency and capacity headroom. Aggressive cleaning can consume the I/O and CPU that keep produce acknowledgments and fetches within their SLO. Conservative cleaning can leave more stale data on disk, increasing capacity and recovery work. A backlog is not merely an operational nuisance: it is deferred storage work that can collide with a traffic peak.

The table is deliberately operational, not a pricing quote. Convert each measured quantity into money only after mapping it to the actual instance, volume, object-storage class, request tier, network path, and retention period.

4An estimation template that survives a budget review

Start with a one-hour window that represents normal traffic, then repeat it during the busiest predictable period. Put these fields in the spreadsheet:

L  = logical producer bytes in the window
R  = cleaner bytes read in the window
W  = cleaner bytes written in the window
Q  = object-storage requests attributable to cleaning
C  = cleaner CPU-seconds in the window
P  = foreground produce/fetch latency percentile

rewrite ratio       = W / L
inspection ratio    = R / L
cleaner cost        = W * storage-write-rate
request cost        = Q * request-rate
contention signal   = P during cleaning - P during comparable idle cleaning

The formula is a measurement template, not a promise that every platform exposes R, W, or Q directly. If the cleaner counters are unavailable, use a controlled test: hold producer and consumer traffic constant, change one cleaner setting, and compare disk bytes, CPU, backlog, and latency. Record the missing counters instead of inventing a precise coefficient.

To turn the result into a monthly estimate, multiply by the number of comparable windows and add the storage footprint that remains during the selected retention boundary. Do not multiply a one-hour peak by every hour unless the peak is genuinely sustained. Also separate one-time catch-up or migration rewrites from recurring compaction work.

The billable boundary matters. With local block storage, the rewrite may be reflected in provisioned capacity, volume throughput, and compute headroom. With object storage, it may be reflected in bytes stored, bytes read, PUT/GET/LIST or metadata requests, lifecycle transitions, and network charges. Provider pricing pages determine which of those dimensions are billable; this article does not assign a universal rate.

5A verification path you can run in an hour

First capture the topic contract and the broker settings that affect eligibility and cleanup. The exact command flags vary by Kafka distribution, so run them against a non-production or read-only administrative path where possible:

Record cleanup.policy, segment.bytes, delete.retention.ms, and any cleaner-related broker settings. Then capture a baseline for 10 minutes, a middle interval while cleaning is active, and a final interval after the cleaner catches up. Align producer bytes, broker disk or object-store bytes, cleaner backlog, CPU, and p95/p99 latency to the same timestamps. If your distribution exposes cleaner metrics through JMX or Prometheus, use those counters; metric names and labels differ by version, so validate the query against the running broker.

The smallest useful experiment changes one variable: lower the update rate in a test topic, increase key cardinality, or temporarily adjust cleaner capacity. It should not change retention, compression, instance type, and consumer behavior at the same time. A one-hour result is enough to reject a bad assumption; it is not enough to establish a universal price.

6When to change the cleanup policy instead

Change policy from the consumer contract, not from a storage graph. delete fits an event history when consumers can lose records after the retention boundary and do not need a latest-value reconstruction. compact fits keyed state when superseded records are no longer useful after the cleaner's semantics have done their job. compact,delete can serve both needs, but its delete-retention and segment-cleanup timing must be understood by downstream consumers.

Changing policy can remove rewrite work, but it can also remove a recovery behavior the application relies on. Before switching, ask:

• Can a new consumer rebuild current state from retained events, or does it require one latest record per key?
• Are tombstones part of the delete contract, and how long must consumers observe them?
• Is the problem recurring rewrite pressure, a temporary cleaner backlog, or an undersized disk and cache boundary?

If the topic is semantically right for compaction, keep the policy and change the cost path. That means comparing cleaner capacity, segment sizing, retention, and storage architecture rather than assuming a policy switch is the only lever.

7When shared storage changes the cost boundary

Once the measurements show that broker-local storage is the constraint, a different architecture becomes relevant: Kafka-compatible brokers that separate request processing from durable shared storage. The requirement is specific. The system must preserve the topic and consumer semantics the application needs while moving persistent data and the associated storage work away from a broker-local disk that is difficult to scale independently.

AutoMQ takes this route with Kafka compatibility and a Shared Storage architecture. Its S3Stream design puts durable stream data on object storage, while the WAL and cache path supports write and read behavior closer to the broker. For a compacted topic, this changes the boundary of the rewrite: broker-local disk is no longer the only durable storage surface, while object writes, reads, metadata, cache pressure, and retention still need measurement.

That distinction prevents an expensive category error. Shared storage can reduce local-disk cleanup pressure and make broker capacity less tightly coupled to retained data, but it does not erase compaction's logical work. You still need to test rewrite ratios, object request volume, catch-up reads, WAL usage, and the latency impact of cleaning. The result also depends on deployment choices such as WAL type, object-storage class, region, access path, and cache sizing.

For a high-update state topic, a fair evaluation compares the whole path:

• Keep Kafka and tune: cleaner budget, segment size, retention, and disk headroom.
• Change the policy: only when the consumer contract permits losing latest-value compaction semantics.
• Evaluate shared storage: when local-disk cleanup, broker scaling, or retained-data placement is the recurring constraint.

No option wins by label. The right choice is the one whose measured work maps cleanly to the budget and the consumer contract.

8The decision to take back to FinOps

Return to the 02:10 alert with five numbers: logical ingress, cleaner read bytes, cleaner write bytes, request volume, and latency while the cleaner runs. If the spreadsheet contains only retained GB, it is describing the final state and ignoring the work required to reach it.

Compaction is not an accidental tax. It is the operational price of maintaining a latest-value view over an append-only log. Model that price, decide whether the view is part of the data contract, and then choose between policy tuning, a different retention model, or a storage architecture with a better boundary. For a representative topic, review the AutoMQ architecture and run a measured evaluation with the same key distribution, update rate, retention, and consumer replay pattern you use in production.

9References

• Apache Kafka log compaction
• Apache Kafka topic configurations
• Apache Kafka broker configurations
• Amazon S3 pricing
• AutoMQ S3Stream shared streaming storage

10FAQ

10.1Why can a compacted Kafka topic use so much CPU?

The cleaner reads eligible segments, identifies records that remain relevant by key, and writes replacement data. Frequent updates, many segments, and a growing cleaner backlog can keep that work active even when the retained state is small.

10.2Does compaction always reduce storage?

No. It can reduce obsolete records after the cleaner runs, but the result depends on update frequency, key distribution, segment eligibility, compression, tombstones, and the retention boundary. During backlog or rewrite, physical storage can be higher than the final compacted state.

10.3Should I replace compact with delete to reduce cost?

Only when consumers can operate without latest-value compaction semantics. A policy change can remove rewrite work while also changing what a new consumer can recover, so treat it as a data-contract decision.

10.4Does shared storage eliminate compaction write amplification?

No. It can move durable storage work away from broker-local disks and change which storage and network counters matter. Logical rewrite work, object requests, metadata operations, cache pressure, and read amplification remain workload-dependent.