Table of Contents
Table of Contents
A replay request sounds simple until an incident asks for a time range that the topic no longer contains. The usual questions are practical: how far back can a consumer start, can a new service rebuild its state, and what happens when the topic holds a key whose value has changed many times? On Amazon MSK, those answers come from the topic cleanup policy and the storage boundary around it. A retention value is therefore a recovery decision with a cost attached.
The useful design unit is a replay window: the evidence an application can still read after an outage, backfill request, or consumer reset. A delete policy preserves records until they become eligible for time or size based cleanup. A compact policy preserves the latest value for each key, which is useful for rebuilding state but does not preserve the event history. Combining compaction and deletion can provide a current snapshot while bounding how long old segments remain. Each policy can be correct; the mistake is choosing one without defining the recovery question first.
1Start with the recovery question
Write the replay requirement before editing a topic. “Keep the data for seven days” is incomplete unless the team also says why seven days is enough and what a consumer must be able to reconstruct. A payments ledger may need every event for audit and reconciliation. A customer-profile topic may only need the latest record per customer. A derived search index may need a rebuild path, but its source of truth may live elsewhere.
Ask four questions for each topic:
- What must be reconstructed? The full event sequence, the latest state by key, or a bounded subset?
- Who replays it? An existing consumer group recovering from lag, a new group doing a backfill, or an operator exporting records?
- How much history is required? Express this as a duration, a byte budget, or a business checkpoint such as “since the last successful snapshot.”
- What is the failure if data is gone? A delayed dashboard is different from an unrecoverable order or an incomplete compliance export.
This separates a replay window from a retention setting. The window is a user-visible guarantee; the setting is one mechanism that may approximate it. Test the guarantee with a consumer that starts at an old timestamp or offset. Do not infer it from the value shown in a Terraform plan or console form.
Retention is also one input to a regional recovery design. If a failover plan uses replication, the retained interval must cover the time needed to promote consumers and validate offsets; the Amazon MSK Replicator vs. MirrorMaker 2 comparison shows why the replication path needs its own evidence.
2Choose the cleanup policy that matches the data
Kafka exposes cleanup.policy with delete, compact, or both values. The Apache Kafka topic configuration reference describes the semantics and the related retention.ms, retention.bytes, and compaction settings. Amazon MSK runs Kafka, but the effective defaults, supported broker configuration, and operational limits still depend on the MSK engine version and cluster configuration. Verify those details against the current MSK configuration properties before changing a production topic.
2.1delete: preserve an event history for a bounded period
Use delete when every record in the window matters. The broker makes log segments eligible for deletion when the configured time or size boundary is reached. Cleanup is asynchronous and segment based, so the oldest readable record will not disappear at an exact second. A retention duration should therefore include margin for segment rolling and operational variance. If a consumer needs a strict seven-day recovery promise, test the observed oldest offset under the real production write rate rather than treating the number as a timestamp-level guarantee.
Time and size boundaries answer different questions. retention.ms limits age; retention.bytes limits the log’s retained size for a partition. A topic with a stable event rate may fit a byte budget, while a traffic spike can consume that budget sooner than the calendar duration suggests. Conversely, a quiet topic can retain less data than expected while still meeting its time target. Set a primary boundary and monitor the other one so the replay promise does not silently change when traffic shape changes.
2.2compact: preserve the latest state by key
Compaction is a state-rebuilding mechanism, not a cheaper event archive. The log cleaner selects eligible segments and removes older records for a key while retaining the latest value. The exact timing is asynchronous, and a consumer can read more than one version while cleaning is pending. Tombstone records are used to represent deletes and are retained long enough for consumers to observe them under the configured compaction and delete settings.
A compacted topic works well for entities such as account settings, device configuration, or product inventory where a new consumer needs the current value for each key. It does not let an auditor prove every intermediate transition. If a consumer needs to calculate “what changed between Tuesday and Thursday,” use an event-retained topic or a separate change log. Treat the key as part of the data contract: a keying mistake can collapse unrelated entities into one state and cannot be repaired by increasing retention.
2.3compact,delete: state plus a bounded history
The combined policy is useful when the topic should converge to a current state while old segments eventually leave the log. It can reduce the amount of long-lived history, but it does not turn compaction into an immediate snapshot. Before relying on it for recovery, prove two paths separately: a new consumer can obtain the latest state, and an incident consumer can still read the required recent history.
The right compact-and-delete settings depend on update frequency, tombstone behavior, and the recovery window. A short delete boundary can remove the older versions before a slow backfill starts. A long boundary can keep storage growing even after compaction has removed duplicate values. Record the intended state and history guarantees in the topic contract so application teams do not infer them from a policy name.
The consumer also needs a valid identity while it replays. When a rebuild runs from a separate account or network, confirm the authentication and authorization path before treating a missing record as a retention failure; the Amazon MSK IAM, SCRAM, and TLS comparison separates those failure modes.
3Understand segment behavior before promising a timestamp
Kafka retention acts on log segments rather than individual records. A segment becomes eligible only after it is closed and meets the relevant policy conditions. Segment rolling settings influence how quickly a topic exposes old data to cleanup; compaction also works on segments and needs cleaner capacity. This is why a topic configured for a nominal duration can show a replayable oldest record that is older than the nominal value, especially after a quiet period or a change in traffic rate.
Use the MSK metrics that match the question you are asking. The Amazon MSK metrics details provide the broker-level signals available for capacity and health analysis. Pair those signals with topic-level measurements from your producer and consumers: bytes written, records per second, oldest available timestamp, consumer lag, and the time needed to drain a replay. A storage graph without a replay test cannot tell you whether the recovery contract is still true.
Avoid setting a very large retention value as a substitute for a backup. Retention protects the log while it is available; it does not create an independent copy, and it does not protect against an application writing bad data under valid credentials. If the recovery plan needs point-in-time restoration or an immutable audit record, design an export or backup path separately and test its restore time.
4Run a replay test that can fail clearly
A useful test is repeatable and has a named pass condition. Produce records with a known key distribution and timestamps. Record the first and last offsets in the test interval, then create a new consumer group that starts from the target timestamp. Measure the oldest record it can read, the number of records returned, and the time required to rebuild the downstream state.
For compacted topics, test both a hot key and a deleted key. The hot key should converge to the latest value. The deleted key should produce a tombstone or an equivalent deletion signal that the consumer handles. For delete-only topics, verify that every event in the interval is present and that offsets are monotonic even when records are produced in bursts. Repeat the test after the topic reaches its normal segment and cleaner cadence; an immediate result after topic creation is not representative.
Keep a small worksheet with these fields:
| Field | Evidence to capture |
|---|---|
| Replay target | Timestamp or source offset and the business recovery objective |
| Policy | delete, compact, or compact,delete plus topic-level overrides |
| Availability | Oldest readable timestamp and offset at test time |
| State result | Event count, final value per key, tombstone handling |
| Drain rate | Records or bytes processed per second during replay |
| Storage guardrail | Topic bytes, broker storage headroom, and alert threshold |
| Rollback | The previous policy and the procedure for restoring it |
A test that cannot state what happens when the target data is already gone is incomplete. Make the failure visible to the recovery runbook: stop the rebuild, identify the missing interval, and switch to the independent source or backup path.
5Put a cost guardrail around the window
Retention cost is workload-specific. Estimate bytes written per partition, replication factor, compression ratio, expected traffic peaks, and the number of topics that share a broker. Then compare the estimate with measured MSK storage and broker utilization. The Amazon MSK pricing page describes the service’s current pricing dimensions; use the region, broker type, storage mode, and data-transfer paths that match your deployment.
A practical guardrail has three levels. First, alert when the oldest readable timestamp falls below the recovery objective. Second, alert when topic or broker storage approaches the tested headroom. Third, require a review when a team asks to extend retention: capture the new replay need, the additional storage estimate, and the test that proves the window. This keeps a one-off backfill request from becoming an unbounded cluster policy.
Long replay windows also expose an architectural boundary. In a conventional broker-local storage design, retained bytes and broker capacity are coupled: keeping more history can require more local storage on the brokers that serve the topic. If storage becomes the constraint while request throughput remains modest, a Kafka-compatible platform with shared object storage and independently scalable broker compute can be part of the comparison. AutoMQ uses a cloud-native storage architecture that separates these dimensions, but it should be evaluated with the same replay, compaction, latency, and recovery tests. Kafka cleanup semantics and the exact workload still determine the result.
The boundary matters: changing storage architecture does not remove the need to define keys, tombstones, replay consumers, or an independent backup requirement. It changes which layer carries the retained bytes and how compute can be sized around them.
6FAQ
6.1Does Amazon MSK retention guarantee an exact replay duration?
No. Retention is evaluated through Kafka log segments and asynchronous cleanup. Treat the configured duration as a target, then validate the oldest readable record under the production traffic shape and segment settings.
6.2Is a compacted MSK topic an event history?
No. Compaction keeps the latest value for a key over time; it is suited to rebuilding current state. Use delete retention or a separate event log when intermediate transitions are part of the recovery or audit requirement.
6.3Should I set both retention.ms and retention.bytes?
They protect against different traffic shapes. You can use both as guardrails, but test which boundary is reached first and document how that affects the replay promise. Avoid claiming a duration that a size limit can cut short during a burst.
6.4Can I change a topic cleanup policy during an incident?
A configuration change may alter future cleanup behavior, but it cannot restore segments that have already been deleted or compacted. Freeze the recovery objective, capture the current offsets, and use an independent source when the target interval is missing.
6.5Does AutoMQ preserve the same compaction behavior as Amazon MSK?
Do not assume semantic equivalence from a storage architecture comparison. Validate the Kafka APIs, cleanup behavior, key and tombstone handling, and replay results for the exact AutoMQ deployment and version you plan to use.
7References
- Apache Kafka topic configurations
- Amazon MSK configuration properties
- Amazon MSK metrics details
- Amazon MSK pricing
A replay window is useful only when an application can prove it. Start with the record or state that must be reconstructed, choose the cleanup policy that preserves that evidence, and run the replay after the topic reaches its normal cleanup cadence. Then put a storage alert and a review gate around the promise. If the window keeps growing until broker-local storage becomes the limiting resource, compare architectures with the same measurements rather than treating retention as a single Kafka property.
