Table of Contents
Table of Contents
A producer acknowledgement answers one narrow question: did the broker accept this record under the configured Kafka contract? It does not, by itself, tell you which storage layer owns the bytes at that point, how a replacement broker will recover them, or what happens when the storage path is unavailable.
That distinction matters when a Kafka deployment moves durable data away from broker-local disks. Apache Kafka’s KIP-1150 frames diskless topics as a storage and operational design question, separate from the producer API. In a diskless Kafka design, the durability boundary crosses a broker, a write-ahead log (WAL), object storage, metadata, and the network between them. A green producer metric can hide a stalled upload, a cache-only copy, or a client that will retry after a leadership change.
The useful question is therefore not “does diskless Kafka support durable writes?” It is: what evidence proves that an acknowledged record remains recoverable through the failures your service must tolerate? This framework turns that question into a testable contract. It starts with Kafka producer semantics, follows the record through a shared-storage path, and ends with rollout gates that an SRE can run without trusting a single dashboard.
1Producer durability starts with an explicit contract
Kafka exposes durability controls at the producer and topic layers, but those controls are often treated as magic settings. acks=all asks the leader to wait for acknowledgement from all in-sync replicas. min.insync.replicas sets the minimum ISR size for a successful write when the topic and producer settings require it. Idempotence prevents retries from creating duplicate sequence numbers when the producer is configured for the idempotent path. These settings define client-visible behavior; they do not replace a storage and recovery test.
Write down what an acknowledgement means for your workload. A useful contract answers four questions:
- Which records are allowed to be acknowledged? Define the required leader, replica, WAL, and object-storage conditions at the acknowledgement boundary.
- What must survive a broker loss? State the accepted recovery point, including whether records acknowledged immediately before the fault must be readable after ownership changes.
- What may the client retry? Record the retry window, delivery timeout, ordering requirement, and duplicate handling at the application boundary.
- How will the team prove it? Name the producer log, broker event, storage metric, metadata record, and replay check that close the evidence chain.
The contract should use observable conditions rather than implementation words. “Flush completed” is useful only when the team can point to the corresponding event and explain what it guarantees. “The record is durable” is stronger than “the request returned”; it requires a defined storage boundary and a recovery observation.
A small producer test can make the contract concrete. Send records with stable keys and sequence markers, record the acknowledgement timestamp and partition, then retain the producer log alongside broker and storage events. After a controlled fault, replay the acknowledged range from a fresh consumer and compare the observed keys and values with the producer ledger. Treat duplicates according to the configured delivery semantics; do not label every retry as loss.
For the exact meaning of acks, min.insync.replicas, and idempotence, use the Apache Kafka producer configuration documentation. The documentation defines the protocol contract. Your test must supply the missing operational evidence.
2Acknowledgement is a point on a path, not a storage location
A record moves through several boundaries before a consumer can replay it. The leader accepts a produce request, appends the record to its active write path, and returns an acknowledgement when the configured condition is met. Followers, WAL storage, object storage, metadata, caches, and clients then contribute to what a replacement can serve. The path is easier to reason about when each boundary has a named observation.
| Boundary | Question to answer | Evidence to capture |
|---|---|---|
| Producer client | Which request and sequence did the client consider acknowledged? | Client callback, sequence number, retry and timeout logs |
| Broker write path | Which partition owner accepted the record, and under what epoch? | Produce response, leader epoch, append or commit event |
| WAL storage | What write is recoverable when the broker process or node disappears? | WAL append, flush, recovery, and backlog metrics |
| Object storage | When can a replacement read the record from the shared durable layer? | Object write or upload event, range read, request status |
| Metadata and ownership | Which broker is allowed to serve the partition after a fault? | Controller event, fencing signal, ownership transition |
| Client replay | Can a fresh consumer read the acknowledged range without an unexplained gap? | Offset range, checksum or key ledger, fetch errors |
This model prevents a common category error: equating an acknowledgement with an object-storage write. Some architectures acknowledge at a local or replicated log boundary and upload older data later. Others place WAL and primary storage in the same object-storage system. Both can be valid, but they produce different failure windows and measurement plans. The test must identify the selected implementation instead of importing assumptions from another Kafka deployment.
The distinction also separates diskless Kafka from Kafka Tiered Storage. KIP-405 describes remote storage for log segments while the active log and broker-local replica model remain part of the design. A diskless architecture changes where the durable stream lives and what a replacement broker has to rebuild. The word “remote” is not enough to infer the recovery boundary.
3Follow the record through the shared-storage mechanism
Once the contract names the boundary, inspect the path that fulfills it. A Kafka-compatible diskless system still has brokers, partition leadership, metadata, caches, and client protocols. The durable stream is separated from broker-local capacity, so the write path includes storage services that a traditional Kafka review might treat as background components.
A practical trace follows this sequence:
Producer request
-> partition leader and Kafka protocol handling
-> WAL append and durability decision
-> acknowledgement returned to producer
-> upload or compaction into object storage
-> metadata update and ownership visibility
-> consumer fetch from cache, WAL, or object storage
The arrows are a measurement plan, not a promise that every implementation emits the same events. For each arrow, record what the platform exposes and what the client can observe. A single end-to-end trace should let an operator answer: “The producer received an acknowledgement at this point; which durable copy could a replacement read after each fault?”
WAL behavior deserves its own row in the test plan. AutoMQ terminology uses WAL storage as the generic layer and distinguishes S3 WAL, Regional EBS WAL, and NFS WAL as concrete implementations. The choice changes latency, fault domains, and recovery dependencies. S3 WAL uses object storage for both the WAL and primary storage, while Regional EBS WAL and NFS WAL introduce a separate low-latency write layer with its own topology and access controls. Avoid attaching a generic latency or durability promise to “WAL” without naming the type.
Caching changes the read side of the contract. A consumer may read hot data from memory or a WAL-adjacent path while an older range requires a catch-up read from object storage. A producer durability test that checks only tailing consumers can miss an object-storage permission problem or an incomplete upload. Include a bounded historical replay after the fault, and retain the exact offset range used for the check.
Metadata is the other half of the path. Shared storage does not remove Kafka leadership, epochs, or fencing. A replacement broker still needs a current ownership decision, and a stale broker must be prevented from continuing to write. Capture controller and broker events around the transition; a record that exists in storage but is served by two owners is not a healthy durability result.
4Turn failure modes into measurable tests
A durability claim is only as useful as the failures behind it. Start with a small matrix where each fault has one hypothesis, one observation window, and one pass line. Keep the workload running so the test sees retries, cache misses, and lag instead of the quiet behavior of an idle cluster.
| Fault | Hypothesis | Measurements | Pass line |
|---|---|---|---|
| Broker process or node loss | A replacement can serve the tested partitions without reconstructing a broker-local retained log. | Detection, fencing, ownership transition, producer errors, replay result | The contract’s acknowledged range is readable and the client returns within the stated service objective |
| WAL storage interruption | The write boundary becomes visible and bounded when its dependency is impaired. | Acknowledgement responses, retry behavior, WAL backlog, recovery events | No record crosses the documented durability boundary without an explicit client-visible outcome |
| Object-storage errors or denied access | Upload and cold-read failures surface before they become silent gaps. | Request errors, retries, upload backlog, historical fetch, alerts | The runbook names the operator action, and replay either passes or fails loudly |
| Controller or metadata disruption | Epoch and fencing rules prevent stale writers after coordination changes. | Quorum state, leader epochs, fencing events, duplicate ownership | One current owner serves each tested partition |
| Network isolation | The storage and client paths match the designed fault domain. | Per-path reachability, cross-zone traffic, retry and timeout logs | The team can state which writes continue, which stop, and how recovery proceeds |
Run a no-fault control with the same keys, partitions, retention, producer settings, and consumer replay. Without a control, an existing lag spike or storage rate limit can be mistaken for damage caused by the injected fault. Repeat the matrix after changing the WAL type, object-storage endpoint, network topology, client version, or metadata configuration; each can move the durability boundary.
Measure producer and consumer outcomes separately. Producer success tells you when the request contract was satisfied. Consumer replay tells you whether the acknowledged range can be found after ownership and cache changes. Keep the producer ledger, broker events, storage request logs, controller state, and replay output in one run record so a reviewer can reconstruct the causal order.
5Cost and compatibility are part of durability
A storage path can be durable and still be operationally wrong for the workload. Object-storage requests, inter-zone traffic, egress, and recovery reads can add cost when the path is retried or a cold cache is rebuilt. Do not turn one test run into a universal price claim. Instead, record the request and byte counters that let FinOps apply the provider’s price sheet to the tested workload and region.
Compatibility belongs in the same worksheet. Producer durability can be invalidated by a client assumption about error codes, ordering, transactions, or metadata refresh. Verify the exact client and library versions, producer settings, transaction usage, and retry policy that the application will run. If a connector or stream-processing job consumes the topic, include its offset and state-rebuild behavior in the replay test; a durable record is not enough if the application cannot resume from its committed position.
This is where a neutral framework becomes useful during vendor evaluation. Ask every candidate to show where an acknowledgement is decided, how the selected WAL type recovers, what metadata evidence fences stale owners, and which cold-read path serves historical records. The answers can differ, but the questions stay stable. For adjacent guidance, compare this worksheet with the Kafka write-path analysis and the failure-injection exercise. Those articles cover related mechanics; this framework keeps the producer acknowledgement and replay evidence together.
6How AutoMQ changes the operating model
The contract above points toward a design that separates broker compute from durable stream storage while preserving the Kafka client surface. AutoMQ is a Kafka-compatible cloud-native streaming platform built around that Shared Storage architecture. Its relevance here is architectural: the broker is not the long-term home of every retained byte, so the durability test follows the WAL, object-storage, metadata, and cache paths explicitly.
AutoMQ uses S3Stream as the storage library behind Kafka-compatible brokers. The architecture overview explains how brokers, KRaft metadata, WAL storage, Data caching, and S3 storage fit together. The WAL storage guide is the right place to verify the selected WAL type and its deployment constraints.
That architecture changes the evidence an operator should collect. A replacement broker must regain metadata ownership, reach the selected WAL and S3 storage, recover data that has not completed its upload path, and serve both tailing and catch-up reads. The test still has to use the Kafka producer contract: acks, ISR requirements, idempotence, retries, and timeouts remain client-visible settings. The shared-storage path changes the mechanisms behind those settings; it does not make the settings optional.
Kafka compatibility also needs an application test. AutoMQ’s compatibility documentation describes the intended boundary. Verify your exact producer library, transactions, ordering assumptions, administrative tools, and replay code against the deployment you plan to operate. A platform-level compatibility statement cannot stand in for an application-level durability drill.
7A rollout checklist that produces evidence
Use the following gates before moving a diskless Kafka workload beyond its evaluation environment:
- Contract gate: producer settings, topic ISR requirements, acknowledged range, retry behavior, and duplicate handling are written down.
- Storage gate: the selected WAL type, object-storage service, credentials, encryption, network path, and fault domain are recorded.
- Ownership gate: controller and broker evidence proves that a stale owner is fenced before a replacement serves writes.
- Replay gate: a fresh consumer can read the acknowledged range, including a bounded historical catch-up read, and the result is reproducible.
- Operations gate: alerts, dashboards, logs, and runbooks identify the next operator action for WAL, object-storage, metadata, and client failures.
- Cost gate: request, byte, recovery, and cross-zone counters are available for the workload and can be reconciled with the provider’s current pricing.
A gate is a decision point, not a formality. If the replay gate fails while the producer metric remains green, preserve the evidence and investigate the storage and ownership path before changing the client configuration. If the cost gate lacks counters, record that as an observability gap rather than presenting an unsupported savings estimate.
7.1Frequently asked questions
7.2Does acks=all prove that a diskless producer write is durable?
No. It proves that the broker met the Kafka acknowledgement condition for the configured ISR. You still need to show where the selected implementation persists the record, how a replacement recovers it, and whether a fresh consumer can replay the acknowledged range after the tested fault.
7.3Is diskless Kafka the same as Kafka Tiered Storage?
No. Tiered Storage can move older segments to remote storage while the active log and broker-local replica model remain. A diskless design moves the primary durability boundary and therefore tests WAL, object storage, metadata, and cache recovery as first-class paths.
7.4Should a test wait for every record to reach object storage before acknowledging it?
That is an implementation and service-level decision. The test should document the actual acknowledgement boundary and verify the recovery behavior that the workload requires. Waiting for an object-storage event is one possible contract; it is not a universal definition of producer durability.
7.5Which WAL type should a production deployment use?
Choose from the workload’s latency, fault-domain, and operations requirements. S3 WAL simplifies the storage layout but depends on object-storage behavior for its write path. Regional EBS WAL and NFS WAL provide different latency and topology characteristics. Confirm the available options and constraints in the product documentation for the deployment you operate.
7.6What should a team measure first?
Start with one representative topic and a bounded ledger of acknowledged records. Capture producer callbacks, broker epochs, WAL and object-storage events, controller transitions, and a fresh-consumer replay. That small run will expose missing evidence before a larger migration hides it in aggregate metrics.
A producer dashboard is a useful starting point, but it cannot close the durability question on its own. Trace one acknowledged record across the write path, inject the failure your service must survive, and require a replay result that another operator can reproduce. When you are ready to apply this worksheet to a Kafka-compatible Shared Storage deployment, start an AutoMQ evaluation with the contract and evidence matrix in hand.
