Durable Writes Without the Disk Wait: Kafka Write Paths on Shared Storage

When an Apache Kafka producer receives a successful response, what has actually become durable? The record may have reached a partition leader, a local log, follower replicas, or a separate WAL (Write-Ahead Log).

That distinction matters when a cluster moves from broker-local disks to shared storage. The architecture can remove a local disk wait from the hot path, but it still needs a precise durability point, a bounded window between that point and background upload, and a replay path when a broker disappears. The useful way to evaluate the design is to follow one record from Produce request to recovery, then attach each latency and failure signal to the step that creates it.

A Kafka acknowledgment is a protocol decision. A storage flush is a physical durability decision. They are related, but they are not the same event.

1When is a Kafka write really durable?

There are three meanings of “durable” that often get collapsed into one sentence. A client can receive an acknowledgment, a cluster can mark a record committed according to its replication protocol, and the underlying storage can confirm that bytes survived the failure model it is designed to handle. A production review should name which of these it means.

Apache Kafka’s replication documentation defines a committed message in terms of the ISR (In-Sync Replicas) for a partition. All replicas in the ISR must have applied the message to their logs before the message is committed. The producer’s acks setting controls how much of that process the client waits for, while min.insync.replicas can reject an acks=all write when too few replicas remain in sync. The details are in the Kafka replication design, producer acks configuration, and min.insync.replicas configuration.

That protocol definition still does not mean that every Produce response waited for an fsync on every replica. Kafka deliberately separates log append, replication, and the scheduling of flushes to disk. Its broker settings include log.flush.interval.messages and log.flush.interval.ms, but those settings describe when log data is flushed to disk; they are not a replacement for the producer’s acknowledgment policy. The broker configuration reference also exposes a checkpoint interval for the log recovery point.

The following checkpoints are easier to reason about than the single word “durable.”

These checkpoints can overlap rather than appear as separate network round trips. The operational point is to identify whether rising latency comes from the leader, followers, a WAL flush, or a remote storage operation.

2The local-disk write path, step by step

In a traditional Kafka deployment, the partition leader owns the first durable-looking decision. A producer sends a batch to that leader. The leader appends the batch to its local log, and follower replicas fetch the leader’s log and append the same ordered records to their own logs. The local append can use the operating system’s page cache, so the path is not equivalent to “write a block and call fsync before doing anything else.”

The acknowledgment policy determines where the producer waits. With acks=1, the leader can respond after its local log append without waiting for followers. A leader failure in the gap before replication can therefore turn a successful response into a record that is not available on the next leader. With acks=all, the leader waits for the current ISR, and min.insync.replicas can add a minimum replica-count requirement. That improves the replication guarantee while making the write path sensitive to follower lag and ISR shrinkage.

This is why the word “flush” is easy to misuse. A disk flush is a storage action. A Kafka commit is a replicated-log action. Kafka’s design explicitly avoids requiring fsync on every write for its normal consistency model; if a broker loses unflushed data, the replica must rejoin only after it has been brought back into sync. A team that changes log.flush.interval.ms to solve an acks=all latency problem is changing a different part of the system.

The local path has a familiar failure boundary:

1. The leader accepts the batch and appends it to its local log path.
2. Followers fetch and append the batch, and the ISR may advance.
3. The producer receives a response based on acks and the current ISR.
4. A broker restart replays its local log and recovers from the last valid recovery point, then catches up before the replica can safely rejoin the ISR.

Each broker also carries partition-local state, so a replacement must recover or move that state before it can carry the same load. The write path, replica health, and recovery path are tied to the same local storage layout.

3What changes when the WAL lives on shared storage

Shared Storage architecture changes the storage boundary rather than removing the need for one. The architecture requirement is straightforward: the fast path needs a confirmed durable write, while the primary object-storage layer can be populated asynchronously and used as the long-lived source for reads and replay. The broker should be able to recover the recent tail without owning the only copy of partition data.

This is the point where AutoMQ is a useful concrete example. AutoMQ replaces Kafka’s native local log storage with S3Stream. Its documented write path is: the Produce request reaches the partition leader, S3Stream appends the data to WAL storage, the client is acknowledged after the WAL write succeeds, and the data is uploaded or compacted into S3 storage in the background. A broker failure can recover data that has not yet reached S3 from the WAL.

The WAL is a fixed-size, cyclic storage area rather than a second long-lived copy of every topic. S3Stream mixes writes from multiple partitions, uses sequential writes and group commit, and can use Direct IO for the selected storage medium. These choices address the latency of remote storage and the number of small write operations. The durable fast path is therefore a storage-layer event, not a broker-local file-format event.

The WAL type still sets the boundary’s behavior. AutoMQ Open Source supports S3-compatible storage as its WAL option, while AutoMQ commercial editions can use other WAL storage choices for different workload and failure-domain requirements. A useful design review asks two separate questions: which medium confirms the write, and where does a broker recover the unuploaded tail? The answer may be S3 WAL, Regional EBS (Elastic Block Store) WAL, or NFS WAL depending on the deployment, and the latency and failure-domain assumptions must follow that choice.

Shared Storage architecture should also be kept distinct from Apache Kafka Tiered Storage. Tiered Storage adds a remote layer after data is written to the primary local log, so the local partition layout and ISR mechanism remain central. Shared Storage replaces that local persistence layer with WAL storage plus object storage, which is why the broker can be stateless with respect to persistent partition data.

4Durability points, latency windows, and replay

Once the WAL becomes the fast persistence point, the write path has at least three windows worth measuring. The first is the request window, from the producer send to the acknowledgment. The second is the upload window, from WAL confirmation until the record and its metadata are available in object storage. The third is the recovery window, from broker failure until a replacement can serve the affected partition from shared data and any surviving WAL tail.

The upload window is not automatically a data-loss window. If the WAL confirmation is the configured durability point and the WAL remains available under the deployment’s failure model, a record can be safe before its object upload finishes. It is still a recovery concern: the platform must retain enough WAL state, metadata, and ordering information to reconstruct the tail. A long upload window can also increase WAL pressure or make catch-up work more expensive, even when the record is already acknowledged.

Replay therefore looks different from a local-disk restart. A traditional broker starts with local log segments and their recovery metadata, then repairs replica divergence through follower catch-up. A shared-storage broker starts with the shared object history and the WAL tail that has not been uploaded. It rebuilds the serving view from those layers and warms caches as needed. The recovery path is less about copying a broker’s entire partition directory and more about reestablishing ownership, metadata, and the hot read path.

That shift changes what a failure test should ask. “Did the broker restart?” is too shallow. The useful questions are whether acknowledged records can be read after the failure, how much unuploaded WAL needs replay, whether object metadata and offsets remain ordered, and when the replacement returns to normal tailing reads. Those are the boundaries that turn a storage architecture into an operational guarantee.

5Metrics to watch and knobs to turn

The fastest way to misdiagnose a shared-storage write path is to look at one end-to-end latency number. Break the request into queue, local processing, remote wait, and response time first. Apache Kafka exposes TotalTimeMs, RequestQueueTimeMs, LocalTimeMs, and RemoteTimeMs for produce requests, along with UnderReplicatedPartitions, UnderMinIsrPartitionCount, ISR shrink rate, and follower MaxLag. The Kafka monitoring reference documents these metrics and their JMX names.

For an AutoMQ deployment, pair those Kafka-facing signals with the storage path. The AutoMQ Prometheus metrics reference lists request time and queue time percentiles, Kafka_logs_flush_time_50p(99p/mean/max)_milliseconds, Kafka_stream_operation_latency_50p(99p/mean/max)_nanoseconds, and stream upload and download byte counters. The combination helps answer whether a spike is created before the WAL, while the WAL is flushing, during object-storage operations, or while the broker is waiting for a response path.

The knobs follow the same boundaries. Use acks and min.insync.replicas to define Kafka’s producer-level durability and availability trade-off. Treat log.flush.interval.ms and log.flush.interval.messages as local log-flush controls, not as a substitute for understanding the acknowledgment path. For shared storage, the larger decision is WAL type and failure domain: a lower-latency block or file-backed WAL may fit an interactive workload, while S3 WAL can simplify a diskless deployment for workloads that can accept its storage latency profile. The right choice comes from the measured write and recovery windows, not from the word “shared.”

6FAQ

6.1Does shared storage make acks=all unnecessary?

No. Shared storage changes the persistence implementation behind the request path. Producer acknowledgment, visibility, ordering, idempotence, and transaction behavior still need to follow the Kafka compatibility contract. The platform must make its storage confirmation line up with the requested semantics; a durable WAL does not turn every producer setting into the same setting.

6.2Is a WAL the same thing as the Kafka log?

No. A WAL is the write and recovery layer that accepts recent data before the primary storage layer has finished its background work. Kafka records still have partitions and offsets, but the storage layout that makes those records durable can be shared across partitions. A file-format inspection may show how bytes are arranged; it does not by itself tell you when a producer acknowledgment is safe.

6.3Can a record be safe before it is in object storage?

Yes, if the configured WAL confirmation is the durability point and the WAL’s failure domain covers the failure you are testing. That statement has a boundary: it says nothing about a failure that removes the WAL before upload, a misconfigured storage service, or a recovery path that cannot replay the unuploaded tail. Validate those cases in a controlled test.

6.4Which metric tells me that a write is slow?

No single metric does. Start with the request-time components, then compare replication wait, flush time, stream operation latency, and upload progress. A single P99 can tell you that users are waiting; the split tells you which part of the write path owns the wait.

7References

• Apache Kafka replication design
• Apache Kafka producer acks configuration
• Apache Kafka broker configuration
• Apache Kafka monitoring
• AutoMQ architecture overview
• AutoMQ WAL storage
• AutoMQ Prometheus metrics

Return to the original question: when the producer sees success, which layer made that promise? If the answer is a named WAL confirmation backed by a tested failure domain, the team can reason about the remaining upload and replay work instead of treating “diskless” as a magic durability claim. To inspect the implementation and run a Kafka-compatible test path, start with the AutoMQ open-source repository.