Kafka without disks sounds like a contradiction because, in traditional Kafka, the broker log is not an implementation detail. It is the center of the system. Producers write records into partition logs on brokers, consumers fetch from those logs, replication protects those logs, retention deletes from those logs, and rebalancing moves those logs between brokers. Remove the broker disk from that model and you have not made Kafka lighter; you have removed the place where Kafka believes durable data lives.
So the short answer is: traditional Apache Kafka cannot become diskless by mounting smaller volumes or turning off local persistence. A Kafka-compatible system can be diskless only if another durable storage path replaces the broker-local log. That replacement has to acknowledge writes safely, serve hot reads efficiently, recover after broker failure, and preserve the Kafka-facing model of topics, partitions, offsets, and ordered appends.
This distinction matters for cloud-native architects and Kubernetes platform teams. A stateless pod is useful only when the state has gone somewhere durable and operationally sane. For Kafka, "somewhere durable" cannot be hand-waved into memory, local ephemeral storage, or a background archive. Diskless Kafka is an architecture, not a deployment flag.
Why Traditional Kafka Depends on Disks
Apache Kafka's default storage model is broker-local. The broker configuration includes log.dirs, the directories where log data is kept. Each broker stores partition replicas in those directories, and the cluster distributes partition leadership and follower replicas across brokers. A topic can have multiple replicas, but each replica is still a local log owned by a broker process and its storage.
That design is powerful because it keeps the hot data path close to the serving process. A broker can append sequentially, read from page cache or disk, enforce retention, and participate in ISR replication without asking a separate shared storage layer to coordinate every operation. Kafka's operational reputation was built on this model: append-only logs, sequential I/O, partition-level parallelism, and replication across machines.
The same design also makes brokers stateful infrastructure. Broker identity, partition placement, disk capacity, and failure recovery are tied together. When a broker runs out of disk, the problem is not solved by adding a random pod. When a node fails, the cluster has to reason about replicas and leadership. When storage grows, retention changes, or partitions are reassigned, the data movement is real.
That is why "Kafka no local disk" is the wrong question unless it is followed by a storage architecture question. The log still needs durable placement. The write path still needs an acknowledgement boundary. The read path still needs a way to serve tail reads and catch-up reads. The metadata layer still needs to know which stream ranges and offsets exist.
For platform teams, the practical consequence is clear:
- If brokers still own durable partition logs on attached volumes, the cluster is stateful even when Kubernetes manages the pods.
- If object storage is used only as an archive for older data, the hot log is still local and broker scaling still involves state.
- If a Kafka-compatible broker can be replaced without moving durable partition data from that broker, the architecture has crossed into diskless or near-diskless territory.
That last line is where the interesting engineering begins.
What Diskless Kafka Requires
Diskless Kafka does not mean storage disappears. It means durable stream data is no longer primarily stored in broker-local log directories. The broker becomes a compute and serving layer, while shared storage takes over the durable log responsibility that local disks used to carry. The hard part is accepting live Kafka writes without losing data or turning every produce request into a slow object storage transaction.
Durable Write Path
Every serious diskless design needs a durable write path before data is considered acknowledged. In many architectures, that role is played by a write-ahead log (WAL). The WAL absorbs incoming writes, provides a durable acknowledgement point, and allows the system to recover unflushed data after a broker failure. Some designs use object storage itself as the WAL; others use block storage, regional block storage, or NFS-like shared file storage before data is uploaded to object storage. The word "diskless" should always be read together with the WAL backend.
Cache and Hot Reads
Traditional Kafka benefits from the operating system page cache and broker-local log segments. A diskless architecture needs another answer for hot reads. Tail consumers expect fresh records to be available quickly, and catch-up consumers can create large read bursts when they replay old offsets.
Good designs separate cache from durability. Cache can be memory, local ephemeral storage, or another evictable layer, but it should not be the durable copy of acknowledged data. A diskless broker may still use local resources for performance, but those resources are not the source of truth.
Object Storage and Metadata
Object storage usually becomes the durable, elastic data repository. It provides shared access, large capacity, and a cost model that fits long retention better than provisioning every broker for peak disk usage. But object storage alone is not enough. The system also needs metadata that maps topics, partitions, offsets, stream ranges, object locations, and ownership state. Diskless Kafka is therefore a storage-system redesign, not an S3 connector attached to Kafka.
Diskless Kafka vs Tiered Storage
Tiered storage is often confused with diskless Kafka because both involve object storage. They solve different problems. Kafka tiered storage, also called remote log storage in Apache Kafka documentation, allows log segments to be copied to remote storage so brokers can reduce local retention pressure. That is useful, especially for long retention and delayed consumers, but it does not remove the local hot log from the broker.
The simplest way to separate the two models is to ask where the active, acknowledged log lives.
| Question | Kafka tiered storage | Diskless shared storage |
|---|---|---|
| Where are recent writes acknowledged? | Broker-local log first | Durable WAL or shared write path |
| Does the broker still own hot partition data? | Yes | No, or only as cache |
| Is object storage primary storage? | Usually remote/cold tier | Yes, for durable stream data |
| Does scaling remove partition data movement? | Not fundamentally | It can, if metadata and storage are shared |
| Is it a small change to classic Kafka? | Closer to incremental evolution | A deeper storage redesign |
Tiered storage is still valuable. It can reduce local disk pressure, extend retention, and make historical replay less expensive. For many existing Kafka clusters, that is the right first step because it preserves the familiar broker model while moving older data to a remote tier.
But tiered storage is not fully diskless. If your main problem is that broker-local state makes scaling, recovery, Kubernetes scheduling, or disk capacity work painful, tiered storage only reduces part of the burden. The broker still owns the hot log. The platform still has to size and operate broker disks for active traffic. Rebalancing can still involve moving local data.
That is why "kafka without disks" should not be used as a synonym for "Kafka with tiered storage." The presence of object storage is not the deciding factor. The deciding factor is whether broker-local disks remain the primary home of active durable log data.
How AutoMQ Approaches Diskless Kafka
Once the requirement is stated clearly, AutoMQ fits into a broader architectural category: Kafka-compatible shared storage. AutoMQ keeps the Kafka protocol surface while replacing Kafka's native broker-local log storage with S3Stream, a storage layer built around object storage and WAL acceleration. In AutoMQ's architecture documentation, object storage is the primary data repository, WAL storage handles the acknowledged write path, and brokers are designed to be stateless relative to durable stream data.
That design is important because it does not pretend that object storage has the same write behavior as local disks. AutoMQ introduces WAL storage specifically to handle the latency and write-efficiency gap between stream ingestion and object storage APIs. Data is written to WAL first, then uploaded to object storage. Hot data and prefetched cold data can be served through cache so that every read does not become a raw object storage read.
The careful part is WAL selection. AutoMQ Open Source supports S3-compatible storage as the WAL option, which creates a minimal architecture that relies on object storage without broker-local data log disks. That can fit latency-insensitive workloads such as logs, monitoring, and analytics pipelines. AutoMQ commercial editions support additional WAL options across cloud providers, including Regional EBS-style block storage and NFS-style file storage, for workloads that need lower write latency or different durability characteristics.
This is also why a responsible diskless Kafka evaluation should not ask whether a vendor says "stateless broker" on a slide. It should ask what happens between a producer send and an acknowledgement:
- What durable system receives the write before the client gets success?
- Is the WAL shared, broker-bound, regional, zonal, object-based, block-based, or file-based?
- What is the expected latency profile for that WAL backend under your workload?
- How does a replacement broker recover unflushed data?
- Which local resources are durable state, and which are only cache?
AutoMQ's answer is not that disks are magic dust to be removed from the universe. The answer is more precise: move durable Kafka log storage out of broker-local data disks, use object storage as the primary repository, and choose a WAL backend that matches the workload's latency and durability needs.
When Diskless Kafka Makes Sense
Diskless Kafka is most compelling when the pain comes from broker state rather than from Kafka APIs. If your teams like Kafka clients, Kafka Connect, Kafka Streams, ACLs, and operational semantics, but dislike provisioning and moving broker-local disks, the architecture may be worth evaluating.
The strongest fit usually appears in cloud and Kubernetes environments:
- Workloads with long retention where object storage economics matter more than keeping every byte on broker-attached block volumes.
- Bursty traffic where compute needs to scale faster than local partition data can be reassigned.
- Kubernetes platforms that want brokers to behave more like replaceable compute units than storage owners.
- Multi-cloud or private cloud designs where S3-compatible storage is already a standard infrastructure layer.
- Cost optimization programs where over-provisioned disks and cross-zone replication traffic are visible budget items.
The weaker fit matters too. If a workload has strict tail-latency requirements, a diskless design needs a WAL backend and cache strategy tested under production-like load. If the object storage service has unpredictable availability or networking, the streaming system inherits that risk. If the team lacks observability across brokers, WAL, object storage, cache, and metadata, failures may become harder to diagnose instead of easier.
Diskless Kafka changes the failure model. Traditional Kafka asks whether enough broker replicas remain in sync. Diskless Kafka asks whether the shared write path, object storage, metadata, and serving compute can recover together. That is not worse by default, but it is different enough that a production decision should include failure injection, broker replacement tests, object storage throttling tests, and cold-read replay tests.
Evaluation Checklist
The fastest way to evaluate "Kafka without disks" is to force precise answers before benchmarks.
Use this checklist in architecture reviews:
| Area | Question to ask | Why it matters |
|---|---|---|
| Durability | Where does an acknowledged write become durable? | This defines the real data-loss boundary. |
| Local disk use | Are broker-local disks durable state or cache? | Cache is replaceable; durable local logs are not diskless. |
| WAL backend | What storage medium backs the WAL? | Latency and failure behavior depend on this choice. |
| Read path | How are tail reads and catch-up reads served? | Object storage needs cache and prefetch discipline. |
| Metadata | Where are offset-to-object mappings maintained? | Shared storage needs accurate shared metadata. |
| Scaling | Does adding or removing brokers move data? | The answer reveals whether brokers are truly stateless. |
| Operations | Can the team observe brokers, WAL, cache, metadata, and object storage together? | Diskless designs fail across layers, not in one process. |
Do not treat "no local disk" as automatically better. Treat it as a claim that has to be decomposed. If the system still depends on broker-local durable volumes, it is not fully diskless. If it uses local ephemeral space as cache while durable writes land in shared WAL and object storage, it may be diskless in the operational sense that matters. Diskless Kafka is not Kafka with the disks unplugged. It is Kafka-compatible streaming with a different durability engine underneath.
For teams evaluating that path, AutoMQ is worth a look because it keeps Kafka compatibility while making object-storage-backed shared storage and stateless brokers the foundation rather than an add-on. The next step is not to accept any vendor's architecture diagram at face value; it is to test your own write latency, replay behavior, broker failure recovery, and object storage limits against the WAL mode you would actually run.
References
- Apache Kafka documentation, broker configuration
log.dirs: https://kafka.apache.org/documentation/#brokerconfigs_log.dirs - Apache Kafka documentation, tiered storage: https://kafka.apache.org/documentation/#tiered_storage
- AutoMQ documentation, architecture overview: https://docs.automq.com/automq/architecture/overview?utm_source=blog&utm_medium=content&utm_campaign=diskless-kafka-can-kafka-run-without-local-broker-disks
- AutoMQ documentation, S3Stream shared streaming storage: https://docs.automq.com/automq/architecture/s3stream-shared-streaming-storage/overview?utm_source=blog&utm_medium=content&utm_campaign=diskless-kafka-can-kafka-run-without-local-broker-disks
- AutoMQ documentation, WAL storage: https://docs.automq.com/automq/architecture/s3stream-shared-streaming-storage/wal-storage?utm_source=blog&utm_medium=content&utm_campaign=diskless-kafka-can-kafka-run-without-local-broker-disks
FAQ
Can Kafka run without local disks?
Traditional Apache Kafka depends on broker-local log storage, so it cannot become diskless by removing disks from the broker. Kafka-compatible diskless architectures are possible when a durable WAL, shared object storage, cache, and metadata layer replace the broker-local log.
Is Kafka tiered storage the same as diskless Kafka?
No. Tiered storage moves older log segments to remote storage, but the broker still keeps the active hot log locally. Diskless Kafka moves durable stream data away from broker-local disks and treats brokers more like replaceable compute.
Does diskless Kafka still need a WAL?
In practice, yes. A WAL gives the system a durable acknowledgement path before data is organized into object storage. The WAL backend can be object storage, block storage, regional block storage, or NFS-style file storage depending on the platform and latency requirements.
Does diskless Kafka mean zero local storage usage?
Not necessarily. A diskless broker may still use memory or ephemeral local space for cache, temporary files, or performance optimization. The key question is whether local storage is durable partition state. Cache does not make the broker stateful in the same way a broker-owned log directory does.
When should teams consider diskless Kafka?
Consider it when broker-local disks are limiting elasticity, Kubernetes operations, retention cost, or recovery speed. Test carefully when workloads have strict tail-latency requirements, heavy replay behavior, or object storage dependencies that need strong observability.