Kafka's local disk model has survived for a good reason. It made sense when Kafka was born: brokers owned partitions, replicas lived on separate machines, and durability came from copying log segments across servers. That design gave the industry a reliable event backbone.
Kafka did not stay in that original environment. The same replication model now runs inside cloud regions where storage, networking, and compute are billed as separate primitives. A broker-local log means every retention decision, scaling event, and availability-zone failure plan becomes a storage placement problem. The diskless Kafka trend is the industry's answer to that mismatch.
The strongest signal is not that one vendor has a clever architecture. It is that several independent teams are moving in the same direction: WarpStream built an object-storage-native Kafka-compatible service, Confluent acquired WarpStream, IBM completed its Confluent acquisition, Aiven introduced Inkless around KIP-1150, and AutoMQ has been running a Kafka-compatible diskless architecture in production.
Why Local Storage Became the Wrong Default
Traditional Kafka treats broker disks as the center of the system. A partition leader writes to its local log, follower replicas copy the data, and consumers fetch from those broker-owned logs. The model is clean when disks are attached to servers and internal network transfer is not an explicit line item. In the cloud, the same model repeats work that the infrastructure layer already performs.
Three cloud realities changed the economics:
- Durability moved below the broker. Object storage services are built to keep data durable across failures. Kafka still needs ordering, offsets, and protocol semantics, but it no longer has to make local broker disks the permanent durability layer for every retained byte.
- Elasticity became a platform expectation. Cloud teams expect compute pools to scale up and down with traffic. Stateful brokers resist that motion because removing or adding capacity often triggers partition movement and replica catch-up.
- Multitenancy raised the cost of overprovisioning. Shared Kafka platforms serve many teams with uneven traffic patterns. Keeping local disks sized for peak retention and failure headroom turns idle capacity into a permanent tax.
The cloud already separated compute from storage for databases, data lakes, warehouses, and queues. Kafka is going through the same adjustment, but it has to preserve a harder contract: clients expect Kafka behavior.
Diskless Does Not Mean One Design
"Diskless Kafka" can hide more than it reveals. Some designs remove local disk only for selected topic types. Some keep Kafka's broker-leader model and change the storage layer underneath. Some move more ordering and indexing responsibility into a separate coordination service. All of them reduce dependence on broker-attached disks, but they do not have the same failure model or migration surface.
The spectrum starts with traditional Kafka: local log segments, broker-to-broker replication, and stateful scaling. Tiered storage moves older data to object storage but keeps the hot path on broker disks. Diskless-topic designs push more data into shared storage, often with topic-level feature boundaries. Fully diskless architectures make brokers mostly stateless and turn partition reassignment into metadata movement rather than data copying.
This distinction matters because buyers do not migrate a diagram; they migrate workloads. A platform team may have append-only telemetry topics, compacted state topics, transactional producers, Kafka Streams applications, Connect jobs, and long-retention replay workloads inside the same estate. A design that works for append-heavy logs can still force a topic-by-topic audit if it does not cover the semantics hidden in the rest of the cluster.
| Architecture direction | What changes | What to verify |
|---|---|---|
| Tiered storage | Older segments move to object storage | Hot-tier sizing and remote-read latency |
| Diskless topics | Selected topics write to shared storage | Topic limits and coordinator availability |
| Kafka-compatible diskless storage | Kafka compute remains; storage moves to WAL plus object storage | WAL latency and semantic coverage |
| Object-storage-native service | Service starts from object storage | API coverage and control-plane dependency |
The useful architecture is the one whose trade-offs match the workload. For a greenfield observability pipeline, API compatibility may matter less than throughput. For a company-wide event backbone, compatibility, failure isolation, and deployment control usually matter more than a storage slogan.
The Market Signals Are Stacking Up
KIP-1150 is important because it moves the diskless conversation into Apache Kafka itself. It signals that the community recognizes a real storage-layer problem rather than a narrow vendor complaint. The exact upstream implementation will continue to evolve, but the direction is clear enough for platform teams to ask what their Kafka estate should look like after local disks are no longer the default durability boundary.
WarpStream's acquisition by Confluent sent a different signal. Confluent built much of the commercial Kafka market around managed Kafka, and buying a Kafka-compatible object-storage-native company showed that diskless architecture had become strategically relevant to the largest Kafka vendor. IBM's completed acquisition of Confluent then placed that combined streaming portfolio inside a larger infrastructure and AI platform story.
Aiven Inkless adds another angle because it connects a managed Kafka provider to the KIP-1150 path. Its public material frames diskless topics as a way to reduce broker storage coupling while keeping the managed Kafka experience. That is valuable for teams already standardized on Aiven, but the evaluation has to include topic limitations and the role of the coordinator in the write path.
AutoMQ represents the production-first version of the trend. It keeps Kafka compatibility as the design center, reuses the Kafka compute layer, and replaces broker-local retained storage with a diskless storage engine built around a WAL and object storage. Public AutoMQ materials reference production deployments at companies such as JD.com, Grab, Tencent Music, LG U+, and HubSpot. They show that diskless Kafka has moved beyond conference speculation and into operational environments where broker failure, scaling, replay, and cost pressure all show up together.
Ursa and other storage-disaggregated streaming efforts make the broader pattern clearer. Streaming vendors are converging on shared durable storage because the operational center of gravity has moved there. The question is how much Kafka behavior you need to preserve while moving there.
The Vendor Map: Same Trend, Different Center of Gravity
A useful vendor comparison starts with two axes: how closely the system preserves Kafka compatibility, and how much production maturity is visible in public materials. Open source posture, deployment model, latency target, and coordination architecture still matter, but the main risk comes first.
WarpStream fits teams that want an object-storage-native service and accept a service-specific architecture. Its acquisition changed the buyer conversation: some teams will like the Confluent and IBM alignment, while others will examine vendor concentration more carefully.
Aiven Inkless fits teams that already want Aiven-managed Kafka and can adopt diskless topics deliberately. The benefit is a managed-service path tied to upstream Kafka. The risk is granular migration planning if the topic model does not cover every Kafka capability a broad platform uses.
KIP-1150 is not a vendor, but it belongs on the map because it validates the direction. It is the community design lane rather than a turnkey production platform, so teams should separate upstream direction from production readiness.
Ursa and adjacent streaming projects show that storage disaggregation is not confined to one ecosystem, though broader ambition can mean a larger migration surface for Kafka-heavy estates.
AutoMQ sits in the quadrant that matters for Kafka platform teams looking for the least semantic disruption: Kafka compatibility plus visible production deployment. It keeps the Kafka compute layer and changes the storage premise, which matters when the goal is to reduce cloud cost, unlock elasticity, and keep Kafka clients, protocols, and operational expectations recognizable.
What Changes for Platform Teams
Diskless Kafka changes the platform team's job. In local-disk Kafka, much of the work is defensive: size disks, watch retention, plan reassignment windows, protect ISR health, avoid replay storms, and keep spare capacity for broker loss. A diskless architecture moves the critical questions to a different layer.
The evaluation checklist looks like this:
- Where is the durability boundary? The answer should be precise. Is a write durable after a WAL append, after object storage commit, or after a coordinator records metadata?
- What happens when object storage slows down? Elevated latency is often harder than total outage because the system keeps running while buffers, caches, and flush queues accumulate pressure.
- Which Kafka semantics are preserved? Transactions, compaction, consumer groups, Streams, Connect, quota behavior, and admin operations should be checked against real workloads.
- How does scaling work under load? The promise of stateless brokers only matters if partition ownership can move without large data-copying events.
- What is the control-plane dependency? A diskless system may reduce data replication while adding a metadata service, managed control plane, or coordinator dependency. That trade should be explicit.
Cost analysis becomes more honest here. Diskless Kafka can reduce storage replication, cross-zone data movement, and overprovisioned compute, but a lower bill is not enough if the architecture introduces a failure mode the team cannot operate.
Why AutoMQ Is a Mature Expression of the Trend
AutoMQ's strongest argument is not that it discovered diskless Kafka before everyone else. The stronger argument is that it made a conservative cut in the architecture. Kafka's compute layer remains because it contains the semantics users rely on; the storage layer changes because cloud economics broke the old model.
That design choice shows up in day-to-day operations. Brokers can be treated more like stateless compute. Partition reassignment is primarily metadata work because retained data is not tied to broker disks. Long retention shifts to object storage instead of expanding local volumes.
There are still trade-offs to test. WAL backend choice affects latency, cache behavior matters for replay-heavy consumers, and object storage performance needs observability. These are bounded engineering questions inside an architecture that already matches the cloud's storage model.
For large Kafka teams, that is the practical appeal. They do not need a philosophical debate about whether Kafka should become diskless. They need a path that preserves the parts of Kafka their applications depend on while removing the cost and operational drag created by broker-owned disks. AutoMQ is one of the clearest production implementations of that path.
The Future Kafka Stack Will Be Less Stateful
The industry is not moving beyond local storage because local disks are bad. It is moving because local disks are the wrong permanent anchor for cloud streaming systems. They make brokers heavy, scaling slow, retention expensive, and multitenant platforms harder to operate. Object storage changes that foundation, but only if the streaming layer handles ordering, latency, caching, and compatibility with enough discipline.
The coming phase of Kafka architecture will likely be mixed. Some teams will use upstream diskless topics as they mature. Some will consume diskless streaming through a managed service. Some will choose object-storage-native platforms for fresh pipelines. Others will adopt AutoMQ because they want Kafka compatibility, open deployment options, and a diskless storage engine that has already met production workloads.
That variety is healthy. It means diskless Kafka is not a marketing phrase looking for a market; it is a response to the same pressure felt across the streaming stack. The old question was how to operate broker disks better. The sharper question is why broker disks should own the future of Kafka at all.