Kafka operators usually search for KIP-1150 after the storage problem has become more concrete than "Kafka is expensive." They have seen broker disks limit retention, partition reassignment stretch across maintenance windows, and multi-AZ replication turn into a standing line item in the cloud bill. The question behind the search is not whether diskless topics sound attractive. It is whether a production Kafka estate can change its storage boundary without breaking the semantics, tooling, and operational habits that applications already depend on.
KIP-1150 matters because it brings that question into the Apache Kafka design conversation. The proposal describes diskless topics: topics whose durable data path moves away from broker-local disks and toward object storage. It asks operators to reconsider which parts of Kafka should remain broker-owned and which parts should become shared infrastructure.
The implementation conversation should start there, with responsibilities. A broker-local log gives operators a familiar failure model: disks fill, replicas fall behind, ISR changes, reassignment moves bytes, and recovery is visible through broker and partition metrics. A diskless topic design changes the map. Object storage, metadata coordination, write buffering, cache policy, network routing, and topic capability boundaries become part of the Kafka operating model.
Why KIP-1150 Is an Operator Question
The most tempting way to read KIP-1150 is as a cost optimization. That is understandable, because local disks and replica movement are painful in cloud environments. Traditional Kafka replicates data between brokers for durability and availability, and multi-AZ deployments can multiply both storage and network traffic. AWS also prices data transfer and S3 operations as explicit services, so every architectural decision around placement, reads, and replication eventually becomes a billable path.
Cost is only the entry point. Operators inherit the consequences of any new write path. If produce acknowledgements depend on an object storage commit, then batching, buffering, retries, and timeout behavior need proof. If a coordinator owns diskless-topic metadata, its availability and recovery behavior need the same scrutiny that teams already apply to KRaft and broker quorum sizing.
The first implementation decision is therefore not "diskless or not." It is a workload classification decision:
- Latency-sensitive transactional streams need evidence for acknowledgement latency, retry behavior, idempotent producer behavior, and transaction boundaries before they should move to a diskless path.
- High-retention telemetry and observability streams often have a stronger case, because storage economics and replay behavior dominate the operating cost.
- Data lake ingestion pipelines may benefit from object-storage alignment, but only if compaction, ordering, schema governance, and downstream replay patterns are validated.
- Shared platform topics require the most caution, because a single topic capability gap can affect many teams that do not know they depend on it.
That classification keeps the discussion honest. KIP-1150 is not a permission slip to move every topic at once. It is a signal that Kafka's storage layer is becoming more flexible, and flexible storage makes workload ownership more important.
Tiered Storage Is Not the Same Decision
Many Kafka teams already use or evaluate Tiered Storage, so it is easy to collapse every object-storage discussion into one bucket. That shortcut causes bad design reviews. Apache Kafka Tiered Storage, associated with KIP-405, keeps the local log as the active write path and offloads older log segments to remote storage. It can reduce local disk needed for long retention, but the broker still owns the hot log.
Diskless topics change the primary storage question. Instead of asking "how much old data can leave the broker," operators ask "what must remain on the broker for Kafka to preserve its behavior?" That can include cache, temporary files, metadata, non-diskless topics, and write buffers used to hide object storage latency.
| Evaluation area | Tiered Storage style question | KIP-1150-style diskless question |
|---|---|---|
| Write path | Does the active segment remain local? | Where is a produce acknowledged as durable? |
| Read path | When do historical fetches go remote? | Which reads hit cache, buffer, or object storage? |
| Scaling | How much data moves during reassignment? | Can ownership move without copying retained data? |
| Failure recovery | Which local replicas are still authoritative? | Which shared storage and metadata state recover the topic? |
| Cost model | How much local retention can be reduced? | Which storage, request, and network paths replace replica storage? |
This distinction is more than terminology. Tiered Storage can be introduced as a retention feature for selected topics. Diskless topics affect the core operating contract of a topic. Treating both as the same migration under-tests writes during partial failure, reads during catch-up, and recovery during a capacity event.
The Production Questions to Ask Before Implementation
A useful KIP-1150 review should look like a launch readiness review, not a feature checklist. The proposal explains an architectural direction, but operators need proof that their platform can absorb the operational changes.
1. What is the durability acknowledgement boundary? Kafka users care about when a record is safe, not where the bytes eventually land. For diskless topics, document the exact path from produce request to durable acknowledgement. Include batching, retries, partial object writes, metadata commits, and what happens when the object store is slow or unavailable. If the design uses a write-ahead log or buffer, specify whether it is local, zonal, regional, or object-storage-backed.
2. Which Kafka semantics are supported without application changes? Compatibility has to cover more than basic produce and consume. Check idempotent producers, transactions, consumer group coordination, offset commits, compaction, retention, ACLs, quotas, MirrorMaker or replication tooling, Kafka Connect, schema registry integrations, and the operational APIs your internal platform already exposes. Any unsupported feature should become a topic placement rule, not a surprise during migration.
3. What happens to ordering during ownership changes? Diskless storage can reduce byte movement, but Kafka applications still depend on partition ordering. Operators need to know how leadership, ownership, fencing, and metadata updates prevent duplicate writers or gaps during broker failure, coordinator failover, rolling upgrades, and scale-in events.
4. How does the network path change? Broker-to-broker replication may shrink, but object storage requests, cache fills, cross-zone reads, and private endpoint routing can still create cost and latency. On AWS, data transfer and S3 pricing should be modeled with the actual deployment topology, not with a generic "object storage is lower cost" assumption.
5. What are the rollback boundaries? Topic-level adoption sounds safer than cluster-wide migration, but rollback can be tricky once producers and consumers have moved traffic. Define whether a diskless topic can be converted back, mirrored to a traditional topic, drained through dual writes, or isolated behind a new cluster. The rollback design should include offset continuity and consumer group behavior.
6. Which metrics prove the design is healthy? Traditional Kafka dashboards focus on broker disk, ISR, under-replicated partitions, request latency, consumer lag, and controller health. Diskless topics add storage-path metrics: write buffer latency, object storage request errors, cache hit rate, remote fetch latency, upload lag, metadata object count, compaction backlog, and recovery progress.
These questions are deliberately operational. A design that looks elegant on a storage diagram can still be hard to run if the failure model is opaque. Kafka operators should demand evidence at the boundary where application expectations meet infrastructure behavior.
A Practical Readiness Scorecard
The fastest way to de-risk KIP-1150 adoption is to turn architecture questions into gates. A scorecard prevents two common failure modes: treating diskless topics as a global switch, and treating one successful benchmark as proof for every workload. The scorecard should be owned by the platform team, but application teams should see the topic placement rules that come out of it.
| Gate | What to verify | Evidence to collect |
|---|---|---|
| Compatibility | Client APIs, transactions, compaction, ACLs, Connect, replication, admin tooling | Integration tests using existing clients and platform automation |
| Latency | Produce p50/p95/p99, fetch latency, catch-up behavior, timeout sensitivity | Workload-specific benchmark with failure injection |
| Cost | Storage, requests, data transfer, cache, compute, operational overhead | Cloud bill model tied to topic throughput and retention |
| Recovery | Broker loss, coordinator failover, object storage throttling, AZ impairment | Game-day results and runbooks |
| Governance | Topic eligibility, rollout policy, rollback path, ownership | Approved platform policy and migration checklist |
| Observability | Kafka metrics plus storage-path and metadata-path metrics | Dashboards, alerts, and incident drill traces |
The important part is the discipline behind the table. A low-risk telemetry topic might pass with a narrower test set because replay delay has modest business impact. A payments topic should require stronger proof around transactions, ordering, timeout behavior, and rollback. Diskless adoption should be boring in production because the interesting questions were already answered in staging.
Where AutoMQ Fits in the Evaluation
Once the platform team has defined the evaluation framework, AutoMQ becomes relevant as one concrete architecture in the broader diskless Kafka category. AutoMQ is a Kafka-compatible streaming platform that keeps Kafka protocol behavior while replacing broker-local retained storage with shared storage built around S3Stream, WAL storage, object storage, and stateless brokers.
That design addresses several of the questions operators ask around KIP-1150-style architectures. Because retained data is placed in shared object storage rather than bound to individual broker disks, scaling and partition reassignment can be treated more like ownership and traffic movement than bulk data copying. Because AutoMQ uses a WAL layer before object storage upload, its write path is designed to avoid exposing raw object storage latency directly to every produce request. Because the architecture is Kafka-compatible, the evaluation can focus on workload semantics, operational boundaries, and migration testing rather than asking application teams to adopt a new streaming API.
AutoMQ is not the same thing as KIP-1150, and it should not be evaluated as if any diskless implementation is interchangeable with another. The useful comparison is at the mechanism level:
- Does the platform preserve the Kafka protocol and the client behavior your applications use?
- Does the write path define a clear durability boundary before acknowledgement?
- Does the storage layer make brokers more replaceable without hiding a fragile metadata dependency?
- Does the deployment model fit your control-plane and data-plane ownership requirements?
- Does the network design reduce the expensive replication paths that pushed the team toward diskless Kafka in the first place?
For teams already studying KIP-1150, AutoMQ is a practical reference point for what a production-oriented shared-storage Kafka architecture can look like. It also separates the strategic direction from the implementation details: Kafka storage is moving away from broker-owned disks, but every platform still has to prove WAL, cache, object storage, metadata, failure recovery, and compatibility.
Migration Planning Without the Drama
The safest adoption plan starts with a topic inventory. Classify topics by latency sensitivity, retention length, replay pattern, compaction need, compliance requirement, and dependency count. The goal is to find topics where diskless storage changes the most expensive part of the system before it touches the riskiest semantics.
After that, build a migration lane rather than a one-time move. A typical lane has four stages: baseline the current topic, run a representative workload, mirror or dual-write during validation, and cut over with a rollback path that preserves offsets or makes offset translation explicit. The team should also test broker restart, scale-out, scale-in, object storage throttling, cache miss storms, consumer catch-up, and failed upgrades.
There is no virtue in making the first diskless topic heroic. Pick a topic with enough traffic to reveal real behavior and enough operational tolerance to survive a controlled rollback. If the team cannot explain what success looks like before the test, the test is not ready.
Procurement and Architecture Review Criteria
KIP-1150 will also show up in vendor and platform evaluations. That is healthy, but procurement language can flatten meaningful differences. "Diskless" may refer to upstream topic types, a managed Kafka feature, a Kafka-compatible object-storage-native platform, or a shared-storage engine with WAL acceleration. Ask for diagrams and failure evidence, not adjectives.
Three artifacts make the review concrete: a write-path diagram that labels the durable acknowledgement point, a failure matrix covering broker, coordinator, object storage, network, and metadata failures, and a cost model separating compute, storage, request, cache, cross-zone transfer, and operations work. If a platform cannot explain those pieces, it is not ready to be compared on total cost.
Diskless topics can reduce disk dependence and data movement, but they do not remove physics. Object storage still has latency, request behavior, endpoint placement, throttling modes, and IAM boundaries. The stronger design exposes those trade-offs clearly and gives operators levers to manage them.
Closing the Loop
The original KIP-1150 search is really a readiness question. Kafka operators need to decide when broker-owned disks are still the right durability boundary and when shared storage is a better operating model. That decision should be made topic by topic, with compatibility tests, failure drills, and a cost model that matches the actual cloud topology.
If your team is comparing KIP-1150-style designs with production-ready Kafka-compatible shared storage, use the scorecard above as the review template. To see how AutoMQ approaches the same storage shift with S3Stream, WAL, object storage, and stateless brokers, read the technical walkthrough here: how AutoMQ implements low-latency diskless Kafka.
References
- Apache Kafka KIP-1150: Diskless Topics
- Apache Kafka documentation: Tiered Storage
- Apache Kafka documentation: Message delivery semantics
- AWS EC2 pricing: Data Transfer
- AWS S3 pricing
- AutoMQ architecture overview
- AutoMQ S3Stream shared streaming storage
- AutoMQ compatibility with Apache Kafka
FAQ
Is KIP-1150 production-ready in Apache Kafka?
Check the Apache Kafka KIP page before making a roadmap decision. Treat KIP-1150 as an architectural proposal and evaluation signal, then validate the specific implementation you plan to run.
Is KIP-1150 the same as Tiered Storage?
No. Tiered Storage moves older log segments to remote storage while the active write path can remain broker-local. KIP-1150-style diskless topics are about changing where topic data is durably stored and how brokers, metadata, cache, and object storage cooperate in the primary topic path.
Does diskless Kafka mean brokers use no disks at all?
Not necessarily. The term usually means retained topic data is no longer primarily owned by broker-local disks. Brokers may still use local or attached storage for cache, metadata, temporary files, logs, or write buffering depending on the implementation.
What should operators test first?
Start with produce acknowledgement latency, consumer catch-up reads, broker failure, coordinator failover, object storage throttling, and rollback. Those tests reveal whether the design preserves the Kafka behavior your applications actually depend on.
Where does AutoMQ fit relative to KIP-1150?
AutoMQ is a Kafka-compatible shared-storage streaming platform, not the upstream KIP itself. It is relevant because it shows one production-oriented way to combine Kafka compatibility, stateless brokers, WAL storage, object storage, and reduced broker-local data movement.
