KIP-1150 is easy to misread if you arrive at it through the lens of a single Kafka cluster. The headline idea, diskless topics, sounds like a storage feature: move more Kafka data away from broker disks and use object storage as the durable layer. For an enterprise Kafka estate, the harder question is not whether object storage is attractive. It is which workloads should move to a different durability model, when that change should happen, and how much operational risk you are willing to take while the Apache Kafka implementation, managed Kafka services, and Kafka-compatible engines continue to evolve.
The Apache Kafka KIP page marks KIP-1150 as accepted, but it also draws an important boundary. Acceptance establishes consensus around the need and end-user requirements for diskless topics; it does not freeze a complete implementation. That distinction matters for platform teams. A design direction can be strategically important before it is a production migration target for every topic in your estate.
Why KIP-1150 Changes the Enterprise Conversation
Traditional Kafka durability is built around broker-local logs and replication between brokers. That model worked because Kafka could turn commodity disks and network replication into a reliable distributed log. In cloud environments, the same model collides with a different cost structure. Block storage, cross-zone network traffic, broker replacement, and partition rebalancing are no longer invisible implementation details; they become line items in the platform budget and risks in the operations calendar.
KIP-1150 addresses that tension by proposing diskless topics as a distinct topic type. The phrase "diskless" does not mean brokers never use disks. It means broker disks stop being the primary durable storage for user data on those topics. The KIP keeps the topic-level framing because large Kafka estates rarely have one latency profile. Fraud detection, metrics ingestion, CDC pipelines, audit logs, feature stores, and replay-heavy analytics may all share one Kafka platform, but they do not share the same tolerance for latency, retention cost, or failover behavior.
That is why the enterprise discussion should start with placement rather than feature adoption. Some topics may continue to require classic broker-local storage because the write path needs the lowest available latency. Other topics may pay a large premium for replication and local disk even though their real value comes from durable retention, large fan-out, and predictable recovery. KIP-1150 gives platform teams a vocabulary for separating those cases instead of treating Kafka storage as one fixed architecture.
The Core Trade-Off Is Not Disk Versus No Disk
The useful distinction is where active data becomes durable. Apache Kafka tiered storage, documented separately from KIP-1150, moves completed log segments to a remote tier such as object storage. That can improve retention economics and reduce pressure on local disks, but the active log still depends on broker-local storage and Kafka replication. Diskless topics aim at a deeper layer of the problem: the active write path and the durability boundary.
For buyers and platform owners, this creates three broad architecture families:
| Architecture path | Durable data placement | Practical implication |
|---|---|---|
| Classic Kafka topics | Broker-local disks plus replica placement | Strong fit for latency-sensitive workloads, but multi-zone replication and rebalancing remain central operating concerns. |
| Kafka tiered storage | Local active segments, remote closed segments | Useful for retention and historical reads, but not a full answer to active replication cost. |
| Diskless or shared-storage designs | Object storage becomes the primary durable layer for topic data | Promising for cost and elasticity, but must be evaluated against latency, semantics, failure handling, and migration maturity. |
The table is intentionally conservative. It does not assume that one path replaces every other path. Enterprise estates usually need a portfolio. The mistake is to compare architectures with a single benchmark and then generalize the result across every topic. A payment authorization topic and a clickstream retention topic can both be "Kafka workloads" while having very different storage economics.
Cost Modeling Starts with Data Movement
Kafka cost analysis often starts with storage price per GiB, but storage is only part of the bill. In multi-AZ deployments, data movement can become the more stubborn cost because replication is continuous. A producer write may enter one zone and then be replicated to brokers in other zones. Consumers may read from followers or leaders depending on configuration, placement, and client behavior. Reassignment, recovery, and rebalancing add more movement at the least convenient times.
Cloud pricing pages make this visible: data transfer within a region but across availability zones is a distinct billing category on AWS. The exact bill depends on region, workload, topology, and managed service details, so a serious evaluation should use your own traffic traces rather than a generic savings percentage. Still, the mechanism is clear. If the platform architecture requires every durable write to become multiple cross-zone broker-to-broker transfers, the cloud network bill becomes part of Kafka's durability model.
A useful cost model separates at least five byte paths:
- Producer ingress into the platform, including whether producers are zone-aware.
- Replication or durability writes performed by the broker layer.
- Remote storage writes and reads, including PUT, GET, request, and retrieval patterns.
- Consumer egress, especially when consumer groups span zones or regions.
- Recovery traffic after broker loss, scale-out, partition movement, or storage rebuilds.
This list prevents a common budgeting error. Teams see object storage and assume the result is automatically lower cost. Object storage can be dramatically more cost-effective for durable retention, but the winning architecture is the one that reduces the expensive byte paths without creating additional latency or recovery problems. KIP-1150 is interesting because it targets the durability byte path directly rather than treating remote storage only as a cold tier.
Migration Risk Is a Topic-Level Problem
The KIP's topic-level design is more than an implementation convenience. It matches the way platform teams control risk. If diskless behavior were only a cluster-wide switch, the migration decision would become too coarse: either every workload accepts the revised trade-off or none of them do. Topic-level adoption lets teams start with workloads whose cost and retention profile warrant the change, then expand based on measured behavior.
That migration plan still needs more than a checklist. Kafka topics carry operational contracts: partition count, ordering assumptions, producer retry behavior, idempotence, transactions, consumer lag expectations, ACLs, quotas, observability, and disaster recovery runbooks. A storage architecture that preserves the Kafka protocol but changes the durability path must be tested against those contracts. The question is not only "does my client connect?" but "does my platform team still understand the failure modes?"
The first production candidates are usually not the most latency-critical topics. Better candidates often share three properties: high retained volume, predictable write patterns, and business tolerance for a slightly different write-path profile. Replay-heavy analytics topics, large observability pipelines, and event archives may fit that pattern. Low-latency command streams, tightly coupled transaction workflows, and topics with fragile consumer behavior deserve a slower path.
A Scorecard for Enterprise Evaluation
A good evaluation framework should work whether you are tracking the upstream Kafka roadmap, reviewing a managed Kafka provider, or considering a Kafka-compatible shared-storage engine. If the framework only works for one vendor's slide deck, it is not a framework; it is a procurement shortcut with a diagram attached.
Use these questions before turning the architectural direction into a migration plan:
- Compatibility: Which Kafka APIs and semantics are preserved for your actual workload? Check producer idempotence, transactions if used, consumer groups, ACLs, quotas, and administrative tooling instead of stopping at basic client compatibility.
- Latency class: Which topics have a strict p99 write budget, and which topics primarily care about durable retention and replay? A mixed estate should not force one latency class onto every workload.
- Failure behavior: What happens when a broker, zone, or storage path fails? The answer should include recovery time, data availability, operational steps, and how much data movement recovery creates.
- Cost boundary: Which byte paths disappear, which remain, and which object-storage operations are introduced? Treat storage price, network transfer, request charges, and operational labor as one model.
- Governance: Who decides topic placement, and how is the decision audited? In a large estate, storage class becomes a platform policy, not a one-off tuning parameter.
- Rollback: Can a topic move back, mirror to another cluster, or run in parallel during validation? A design with no credible rollback path belongs in a lab until the team has stronger evidence.
The scorecard also protects teams from over-indexing on KIP status. An accepted proposal can validate the direction of travel, while a production estate still needs evidence from software versions, provider SLAs, test results, and runbooks. Architecture adoption is not a vote on whether the KIP is good. It is a decision about when a specific implementation is mature enough for a specific class of topics.
How AutoMQ Fits the Evaluation
Once the evaluation is framed around durability placement, network movement, and topic-level workload fit, AutoMQ becomes relevant as one concrete architecture in the broader shared-storage category. AutoMQ is a Kafka-compatible streaming system that keeps the Kafka protocol surface while moving durable data into object storage through a shared-storage architecture. Its brokers are designed to be more stateless than traditional Kafka brokers, so scaling and recovery are less tied to local disk ownership.
That does not make every migration automatic. The same scorecard still applies: compatibility, latency, failure behavior, cost boundary, governance, and rollback all need to be tested against your workloads. The useful point is that AutoMQ gives platform teams a way to evaluate the storage direction behind KIP-1150 without waiting for every upstream implementation detail to land in the exact form they may eventually use. For teams already pressured by cross-zone replication cost or broker-local storage operations, that can turn an abstract architectural debate into a measurable proof of concept.
AutoMQ's documentation describes its shared-storage architecture, Kafka compatibility positioning, and approach to reducing cross-zone traffic. Those claims should be read the way any platform claim should be read: as inputs to a test plan. Start with a topic class where the business problem is concrete, such as high-volume retention or cross-AZ transfer cost. Reproduce the write rate, retention, consumer fan-out, and failure cases. Then compare the operational result against your current Kafka baseline, not against a theoretical cluster.
If your team is evaluating this architecture path, the most useful next step is to make the trade-off explicit. Pick 3 representative topics, classify their latency and retention requirements, and run the cost-boundary exercise above. To see how a Kafka-compatible shared-storage implementation approaches that model, review the AutoMQ architecture documentation: AutoMQ Shared Storage Architecture.
References
- Apache Kafka Wiki: KIP-1150: Diskless Topics
- Apache Kafka Documentation: Tiered Storage Overview
- AWS: Amazon EC2 On-Demand Pricing, Data Transfer
- AutoMQ Documentation: Architecture Overview
- AutoMQ Documentation: Eliminate Inter-Zone Traffics
- AutoMQ Documentation: Difference with Apache Kafka
FAQ
Is KIP-1150 already implemented in Apache Kafka?
The Apache Kafka KIP page lists KIP-1150 as accepted, but the page also states that acceptance does not require code or documentation changes by itself. It establishes agreement on the need and end-user requirements, while implementation details are handled separately.
Does diskless Kafka mean brokers use no disks at all?
No. In the KIP framing, diskless means broker disks are no longer the primary durable storage of user data for diskless topics. Brokers may still use local resources for caching, buffering, logs, and operational needs.
How is KIP-1150 different from Kafka tiered storage?
Tiered storage moves completed log segments to remote storage while active segments still rely on the broker-local log. KIP-1150 addresses diskless topics, where the durability model for topic data changes more directly. The difference matters because active replication cost and retention cost are not the same problem.
Should every Kafka topic move to a diskless or shared-storage model?
No. Enterprise Kafka estates usually contain multiple workload classes. Ultra-low-latency topics may remain better suited to classic topics, while high-retention and replication-cost-heavy topics may be stronger candidates for a shared-storage or diskless path.
Where should a platform team start?
Start with topic classification. Pick representative topics, measure write volume, retention, fan-out, latency budget, and cross-zone traffic, then test candidate architectures against those measured requirements. The right first topic is usually one where the current cost or operational pain is large and the latency contract is well understood.
