KIP-1150 is not a small Kafka storage tweak. It asks a bigger question that many platform teams have already been asking in their own architecture reviews: what happens when Kafka topics no longer depend on broker-local disks as the durable system of record? That question sounds abstract until it reaches production planning, where the hard parts become familiar: client compatibility, failure recovery, network cost, operational ownership, and the migration path from the clusters already running today.
The reason diskless topics attract attention is easy to understand. Kafka has always tied a partition replica to a broker and its storage. That design is operationally clear, but cloud infrastructure has changed the economics around it. Local disks, block volumes, replication traffic, and rebalancing time all become visible line items or operational constraints. A proposal that moves durable log storage away from broker disks is therefore not only a Kafka internals discussion. It is a platform decision.
Why KIP-1150 Changes the Evaluation
The Apache Kafka proposal for diskless topics describes a model where the broker no longer needs to store topic data on local disks for selected topics. In broad terms, the direction is to place the authoritative log data in shared storage while brokers continue serving Kafka protocol operations. That shift is different from classic broker scaling, where more data usually means more broker storage, more replica placement work, and more time spent moving partition data during operational events.
Tiered storage already changed part of this equation by letting Kafka move older log segments to remote storage. It helps with retention-heavy workloads because brokers do not need to keep all historical data on local volumes. Diskless topics push the separation further: the topic's durable log path is designed around shared storage from the start, not as a secondary tier for cold data. Mixing those two ideas leads to bad architecture reviews. Tiered storage asks, "Where should old segments live?" Diskless topics ask, "Does the broker need to own durable topic storage at all?"
That distinction matters because the trade-offs move from storage capacity to the full data path. A diskless design has to answer how produce latency is protected, how consumers read hot and cold data, how failures are masked, and how Kafka's existing semantics remain intact. The strongest proposal-to-production review treats KIP-1150 as a change in failure domains, not as a way to remove disks from a bill of materials.
The Questions to Ask Before a Pilot
A pilot should not begin with a benchmark alone. Benchmarks are useful, but they rarely expose the risks that decide whether a new Kafka architecture can be adopted by a platform team. The first review should identify the invariants that the business expects Kafka to preserve.
Start with the workload, not the feature. A diskless topic is more attractive when storage growth, recovery time, or cross-zone data movement is already painful. It is less attractive when the workload is small, the operational model is stable, and the team has no clear cost or elasticity problem. The proposal becomes easier to evaluate when the pilot has a specific production target, such as long-retention topics, bursty ingest, multi-tenant platform topics, or clusters where reassignment time blocks routine maintenance.
The pre-pilot checklist should cover five areas:
- Kafka semantics. Producers, consumers, transactions, ACLs, quotas, compaction, retention, and offset behavior should be checked against the client and framework versions in use. Compatibility is not a slogan; it is a matrix.
- Latency envelope. Shared storage changes where durable writes land. The design must explain the write-ahead path, cache behavior, and tail-latency controls before it can be trusted for hot topics.
- Failure recovery. Broker loss, zone impairment, storage API throttling, and metadata-plane events need separate recovery expectations. A single "it recovers" answer is not enough.
- Network boundary. The review should model traffic between clients, brokers, zones, and storage. Cloud networking often becomes the hidden cost center in Kafka.
- Operating model. The team needs to know who owns upgrades, observability, capacity planning, incident response, and rollback when disk ownership moves out of the broker.
This is where many architecture reviews become too optimistic. They compare steady-state throughput but skip the moment when a broker fails, a zone becomes noisy, or a consumer suddenly replays a large window. Diskless topics are a storage architecture decision, and storage architecture is tested under disruption.
A Production Evaluation Framework
The practical way to evaluate diskless topics is to separate "Kafka compatibility" from "production equivalence." Compatibility means Kafka clients and applications can speak the protocol and rely on expected semantics. Production equivalence means the platform can meet the same recovery, governance, cost, and observability standards that the existing Kafka service already carries.
| Evaluation area | What to verify | Evidence to collect |
|---|---|---|
| Client behavior | Producer acks, consumer groups, transactions, admin APIs, security controls | Integration tests with real client versions |
| Write path | Durable acknowledgement point, WAL behavior, cache hit patterns | p50/p95/p99 latency under normal and failure conditions |
| Read path | Hot reads, replay reads, catch-up after downtime | Consumer lag recovery and remote-read behavior |
| Cost model | Compute, storage, requests, inter-zone or inter-region data transfer | Cloud bill model tied to workload assumptions |
| Recovery | Broker loss, zone degradation, storage throttling, metadata failure | Game-day results and rollback procedure |
| Operations | Monitoring, alerts, scaling, upgrades, incident ownership | Runbooks and SLO dashboards |
The table looks ordinary, but the order matters. Teams often start with cost because storage separation promises better resource utilization. That is understandable, especially for FinOps teams facing Kafka clusters that are overprovisioned for storage or replication. Still, cost is only useful after semantics and recovery are clear. A lower infrastructure bill does not help if the platform team cannot explain when a record is durable or how a consumer replay behaves during a storage-side event.
The cost model deserves its own pass because diskless architecture changes multiple cost drivers at once. Broker disks may shrink. Compute can scale with request load instead of retained bytes. Object storage can absorb durable capacity. At the same time, request charges, cache sizing, and network topology become more important. On AWS, for example, data transfer and managed service pricing vary by path and service, so an honest model should avoid a single blended number. Build the model from the actual paths your records travel.
From Proposal to Production Rollout
A safe rollout treats diskless topics as a new class of topic, not a global replacement switch. The first candidate should have meaningful storage pressure but clear blast-radius control. Internal platform topics, non-critical analytics feeds, or topics with strong replay tolerance are often better early candidates than the most latency-sensitive payment or order streams.
The migration plan should be explicit about dual running. Even if the final target is a diskless topic, the team needs a period where producers, consumers, metrics, and rollback procedures are all observable. That does not always mean dual-writing every record. In some cases it means mirroring, shadow consumers, or a replayable cutover window. The right approach depends on whether the topic has strict ordering expectations, how consumers commit offsets, and whether downstream systems can tolerate duplicate processing during a rollback.
The rollout path should have gates:
- Lab validation. Prove client compatibility and basic performance with the exact libraries, security settings, and message sizes used in production.
- Failure testing. Kill brokers, isolate zones where possible, throttle storage paths if the environment allows it, and verify that alerts explain the incident rather than only reporting symptoms.
- Cost rehearsal. Run enough traffic to validate storage, request, and network assumptions. A spreadsheet that has never met real traffic is a guess.
- Limited production. Move one controlled topic class, watch consumer lag and error budgets, and keep rollback boring.
- Policy update. Define which topic types should use diskless storage by default and which should stay on the existing architecture.
The final gate is governance. Once platform teams support more than one storage model, application teams need clear defaults. Without policy, every new topic becomes a debate. With policy, the platform can say that high-retention analytical topics, elastic ingest topics, or specific multi-tenant workloads use the diskless path, while ultra-low-latency workloads remain on the proven local-disk path until the evidence says otherwise.
Where AutoMQ Fits
After the evaluation framework is clear, AutoMQ becomes relevant as one implementation path for teams that want Kafka-compatible streaming with shared storage as the architectural baseline. AutoMQ is a Kafka-compatible cloud-native streaming system that separates compute from storage, uses object storage as the durable storage layer, and keeps brokers stateless enough to scale and recover without moving large local replicas.
That architecture maps directly to several questions in the checklist. If the problem is slow partition reassignment, the important capability is not a faster reassignment script; it is avoiding large broker-to-broker data movement during scaling and recovery. If the problem is cloud traffic cost, the important capability is not only lower storage spend; it is controlling the data paths that create cross-zone traffic. AutoMQ's public documentation describes a shared-storage architecture and a zero inter-zone traffic design for supported deployment patterns, which are the kinds of claims teams should verify against their own workload and cloud topology.
AutoMQ is not a reason to skip the production review. It is a candidate architecture to put through it. The right test is whether existing Kafka clients continue to behave correctly, whether latency fits the topic class, whether operational recovery is simpler in practice, and whether the cloud bill reflects the expected shift from broker-attached storage to shared object storage. Those are measurable questions, which is exactly why they belong in the proposal-to-production checklist rather than in a vendor comparison spreadsheet.
The most useful AutoMQ pilot mirrors the rollout path above. Pick a topic class where the current Kafka architecture creates a visible constraint. Model the cost drivers before the test. Run the client compatibility suite. Break brokers on purpose. Measure consumer catch-up after a replay. Then decide whether the architecture is ready for broader production use. That process is slower than a quick benchmark, but it produces an answer platform owners can defend.
If your team is turning KIP-1150 research into a real platform decision, use the checklist above as a pilot design, then compare it with a Kafka-compatible shared-storage implementation you can test today. AutoMQ's architecture overview is a practical starting point: https://docs.automq.com/automq/what-is-automq/overview?utm_source=blog&utm_medium=cta&utm_campaign=gs100-0017
References
- Apache Kafka KIP-1150: Diskless Topics: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A%2BDiskless%2BTopics
- Apache Kafka KIP-405: Kafka Tiered Storage: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A%2BKafka%2BTiered%2BStorage
- Apache Kafka Documentation: https://kafka.apache.org/documentation/
- AWS EC2 On-Demand Pricing, including data transfer notes: https://aws.amazon.com/ec2/pricing/on-demand/
- AutoMQ overview: https://docs.automq.com/automq/what-is-automq/overview?utm_source=blog&utm_medium=reference&utm_campaign=gs100-0017
- AutoMQ zero inter-zone traffic overview: https://docs.automq.com/automq-cloud/eliminate-inter-zone-traffics/overview?utm_source=blog&utm_medium=reference&utm_campaign=gs100-0017
FAQ
Is KIP-1150 the same thing as Kafka tiered storage?
No. Tiered storage moves older log segments to remote storage while brokers still use local storage for the active log path. Diskless topics are about designing selected topics so durable topic storage does not depend on broker-local disks in the same way. The operational questions overlap, but the architecture review should treat them separately.
Does a diskless topic automatically reduce Kafka cost?
No. It can reduce certain cost drivers, especially when broker storage, overprovisioning, or data movement dominate the bill. The final result depends on object storage cost, request volume, cache design, network paths, and operational overhead. A good evaluation uses workload-specific assumptions rather than generic savings claims.
What workloads are good early candidates?
Good candidates usually have visible storage pressure, elastic throughput, or long retention, while also having manageable blast radius. Topics that are highly latency-sensitive, hard to replay, or tied to complex exactly-once workflows should wait until the team has stronger evidence from lower-risk workloads.
How should teams compare KIP-1150 with Kafka-compatible systems such as AutoMQ?
Compare the production properties, not only the feature names. Check protocol compatibility, write durability, read performance, failure recovery, network topology, cost model, observability, and migration tooling. AutoMQ is relevant when the team wants a Kafka-compatible shared-storage architecture available as a product path rather than waiting for a specific upstream Kafka feature to mature.
What is the most common mistake in diskless Kafka evaluations?
The most common mistake is treating diskless storage as a storage-only decision. The broker data path, cloud network path, cache behavior, metadata operations, and rollback model all change the production risk profile. The checklist should follow the record from producer acknowledgement through consumer replay, including the failure cases in between.
