Teams searching for KIP-1150 are rarely asking whether object storage is interesting. That question has already been answered by years of Kafka cost reviews, cloud network bills, and operational pain around broker-local disks. The harder question is whether a Kafka application that depends on ordering, transactions, compaction, consumer groups, quotas, ACLs, and operational tooling can move to diskless topics without turning a storage optimization into an application migration.
The Apache Kafka community has accepted KIP-1150: Diskless Topics, which records agreement on the direction: reduce the role of broker disks as the primary durable store for user data and use object storage as part of the durability model. That acceptance matters, but it is not the same thing as a finished production feature in every Kafka release. The follow-up design work, including KIP-1163: Diskless Core, carries the implementation details that determine what platform teams can safely adopt.
That distinction is where compatibility risk lives. A diskless topic can preserve the Kafka protocol at the client boundary while changing the internal path for writes, reads, offsets, batch metadata, and recovery. For platform owners, the right evaluation does not start with "does it use S3?" It starts with "which Kafka promises still hold for my application, and which operational assumptions have moved?"
What KIP-1150 changes and what it does not
KIP-1150 is explicit about the meaning of diskless: it does not mean there are literally no disks in the system. Broker disks may still be used for metadata, temporary buffering, or cache. The change is that broker-local disks are no longer the primary durable storage layer for user records. That is a meaningful architectural shift because traditional Kafka couples the broker to three duties at once: serving client requests, owning partition leadership, and persisting replicated log data on local block storage.
Diskless topics try to break that coupling. KIP-1150 frames diskless topics as a separate topic type that can operate alongside classic topics and use object storage for durable data. KIP-1163 goes further into the proposed mechanics: data can be stored in object storage, local disk can behave as cache, and replication can be delegated to the storage backend rather than performed through Kafka broker-to-broker replication. This is why the cost argument is so strong in cloud deployments. On AWS, data transferred across Availability Zones in the same Region is charged at \$0.01/GB in each direction for common EC2-related paths, while data transferred within the same Availability Zone is free according to the EC2 on-demand pricing page.
Cost pressure explains the interest, but compatibility determines adoption. A payment pipeline using idempotent producers and transactions has a different risk profile from a clickstream pipeline with append-only analytics events. A compacted changelog topic for stream processing has different requirements from a short-retention metrics topic. Diskless topics may eventually become broad infrastructure, but migration planning still has to classify topics by the semantics the application actually uses.
The compatibility surface is wider than the client protocol
Kafka compatibility is often reduced to client compatibility: can the existing producer and consumer libraries connect, produce records, fetch records, commit offsets, and handle metadata responses? That is necessary, but it is too narrow for diskless evaluation. Kafka applications depend on behavior that emerges from the interaction between the client protocol, broker internals, storage layout, metadata, and operational tooling.
The risk areas usually fall into five groups:
- Write-path semantics. Idempotence, transactions, ordering, acknowledgements, producer retries, and rack-aware routing need explicit testing. A protocol-compatible write path can still have different latency and failure timing.
- Read-path behavior. Consumer lag, catch-up reads, remote fetch behavior, cache hit rate, and fetch isolation affect tail latency and recovery after consumer downtime.
- Topic-level features. Compaction, retention, deletion, timestamp handling, tiered storage interaction, and topic-type conversion are not side details when applications depend on them.
- Operational tools. MirrorMaker, Kafka Connect, schema registries, stream processing frameworks, rebalancing tools, ACL workflows, and observability dashboards may assume classic topic behavior.
- Failure recovery. Object storage outage behavior, coordinator state, metadata reconciliation, downgrade paths, and disaster recovery all need runbooks before production traffic moves.
The trap is to treat these as edge cases. They are not edge cases for the teams that run Kafka as a platform. Kafka estates usually contain hundreds or thousands of topics with different owners and assumptions. The storage layer can change centrally, but application risk is distributed across every producer, consumer, connector, and operational workflow.
A topic-by-topic migration model
A practical evaluation starts by separating topics by risk instead of by owner team. The first bucket is usually append-heavy, replay-tolerant data: logs, metrics, clickstreams, CDC staging topics, and analytics buffers. These topics often care about throughput, retention cost, and replay economics more than single-digit millisecond latency. Diskless topics can be attractive here because the application contract is closer to durable append and fetch.
The second bucket is stateful application infrastructure: compacted topics, Kafka Streams state stores, transactional outboxes, exactly-once pipelines, and topics used as coordination or recovery logs. These workloads should move later because they depend on the finer parts of Kafka semantics. The issue is not that diskless architecture can never support them. The issue is that every implementation must prove those semantics under failures, upgrades, and degraded storage paths.
The third bucket is operational metadata and shared platform dependencies: internal topics, offsets, transaction state, schema and connector dependencies, and governance workflows. These should not be pulled into a storage migration casually. Even when a platform can host multiple topic types in the same cluster, internal and application-visible topics deserve separate policies.
| Topic class | Typical examples | Compatibility questions | Suggested migration posture |
|---|---|---|---|
| Append-heavy data | Logs, metrics, clickstream, analytics staging | Can the workload tolerate higher write or read tail latency? Are replay and retention behavior verified? | Good first candidate after benchmark testing |
| Stateful processing | Compacted changelogs, Kafka Streams state, transactional pipelines | Are transactions, compaction, and recovery semantics fully supported and tested? | Move after semantic validation |
| Platform internals | Offsets, transaction state, connector control topics, governance logs | Does the platform vendor recommend moving these? Is rollback defined? | Keep conservative until documented |
This classification changes the conversation with application teams. Instead of asking them whether they "support diskless Kafka," ask which Kafka features their topics use and what failure behavior they expect. Most teams can answer that with logs, configs, and integration tests. That turns a vague architecture decision into an inventory-driven migration plan.
Cost wins can hide latency and request-shape changes
Object storage is excellent at durable, large-scale storage, but it is not a drop-in local disk. The KIP-1163 design notes that remote storage may increase Kafka request latency and end-to-end data latency compared with local disks. That does not make diskless topics unsuitable. It means the evaluation has to model both cost and request shape.
The most important latency question is not average latency. It is the shape of tail latency when the cache is cold, when a consumer catches up from hours behind, when an object storage request is retried, or when metadata coordination is under load. Kafka platform owners already know this pattern from tiered storage: the happy path can look clean while the worst operational moments reveal whether the abstraction is solid.
Cost has the same subtlety. Removing broker-to-broker replication traffic can reduce a painful line item in cloud bills, especially in multi-AZ deployments. But object storage introduces its own request, retrieval, lifecycle, and data transfer model. The right comparison therefore includes:
- Cross-AZ replication and client traffic avoided by the diskless architecture.
- Object storage PUT, GET, list, lifecycle, and retrieval behavior under the expected topic and partition count.
- Broker cache sizing, miss rate, and cold-read frequency.
- Retention and compaction behavior, especially for topics with small batches or many partitions.
- Operational savings from faster scaling, smaller broker disks, and reduced data rebalancing.
The conclusion may vary by workload. A high-throughput append-only analytics topic can be an excellent diskless candidate. A low-latency trading workflow or a compacted state topic may require classic topics, a different diskless implementation, or a phased rollout. The point is to avoid collapsing every topic into a single architectural answer.
Production readiness scorecard
Once topic inventory is complete, the readiness review should move from "architecture preference" to evidence. A vendor page, KIP text, or benchmark is useful input, but the platform team needs a scorecard that can be repeated across workloads and regions.
Use the scorecard as a gate, not as a slide. For each candidate topic family, run a focused test that covers normal throughput, burst traffic, consumer catch-up, broker restart, storage throttling, metadata service disruption, credential rotation, and downgrade planning. The test does not need to mimic every application. It needs to represent the failure modes that would be expensive to discover after migration.
Compatibility should also be verified above Kafka clients. Kafka Connect tasks, Flink jobs, Debezium connectors, MirrorMaker flows, audit pipelines, and observability exporters may all work through standard Kafka APIs while still depending on timing, topic configs, or metadata assumptions. A storage architecture can be protocol-compatible and still require updates to dashboards, SLOs, alert thresholds, or capacity models.
Security and governance deserve the same treatment. Diskless topics add object storage permissions, encryption configuration, bucket policies, lifecycle rules, and sometimes additional coordinator or metadata components. That means the security review should include both Kafka-layer permissions and cloud-layer access boundaries. If the organization uses BYOC, the evaluation also needs to confirm where data resides, who can access the object store, and how operational control is separated from business data access.
Where AutoMQ fits this evaluation
After the evaluation framework is clear, AutoMQ belongs in the conversation as one implementation path for teams that want Kafka-compatible streaming with an object-storage-centered architecture. AutoMQ is a cloud-native Kafka-compatible system that keeps the Kafka API and ecosystem compatibility focus while redesigning the storage layer around shared storage, stateless brokers, and object-storage-backed durability. Its documentation describes compatibility with Apache Kafka clients and ecosystem components, including connectors, proxies, and monitoring integrations.
That design changes the migration question. Instead of enabling a separate topic type inside a classic Kafka cluster, AutoMQ treats shared storage as the foundation of the system. Brokers can be more stateless because retained data is not anchored to broker-local disks. That can reduce data movement during scaling and recovery, while object storage becomes the durable storage substrate. AutoMQ also emphasizes BYOC and software deployment models for teams that need business data to remain in their own cloud account.
AutoMQ should still be evaluated with the same scorecard. Kafka-compatible does not remove the need to test transactions, compaction, connector behavior, latency, recovery, and operational workflows for your workload. It does mean the architecture is already designed around the diskless direction that KIP-1150 formalizes in the Apache Kafka community. For teams comparing upstream diskless topics, managed Kafka services, and Kafka-compatible engines, that architectural maturity is a concrete dimension to test rather than a marketing label.
If your team is already building a compatibility matrix for KIP-1150, run the same matrix against AutoMQ and include cloud network costs, object storage behavior, recovery time, and tooling compatibility in the same worksheet. You can start from the AutoMQ documentation and use the Kafka compatibility notes as the first checkpoint before running a workload-specific proof of concept.
A decision rule for platform teams
Diskless topics are not mainly a storage feature. They are a change in where Kafka places durable truth, how brokers recover, how traffic crosses cloud topology, and how application semantics are preserved while the storage layer moves. That is why KIP-1150 is important and why it should not be adopted as a blanket checkbox.
A useful rule is simple: move the topics whose application contract is already close to durable append and replay first, and hold back the topics whose contract depends on the deepest Kafka semantics until those semantics are proven in your chosen implementation. This rule is conservative, but it keeps the cost optimization from outrunning the application contract.
The teams that get the most value from diskless Kafka will not be the teams that ask "can we turn it on?" They will be the teams that know which topics are safe, which topics need more evidence, and which topics should stay on a different path.
References
- Apache Kafka: KIP-1150: Diskless Topics
- Apache Kafka: KIP-1163: Diskless Core
- AWS EC2 pricing: Data Transfer within the same AWS Region
- AWS S3 documentation: What is Amazon S3?
- AutoMQ documentation: Compatibility with Apache Kafka
- AutoMQ documentation: Experience AutoMQ
FAQ
Is KIP-1150 already a production feature in Apache Kafka?
KIP-1150 is accepted as an Apache Kafka improvement proposal, but the KIP itself states that acceptance records agreement on the need and end-user requirements rather than all implementation details. Follow-up KIPs such as KIP-1163 define core behavior, APIs, and operational details. Platform teams should verify the release status of the specific Kafka distribution they plan to run.
Does diskless Kafka mean brokers use no disks at all?
No. In KIP-1150, diskless means broker disks are not the primary durable storage for user data. Brokers may still use disks for metadata, cache, or temporary buffering. The architectural change is about the source of durable truth, not the literal absence of every disk device.
Which Kafka topics should move first?
Append-heavy topics with replay-tolerant workloads are usually better first candidates than transactional, compacted, or state-store topics. Good examples include logs, metrics, clickstream events, and analytics staging topics, provided latency and recovery tests pass. Topics that carry application state or coordination semantics deserve stricter validation.
How should AutoMQ be compared with KIP-1150?
Compare them by workload evidence, not by labels. KIP-1150 is the upstream Apache Kafka direction for diskless topics. AutoMQ is a Kafka-compatible implementation built around shared storage and object-storage-backed durability. The useful comparison is whether each option satisfies your compatibility, latency, cost, recovery, governance, and operational requirements.
