Searches for kip 1150 architecture kafka usually come from teams that know Kafka well enough to be worried. They are not asking whether Kafka can process streams. They are asking whether the storage model behind a production Kafka estate still makes sense when brokers run across cloud availability zones, object storage is mature, and platform teams are being asked to cut cost and operational drag. KIP-1150 matters because it makes that question explicit inside the Apache Kafka community: should Kafka add a diskless topic model where broker-local disks stop being the primary durable home for user data?
The answer for buyers is more nuanced than a feature checklist. KIP-1150 is accepted as a direction and set of end-user requirements, while implementation details are expected to continue through follow-up KIPs. An accepted KIP is a strong community signal, but it is not the same thing as a production-ready upstream capability in the Kafka version you run today. A buyer still has to decide which problem they need to solve now and which compatibility promises are non-negotiable.
The strongest way to use KIP-1150 in a buying process is to treat it as a lens rather than a product requirement. It clarifies the pressure points: active-segment replication cost, cross-zone data movement, broker replacement, scaling, and the need to choose latency and cost trade-offs per topic. Once those pressures are visible, the decision becomes practical. Keep local-disk Kafka where it is the right fit, use tiered storage where retention cost is the main issue, and evaluate shared-storage or diskless Kafka-compatible architectures where broker-local data ownership is the operational constraint.
Why KIP-1150 Changes the Kafka Architecture Conversation
Traditional Kafka is a shared-nothing system. Each broker owns local log replicas, leaders serve writes and reads, followers replicate data, and partition reassignment moves data between broker disks. That design has a clear virtue: it is understandable, battle-tested, and deeply aligned with Kafka's original operating assumptions. When storage and compute live on the same machines, replication is the mechanism that gives the system durability and availability.
The cloud changes the cost surface underneath that design. Block storage, cross-AZ network transfer, and retained data are billed as separate infrastructure resources. A three-replica Kafka topic deployed across zones is no longer only a durability setting; it is also a recurring storage and network pattern. Tiered storage helps by moving completed segments into remote storage, but the active write path and recent local log still depend on broker-local storage and replication.
KIP-1150 points at the part tiered storage does not fully remove. Its motivation section states that tiered storage uses object storage for inactive segments, while active segments still need replication and durable storage. It also frames diskless topics as a way to make broker disks less central by writing through to object storage and using local disk mainly for caching or short-lived staging. That is the architectural shift buyers should focus on: the durable center of gravity moves away from the broker.
| Architecture option | Durable data ownership | Best buyer question |
|---|---|---|
| Local-disk Kafka | Broker-local replicas own active and retained log data | Is operational familiarity more important than cloud elasticity? |
| Kafka tiered storage | Broker-local storage owns active data; remote storage holds completed segments | Is long retention the main cost driver? |
| Diskless or shared storage | Durable stream data is externalized from broker-local disks | Is data movement during scaling, replacement, or recovery the bottleneck? |
This table is deliberately not a ranking. Local-disk Kafka remains a good fit for some low-latency, stable-capacity, self-managed environments. Tiered storage is a serious answer when retention dominates cost. Diskless and shared-storage designs become compelling when the expensive part of operating Kafka is no longer keeping bytes for a long time, but constantly tying those bytes to specific brokers.
The Storage Constraint Behind Cloud Kafka
Most Kafka cost reviews begin with instance size, disk size, and retention period. Those are visible numbers, so teams optimize them first. The less visible number is the cost of keeping the storage model stable while the workload changes. When a broker is replaced, when partitions are rebalanced, when a workload grows faster than expected, or when a team wants to shift traffic across zones, broker-local ownership turns infrastructure change into data movement.
That data movement has several forms. Replica fetch traffic crosses brokers, cross-zone deployments may create charges, reassignment pushes retained segments to target brokers, and recovery requires catch-up before the cluster is balanced again. These are consequences of making broker-local replicas the unit of durability.
Cloud buyers should separate four questions that are often blended together:
- What is the latency target for active writes and tail reads? A design that optimizes storage cost but harms the hot path may be unacceptable for transaction, fraud, ad-tech, or operational telemetry workloads.
- What is the retention and backfill pattern? A workload with frequent historical reads behaves differently from a workload where old data exists mostly for compliance or replay.
- What is the scaling pattern? Stable clusters can tolerate more manual storage planning than elastic platforms that add and remove capacity frequently.
- What is the governance boundary? BYOC, private deployment, data residency, encryption control, and network isolation can matter as much as raw cost.
The architecture review gets cleaner once those questions are separated. A stable cluster may need retention economics. A platform team serving many tenants may care more about provisioning, quota isolation, and failure recovery. A FinOps team may care about cross-zone transfer and duplicated block storage. A CTO may care whether adopting a diskless direction creates future migration risk or reduces it.
Local Disk, Tiered Storage, and Shared Storage
The phrase "diskless Kafka" can mislead buyers if it is taken literally. KIP-1150 itself clarifies that diskless does not mean there are no disks anywhere. It means broker disks are no longer the primary durable storage for user data. The distinction is important because production systems still need caches, metadata, write-ahead protection, temporary staging, and recovery logic. Removing durable ownership from broker disks does not remove storage engineering; it moves the hardest storage questions into a different layer.
Tiered storage sits in the middle of this discussion. Apache Kafka documentation describes tiered storage as a two-tier model: local storage on Kafka brokers and remote storage for completed log segments. That is useful because older data can move to lower-cost storage, and backfill or recovery reads can come from the remote tier. It does not fully convert brokers into stateless compute because local storage still participates in the active log path.
Shared-storage Kafka-compatible systems take a more aggressive position. Durable stream data lives in object storage or another shared durable layer, while brokers focus on Kafka protocol handling, leadership, caching, and coordination. The architecture still needs a low-latency write path, metadata fencing, object layout, recovery rules, and observability. The payoff is that broker replacement and partition movement can become less dominated by copying retained data between disks.
This is where buyers should be careful with optimistic claims from any provider. The architecture category is promising, but implementation details decide whether it is usable. Ask how the system acknowledges writes, what happens when object storage is slow, how read-after-write behavior is protected, how transactions and consumer group semantics are handled, and how the system behaves during a zone or storage-service incident. A storage model that looks elegant in a diagram still has to survive the boring parts of production.
Evaluation Checklist for Platform Teams
KIP-1150 is useful because it gives buyers a shared vocabulary for evaluation. Instead of asking whether a platform is "diskless," ask whether it changes the specific operational model that hurts your team. The checklist below is intentionally written for architecture review meetings, not marketing comparisons.
| Review area | What to validate | Why it matters |
|---|---|---|
| Kafka compatibility | Producer, consumer, admin, Kafka Connect, Streams, transactions, idempotence, offsets, and security behavior | Application rewrites can erase infrastructure savings |
| Write path durability | WAL design, acknowledgment point, recovery after broker loss, and object-storage upload behavior | Diskless designs still need a trusted path for acknowledged writes |
| Cost model | Block storage, object storage, requests, cross-AZ traffic, cache hit rate, and retention | Moving data to object storage changes the bill, but does not make it disappear |
| Elastic operations | Broker replacement, partition reassignment, scaling time, and traffic rebalancing | The main benefit should show up during change, not only steady state |
| Governance | VPC boundary, IAM, encryption, audit logs, data residency, and private connectivity | Enterprise buyers often reject architectures that solve cost but weaken control |
| Migration and rollback | Topic migration path, dual-write or linking strategy, consumer offset handling, and cutover testing | A good target architecture still needs a low-drama path to adoption |
The table should lead to tests, not only discussion. Run a proof of concept with the topics that make your current Kafka estate uncomfortable: high-retention topics, high-ingress topics, bursty topics, and topics with strict consumer-lag requirements. Measure broker replacement, partition movement, consumer catch-up, and cost under realistic producer and consumer behavior. Do not let a synthetic throughput test become the whole decision.
The buyer question is not "Should we wait for upstream Kafka to become diskless?" The better question is "Which storage ownership model matches the operational pressure we have now, and how do we preserve Kafka semantics while changing it?"
That framing avoids two common mistakes. The first is freezing every Kafka architecture decision until upstream implementation details settle. The second is treating every diskless or shared-storage product as interchangeable. KIP-1150 validates the problem space, but production readiness still belongs to the implementation you deploy.
How AutoMQ Fits After the Neutral Review
After the evaluation framework is clear, AutoMQ becomes relevant as a Kafka-compatible shared-storage architecture rather than as a generic managed Kafka replacement. AutoMQ keeps the Kafka protocol surface while replacing Kafka's broker-local log storage with S3Stream, backed by object storage and a WAL layer. Brokers become stateless in the sense that durable stream data is not permanently attached to broker-local disks; they still handle Kafka-facing compute, cache, leadership, and coordination.
That design is aligned with the same pressure KIP-1150 highlights, but it is not dependent on waiting for diskless topics to land in upstream Kafka. For buyers, the practical question is whether a Kafka-compatible shared-storage implementation can preserve the operational contract their applications expect while changing the storage economics and recovery behavior underneath. AutoMQ's architecture documentation describes object storage as the primary data repository and WAL storage as the layer that protects the write path before data is stored into object storage.
The important operational effect is not a slogan like "no disks." It is a different failure and scaling shape. If a broker fails, durable data is not stranded on that broker's local disk. If capacity changes, the system can focus more on serving ownership, metadata, cache warming, and traffic balance instead of moving large retained logs. If a workload has heavy cross-zone replication cost, a design that avoids broker-to-broker replica traffic can change the FinOps conversation.
AutoMQ also has buyer relevance around deployment boundary. Some teams want Kafka compatibility but do not want data to leave their cloud account or VPC. A customer-controlled deployment model, combined with Kafka-compatible clients and shared object storage, lets the architecture review include security and governance alongside cost and elasticity. That matters because a strong storage design on paper will fail procurement if it moves data control to the wrong place.
The conservative adoption path is to start with a workload class, not the whole Kafka estate. Pick topics where local-disk ownership is painful: large retention, frequent scaling, expensive cross-zone traffic, or slow recovery. Validate client compatibility, consumer lag, failover, observability, and rollback. If the architecture passes, expand by workload pattern rather than by organization chart.
Migration and Risk Review
Migration planning is where architecture diagrams meet organizational reality. A Kafka-compatible endpoint helps, but compatibility is broader than wire protocol. Teams need to validate client versions, serializers, schema registry behavior, ACLs, quotas, transactions, consumer offsets, Connect jobs, Streams state stores, linking strategy, and dashboards.
The lowest-risk plan usually has three stages. First, mirror or dual-write a non-critical workload and compare behavior under normal traffic. Second, run failure drills: broker loss, consumer catch-up, object-storage latency, network interruption, quota pressure, and rollback. Third, migrate a workload whose pain is visible enough to justify the architecture change.
Buyers should also ask what happens if the experiment fails. Can topics move back? Can consumers resume from known offsets? Are alerts understandable to the SRE team? A good diskless or shared-storage system should reduce day-two operations, but the first migration still needs discipline.
Use the checklist with one painful workload class, then test failure and scaling before critical traffic. If broker-local storage constrains your Kafka estate, explore AutoMQ through the AutoMQ Cloud Console and validate shared storage in a controlled environment.
References
- Apache Kafka KIP-1150: Diskless Topics: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A+Diskless+Topics
- Apache Kafka KIP-405: Kafka Tiered Storage: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage
- Apache Kafka Tiered Storage documentation: https://kafka.apache.org/39/operations/tiered-storage/
- Apache Kafka KRaft documentation: https://kafka.apache.org/documentation/#kraft
- Apache Kafka message delivery semantics: https://kafka.apache.org/documentation/#semantics
- AWS EC2 data transfer pricing: https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer
- AWS S3 pricing: https://aws.amazon.com/s3/pricing/
- AutoMQ architecture overview: https://docs.automq.com/automq/architecture/overview?utm_source=blog&utm_medium=reference&utm_campaign=aivk-0004
- AutoMQ cross-AZ traffic cost guidance: https://docs.automq.com/automq-cloud/best-practice/save-cross-az-traffic-costs-with-automq?utm_source=blog&utm_medium=reference&utm_campaign=aivk-0004
- AutoMQ GitHub repository: https://github.com/AutoMQ/automq
FAQ
Is KIP-1150 already available as a complete Apache Kafka feature?
No. KIP-1150 is marked accepted, but the KIP says acceptance establishes the need and end-user requirements rather than final implementation details. Buyers should check the exact Kafka release, follow-up KIPs, and implementation they plan to use before making production assumptions.
Is diskless Kafka the same as tiered storage?
No. Tiered storage moves completed log segments to remote storage while local broker storage remains part of the active log path. Diskless or shared-storage designs aim to make broker-local disks stop being the primary durable storage for user data.
Does diskless mean there are no disks?
No. Diskless mainly means broker disks are no longer the durable home for user records. Production systems may still use disks or memory for cache, metadata, staging, WAL-related paths, and operational needs.
When should buyers evaluate AutoMQ?
Evaluate AutoMQ when Kafka compatibility is important but broker-local storage is creating cost, scaling, recovery, or cross-AZ traffic problems. The proof of concept should use real topic patterns and validate write durability, consumer behavior, failover, governance, and rollback.
What is the safest next step?
Start with one painful workload class, then test the architecture under failure and scaling conditions before touching business-critical traffic. The proof should include real client behavior, realistic retention, failure drills, and rollback criteria.
