Teams rarely search for diskless kafka because they dislike disks in the abstract. They search for it after Kafka storage has started to control decisions that should belong to the platform: how fast a broker can be replaced, how much retention a cluster can afford, how much data crosses Availability Zone boundaries, and how long a capacity change takes to finish. The word "diskless" is shorthand for a deeper question: can Kafka-compatible streaming keep the operational contract of Kafka while moving durable data away from broker-local storage?
That question is a modernization problem, not a naming debate. A shared-storage architecture may reduce the amount of durable data tied to each broker, but it also changes where write durability, cache behavior, recovery, and governance must be validated. The right evaluation starts with production evidence. If a platform cannot prove the same application-facing semantics under failure, the fact that it uses object storage is not enough.
Why Shared Storage Enters the Kafka Roadmap
Traditional Kafka binds three responsibilities tightly together: brokers accept requests, brokers coordinate partition leadership, and brokers store replicated log data on attached disks. That coupling is operationally clear, which is one reason Kafka has been so successful. It also means that compute scaling, storage scaling, replica placement, and failure recovery are entangled. When retention grows faster than CPU usage, the cluster still needs broker storage headroom. When a broker is replaced, the system has to reason about data ownership as well as compute replacement.
Cloud infrastructure makes this coupling more visible. Compute, block storage, object storage, private connectivity, and inter-zone traffic are priced and scaled separately. A design that was reasonable in a data-center environment can become expensive or slow to operate when every replicated byte and every recovery path has a billable route. Tiered Storage helps by moving older segments to remote storage, but it does not automatically make active brokers stateless. Shared-storage or diskless designs go further by making remote durable storage a central part of the write and recovery model.
The modernization opportunity is real, but it needs guardrails. A platform team should not ask only whether a vendor or project supports "Kafka on object storage." It should ask what remains Kafka-compatible, what becomes shared infrastructure, what moves onto the cloud bill, and what new failure modes appear. Those questions turn a storage trend into a validation plan.
Step 1: Define the Kafka Contract You Cannot Break
The first validation step is semantic, not architectural. Applications do not care whether a record sits on local NVMe, EBS, S3, or another storage service. They care whether producers receive acknowledgments at the right time, consumers can fetch by offset, group coordination behaves predictably, security policies apply, and operational tools still work. If those behaviors change, the migration is no longer a storage modernization. It is an application compatibility project.
Define the contract at three levels. At the client level, test producer idempotence, consumer groups, offset commits, transactions if used, authentication, authorization, and admin APIs. At the topic level, test retention, compaction, partition expansion, leader movement, and replay. At the platform level, test upgrades, metrics, audit trails, quota behavior, and incident runbooks. The goal is not to test every feature in the abstract; it is to prove the behaviors your estate actually depends on.
This is where Apache Kafka's own direction matters. KIP-1150, Diskless Topics, signals that the Kafka community sees value in separating topic storage from broker-local disks. It should not be treated as a shortcut around validation. A KIP describes an accepted direction for Kafka; a production modernization still depends on the implementation you deploy, the feature set you need, and the failure cases you test.
Step 2: Draw the Durable Write Path
Shared storage changes the most important line in the architecture: the point where a write becomes durable. In classic Kafka, platform teams can reason about leader append, follower replication, in-sync replicas, and acknowledgment settings. In a diskless or shared-storage design, the proof has to include the broker, WAL or staging layer, object storage commit, metadata visibility, and cache. A whiteboard diagram is useful only if it answers one question: what must complete before the producer sees success?
The validation trace should include both the normal path and the ugly path. Kill the active broker after acknowledgments. Delay object storage operations. Force a cold cache read after leader movement. Interrupt an upload or storage commit if the implementation exposes that state. Then check acknowledged records, offsets, consumer visibility, duplicate behavior, and alerting. The strongest designs make these transitions understandable enough for SREs to rehearse, not only for architects to admire.
A practical write-path review usually asks:
- Where is the first durable copy of an acknowledged record?
- Can another broker recover ownership without replaying a large local replica?
- How are metadata and data commits ordered so offsets do not point at unreadable data?
- What happens when object storage is slow, throttled, or temporarily unavailable?
- Which local disks remain in use for WAL, cache, logs, or operating-system state?
Those questions are intentionally concrete. "Uses object storage" is not an answer. The answer has to describe the failure boundary and the backpressure behavior.
Step 3: Separate Tiered Storage From Diskless Operation
Tiered Storage and diskless Kafka are often discussed together because both use remote storage. They solve different problems. Tiered Storage is mainly a retention and cost optimization for older log segments. Active writes can still depend on broker-local storage and broker-to-broker replication. Diskless or shared-storage operation changes the active durability path, which means it affects recovery, scaling, and failure handling more directly.
This distinction matters during modernization planning because the acceptance criteria differ. A Tiered Storage rollout should validate remote reads, segment lifecycle, retention, fetch latency for older data, and catch-up consumers. A shared-storage rollout must also validate write acknowledgment, WAL durability, ownership transfer, cache coherency, and storage-service dependency. Putting both under one "remote storage" checklist hides the risk that actually changes.
For many estates, the answer is not one architecture for every topic. Audit logs with long retention may benefit from shared storage earlier than compacted state topics. Analytics fan-out may need read-cache and request-cost modeling before migration. Transactional pipelines deserve explicit tests for idempotence, fencing, and rollback. Modernization works better when topics move by contract, not by cluster size.
Step 4: Model the Cloud Bill by Byte Path
Cost validation should start with byte movement, not unit prices. A Kafka platform spends money on broker compute, attached storage, remote storage, request operations, data transfer, private connectivity, monitoring, and engineering time. Shared storage can reduce some of those costs by removing large local replicas from broker lifecycle and reducing application-layer replication traffic. It can also introduce object storage requests, cache misses, endpoint routing choices, and governance work.
Build a byte-path model before building a savings claim:
| Cost area | What to measure before migration | What changes in shared storage |
|---|---|---|
| Write durability | Producer ingress, replica traffic, and cross-AZ routes | WAL and shared-storage writes replace some local replica movement |
| Retention | Hot data, retained history, and replica multiplier | Durable history can live in object storage with different lifecycle controls |
| Read fan-out | Consumer placement, catch-up reads, and replay frequency | Cache hit rate and remote fetch behavior become first-class cost drivers |
| Recovery | Replica rebuild time and data copied during replacement | Broker replacement can be lighter when durable data is outside the broker |
| Operations | Reassignment windows, disk expansion, and balancing work | Scaling shifts toward compute, cache, and storage-policy management |
AWS documentation is useful here because it separates data transfer, storage, and service-specific pricing. Amazon MSK documentation is also useful for understanding where managed Kafka features such as Tiered Storage apply. The exact numbers will depend on region, topology, and workload, so the public pricing page is not a substitute for your own bill simulation. It is the source of the variables your simulation must include.
Step 5: Run a Migration Drill Before a Migration Plan
The most common modernization mistake is treating validation as a procurement exercise and migration as a later implementation detail. Shared storage changes enough operational assumptions that the migration drill should happen before final architecture approval. The drill does not need every topic. It needs one representative topic class, real clients, realistic retention, the intended security model, and a rehearsed rollback.
The drill should cover a full operating cycle. Mirror or dual-write where appropriate. Move one consumer group intentionally, then move it back. Recreate a broker or container. Induce storage-path errors in a controlled environment. Compare lag, throughput, error rates, and offset behavior. Check whether observability tells the story quickly enough for an on-call engineer who did not design the platform.
The output should be a decision record, not a demo note. It should say which topic class passed, which behaviors were conditional, which runbooks changed, and which owners accepted the new responsibilities. That record becomes the modernization contract between platform engineering, application owners, security, and FinOps.
Step 6: Score Readiness Across Owners
A useful readiness scorecard is short, but it cannot be vague. Each category should have evidence and an owner. Platform engineering owns Kafka compatibility and performance tests. SRE owns failure drills and alerting. Security owns object storage permissions, encryption, retention, and audit boundaries. FinOps owns the cost model and topology assumptions. Application owners own consumer behavior, rollback, and acceptable migration windows.
Use a simple rating for each category: pass, conditional pass, or fail. A conditional pass is not a political compromise. It names missing evidence. For example, append-heavy ingestion may pass after acknowledgment and broker-loss tests, while compacted topics remain conditional until tombstones and restore behavior are proven. Long-retention audit topics may pass cost validation but stay conditional on deletion policy and audit review.
This shared scorecard prevents a subtle failure mode: the storage platform looks ready to one team and risky to another because each team evaluated a different system. The SRE sees recovery. FinOps sees traffic. Security sees object permissions. Application teams see client semantics. A modernization plan is ready only when those views describe the same architecture.
How AutoMQ Fits the Evaluation
After the validation framework is in place, AutoMQ is a concrete Kafka-compatible shared-storage implementation to test against it. AutoMQ uses an S3Stream Shared Storage architecture that moves durable stream data to S3-compatible object storage, while WAL storage and data caching support Kafka-like write and read behavior. Its Kafka compatibility documentation, BYOC and software deployment options, and inter-zone traffic guidance map to the scorecard categories above.
The right way to evaluate AutoMQ is the same way you would evaluate any serious modernization candidate: with representative clients, real topic settings, failure drills, byte-path modeling, and governance review. The product fit is strongest when broker-local storage, cross-AZ traffic, slow recovery, or retention growth are already shaping your Kafka roadmap. In that situation, AutoMQ gives platform teams a way to test whether Kafka-compatible semantics can be preserved while the durable storage boundary moves out of the broker.
The original diskless kafka search usually starts with curiosity about architecture. It should end with evidence. If your team is ready to validate a shared-storage Kafka-compatible path with real workloads, start from the AutoMQ Cloud deployment path and bring one representative topic class, one failure drill, and one byte-path cost model to the proof of concept.
References
- Apache Kafka KIP-1150: Diskless Topics: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1150%3A%2BDiskless%2BTopics
- Apache Kafka documentation, Tiered Storage: https://kafka.apache.org/documentation/#tiered_storage
- Amazon MSK Developer Guide, Tiered Storage: https://docs.aws.amazon.com/msk/latest/developerguide/msk-tiered-storage.html
- AWS architecture blog, overview of data transfer costs: https://aws.amazon.com/blogs/architecture/overview-of-data-transfer-costs-for-common-architectures/
- AWS S3 Pricing: https://aws.amazon.com/s3/pricing/
- AutoMQ documentation, S3Stream Shared Streaming Storage: https://docs.automq.com/automq/architecture/s3stream-shared-streaming-storage/overview?utm_source=blog&utm_medium=reference&utm_campaign=gs100-0009
- AutoMQ documentation, Compatibility with Apache Kafka: https://docs.automq.com/automq/what-is-automq/compatibility-with-apache-kafka?utm_source=blog&utm_medium=reference&utm_campaign=gs100-0009
- AutoMQ documentation, Eliminate Inter-Zone Traffic: https://docs.automq.com/automq-cloud/eliminate-inter-zone-traffics/overview?utm_source=blog&utm_medium=reference&utm_campaign=gs100-0009
FAQ
Does diskless Kafka mean brokers have no disks at all?
No. Brokers still need disks for the operating system, logs, cache, metadata, or WAL depending on the implementation. The meaningful change is that broker-local disks stop being the primary durable home for user topic data.
Is shared-storage Kafka the same as Tiered Storage?
No. Tiered Storage usually moves older segments to remote storage while the active log can still depend on local broker storage. Shared-storage or diskless operation moves the active durability model toward shared storage, so write-path and recovery validation become more important.
What should a platform team validate first?
Start with Kafka compatibility and write durability. If producers, consumers, offsets, security, and recovery do not behave correctly, cost savings do not matter. After that, validate read paths, cloud network cost, governance, and migration rollback.
Which workloads are usually better first candidates?
Append-heavy ingestion, long-retention audit logs, and workloads with expensive broker storage pressure are often reasonable first candidates. Compacted topics, Kafka Streams changelogs, and transactional pipelines need deeper semantic tests before migration.
How should AutoMQ be included in a proof of concept?
Use the same scorecard for AutoMQ as for any shared-storage Kafka candidate. Test representative clients, real topic configurations, broker failure, storage-path behavior, inter-zone traffic assumptions, governance controls, and rollback before expanding the migration.
