Searches for "Kafka on S3" usually start with a practical hope: keep Kafka's API and ecosystem, but stop treating local broker disks as the permanent center of the architecture. The phrase is broad enough to describe several different patterns, from exporting topic data into S3 for analytics to running a Kafka-compatible system whose durable stream state is backed by object storage. Those patterns solve different problems. Confusing them is how teams end up with a pipeline that archives data well but still leaves the production streaming cluster with the same broker recovery, rebalancing, and cloud network cost profile.
The important distinction is whether S3 is a copy target or part of the commit path. A connector that writes records to S3 can be excellent for data lake ingestion, compliance retention, and replay into batch systems, but the Kafka cluster still depends on broker-local logs for serving producers and consumers. Tiered storage changes the retention model by offloading older log segments to remote storage while keeping the hot path on brokers. A shared-storage streaming architecture goes further: brokers become much less stateful because durable stream data is placed outside the broker lifecycle.
That difference matters because platform teams are not only asking where bytes sit. They are asking what happens during a broker failure, how fast capacity can change, which charges appear in the cloud bill, whether existing clients continue to work, and what migration rollback looks like. A useful evaluation starts with those operating questions rather than with a storage brand name.
Why "Kafka on S3" Is an Architecture Question
Apache Kafka was designed around an append-only log stored on brokers, with replication across brokers for durability and availability. That model has served the industry well because it gives Kafka clear ordering, retention, and consumer offset semantics. The cost of the model is that persistent state is tied to broker placement. When a broker fails, scales down, or needs disk changes, the system has to reason about replicas, leaders, partition movement, and the network impact of moving data between machines.
Cloud object storage changes one part of that equation but not all of it. Amazon S3 Standard is designed for very high durability, and AWS documents its availability, request, storage, and data transfer pricing as separate dimensions. That does not automatically make a streaming system production-ready on S3. Object storage has different latency, request, and consistency characteristics from local disks. A streaming engine has to decide how it commits writes, caches hot reads, handles metadata, and prevents object-store interactions from leaking into producer and consumer latency.
This is why "Kafka on S3" should not be reduced to "Can records be written to S3?" The stronger question is: which part of Kafka's persistence model is being replaced, and which part is still bound to broker-local state?
| Pattern | What S3 does | What remains on brokers | Main use case |
|---|---|---|---|
| S3 connector export | Receives copied records | Durable Kafka log and replication | Analytics, lake ingestion, archive |
| Kafka tiered storage | Stores remote log segments | Hot log, leaders, replicas, active serving path | Longer retention with lower local disk pressure |
| Shared-storage engine | Stores durable stream data | Compute, protocol handling, cache, coordination | Elastic Kafka-compatible streaming on cloud primitives |
The table is not a ranking. Each pattern can be the right answer when the problem is scoped correctly. The mistake is buying an export path when the real pain is broker recovery, or adopting tiered storage when the target is independent compute and storage scaling.
The Production Questions Export Does Not Answer
Connector-based export is intentionally downstream of Kafka. It reads records after Kafka has accepted them and writes those records into S3 using files, partitions, and formats that downstream systems can consume. That is useful, but it does not change the commit, replication, or failure behavior of the Kafka cluster itself. If the platform team is trying to reduce operational risk in the streaming layer, the connector is solving a neighboring problem.
The same boundary appears in cost analysis. Export can reduce custom ETL work and make historical data easier to query, but it does not remove broker disk, inter-broker replication, or cross-AZ traffic caused by the active Kafka deployment. In a multi-AZ cluster, producers, leaders, followers, and consumers may still create traffic patterns that are charged separately by the cloud provider. AWS pricing pages make it clear that storage, requests, and data transfer are distinct billing categories, so the right cost model has to include all three.
Four questions separate a storage export project from a persistent stream storage redesign:
- What is the source of truth? If Kafka is still the durable source and S3 is a copy, broker-local persistence remains the production dependency.
- What is the recovery unit? If a broker failure still triggers replica catch-up or partition movement, object storage has not removed state from the broker lifecycle.
- What is the write path? If producer acknowledgments depend only on local broker logs, S3 is not part of the streaming commit path.
- What is the network boundary? If replicas and clients still cross availability zones in the same way, the cloud bill may not improve where the team expects it to.
These questions are uncomfortable because they force teams to separate data placement from system behavior. A file in S3 is not the same thing as a stream whose durability, recovery, and serving model are built around shared storage.
A Practical Evaluation Framework
The first evaluation layer is compatibility. Kafka compatibility is not a vague promise that clients can connect. Platform owners need to test the specific protocol versions, security model, ACL behavior, idempotent producers, transactions if used, consumer group behavior, Kafka Connect, stream processing frameworks, schema tooling, and operational scripts that exist in their estate. A system can be Kafka-compatible for common workloads and still require careful validation for edge cases such as transactional processing or custom admin automation.
The second layer is latency. Object storage is not local NVMe, and pretending otherwise weakens the design review. A serious architecture explains its write-ahead log path, batching, cache design, and failure handling. It should also separate average latency from tail latency. In production, the p99 behavior during object-store throttling, cache misses, broker replacement, and consumer rewind is usually more important than a happy-path benchmark.
The third layer is cost. Kafka cost is not only storage capacity. It includes compute, block storage, object storage, API requests, inter-AZ data transfer, cross-region replication if used, observability, support, and the engineering time spent on rebalancing and recovery. A platform team should model at least three workload shapes: steady ingest with short retention, bursty ingest with long retention, and high fan-out reads. These shapes stress different parts of the system, so a single cost number is usually misleading.
The fourth layer is migration risk. A clean migration plan defines how topics move, how producers and consumers cut over, how offsets are handled, how rollback works, and which workloads are excluded from the first phase. For Kafka-compatible alternatives, the strongest migration story is often boring in a useful way: keep client protocols stable, avoid application rewrites, and prove behavior under mirrored traffic before changing the production endpoint.
The fifth layer is governance and operations. Object storage introduces IAM policies, bucket configuration, encryption, lifecycle rules, access logs, and cloud-region dependencies. Those are strengths when they are designed into the platform; they become risks when they are treated as implementation details. The operations team should know who can access stored stream data, how deletion and retention are enforced, how metadata is protected, and how incidents are diagnosed across brokers, storage, and network paths.
Where Shared Storage Changes the Trade-Off
Shared storage is compelling because it changes the unit of failure. In a broker-local design, the broker is both compute and a container for durable state. In a shared-storage design, the broker can focus on protocol handling, caching, coordination, and serving traffic while durable stream data lives in a storage layer built for persistence. That does not make the system magically simpler; it moves complexity into the storage engine, WAL, cache, and metadata design. The trade is worthwhile only if those pieces are engineered and observable.
The biggest operational effect is elasticity. When brokers are less stateful, adding or replacing broker capacity does not require the same volume of data movement. That is especially relevant for cloud deployments where workloads are spiky and infrastructure is expected to scale through APIs. In a traditional Kafka deployment, the cluster may have enough CPU but not enough disk, or enough disk but the wrong partition placement. Shared storage aims to reduce those coupled constraints.
The second effect is recovery behavior. If a broker disappears and the durable stream is not trapped on that broker's disk, replacement can focus on metadata, cache warming, and ownership transfer rather than restoring local log replicas. Teams should still test recovery under load, but the architectural goal is clear: broker failure should not become a large data relocation event.
The third effect is cost visibility. Once storage and compute scale independently, FinOps teams can reason about them separately. This is not the same as claiming every workload becomes lower cost. Object storage request rates, read fan-out, cache hit ratio, and network placement still matter. The difference is that the cost model becomes closer to the actual workload shape instead of being dominated by over-provisioned broker disks.
How AutoMQ Fits the Evaluation
After those criteria are clear, AutoMQ belongs in the discussion as a Kafka-compatible cloud-native streaming platform built on object storage. Its public documentation describes AutoMQ as fully compatible with Apache Kafka and built on S3, with a storage-compute separated architecture. That means the relevant evaluation is not "Does AutoMQ export Kafka data to S3?" but "Can a Kafka-compatible shared-storage architecture meet this workload's latency, compatibility, recovery, and governance requirements?"
For teams evaluating multi-AZ cost, AutoMQ's documentation also describes an S3-based multi-AZ architecture designed to avoid server-side replica replication traffic and producer cross-AZ write traffic under specific deployment conditions. That claim should be tested against the team's own availability-zone layout, producer and consumer placement, and workload balance. It is a strong example of why the network boundary belongs in the architecture review instead of being left to the end of procurement.
The practical way to evaluate AutoMQ, or any Kafka-compatible shared-storage system, is to run a workload proof rather than a slide comparison. Start with a small set of representative topics, include both hot reads and historical reads, keep existing clients and ACL patterns, and record broker replacement behavior under load. Then compare the result against the current Kafka baseline across latency, recovery time, operational steps, and cloud bill categories.
Buyer Checklist for Persistent Stream Storage
A serious architecture review should produce artifacts that operations, security, and finance can all read. The checklist does not need to be long, but it needs to be concrete enough to prevent teams from debating abstractions.
- Compatibility evidence: client libraries, protocol features, security settings, admin tools, Connect jobs, and stream processors tested against the target platform.
- Latency evidence: p50, p95, and p99 behavior for produce, consume, catch-up reads, broker restart, and storage-layer stress.
- Cost evidence: compute, storage, request, and network charges modeled separately, with assumptions visible.
- Recovery evidence: broker loss, AZ impairment, metadata backup, object-store access failure, and rollback procedure tested.
- Governance evidence: encryption, IAM, audit logging, retention, deletion, and operational ownership documented.
That checklist turns "Kafka on S3" from a search phrase into an engineering decision. It also keeps the vendor conversation respectful and factual. Managed Kafka services, self-managed Kafka with tiered storage, Kafka-compatible engines, and connector-based lake pipelines all have valid roles. The right choice depends on which production constraint is actually binding.
The useful next step is to test the architecture boundary, not the slogan. If your team is evaluating whether a Kafka-compatible shared-storage design can reduce broker state and cloud network exposure, review the AutoMQ architecture overview and run the same checklist against your own workload: Explore AutoMQ's Kafka-compatible architecture.
References
- Apache Kafka documentation: Tiered Storage: https://kafka.apache.org/40/operations/tiered-storage/
- Apache Kafka KIP-405: Kafka Tiered Storage: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage
- AWS S3 pricing: https://aws.amazon.com/s3/pricing/
- AWS EC2 On-Demand pricing and data transfer notes: https://aws.amazon.com/ec2/pricing/on-demand/
- AWS S3 User Guide: https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html
- AutoMQ documentation: What Is AutoMQ: https://docs.automq.com/automq/what-is-automq/overview?utm_source=blog&utm_medium=reference&utm_campaign=gs100-0047
- AutoMQ documentation: Save Cross-AZ Traffic Costs with AutoMQ: https://docs.automq.com/automq-cloud/best-practice/save-cross-az-traffic-costs-with-automq?utm_source=blog&utm_medium=reference&utm_campaign=gs100-0047
FAQ
Is Kafka on S3 the same as exporting Kafka data to S3?
No. Exporting Kafka data to S3 creates a downstream copy for analytics, retention, or lake ingestion. Kafka on S3 can also refer to architectures where object storage participates in the streaming storage layer itself. The production consequences are very different.
Does tiered storage make Kafka diskless?
Not by itself. Kafka tiered storage offloads older log segments to remote storage, but the active hot path and broker-local state still matter. It is useful for retention and disk pressure, but it is not the same as a shared-storage architecture with stateless brokers.
What should teams test before adopting a Kafka-compatible shared-storage platform?
Test compatibility, write latency, tail latency, consumer catch-up, broker replacement, IAM and encryption, observability, and cloud network cost. Use representative topics and clients rather than synthetic produce-only tests.
Where does AutoMQ fit?
AutoMQ fits the shared-storage category: a Kafka-compatible streaming platform built on object storage with a storage-compute separated architecture. It should be evaluated with the same workload-specific checklist used for any production streaming platform.
