Most Redpanda architecture evaluations arrive at the same cloud question: where does durable streaming data really live? The answer shapes broker replacement, scaling, recovery, retention, and how much cloud infrastructure work the platform team has to absorb. Redpanda and shared-storage Kafka can both serve Kafka-compatible workloads, but they make different bets about the boundary between compute and durable storage.
Redpanda is not Apache Kafka with a different packaging layer. Its documentation describes a Kafka API-compatible streaming platform with partitions implemented as Raft groups and data persisted on local disk before replication across Redpanda brokers. Shared-storage Kafka keeps the Kafka-facing contract, but moves durable log storage away from broker-local disks into shared cloud storage, usually with a WAL and cache layer to protect the hot path. The architectural divide is local ownership of state versus shared ownership of state.
The Storage Question Behind Kafka-Compatible Platforms
Kafka-compatible systems inherit a demanding contract. Producers expect ordered appends per Partition, consumers expect Offset-based replay, and operators expect Topic configuration, retention, replication behavior, and failure handling to remain legible under load. A platform can change its internal engine, but it cannot casually change these expectations without turning an infrastructure migration into an application migration.
That is why storage architecture matters so much. In a local-storage design, the Broker owns local log data, or at least owns the hot durable state needed to serve and recover a Partition. This keeps the write path close to the compute node, but it also makes capacity, replacement, and rebalancing tied to where bytes are placed. In a Shared Storage architecture, the Broker is closer to a compute role. Durable data is stored in a shared storage layer, and broker changes become less coupled to copying local log segments between machines.
Tiered Storage sits between those two worlds and is often the source of confusion. Redpanda Tiered Storage, like Kafka Tiered Storage in the broader ecosystem, can offload older log segments to object storage. That improves retention economics and makes historical data available from a remote tier. It does not automatically make brokers stateless, because the hot path and recent data still depend on local broker storage.
Redpanda Architecture At A High Level
Redpanda's architecture is built around a Kafka API-compatible surface and a storage engine that is not the Apache Kafka broker implementation. Redpanda documentation describes Topics as collections of Partitions, each Partition as a Raft group, and replication as a consensus process among replicas. Producers write to the Partition leader, and data is appended to local disk. Followers replicate from the leader, and leadership can move as cluster state changes.
This design gives Redpanda a coherent internal model. Raft is used for replication and consistency, and the local disk remains a central part of the log path. For operators, the key point is that durable state is associated with brokers and replicas. When capacity changes, the cluster still has to reason about where Partition replicas live, how much local disk is available, and how leadership and replicas are balanced across nodes.
Redpanda Tiered Storage changes retention behavior by offloading log segments to object storage. The official Tiered Storage documentation says it offloads log segments to object storage, lets operators specify how much data remains in local storage, and uses remote read to fetch data from object storage when needed. That is a useful feature for long retention and catch-up reads. It is not the same as a stateless broker model, because local storage still participates in the active write and read path.
The right mental model is "local-first with a remote tier." Recent data is local, older data can live in object storage, and recovery still needs to account for broker-local state.
Shared-Storage Kafka At A High Level
Shared-storage Kafka starts from a different premise: Kafka compatibility should not require every Broker to be the long-term owner of its data. The Broker handles protocol, request processing, coordination, caching, and scheduling. The durable log is stored in shared storage, typically S3-compatible object storage or cloud object storage. A WAL absorbs the latency and IOPS mismatch between streaming writes and object storage APIs, while cache accelerates reads.
This is where AutoMQ is a useful example, not because every reader must choose it, but because its public architecture is explicit about the pattern. AutoMQ documents S3Stream as a shared streaming storage library that replaces Kafka's native log storage with WAL storage, S3 storage, and Data caching. Its architecture overview describes object storage as the primary repository and broker nodes as stateless because durable data is not bound to local disks.
The write path in this model has two jobs: make the append durable quickly enough for streaming workloads, and organize data into object storage efficiently enough that storage and API cost do not dominate the system. The WAL handles the first job. Object storage handles the long-lived repository.
AutoMQ Open Source and AutoMQ commercial editions differ in WAL options. AutoMQ Open Source supports S3-compatible storage as the WAL option, while AutoMQ commercial editions document additional WAL storage choices across cloud providers. That boundary matters in technical writing because "WAL" is not one latency or durability profile. It is an architectural layer whose implementation has to be selected for the workload.
The Read Path: Tailing Reads Are Not Catch-Up Reads
Read behavior is where shared storage is often misunderstood. A design that reads every consumer request directly from object storage would struggle with small requests and hot partitions. Practical shared-storage Kafka designs separate Tailing Read from Catch-up Read.
Tailing Read is the read of fresh data near the end of the log. It usually wants low latency and predictable behavior because it sits on the steady-state path for real-time applications. Cache is the natural serving layer here, backed by recent writes and hot data. Catch-up Read is different. It happens when consumers replay older offsets, recover from downtime, or scan historical retention. That path can prefetch from object storage into cache and trade slightly different latency behavior for elasticity and lower-cost retention.
Redpanda Tiered Storage also has a remote read path for historical data, so the existence of object storage in the read path is not unique to shared-storage Kafka. The distinction is where object storage sits in the architecture. In Redpanda Tiered Storage, object storage is a retention tier behind a local-storage broker model. In shared-storage Kafka, object storage is part of the primary durable storage design, with cache and WAL built around that assumption.
That difference shows up during operational events. If a Broker fails in a local-storage system, the cluster cares about which replicas were on that broker and how they are restored or rebalanced. If a stateless Broker fails in a Shared Storage architecture, another Broker can take over the compute role because durable data is already in shared storage. Metadata, cache warm-up, and workload-specific behavior still need testing, but the recovery task is no longer primarily local disk reconstruction.
Scaling, Recovery, And Cost Implications
The cloud does not make local storage wrong. It makes local storage more visible. Every local-disk streaming architecture has to plan for broker size, attached volume size, replica placement, retention headroom, rebalancing bandwidth, and failure recovery. Those constraints become harder when traffic is bursty, retention grows faster than compute, or the team wants to scale down after peak windows.
Shared-storage Kafka changes the shape of those constraints:
- Scaling: Adding or removing Brokers can be closer to changing compute capacity because durable data is not permanently attached to the Broker being added or removed.
- Recovery: Broker replacement can avoid copying large local log segments before the node becomes useful again, although cache warm-up and metadata correctness still matter.
- Retention: Object storage can hold long-lived data without forcing local disks to grow with the full replay window.
- Cost visibility: The bill separates compute, WAL storage, object storage, cache, and network behavior more clearly than a single broker-local capacity envelope.
This is not a claim that shared-storage Kafka wins every benchmark. Workloads with extreme low-latency tail reads, small messages, aggressive compaction, or unusual fan-out should be tested against the exact implementation. Workloads dominated by long retention, frequent scaling, and large replay windows may value the architectural separation more than a small difference in steady-state local write behavior.
The same discipline applies to Redpanda. Redpanda can be a strong fit when a team wants a Kafka API-compatible platform with a local-storage engine and a Tiered Storage option for historical data. The evaluation becomes weak when Tiered Storage is treated as equivalent to stateless shared storage.
How To Compare The Two Architectures
A fair comparison starts with the workload, not with the diagram. If the main pain is long retention on expensive local disks, Redpanda Tiered Storage may reduce the local footprint enough. If every scaling or recovery event behaves like a storage movement project, Tiered Storage may not change the part of the system that hurts.
| Question | Redpanda Architecture | Shared-Storage Kafka |
|---|---|---|
| Where is the active durable state? | Broker-local storage, replicated through Raft groups. | Shared storage, usually object storage, with WAL for durable appends. |
| What does object storage do? | Acts as a Tiered Storage target for offloaded log segments. | Acts as the primary durable repository in the storage layer. |
| Are Brokers stateless? | No, broker-local storage remains part of the active model. | Yes in designs such as AutoMQ, where durable data is in shared storage. |
| What changes during scale-out? | The cluster must still account for local replicas, placement, and balance. | Compute capacity can change with less data recopy because storage is shared. |
| What should be benchmarked? | Kafka API compatibility, Raft replication behavior, local disk, and tiered reads. | WAL latency, cache hit behavior, object storage throughput, and recovery. |
This table is mechanism-first. Cloud-native streaming is not a badge. It is a set of operational properties: elastic compute, durable shared storage, clear data-plane ownership, and failure recovery that does not require large local data movement.
Where AutoMQ Fits
AutoMQ fits the shared-storage Kafka category. It is Kafka-compatible, uses S3Stream to replace Kafka log storage, writes data through a WAL layer, persists data to S3-compatible object storage, and uses Data caching for read acceleration. Its GitHub repository also describes AutoMQ as a diskless Kafka implementation on S3-compatible storage.
For cloud and BYOC evaluations, the point is the operating model. AutoMQ BYOC keeps the control plane and data plane in the customer's cloud account, while the storage design separates Broker compute from durable data. That combination is relevant when the buyer cares about Kafka compatibility, account-level data control, and scaling behavior at the same time.
There are still evaluation questions. Which WAL option matches the latency target? How does cache behave under the actual fan-out pattern? What happens during a Broker failure, object storage throttling, or a replay-heavy incident? Does the target preserve idempotent producers, transactions, Kafka Connect, Schema Registry integration, and monitoring assumptions? Shared storage changes the failure model, but it does not remove the need for workload testing.
Choosing The Right Architecture
Choose Redpanda's architecture when the workload benefits from a Kafka API-compatible engine with broker-local storage, Raft-based replication, and Tiered Storage for retention. This can fit teams that want local disk in the hot path, predictable cluster sizing, and an operational model they can benchmark directly against latency and throughput goals.
Choose shared-storage Kafka when the main constraint is cloud elasticity rather than broker engine identity. If the team needs long retention without growing broker disks, faster broker replacement, scale-in after burst windows, or a BYOC data plane with less local state to manage, the stateless broker pattern deserves a serious test. AutoMQ is one example in that category.
The cleanest decision is workload-specific. Redpanda architecture and shared-storage Kafka are not interchangeable labels for "Kafka alternative." Redpanda extends a local-storage broker model with Tiered Storage. Shared-storage Kafka moves the durable center of gravity into object storage and uses WAL plus cache to make that practical for streaming. Once that distinction is clear, the migration question becomes less emotional and more useful: which storage boundary makes next year's operations easier than this year's?
If your evaluation is pointing toward Kafka-compatible shared storage, start with the AutoMQ architecture overview and run the proof with your real Topics, client versions, retention, replay patterns, and failure drills.
FAQ
Is Redpanda shared storage?
Redpanda supports Tiered Storage, which offloads log segments to object storage and supports remote reads. That is different from a Shared Storage architecture where object storage is the primary durable repository behind stateless Brokers.
Is Redpanda Tiered Storage the same as shared-storage Kafka?
No. Tiered Storage extends a local-storage broker model with an object storage tier for older log segments. Shared-storage Kafka changes the primary storage model by moving durable data into shared storage and using WAL plus cache around it.
What is a stateless Kafka broker?
A stateless Broker does not own durable local log storage as the source of truth. It can process Kafka protocol requests and cache data, but durable data lives in shared storage, so broker replacement and scaling require less local data movement.
Why does shared-storage Kafka need a WAL?
Object storage is durable and elastic, but streaming writes require low-latency append behavior and efficient batching. A WAL gives the system a durable write path before data is organized and uploaded into object storage.
When should Redpanda be preferred?
Redpanda can fit when the team wants a Kafka API-compatible platform with a local-storage hot path, Raft replication, and Tiered Storage for retention. The decision should be based on compatibility, latency, recovery, and operational tests under the real workload.
When should AutoMQ be evaluated?
Evaluate AutoMQ when Kafka compatibility, shared object storage, stateless broker operations, and BYOC-style data control are all part of the requirement. Treat it as a workload test, not as a diagram-only architecture choice.
References
- Redpanda Docs: Architecture
- Redpanda Docs: Use Tiered Storage
- Apache Kafka Docs: Replication
- Apache Kafka Docs: Tiered Storage
- AutoMQ Docs: Architecture overview
- AutoMQ Docs: S3Stream shared streaming storage
- AutoMQ GitHub: AutoMQ repository