Cloud Kafka often becomes expensive in a way that is hard to see from the first sizing spreadsheet. The team buys broker instances for traffic, attaches disk for retention, adds replication for availability, and then discovers that each operational change has a storage shadow. Adding brokers often means moving partition replicas. Shrinking the cluster means finding a safe destination for the data that lived on those brokers.
That is the storage-compute coupling problem. In traditional Kafka, a broker is both a networked compute process and the durable home of local log replicas. This design has served Kafka well, but it was shaped by a world where disks were attached to servers. In elastic cloud infrastructure, compute and storage are already separate services. Kafka's broker-local disk model can pull them back together.
Storage-compute separation asks a different question: what if broker capacity followed traffic while storage capacity followed retention? That shift changes how architects think about expansion, contraction, recovery, and cost allocation.
How traditional Kafka couples storage and compute
Kafka stores records in partition logs. In the classic shared-nothing model, those logs are stored on broker-local disks. Topics are split into partitions, partitions have replicas, and each replica is placed on a broker. One replica acts as leader for a partition, while follower replicas fetch from the leader and persist their own local copies. Kafka's official design documentation describes replication as copying a partition log across multiple servers for durability and availability.
The coupling comes from the unit of placement. A broker is not only a process that accepts produce and fetch requests. It is also the location of some persistent replicas. When a topic's retention grows, the cluster needs enough disk across brokers. When traffic grows, the cluster needs enough CPU, network, page cache, and disk throughput across brokers. Those two requirements are planned together because they share the same node boundary.
This coupling is not a bug in Kafka. It is a deliberate architecture with useful properties:
- The data path is direct: clients talk to brokers, and brokers write local logs.
- Durability is handled through replicated logs and in-sync replicas.
- Local storage gives predictable behavior for hot sequential I/O.
- Operators can reason about placement by looking at brokers, racks, disks, and partitions.
The same design becomes less flexible when workloads have uneven curves. A platform may need more write throughput for a daily event burst but not more long-term retention. Another workload may need more retention for audit or replay but only modest traffic. With broker-local disks, those independent demands turn into a single broker-sizing problem.
Why coupling hurts cloud elasticity
The cloud makes underused infrastructure visible. Idle vCPU, reserved block storage, cross-zone replication, and recovery bandwidth all show up as separate line items. Kafka's storage-compute coupling matters because it can force the platform team to buy resources for the most constrained dimension and carry the unused capacity in the others.
Expansion becomes data-heavy
Adding brokers to a traditional Kafka cluster gives the cluster more potential capacity, but existing data does not automatically redistribute itself across those brokers. Operators typically use partition reassignment or balancing tooling to move replicas. Apache Kafka's operations documentation describes partition reassignment through generated or custom reassignment plans, which is a clear signal that expansion involves placement work rather than only adding processes.
That movement consumes network and disk I/O. During the copy, the cluster is doing production work and rebalancing work at the same time. Throttles can protect client traffic, but throttling also stretches the operation. The bigger the retained data set, the more expansion becomes a storage migration project.
This is the first elasticity mismatch: traffic may need compute quickly, while local replicas require time to move.
Shrinking is operationally harder than growing
Scaling down is where the coupling becomes more visible. A broker with idle CPU may still hold partition replicas that are needed for retention and availability. Removing it safely requires relocating those replicas and verifying that the resulting placement still satisfies replication, rack awareness, and capacity constraints.
This creates a practical asymmetry. Teams are willing to add capacity before a peak because the risk of being short is obvious. They are more cautious about removing capacity after the peak because the shrink operation touches durable data. Over time, that bias turns into over-provisioning.
The waste is not limited to compute. Broker-local disks are often sized for retention headroom, replication factor, and operational safety margin. If compute demand falls faster than retained data, those disks can keep broker count high.
Recovery competes with production traffic
Broker and disk failures also expose the combined failure domain. Kafka can elect leaders from in-sync replicas, but returning the cluster to the desired replica layout may require rebuilding data onto replacement brokers or disks. That rebuild has the same resource conflict as expansion: network, disk, and page cache are shared with client traffic.
For workloads with large retention, recovery is not only about detecting a failed broker. It is also about how long the system spends reconstructing placement. The user-visible outage may be short, while the operational recovery window remains long and resource intensive.
The lesson for cloud Kafka architecture is not that local disks are unusable. It is that they bind recovery, elasticity, and cost to the same physical unit.
What storage-compute separation changes
Storage-compute separation moves durable log ownership out of broker-local disks and into a shared storage layer. Brokers remain the Kafka-facing compute layer: they handle client connections, protocol handling, partition leadership, batching, fetch serving, group coordination, quotas, and runtime caches. The storage layer becomes responsible for durable capacity and long-lived log data.
The goal is not to remove state from the system. Kafka still needs metadata, epochs, ordering, fencing, offsets, and durability. The goal is to make durable data independent of a specific broker's local disk. A replacement broker should be able to take over compute responsibility by using metadata and shared durable state, rather than waiting for the full data set to be copied to its local storage first.
This changes the scaling model:
| Dimension | Broker-local Kafka | Storage-compute separated Kafka |
|---|---|---|
| Traffic growth | Add brokers, then rebalance replicas to use them well | Add compute capacity for leaders, connections, and fetch load |
| Retention growth | Add disk to brokers or add brokers for disk capacity | Increase shared storage usage independently |
| Scale down | Move replicas away from brokers before removal | Remove compute once workload and ownership rules allow |
| Recovery | Elect leaders, then rebuild local replica placement | Reassign compute against shared durable data |
| Cost planning | Broker size blends CPU, network, disk, and retention | Compute and storage can be budgeted as separate curves |
The important phrase is "separate curves." Compute should scale with traffic. Storage should scale with retention. When those curves are independent, architects can ask cleaner questions. How many broker processes do we need for peak produce and fetch load? How much durable storage do we need for seven-day, thirty-day, or one-year retention? Which workloads need hot cache, and which mainly need reliable replay?
Storage-compute separation is also different from treating object storage as a cold archive. Apache Kafka tiered storage keeps an active local tier while older log segments can move to remote storage. That can reduce disk pressure for long retention, but it does not by itself make brokers independent of local persistent state. A shared-storage Kafka design goes further: object storage or another shared durable layer becomes part of the primary storage architecture.
What has to be true for separation to work
The phrase "decoupled storage and compute" is easy to draw and difficult to implement well. A Kafka-compatible system has to preserve the semantics that Kafka users rely on while changing the internal storage substrate. Several requirements matter more than the label.
First, acknowledgments must represent real durability. If a producer receives a successful acknowledgment, the data must be protected according to the system's durability design. Object storage has different latency and request characteristics from a local append-only disk, so many architectures introduce a write-ahead path, batching, caching, or segment layout strategy to bridge that gap.
Second, leadership and fencing must remain strict. Shared storage cannot become shared confusion. Only the valid owner for a partition epoch should append, and a broker that has lost authority must not continue writing. Metadata, storage ownership, and broker runtime state have to agree during failover.
Third, read performance needs a deliberate hot path. Kafka workloads are not all archive reads. Many consumers read the tail of the log with low lag, while others perform catch-up reads after downtime or replay. A separated architecture needs caching, prefetching, and object layout choices that serve both patterns without exposing a new API to clients.
Fourth, operations must become simpler in practice, not only in diagrams. If every scale event requires a complex manual sequence, the architecture has not delivered elasticity. The control plane should make reassignment, balancing, and recovery more compute-oriented.
How AutoMQ applies storage-compute separation to Kafka
Once the problem is framed this way, AutoMQ fits as one implementation of the separated model: a Kafka-compatible compute layer backed by an object-storage-based stream storage layer. AutoMQ keeps the Kafka protocol and ecosystem compatibility at the broker boundary, while replacing broker-local log storage with S3Stream, its shared streaming storage abstraction.
AutoMQ documentation describes S3Stream as a component that offloads Kafka's built-in log storage to cloud storage and follows the idea of decoupling storage and computation. In the architecture overview, AutoMQ also describes broker nodes as stateless because S3Stream replaces native local log storage. The practical meaning is that broker nodes can be treated more like replaceable compute workers, while durable log data is held in object storage and associated storage components.
That does not mean object storage is magically identical to a local disk. AutoMQ's S3Stream design includes a write-ahead log path for low-latency persistence and recovery, plus S3 storage for the main durable data layer. The details matter because Kafka workloads are latency sensitive at the tail and storage intensive over time.
This is also why AutoMQ should be evaluated as a Kafka architecture choice rather than as a storage plugin. The architectural claim is broader than "put Kafka data in S3." It is:
- Preserve Kafka client and ecosystem behavior at the protocol layer.
- Move durable log storage away from broker-local disks.
- Use object storage as the scalable storage foundation.
- Make broker scaling and recovery less dependent on large local data movement.
- Let operators plan compute and retention with fewer hidden dependencies.
For teams already running Kafka, this is the right level of evaluation. The question is not whether object storage is less costly per GiB than high-performance block storage in a specific region. That can be checked from cloud pricing pages. The deeper question is whether the platform's operational model should continue to make every scale event a broker-and-disk event.
Migration and evaluation checklist
A storage-compute separated Kafka architecture is most compelling when the current pain is architectural rather than incidental. If a cluster is small, steady, and has short retention, traditional Kafka may be perfectly reasonable. If the cluster has large retained data, bursty traffic, frequent scaling events, or long recovery windows, the evaluation becomes more interesting.
Start with workload shape. Separate sustained traffic, burst traffic, fan-out reads, retention, and replay needs. Many Kafka sizing discussions blur these together, but separated architecture benefits depend on whether those curves actually diverge.
Then examine operational friction:
- How often do you add brokers for traffic rather than retention?
- How often do you keep brokers because of disk usage rather than CPU or network usage?
- How long do partition reassignment and post-failure rebuild operations take?
- How much headroom is reserved because shrink operations feel risky?
- Which workloads would benefit from faster replacement of broker compute?
After that, test compatibility at the boundary that matters: Kafka clients, Kafka Connect, Kafka Streams, security configuration, observability, operational tooling, and disaster recovery procedures. A system can be architecturally elegant and still fail the migration if the surrounding platform contract changes too much.
Finally, model cost as two curves rather than one blended broker bill. Compute should be estimated from produce throughput, fetch throughput, connection count, partition leadership, and cache needs. Storage should be estimated from retained bytes, replication or durability model, object storage request patterns, and recovery requirements. This makes the trade-off visible even before a proof of concept.
When separation matters most
Storage-compute separation matters most when Kafka has become a platform service rather than a single application dependency. Platform teams need to absorb changing workload shapes without making every tenant's retention policy and traffic pattern fight for the same broker resources.
It is especially relevant for:
- Multi-tenant Kafka platforms where some tenants need traffic elasticity and others need long retention.
- Event pipelines with bursty ingest but predictable retention.
- Clusters where partition reassignment is avoided because retained data is large.
- Cloud deployments where block storage, cross-zone replication, and idle compute are visible cost drivers.
- Teams that want Kafka compatibility but a more cloud-native operational model.
The architectural test is straightforward. If your Kafka costs and operations are dominated by the fact that data lives on broker-local disks, then tuning broker sizes may only postpone the problem. Storage-compute separation changes the unit of scaling. That is why it matters for cloud Kafka.
References
- Apache Kafka Design: Replication
- Apache Kafka Operations: partition reassignment
- Apache Kafka Operations: Tiered Storage
- AutoMQ Documentation: Architecture Overview
- AutoMQ Documentation: S3Stream Overview
- AutoMQ Documentation: S3 Storage
- AutoMQ Documentation: Difference with Apache Kafka
FAQ
Is storage-compute separation the same as Kafka tiered storage?
No. Tiered storage typically keeps a local broker tier for active data and moves older segments to remote storage. Storage-compute separation is a broader architecture where durable log ownership is not bound to broker-local disks. Tiered storage can reduce local disk pressure, but it does not automatically make brokers stateless.
Does a separated Kafka architecture remove brokers?
No. Producers and consumers still talk to brokers through Kafka APIs. Brokers remain the compute layer for protocol handling, leadership, fetch serving, coordination, and runtime caching. The change is where durable log data lives and how broker replacement works.
Does object storage make Kafka slower?
Object storage has different latency and request behavior from local disks, so a serious design needs a write path, caching, batching, and object layout suited to streaming. The right question is not raw object storage latency in isolation, but whether the system preserves Kafka-facing latency and durability for the workload being evaluated.
When should a team evaluate AutoMQ?
Evaluate AutoMQ when Kafka compatibility is required but broker-local storage is causing cost, scaling, or recovery friction. It is most relevant when traffic and retention scale differently, when partition movement is painful, or when cloud cost visibility makes over-provisioned broker resources hard to ignore.