Table of Contents
Table of Contents
A broker can be elastic without being stateless. It can also be stateless without scaling itself when traffic changes. Those two words describe different properties, but production designs often use them as if they were synonyms. That is where runbooks become vague: a team expects a failed broker to be replaced like a disposable compute instance, then discovers that its local partitions still have to be recovered before the cluster is healthy.
The distinction matters whenever the workload changes shape or a node disappears. Elasticity asks how easily you can add or remove broker compute as traffic changes. Statelessness asks whether a broker owns the authoritative durable data needed to serve its partitions after a restart. The first is about capacity over time; the second is about where durable state lives. Failover, scaling, replacement, and upgrades follow from those two answers.
1Two words that got blended in production conversations
An elastic broker is a broker whose serving capacity can be increased or reduced as the workload changes. The mechanism might be manual capacity planning, an Auto Scaling Group (ASG), a Kubernetes controller, or a platform scheduler. “Elastic” says that the compute fleet can change. It does not say that adding a broker is fast, that partitions move without copying data, or that the broker can start without local storage.
A stateless broker does not keep broker-local durable partition data as the source of truth. The broker can still hold request state, metadata, open connections, caches, and in-flight writes. “Stateless” means that the durable stream data required after a replacement is held in a shared or external storage layer, so a replacement broker does not need to reconstruct the entire partition history from a disk that belonged to the failed process.
This gives operators a useful separation:
- Elasticity: Can the amount of broker compute follow traffic, and what work must happen before the added capacity is useful?
- Broker state: Does a broker replacement need the old node's local durable data, or can a fresh node read the authoritative stream from shared storage?
The distinction is more than terminology. It predicts what an incident commander will be asked to protect. An elastic, stateful cluster still has to protect local disks and replica recovery while changing capacity. A stateless cluster shifts the critical dependencies toward shared storage, WAL (Write-Ahead Log) durability, metadata coordination, network access, and cache warm-up.
2Elastic means capacity follows time
Elasticity is about the relationship between traffic and compute. When traffic rises, a platform with elastic brokers can add serving capacity. When traffic falls, it can remove capacity and release resources. That sounds straightforward until the partitions assigned to the new broker have to become useful, because the time to add a node is only one part of the scale-out path.
In a Shared Nothing architecture, each broker usually manages local log files and participates in replication for the partitions it owns. Adding a broker therefore has two separate steps: create the process and move ownership. The second step may involve copying partition data, catching up a replica, throttling movement to protect client traffic, and checking that the destination is sufficiently synchronized. The platform may be able to create instances quickly while the Kafka cluster remains constrained by data movement.
The runbook records the whole path:
- Trigger: identify the traffic, throughput, connection, or resource signal that requests capacity change.
- Provision: create a broker with the correct network, identity, storage, and configuration.
- Assign: move partitions or leadership to the new broker and observe replication or reassignment work.
- Verify: confirm request health, partition leadership, replica state, consumer progress, and headroom before declaring the change complete.
This runbook applies even when the platform automates the first two steps. Automation can start a node; it cannot make a data movement constraint disappear. The capacity decision should therefore use “time until useful serving capacity,” not only “time until a VM or pod exists.”
3Stateless means the broker holds no authoritative partition data
Statelessness changes the replacement problem because the broker is no longer the only place where its partition history can be recovered. A failed broker still affects client connections, partition leadership, caches, in-flight requests, and possibly a WAL or metadata transition. A stateful replacement is data-aware: the operator checks replica availability, leadership, copy volume, and disk consistency. A stateless replacement is access- and coordination-aware: the operator restores compute identity, shared-data access, ownership or leadership, and cache readiness.
That does not make statelessness a promise of instant failover. Failure detection, controller decisions, client metadata refresh, storage availability, WAL durability, and cache state still set the recovery boundary. A stateless design removes broker-local data movement from one part of the path; it does not remove every other part.
The failure domain changes with it. A stateful broker couples a process, its local storage, and the partitions assigned there. A stateless broker reduces that coupling, but introduces a stronger dependency on the shared storage and metadata path. That dependency needs its own redundancy, access monitoring, quota checks, and failure drills. The right question is not “Does the broker have no state?” It is “Which state remains after the broker is replaced, and which system owns it?”
4The failover runbook changes before the alert fires
The first operational difference appears in the incident checklist. For a stateful broker, begin by protecting the replicas and the storage path. For a stateless broker, begin by protecting the shared storage and metadata path while treating the failed broker as replaceable compute. Both designs still need fencing, leadership checks, and client validation, but the evidence that matters is different.
| Event | Stateful broker with local durable data | Stateless broker with shared durable data |
|---|---|---|
| Broker failure | Fence the node, check leader availability, ISR (In-Sync Replicas), and under-replicated partitions. | Fence the node, check metadata ownership, shared storage access, WAL durability, and leader availability. |
| Replacement | Provision compatible storage, restore or rebuild the replica, wait for catch-up, then verify partition health. | Provision compatible compute, restore access and identity, reassign ownership or leadership, then verify cache and request health. |
| Scale-out | Add a broker, move partitions, and account for replica copy and recovery pressure. | Add compute, assign traffic, and account for shared storage throughput, metadata coordination, and cache warm-up. |
| Upgrade | Drain or roll one broker at a time while preserving replica availability and storage compatibility. | Roll compute while preserving shared storage, metadata, protocol, and cache or WAL compatibility. |
The table is a runbook boundary, not a promise that every platform implements each step in the same way. Automation may handle snapshots, replica placement, or ownership changes, but the operator still needs to know what it watches and what evidence closes the step.
4.1Scaling
For stateful brokers, scale-out is complete only after the new broker serves useful partitions without pushing replica recovery into the danger zone. Watch reassignment throughput, network and disk utilization, ISR changes, request latency, and consumer lag. A scale-out that improves broker CPU but creates a long replication queue has moved the bottleneck rather than removed it.
For stateless brokers, the data copy may no longer be the dominant step. Watch shared-storage request latency and errors, WAL pressure, metadata convergence, connection distribution, cache misses, and the load placed on the new broker. A fresh node can accept ownership before its cache is warm, so “assigned” and “serving at the target level” are separate states.
4.2Replacement
A stateful replacement runbook should define the acceptable replica state before the failed broker is removed permanently. Fencing, duplicate identity handling, compatible storage, placement, and catch-up capacity all matter. The close condition is healthy replication and client behavior, not a green instance status.
A stateless replacement runbook should define the storage and metadata contract before the new compute starts. Check the service identity, object-storage or shared-storage permissions, network route, WAL configuration, listener addresses, controller membership, and client metadata refresh. Then exercise produce and fetch paths, including a partition whose data is not already hot in memory. That last check distinguishes a process restart from a recovery path that works only because the cache survived.
4.3Upgrades
Stateful upgrades are storage operations as much as process operations. A rolling sequence must preserve enough replicas and leadership to keep the topic available while each broker restarts or changes version. The rollback plan must account for data format, replication state, configuration compatibility, and any reassignment work already in flight. “The old image is available” does not prove that rollback is safe.
Stateless upgrades can isolate more of the change to compute, which makes rolling replacement easier to reason about. The boundary moves to the shared storage API, WAL format, metadata schema, controller behavior, and client protocol compatibility. Test a canary, validate produce and fetch, observe controller and storage errors, and only then continue the rollout. The broker's lack of local durable history narrows the data movement problem, but it does not remove version-coupling checks.
5How AutoMQ fits the distinction
The architecture requirement is now concrete: a platform must let compute capacity change independently of durable stream data, while keeping the Kafka client and protocol contract that existing applications rely on. It must also expose enough storage, metadata, and recovery signals for an operator to verify the new failure path. That is the point at which a Kafka-compatible platform such as AutoMQ becomes a bounded example rather than a definition of the terms.
AutoMQ's Shared Storage architecture uses S3Stream to move Kafka log storage into a shared object-storage-backed layer, with WAL storage and data caching supporting the write and read paths. Its documentation describes the result as stateless Broker nodes, and connects that design to partition reassignment and scaling behavior. In this model, a broker replacement does not begin by copying the full historical log from the failed broker's local disk. It begins by restoring compute, metadata, storage access, and serving readiness.
AutoMQ also documents scale-out and scale-in in seconds and partition reassignment in seconds under its shared storage model. Those statements describe the architecture and documented operating path, not a universal RTO for every deployment. Actual recovery still depends on the chosen WAL type, object storage, controller quorum, network, client behavior, workload shape, and the checks in the runbook. The platform changes which work is fundamental; it does not make measurement optional.
For deployment choices, keep the product boundary visible. AutoMQ Open Source uses S3-compatible storage as its WAL option, while AutoMQ commercial editions can support additional WAL storage forms according to deployment and workload. That choice affects latency, durability, resource dependencies, and recovery testing. “Stateless” is not a storage performance claim.
6A two-axis way to compare platforms
Use two axes when comparing a Kafka platform. The horizontal axis is elasticity of broker compute, from fixed or manually changed capacity to capacity that follows traffic. The vertical axis is broker-local durable state, from a broker that depends on its local partition data to one that can be replaced while the authoritative stream remains in shared storage.
This produces four useful categories:
- Stateful and fixed: capacity changes are infrequent, and both compute and data placement are tied to broker instances. The runbook prioritizes headroom and careful maintenance windows.
- Stateful and elastic: compute can change, but local data still has to be replicated, moved, or reattached. The runbook gains automation but keeps data-movement constraints.
- Stateless and fixed: durable data is separated from brokers, but scaling still depends on a human or an external scheduler. Replacement can be simpler while capacity planning remains manual.
- Stateless and elastic: compute and durable stream data can change independently. The runbook focuses on routing, metadata, shared-storage health, WAL behavior, and cache readiness.
AutoMQ is an example of the fourth category when deployed with its Shared Storage architecture and supported scaling path. The comparison still needs a workload-specific check. If the workload is dominated by shared-storage throughput, cache misses, controller coordination, or strict latency requirements, stateless compute alone does not settle the decision. If broker-local data movement is the main obstacle to replacement and capacity changes, the separation can remove a large part of the operational work.
When an incident starts, ask what a replacement must carry, what data must move, what dependency must remain available, and what observation closes the runbook. Those answers reveal the operating model better than either label.
7FAQ
7.1Is an elastic broker the same as a stateless broker?
No. Elastic describes the ability to change broker compute as traffic changes. Stateless describes whether a broker owns authoritative durable partition data. A platform can have one property without the other.
7.2Does a stateless broker have no state at all?
No. It can hold metadata, connections, caches, in-flight requests, and temporary write state. The precise claim is that broker-local durable partition data is not the authoritative source needed to recover the stream after replacement.
7.3Does statelessness guarantee faster Kafka failover?
No. It can remove full local-log movement from the replacement path, but recovery still depends on failure detection, metadata coordination, shared storage, WAL durability, client refresh, and cache readiness. Measure the complete path in the target deployment.
7.4What should a stateless broker replacement runbook include?
Include fencing, compute provisioning, identity and permissions, network access to shared storage, metadata or ownership convergence, WAL and object-storage health, cache warm-up, produce and fetch checks, and client metadata refresh.
7.5Where does AutoMQ fit?
AutoMQ is a Kafka-compatible streaming platform that uses Shared Storage architecture and stateless Broker nodes. It is a bounded example of combining statelessness with elastic scaling, subject to the selected WAL, object storage, deployment mode, workload, and recovery validation.
8References
- Apache Kafka documentation: Replication
- Apache Kafka documentation:
min.insync.replicas - Apache Kafka documentation: Upgrading
- AutoMQ architecture overview
- AutoMQ stateless Broker documentation
- AutoMQ partition reassignment in seconds
- AutoMQ scale-out and scale-in in seconds
- AutoMQ S3Stream shared streaming storage
A broker that looks replaceable on a diagram is only operationally replaceable when the storage, metadata, and client paths have been tested together. If you want to evaluate that boundary with your workload, start with the AutoMQ GitHub repository and measure the recovery steps that matter to your team.
