Table of Contents
Table of Contents
“Scale Apache Kafka” sounds like a single action until someone asks what is actually full: broker CPU, a hot partition, slow consumers, or retained data and replica traffic consuming the storage and network budget. Adding brokers, adding partitions, and changing neither solve different problems.
The safest Kafka scaling decision starts with the constrained resource, not with the size of the cluster. A broker adds serving capacity and failure-domain room. A partition adds a unit of parallelism, but also more metadata and placement work. A consumer-side change may be the right answer when the cluster is healthy and the application is the slow part. When the real constraint is broker-local storage or data movement, any of those actions can become an expensive way to postpone an architectural decision.
1Scaling has three levers, not one
Kafka exposes several resources that look related in a dashboard but behave differently under load. A producer writes to a topic partition; a partition leader handles the write path; replicas add durability work; and a consumer group divides partitions among its members. The broker is where many of those paths meet, which is why broker CPU or network pressure is often the first visible symptom even when the cause lives elsewhere.
Think of the cluster as four questions rather than one capacity number:
- Can the brokers serve the work? Request handling, compression, replication, networking, page cache, and protocol coordination consume broker resources.
- Can the workload be divided? A topic with too few partitions limits producer and consumer parallelism, even if the brokers have spare capacity.
- Can consumers keep up? Consumer processing time, downstream rate limits, fetch settings, and group membership can create lag without a broker shortage.
- Can the storage and network model absorb the change? Retention, replica placement, disk throughput, traffic between Availability Zones (AZs), and reassignment work may dominate the result of a scale-out.
The distinction matters because a change at one layer does not automatically create capacity at another. Adding a broker does not make a single hot partition parallel. Adding partitions does not make a slow downstream database process records faster. Increasing consumer instances does not remove a disk write bottleneck. The first useful action is to map the symptom to the resource that can physically relieve it.
2Broker-bound, partition-bound, or storage-bound
The broker is the right scaling unit when pressure is broad. Look for several brokers showing sustained request queue, CPU, network, or disk-bandwidth pressure at the same time, with traffic distributed across the leaders they serve. AWS describes broker instance selection in terms of resources such as vCPU, memory, storage, and network, which is a useful reminder that “broker capacity” is a bundle of constraints rather than a single throughput label.
Adding brokers helps when the added nodes can receive useful leaders or replicas and the cluster has a controlled way to redistribute work. It does not help when one key pins most writes to one partition, when the consumer group is the bottleneck, or when the added nodes remain empty because reassignment cannot finish within the operational window. In a broker-local storage model, making those nodes useful may also require copying retained log data, so the act of scaling competes with the traffic it is meant to protect.
Partitions are the right lever when parallelism is the missing resource. A topic with fewer partitions than the desired producer or consumer concurrency cannot use all available workers, because each partition has one leader at a time and a consumer group assigns a partition to one consumer within that group. The evidence is usually specific: a topic or workload has growing lag or producer contention while its assigned brokers retain headroom, and the workload can be split safely by key and ordering requirements.
More partitions are not a free performance switch. They increase metadata, file or object management, scheduling, recovery, and monitoring work. Expanding a topic can also change how keys are mapped to partitions, which is why teams must validate ordering assumptions and key distribution before treating partition count as a reversible tuning knob. If a single key remains hot, adding empty partitions changes the ceiling without moving the hot key's traffic.
Consumers are the right lever when the broker is delivering data but the application is not draining it. Compare consumer processing time, fetch behavior, downstream throughput, rebalance activity, and lag by partition. A consumer group with fewer active members than useful partitions may need more workers; a group blocked on an external service may need backpressure or downstream capacity instead. Adding brokers to that situation adds infrastructure but leaves the queue at the same endpoint.
Storage-bound and network-bound symptoms need their own decision. Disk utilization can rise because retention is longer, compaction is slower, replicas are catching up, or the write path is limited by disk throughput. Network pressure can come from client traffic, replica traffic, catch-up reads, or cross-AZ placement. A broker count change may reduce per-node pressure, but it can also multiply the amount of data that must move during reassignment and the network paths that must be monitored.
3The decision rule for each lever
Use a short evidence window that includes the incident or peak period. Compare per-broker request latency, CPU, network, disk I/O, leader distribution, topic throughput by partition, consumer lag, and reassignment activity over the same interval. Cluster averages are useful for trend lines, but they can hide the one partition or broker that is carrying the problem. A broader diagnostic workflow is available in Diagnosing Kafka Saturation Before Adding More Brokers; the storage-movement angle is covered in How Shared Storage Changes Partition Reassignment Bottlenecks.
The decision can then be made with four tests:
| Observed constraint | First lever to test | Why it fits | What would make it the wrong fix |
|---|---|---|---|
| Broad broker CPU, request, or network pressure | Add or resize brokers | More serving capacity can spread leaders and requests | The pressure is isolated to one hot partition or a downstream consumer |
| Too little parallelism for a topic or consumer group | Add partitions, then add consumers as needed | More partitions create more units that producers and consumers can use | Key ordering, hot-key skew, metadata scale, or reassignment risk is not acceptable |
| Consumer processing or downstream service is slow | Add consumer capacity or fix the downstream path | Lag is created after the broker has delivered the records | Broker metrics show the broker, not the consumer path, is saturated |
| Retention, replica movement, or cross-AZ transfer dominates | Change retention/storage/network design | The constraint is in the data path, not the count of serving nodes | The workload actually needs more compute or partition parallelism |
| A short-lived traffic spike has already ended | Do neither, or use temporary capacity with a removal plan | The system may recover without a permanent topology change | The spike is becoming a sustained SLO or capacity breach |
The table is a starting rule, not an automation policy. A broker expansion still needs placement, replication, throttling, rollback, and observation criteria. A partition expansion still needs key and ordering validation. A decision to do neither still needs a bounded observation period and a clear trigger for action. The important change is that each action has a falsifiable reason.
4When scaling hides the real issue
The most common anti-pattern is multiplying infrastructure before identifying what the additional capacity can serve. Four versions show up repeatedly in production.
Adding brokers for a hot partition. If one partition receives most records for a popular key, the leader for that partition remains the bottleneck. Added brokers may carry other partitions while the hot leader stays unchanged. Fix the key distribution, topic design, or producer behavior first; only add brokers when the evidence shows a broader broker constraint.
Adding partitions for a slow consumer. A topic can have enough partitions while its consumers spend most of their time waiting on a database, API, or CPU-heavy transformation. More partitions can create more concurrent work and more downstream contention. Measure end-to-end processing time before increasing the topic's parallelism.
Adding brokers because disk is full. More brokers may create space, but they also introduce another reassignment and recovery surface. First separate retained data growth, replica catch-up, compaction, segment sizing, and actual write throughput. If retention is the requirement, changing the storage model or lifecycle policy may be more direct than buying more serving nodes.
Doubling everything after a peak. A temporary spike does not prove that the steady-state topology is undersized. If the peak has ended and SLOs are recovering, immediate broker and partition growth may create idle capacity, more placement work, and a harder future scale-down. Record the trigger, watch the recovery, and make the next change proportional to the measured constraint.
These anti-patterns share one failure mode: they make the dashboard look quieter without removing the work that caused the symptom. Scaling is valuable when it changes the constrained resource. It is wasteful when it only spreads unrelated work across more boxes.
5What changes when brokers and data are decoupled
The decision rules become different when the platform separates serving compute from durable stream storage. In traditional Apache Kafka, the Shared Nothing architecture places partition data on broker-local storage and uses replication to provide durability and availability. A broker therefore represents both request-serving capacity and a slice of the durable dataset. Adding or removing brokers can require data movement before the changed topology is useful.
Tiered Storage addresses part of that pressure by moving older log segments to remote storage, but it does not automatically remove the local active log or replica placement model. The question remains: how much of the workload is still tied to broker-local storage when a broker changes shape?
AutoMQ is a Kafka-compatible streaming platform that uses a Shared Storage architecture. Its S3Stream storage layer combines WAL storage, data caching, and object storage, so durable stream data is not treated as the long-lived possession of one broker's local disk. That is why AutoMQ's architecture overview connects stateless brokers with automatic scaling and traffic rebalancing rather than treating each broker change as a storage migration.
The architecture does not make partition design irrelevant. It changes the boundary of the broker decision. You still need enough partitions for the desired workload parallelism, enough consumer capacity for the processing path, and enough WAL, cache, and object-storage capacity for the latency and durability requirements. What changes is that broker scale-out can focus more directly on serving compute and traffic ownership instead of adding local disk capacity and then moving the retained log to match it.
That separation also changes how to evaluate a scale event. Ask whether the event is moving metadata and traffic, or copying durable data. AutoMQ documents partition reassignment in seconds and continuous self-balancing as consequences of that storage model. Those claims still need workload-specific validation: test the Kafka clients, security settings, retention shape, read patterns, failure recovery, and cloud network topology that matter to your cluster.
The practical rule is direct: choose the lever that changes the constrained resource, then verify that the architecture does not make the change itself the next bottleneck. If the problem is serving compute, add brokers. If it is parallelism, add partitions and consumers together. If it is application processing, fix that path. If storage ownership and movement dominate every topology change, evaluate a platform that separates compute from durable storage instead of repeatedly scaling the wrong bundle.
6FAQ
6.1When should I add Kafka brokers?
Add brokers when several brokers show sustained pressure in the resource that limits the workload, and when leaders or replicas can be distributed to the added nodes without creating an unsafe reassignment window. Confirm that the pressure is broad rather than a single hot partition, slow consumer, or retention problem.
6.2Is adding partitions better than adding brokers?
Neither is generally better. Partitions increase workload parallelism; brokers add serving and failure-domain capacity. Add partitions when the topic cannot expose enough independent work to producers or consumers. Add brokers when the existing partition set is broadly stressing broker resources and can be redistributed safely.
6.3Can I add partitions to fix a hot partition?
Not by itself. A hot key can continue mapping to one partition after the topic is expanded, depending on the producer's partitioning behavior. Inspect key distribution and ordering requirements before changing the partition count.
6.4When should I do neither?
Do neither when the pressure is temporary and recovering, the broker has headroom, or the evidence points to client configuration, consumer processing, a downstream service, or retention policy. Set an observation window and a concrete threshold for revisiting the decision.
6.5Does Shared Storage remove the need for Kafka capacity planning?
No. It changes which resources are coupled. Partition parallelism, consumer capacity, WAL and cache behavior, object-storage access, metadata scale, network paths, and failure recovery still need measurement. The benefit is that broker compute can be evaluated more independently from durable data placement.
7References
- Apache Kafka Documentation: Introduction to Kafka
- Apache Kafka Documentation: Replication
- Apache Kafka Documentation: Topic configuration
- Apache Kafka Documentation: Consumer groups
- KIP-405: Kafka Tiered Storage
- Amazon MSK broker instance types
- AutoMQ S3Stream shared streaming storage
- AutoMQ stateless brokers
- AutoMQ compatibility with Apache Kafka
If a Kafka scaling alert is asking for “more capacity,” start by identifying which resource is actually constrained. Then model the smallest change that can relieve it. To evaluate a Kafka-compatible Shared Storage deployment against your own workload, explore AutoMQ.
