Title: MSK Scaling: Why Scaling Kafka on AWS Is Still Hard
Description: Understand Amazon MSK scaling challenges around partitions, brokers, storage, rebalancing, traffic spikes, and shared storage alternatives.
Keywords: msk scaling, amazon msk scaling, aws msk scaling, kafka scaling aws, msk autoscaling
User Search Queries: msk scaling; amazon msk scaling; AWS MSK scaling; Kafka scaling AWS; MSK autoscaling; scale Amazon MSK cluster
Search for "MSK scaling" and you will find a familiar split. Amazon MSK gives you managed operations for Apache Kafka on AWS, with documented ways to expand broker count, change broker size, increase storage, and configure storage auto scaling. Yet production teams still treat Kafka scaling as a maintenance event, not as a casual cloud elasticity setting. The reason is not that the AWS console is missing a button. Kafka's original scaling model binds traffic, partitions, and durable log data to brokers, so the operational work begins after infrastructure capacity exists.
That distinction matters for architects, SREs, and FinOps teams. If a workload has a predictable weekly curve, you can plan broker capacity and storage headroom with enough discipline. If the workload has bursts, tenant-level hot spots, retention surprises, or a hard scale-in requirement after peak traffic, the hard part is no longer provisioning a node. The hard part is moving risk around without creating a second incident.
What scaling means in Amazon MSK
Amazon MSK scaling is not one operation. It is a set of operations that touch different bottlenecks. A broker-count change gives the cluster more broker nodes. A broker-size change gives each broker more CPU, memory, and network capacity. A storage change gives each Standard broker more EBS capacity. Partition changes increase parallelism, but they also affect ordering, metadata, consumer assignment, and broker load.
Those levers solve different symptoms:
- Add brokers when you need more aggregate capacity or want to spread leaders and replicas across more nodes.
- Change broker size when the existing broker shape is the bottleneck and a rolling workflow is acceptable.
- Increase storage when retention or write volume pushes disks toward their limits.
- Add partitions when topic-level parallelism is the limiting factor, with care around key ordering and consumer behavior.
- Reassign partitions when broker-level load or storage distribution is uneven after the physical cluster changes.
AWS documents the broker-count path clearly: to expand an MSK cluster, the cluster must be ACTIVE, the target broker count must be greater than the current count, and the target must be a multiple of the number of Availability Zones. AWS also points users from broker expansion to partition rebalancing, because added brokers do not automatically mean existing topic load has moved. That is the first place where "managed Kafka" still behaves like Kafka.
The storage lever has its own shape. AWS states that Standard broker storage can be increased but not decreased. Storage remains available during the scale-up operation, but after each storage scaling event MSK requires a cooldown period before storage can be scaled again. Automatic storage scaling can expand storage in response to usage, but AWS notes that it does not reduce storage when usage falls. For FinOps, that means disk growth is operationally convenient in one direction and structurally sticky in the other.
The quota model reinforces the same planning problem. AWS publishes quotas for brokers per account, brokers per cluster, and storage per broker, and its best-practices page gives partition-per-broker guidance as both a sizing recommendation and a precondition for some update operations. A quota increase is another dependency in the capacity plan, not an autoscaling mechanism.
The hidden work behind adding brokers
Apache Kafka's own operations documentation is blunt about cluster expansion: adding servers is easy, but added servers will not automatically be assigned existing data partitions, so they will not do useful work for existing load until partitions move. Under the covers, Kafka adds the destination server as a follower for a partition being migrated, lets it replicate the existing data, waits for it to join the in-sync replica set, and then removes the old copy. That design is reliable, but it is not lightweight.
In MSK, the same mechanism shows up as partition reassignment. AWS recommends using kafka-reassign-partitions.sh to move partitions within an MSK Provisioned cluster, and it explicitly warns that after adding brokers and reassigning existing partitions, the cluster spends resources replicating data from broker to broker. AWS also recommends avoiding this option when CPU utilization is already high because replication adds CPU load and network traffic.
That is why the timing of an MSK scaling event is so sensitive. You might add brokers because the cluster is hot, but the most direct way to move existing traffic onto those brokers can make the existing brokers hotter before it makes them cooler. If CPU, disk I/O, replication lag, or network capacity is already near the edge, reassignment can compete with the client traffic you are trying to protect.
| Scaling task | What changes immediately | What still needs operational work |
|---|---|---|
| Add brokers | Broker capacity exists | Existing partitions must be assigned or reassigned to use it |
| Change broker size | Per-broker resources increase through a rolling process | Leaders move while brokers are taken offline in sequence |
| Increase storage | EBS capacity grows | Storage cannot be reduced later; cooldown affects repeated changes |
| Add partitions | Topic parallelism increases | Existing data is not redistributed; clients and consumers react to new metadata |
| Remove brokers | Infrastructure shrinks | Partitions must be evacuated first, making scale-in riskier |
The table is not an argument against MSK. It is a reminder that MSK manages Kafka infrastructure, while Kafka's partition and log placement rules still define the scaling physics.
Partition reassignment is the scaling tax
Partition reassignment sounds like a background maintenance task until you look at what it contends with: replication bandwidth, disk throughput, CPU, controller metadata updates, and broker request handling capacity. It also interacts with leader placement, in-sync replica health, and consumer lag. In a quiet cluster, that can be manageable. In a cluster being scaled because of a traffic spike, the same task can land at the worst possible time.
AWS best practices expose the operational assumptions. MSK recommends keeping broker CPU headroom so the cluster can tolerate broker failures, patching, rolling upgrades, broker-size changes, and version upgrades. AWS also describes partition reassignment after broker expansion as a path that can significantly increase load at first. Scaling capacity is safest when the cluster already has enough headroom to pay the reassignment cost.
Kafka gives operators throttles for reassignment traffic, and those throttles are important. But throttling creates a trade-off rather than removing the cost. Set the throttle too high and reassignment competes with client traffic. Set it too low and the operation takes longer, or may not make progress if incoming write throughput is higher than the migration budget. This is why experienced Kafka SREs often treat rebalancing as a controlled change window.
Consumer behavior adds another layer. Adding partitions can increase parallelism for future writes, but Kafka documentation notes that increasing partition count can change key distribution when a hash-based partitioning scheme is used. For keyed workloads where ordering matters, "add partitions" is a data-model decision.
Why scale-in is harder than scale-out
Scale-out is uncomfortable because added brokers need useful work. Scale-in is harder because removed brokers must have no irreplaceable work left. Their partitions need to move elsewhere, and the remaining cluster needs enough capacity to absorb the traffic and storage footprint. If the remaining cluster is barely large enough, scale-in can remove the headroom needed to complete the move safely.
Storage behaves similarly. Increasing MSK Standard broker storage is operationally supported, but AWS says storage cannot be decreased. If you need smaller storage, AWS points to migration to a smaller-storage cluster. From a FinOps perspective, this turns storage oversizing into a long-lived cost unless you accept a migration project. Autoscaling storage helps avoid running out of disk, but it does not make storage elastic in both directions.
Broker count, broker size, partition count, storage size, and quotas all converge into one discipline: capacity planning. The important question is not "Can MSK scale?" It can, within documented operations and quotas. The better question is "Which resource becomes stateful debt after the scale event?" Local log data, expanded EBS volumes, partition counts, and reassignment plans all create future work.
What a traffic spike does to the plan
Traffic spikes expose the gap between infrastructure elasticity and Kafka elasticity. In a stateless web tier, scaling out usually means adding instances and letting a load balancer route traffic. In Kafka, traffic is bound to partitions, partitions have leaders, leaders live on brokers, and replicas are backed by broker-local storage. Adding brokers gives you places to move load, but the movement is not free.
A practical MSK runbook for traffic spikes usually needs at least four checks before action:
- Is the bottleneck CPU, network, disk throughput, partition skew, request latency, or storage capacity?
- Is there enough CPU and network headroom to run reassignment while serving producers and consumers?
- Are the hottest topics safe to repartition, or would key-ordering expectations break?
- If storage expands during the event, what is the long-term cost of the higher floor?
That runbook is why "MSK autoscaling" can be a misleading search phrase. MSK supports storage auto scaling for Standard brokers, and AWS provides APIs and workflows for other scaling dimensions, but general Kafka capacity is not automatically elastic like stateless compute. Storage auto scaling solves one failure mode. It does not rebalance partition leaders, decide safe throttles, reduce storage, or make scale-in trivial.
How shared storage changes Kafka scaling
The architectural alternative is to change what a broker owns. In traditional Kafka, a broker owns compute responsibility and local durable log segments. In a shared-storage Kafka-compatible architecture, brokers still serve Kafka protocol traffic, but durable log data is stored in a shared layer rather than anchored to one broker's disk. That changes the scaling unit from "move the data to the broker" to "move ownership and traffic to a broker that can read the shared data."
This is where AutoMQ enters the discussion as an architecture category, not as a magic replacement for operational judgment. AutoMQ is a Kafka-compatible cloud-native streaming system that offloads Kafka's log storage layer to object storage through S3Stream and uses stateless brokers through storage-compute separation. Its documentation describes partition reassignment as synchronizing only the data not yet uploaded to object storage rather than copying the full local dataset.
That difference directly targets the scaling tax. If durable data is already in shared object storage, reassignment does not need to replicate the whole retained log from one broker disk to another. The remaining work is metadata, ownership, traffic placement, and data still in the write-ahead path. AutoMQ's public docs include specific claims about second-level partition reassignment and scale-out/in behavior; those claims should be evaluated against your workload, but the mechanism is the important part here.
The trade-off deserves a sober reading. Shared storage architectures introduce design requirements around write-ahead logging, object-store access patterns, cache behavior, and failure recovery. AutoMQ's S3Stream documentation explains that object storage alone has latency and IOPS characteristics that are not ideal for streaming writes, so AutoMQ combines WAL storage with object storage. The architecture is not "put Kafka logs on S3 and call it done." It reworks the storage layer so brokers stop being the durable-data boundary.
When MSK is still the right answer
For many AWS teams, MSK remains a sound default. If you want managed Apache Kafka, have stable workloads, operate within documented quotas, and can plan rebalancing windows, MSK removes a lot of infrastructure work. The problem starts when the business expects Kafka to behave like a stateless cloud service while the workload still follows broker-local storage rules.
The decision framework is straightforward:
| If your main constraint is... | MSK scaling question | Architecture question |
|---|---|---|
| Predictable growth | Can we add brokers and rebalance during planned windows? | Probably manageable with disciplined operations |
| Burst traffic | Can reassignment finish before the spike hurts users? | Consider whether broker-local data movement is acceptable |
| Cost after peak | Can we scale in without risky evacuation work? | Shared storage may reduce the scale-in penalty |
| Retention growth | Can we tolerate one-way storage expansion? | Separate storage economics from broker count |
| Hot partitions | Can we move leaders or repartition safely? | Look for faster ownership and traffic balancing |
This is also where FinOps and SRE concerns meet. SREs care about reassignment blast radius, rolling workflows, and consumer lag. FinOps cares about idle brokers, sticky storage expansion, and overprovisioning for peak. Broker-local Kafka often asks both groups to accept the same compromise: keep more headroom than average traffic needs because emergency scaling consumes capacity while it creates capacity.
FAQ
Does Amazon MSK support scaling?
Yes. Amazon MSK supports documented operations for expanding broker count, changing broker type, increasing Standard broker storage, and configuring automatic storage scaling. The important nuance is that these operations do not remove Kafka's need for partition placement, leader distribution, and data movement.
Does adding MSK brokers automatically rebalance existing partitions?
No. AWS points users to partition reassignment after adding brokers, and Apache Kafka documentation explains that added servers will not automatically be assigned existing data partitions. Without reassignment or new partitions/topics being placed on added brokers, the added brokers may not carry existing workload.
Why is MSK storage scaling one-way?
AWS documentation states that you can increase EBS storage per Standard broker but cannot decrease it. Automatic storage scaling can increase capacity in response to usage, but MSK does not reduce cluster storage when usage falls. Reducing storage requires migration to a cluster with smaller storage.
Is MSK autoscaling the same as Kafka autoscaling?
Not fully. MSK storage auto scaling addresses disk capacity for Standard brokers. General Kafka elasticity also needs traffic balancing, partition reassignment, broker headroom, consumer behavior, and sometimes topic partition changes. Those are Kafka-level concerns, not only infrastructure-level concerns.
How does AutoMQ reduce scaling friction?
AutoMQ uses a Kafka-compatible shared-storage architecture with stateless brokers. Because durable log data is stored in shared storage rather than tied to one broker disk, scaling and partition reassignment can focus more on ownership and traffic movement instead of copying full retained logs between brokers.
Should every MSK user move to shared storage Kafka?
No. Stable workloads with mature Kafka operations may be well served by MSK. Shared storage becomes more interesting when scaling speed, scale-in safety, storage elasticity, and peak-driven overprovisioning are recurring business problems. The right test is your workload's operational pattern, not a generic product comparison.
References
- AWS Documentation: Expand the number of brokers in an Amazon MSK cluster
- AWS Documentation: Scale up Amazon MSK Standard broker storage
- AWS Documentation: Automatic scaling for Amazon MSK clusters
- AWS Documentation: Amazon MSK quota
- AWS Documentation: Best practices for Amazon MSK Standard brokers
- AWS: Amazon MSK pricing
- Apache Kafka Documentation: Basic Kafka Operations: Expanding your cluster
- Apache Kafka Documentation: Basic Kafka Operations: Limiting bandwidth usage during data migration
- AutoMQ Documentation: Stateless Broker
- AutoMQ Documentation: S3Stream: Shared Streaming Storage
- AutoMQ Documentation: Partition Reassignment in Seconds
- AutoMQ Documentation: Scale-out/in in Seconds
- AutoMQ: Product site
If your MSK scaling runbooks are starting to look like capacity planning documents plus migration plans, the next useful step is architectural rather than cosmetic: compare broker-local Kafka operations with a Kafka-compatible shared-storage design, and test the difference with your own partition count, retention, and traffic pattern.