Table of Contents
Table of Contents
When a Kafka producer starts blocking in send(), the reflex is often to add brokers. That can be the right answer when broker capacity is the limiting resource. It is the wrong first move when a short burst has filled a client buffer, one tenant is consuming its fair share, or the application is offering work faster than the pipeline can accept it.
Kafka backpressure is the control signal that appears when offered load reaches a boundary. The boundary may be producer memory, a broker request or network budget, a storage write path, or a quota assigned to a client group. A healthy system makes that signal visible, bounds how long callers wait, and slows the source at a rate the downstream path can sustain. Treating every pressure signal as a scaling failure adds cost and concurrency while making the queue harder to drain.
1Backpressure is a control mechanism, not a diagnosis
Backpressure describes a direction of information flow. The broker cannot accept produce work as fast as the producer offers it, so the client accumulates records, waits for capacity, or receives a throttle. That signal can travel through the producer thread into an application queue, an HTTP worker pool, a job scheduler, or the service that feeds the producer. The first visible symptom may be a full buffer or rising send latency even though the limiting resource is several hops away.
That makes backpressure different from consumer lag. Consumer lag measures the distance between records available in a Partition and a consumer group's progress. Producer backpressure happens before a record is accepted into the Kafka log, so a producer can be under pressure while consumers are caught up. The two can coexist, but one metric does not diagnose the other.
It is also different from a retry storm. Retries add work after requests fail or time out; backpressure is the deliberate reduction of offered work when the receiver cannot safely accept more. A retry policy can amplify pressure, but a bounded producer queue and a rate limit can prevent the initial mismatch from becoming a second workload.
A useful operating definition is goodput: the rate of records that the application successfully sends and the broker accepts within the required delivery and memory budget. A higher attempted send rate is not higher goodput if it produces long waits, repeated failures, or a queue that cannot catch up. Keep accepted work moving at a sustainable rate, then add capacity when the evidence says capacity is the constraint. When consumer capacity is part of the service contract, this control keeps accepted work within a rate the downstream path can process. It does not guarantee that consumers will never lag.
2The pressure chain runs from broker to producer to upstream
A produce request crosses limits at the broker and the client before the application sees success. The broker processes the request, routes it to the Partition leader, applies any quota, and completes the acknowledgment path. The client holds the Record in memory, forms batches, maintains in-flight requests, and waits for the response. When broker service rate falls below offered rate, the difference has to live somewhere.
In a Java KafkaProducer, buffer.memory is the memory available for Records waiting to be sent. Apache Kafka describes it as a rough guide rather than a hard process-memory ceiling because compression and in-flight requests use additional memory. When Records arrive faster than delivery, the producer waits for buffer space for up to max.block.ms, then fails the call with an exception. For send(), max.block.ms also bounds time spent waiting for metadata, so a timeout needs to be read with metadata, request latency, and buffer metrics.
The pressure continues upstream when the caller waits or receives an error. A web request can hold a worker, a stream processor can stop fetching input, or a scheduler can accumulate jobs. That is often the desired signal: the source learns that its downstream contract is full. The operator's job is to make the wait bounded and observable, then choose the scope of control.
Measure three rates and queues:
- Offered rate: Records or bytes submitted, including application queues.
- Accepted rate: Records or bytes acknowledged under the chosen delivery semantics.
- Waiting work: Buffer usage, queue depth, send wait time, quota throttle time, and failed sends.
A brief gap that closes may need burst absorption. A persistent gap with rising broker service time across producers points toward capacity. One throttled client group with healthy peers points toward fairness.
3Tune the producer side as a control loop
The producer settings work together. Use one for burst memory, one for caller waiting, and one for shaping offered load. Changing a value without deciding what happens at the next boundary usually makes the failure slower rather than safer.
3.1Use buffer.memory for bursts, not a permanent mismatch
A buffer absorbs a burst when the broker catches up before the memory budget is exhausted. A rough planning model is:
required buffer ≈ (incoming rate - accepted rate) × burst duration
Use the positive rate difference only while incoming work exceeds accepted work, and leave room for compression, in-flight requests, serialization, and the rest of the process. A larger buffer changes how long the producer can hide a mismatch; it does not change long-term broker service rate. If the mismatch persists, more Records wait and memory and freshness risk increase.
3.2Use max.block.ms to protect the caller's deadline
Choose max.block.ms from the application contract. A bounded request handler should not hold a worker for most of its deadline while waiting for Kafka. A batch job may accept a longer wait when its queue and error behavior are designed for it. The timeout path needs a policy such as reject, shed, pause upstream input, or retry under a separate budget.
Raising the value can reduce immediate exceptions during a brief shortage, but can hide pressure. Lowering it exposes overload sooner, but can turn a recoverable pause into application errors. Use the value that fits the business deadline and recovery action.
3.3Rate-limit the source that changes offered load
An application-level rate limiter shapes a known producer fleet or planned burst. Limit Records, bytes, or both, and scope the limit to a tenant, job type, endpoint, or process. A token bucket can allow a short burst while keeping the long-term rate near the measured goodput envelope.
React to send wait time, failed sends, quota throttle time, and broker produce latency. Use gradual changes and a clear upper bound; a limiter that swings between zero and an unbounded rate can create oscillation. Keep it close to the source because the broker sees useful and abusive bursts only after both consume broker work.
| Control | Primary job | Boundary |
|---|---|---|
| Application rate limiter | Shape offered Records or bytes | Cannot enforce fairness across applications |
buffer.memory | Absorb a bounded producer burst | Does not increase broker capacity |
max.block.ms | Bound producer caller wait | Does not drain a full buffer faster |
| Broker quota | Protect shared broker resources | Does not remove CPU, network, or storage limits |
4Broker quotas provide fairness at the cluster boundary
A local rate limiter protects one application. It cannot protect other tenants when independent producer fleets share a cluster. Kafka broker quotas fill that gap by limiting resources for a client group. Apache Kafka documents network bandwidth quotas as byte-rate thresholds and request rate quotas as a percentage of request-handler and network-thread time. Clients that exceed the configured quota are throttled.
Use quotas when the question is “who should get how much of a shared resource?” User and client.id groups can give an ingestion service a bounded share or contain a noisy batch job. Identity must be deliberate: different identities can fragment policy, while one shared identity can let a tenant consume the group's allowance.
Quotas are enforced per broker. Uneven Partition placement or concentrated traffic can leave one leader hot while the quota remains fair at the client-group boundary. Inspect traffic distribution before concluding that a larger cluster or a higher quota will improve goodput. A quota is a fairness contract and safety ceiling, not a replacement for source shaping or capacity planning.
5Scale when capacity is the constraint, slow down when load is
Add broker or Partition capacity when the produce path is consistently saturated, the workload needs a higher accepted rate, traffic can use added parallelism, and the change affects the limiting resource. Scaling does not help a single hot Partition if the same key pattern still sends work to one leader.
Slow producers when the mismatch is temporary, intentional, or local. A scheduled export, a tenant over its share, a downstream system that cannot accept more events safely, or too many concurrent sends can all be cases where slower offered load preserves more useful work than more brokers. The decision is about where to place the queue and who owns the wait.
The curve is a mental model, not a benchmark. Near the service boundary, accepted goodput approaches a plateau while waiting time and queue depth rise. Beyond it, extra input buys pressure. Scaling is justified when that plateau is below the required goodput and the bottleneck can use more capacity.
6What AutoMQ changes, and what it does not
The control loop remains necessary on a Kafka-compatible platform. AutoMQ uses a Shared Storage architecture that separates broker compute from durable stream storage, with object storage as the persistent data layer and WAL storage according to the deployment model. That can change the pressure boundary when broker-local data ownership, local log capacity, or data movement during broker changes is the recurring constraint.
Shared storage does not remove flow control or make object storage, WAL storage, network paths, broker CPU, quotas, or producer memory unlimited. A producer can still fill buffer.memory, wait for max.block.ms, or receive a broker throttle. A workload that writes faster than the selected storage and broker path can sustain still needs to slow down, shed work, or add capacity at the limiting layer.
AutoMQ is relevant when broker scaling is repeatedly entangled with moving durable Partition data or sizing local storage for retention. With Kafka-compatible clients and a more stateless broker model, teams can test compute scaling and storage pressure as separate hypotheses. Compare offered rate, accepted goodput, producer wait time, throttle time, broker latency, WAL or object-storage write behavior, and the effect of adding compute. The AutoMQ architecture documentation describes the storage and broker model; use your producer workload to verify where pressure appears.
7A practical decision loop for a full producer buffer
When producers block or are throttled, walk the path in this order:
- Locate the queue. Check producer buffer availability, application queue depth, send wait time, and broker throttle time. Do not infer consumer lag from a producer-side symptom.
- Compare rates. Put offered Records or bytes beside accepted Records or bytes over the same interval. A widening gap means the path accepts less than the source offers.
- Scope the pressure. Compare clients, users,
client.idgroups, brokers, and Partition leaders. One noisy group calls for fairness control; a cluster-wide rise calls for a capacity investigation. - Choose the control. Use an application limiter for source shaping, a broker quota for fairness,
buffer.memoryfor a bounded burst, andmax.block.msfor a bounded wait. - Scale with evidence. Add capacity only when metrics show a saturated resource that can use the added parallelism. Check whether accepted goodput improved.
This keeps scaling in its proper role. It raises sustainable service capacity; it does not hide a workload with no safe place to wait.
Backpressure has done its job when producers stay within memory and latency budgets, noisy clients are contained, and accepted goodput remains close to what the downstream path can sustain. If a short burst fills the buffer, slow the producer before buying permanent capacity. If broker latency stays high across clients and the workload needs more accepted rate, scale the limiting resource. If local storage coupling makes that decision expensive, evaluate a storage architecture that changes the capacity boundary while keeping the control loop visible.
If you want to test that boundary with the Kafka clients and producer policies you already run, try AutoMQ on GitHub. Reuse the same load shape, quota policy, buffer.memory, max.block.ms, and goodput checks so the result answers an operational question rather than a marketing one.
8FAQ
8.1What is Kafka backpressure?
It is the signal that producers offer Records faster than the broker path can accept them. It can appear as producer buffering, blocking in send(), broker throttling, application queue growth, or upstream rate reduction.
8.2What causes a Kafka producer buffer to fill?
The buffer fills when Records arrive faster than the client can deliver them. Broker latency, storage or network pressure, quotas, metadata delays, and too much producer concurrency can create the mismatch. A larger buffer absorbs a short burst but cannot raise long-term accepted rate.
8.3Should I increase buffer.memory or max.block.ms?
Increase buffer.memory for a bounded burst that the broker can drain within the memory and freshness budget. Set max.block.ms from the caller's deadline and the action after timeout. Pair either change with offered-versus-accepted rate and send-wait measurements.
8.4When should I use Kafka broker quotas?
Use quotas when independent producers share a cluster and one client group needs a resource ceiling or fair share. Use an application rate limiter when one source needs smooth traffic shaping. Quotas protect the cluster boundary; they do not replace local backpressure handling.
8.5Does AutoMQ eliminate Kafka backpressure?
No. Shared Storage can change the relationship between broker compute, durable data, and scaling work, but producers still face memory, network, storage, quota, and broker capacity limits. The same producer control loop remains necessary.
