Table of Contents
Table of Contents
A rolling Kafka upgrade can finish with every broker reporting the target version while the upgrade is still unsafe. The hidden failures tend to appear later: a retained segment cannot be read, a controller quorum is not fully compatible, or a producer reconnects to an address that the load balancer never meant to expose. The version change is complete. The system change is not.
The useful definition of “zero downtime” is stricter. The workload must keep meeting its availability and data continuity objectives while brokers are changed, connections are reset, leadership moves, and the cluster returns to steady state. A reconnect, consumer-group rebalance, or retry can still occur during that process.
The test plan has to follow the failure path. Kafka upgrades cross three boundaries:
- Data layout: existing log directories, segments, indexes, checkpoints, and controller metadata must remain readable.
- Protocol state: old and target-version brokers, clients, and feature settings must coexist during the transition.
- Connection topology: clients must discover the right broker endpoints and reconnect through the network path that production actually uses.
1A rolling restart proves less than teams think
The Apache Kafka upgrade guidance is built around compatibility during a staged change. Operators normally upgrade brokers in a rolling sequence, keep the inter-broker protocol at a compatible level while versions coexist, and activate target behavior only after the cluster is ready. That sequence protects the control path, but it is not a complete production test.
A process-level check answers one narrow question: can this broker start with these files and join the cluster? It does not answer whether a consumer can fetch from an older retained segment, whether a compacted topic can continue cleaning, whether a client can reach every address returned in metadata, or whether the rollback path still works after a feature has been enabled.
| Boundary | State that can be hidden during a restart | What a useful test must prove |
|---|---|---|
| Data layout | Active and closed segments, index files, retention and compaction state, local checkpoints | The target binary can read the data that production will actually open after the restart |
| Protocol | Inter-broker requests, message format, feature flags, controller and broker version mix | Both versions can complete the operations the workload uses |
| Connections | Long-lived TCP sessions, DNS or load-balancer routing, advertised listeners, TLS names, idle timeouts | A client can discover, connect, fail over, and resume from the real network path |
| Rollback | Old binary, old protocol setting, data formats already activated, current leadership | The team knows which reversals are safe and where the plan becomes forward-only |
An upgrade procedure says what to restart. An upgrade proof shows that the state, contracts, and routes around that restart behave as expected.
2Test the data layout, not only process startup
Broker-local storage creates a broad test surface because the broker owns both the serving process and the durable files it must reopen. Kafka log data is organized into segments and associated indexes. A restart may reopen an active segment, a retention boundary, or a compacted topic with cleanup pending. A clean restart of a fresh topic says little about those cases.
A useful layout test matrix starts with the states that steady-state smoke tests avoid:
| Test state | Why it matters | Evidence to collect |
|---|---|---|
| Active segment receiving writes | The broker must reopen a file that was not at a natural roll boundary | Produce and fetch continuity, segment recovery logs, record ordering |
| Closed segment with indexes | Historical reads exercise more than the newest append path | Fetch from older offsets and verify the expected records are returned |
| Retention boundary | Deletion can race with restart and leave a different set of files to inspect | Topic remains readable at the intended retention boundary; no unexpected fetch errors |
| Compacted topic | Compaction changes which records and indexes survive over time | Key-based reads, cleaner progress, and consumer behavior after restart |
| Dirty or interrupted shutdown | Recovery code sees a different state than a graceful rolling stop | Broker recovery completes, leaders return, and the affected partitions serve reads and writes |
| KRaft metadata recovery | Controller snapshots and metadata logs are a separate durability path | Quorum members rejoin, metadata becomes current, and topic operations work |
The goal is to make retained data shape part of the test fixture. A staging cluster with only fresh topics has removed the state that makes an upgrade risky.
Treat data format changes as an explicit gate. Kafka exposes separate controls for inter-broker protocol and message format, so record what stays held back during the rolling phase and when it changes. “Restore the previous image” is incomplete if the cluster has crossed a format boundary that the old image cannot safely interpret. Read the message format reference and inter-broker protocol reference for the version path being tested.
Apply the same discipline to KRaft. Broker data and controller metadata are different recovery paths, so include controller restarts, quorum membership, metadata propagation, and a topic administration action. Serving an existing partition is insufficient if the cluster cannot commit a metadata change.
3Protocol compatibility needs its own matrix
“The client connected” is too small a protocol test. A client may open TCP and still fail on a versioned request, transaction, metadata refresh, or authentication negotiation.
Keep rolling and feature activation separate. Test the mixed-version state first, then test activation as a second change. This distinguishes a binary rollout failure from an activated-behavior failure.
A practical protocol matrix includes:
- An older producer writing to a target-version broker, including retries and the configured acknowledgment policy.
- A target-version producer writing while an older broker remains in the cluster.
- Consumers fetching existing data, committing offsets, and recovering after a broker connection closes.
- Admin operations that create or alter topics, quotas, and consumer groups while controller leadership changes.
- Transactional or idempotent workloads if the estate uses them, with the same producer and consumer client versions used in production.
- Connectors or stream processors that issue their own admin, metadata, or group-coordination requests.
For each row, capture request outcomes, client and broker logs, leader and ISR state, and when the client becomes healthy. These observations are evidence for the exact combination under test, not a universal recovery number.
Feature activation deserves separate approval. The target binary can run while the cluster stays at the older compatibility level, preserving a rollback window. When the setting changes, record the activation rollback boundary and rerun the client matrix.
4Load balancers do not carry Kafka failover for free
Kafka clients use bootstrap servers to discover the cluster, then use broker addresses returned in metadata. The broker's advertised listener configuration therefore matters more than the first connection address. A load balancer in front of one bootstrap address cannot repair a bad broker address in metadata.
A health check can reach the load balancer while the client receives an unusable broker endpoint. The address may work from an operator shell but fail from the application subnet, or its TLS name may not match the certificate. The broker can be healthy while the application is not.
Test the whole sequence over the production route:
- Drain or restart one broker while the workload is producing and consuming.
- Observe the existing client connection close, stall, or receive an error.
- Confirm that the client refreshes metadata rather than staying pinned to the old endpoint.
- Verify that the advertised broker address is reachable from the client network and passes TLS or SASL checks.
- Confirm that the producer resumes according to its retry and delivery policy and that consumers resume from their expected positions.
- Repeat the test with the load balancer's connection-draining and idle-timeout behavior enabled.
Long-lived Kafka connections make this matrix necessary. A load balancer can report a healthy backend while an existing socket still points at a draining broker. DNS may work for subsequent connections while existing sockets remain unchanged, and a per-broker address may work inside one network but fail in another. These are topology facts.
If the load balancer is only a bootstrap endpoint, say so in the runbook and verify that every advertised broker endpoint is independently routable. If it routes broker traffic, test each broker path for draining, health transitions, certificates, security groups, and reconnects. One successful bootstrap connection does not prove the cluster path.
5The shared-storage boundary changes the test, not the need for one
The deepest upgrade question is whether a broker restart also requires durable partition data to move with it. In a broker-local layout, compute replacement and data movement are coupled, so capacity and reassignment work can enlarge the recovery surface.
A Kafka-compatible Shared Storage architecture changes that boundary. AutoMQ documents S3Stream as a replacement for Kafka's native log storage, with object storage as the primary data layer and WAL for write acceleration and recovery. Brokers become stateless with respect to durable stream data, so an upgrade can focus on binary compatibility, metadata, clients, and storage backends instead of copying retained logs between broker disks. See the AutoMQ Shared Storage architecture and stateless broker documentation.
That shift removes one class of data-copy work, but it creates a clear test boundary. Verify the selected WAL backend, object-storage endpoint, credentials, network path, cache behavior, and KRaft metadata. Then run the same producer, consumer, protocol, and load-balancer tests. AutoMQ's Kafka compatibility documentation describes the model; your team still has to prove its own features and client versions.
This is the shared-storage boundary in practical terms:
| Still required in any architecture | Potentially reduced by Shared Storage |
|---|---|
| Mixed-version protocol tests and client reconnect tests | Copying retained partition data during broker replacement |
| KRaft metadata and controller-quorum validation | Treating each broker disk as the only recovery location for a partition |
| WAL, object-storage, credentials, and network checks | Coupling compute scaling directly to local data capacity |
| Load-balancer and advertised-listener validation | Assuming a changed broker must first receive the entire retained log |
Shared storage narrows the data-movement problem while leaving Kafka protocol and network contracts visible for testing.
6Rollback gates that make an upgrade reversible
Rollback works when each stage has a stopping condition and the team knows whether it is reversing a process, protocol, or data change. “Rollback if unhealthy” is not observable evidence.
| Gate | Pass condition | Stop or rollback trigger |
|---|---|---|
| Preflight | Backups or recovery artifacts, capacity headroom, client inventory, listener map, and version-specific compatibility settings are recorded | An owner cannot explain how to restore the current state |
| First broker | Target binary starts, reads its existing data, rejoins the cluster, and returns leadership or replica health to the expected state | Data read errors, persistent ISR loss, metadata errors, or an unreachable advertised endpoint |
| Mixed-version canary | Production-shaped producer, consumer, admin, and connector tests pass with old and target-version brokers present | Client errors, retry growth, offset or transaction failures, or controller instability |
| Full rolling pass | All brokers reach the target binary and the client reconnection test passes through the production network path | Any workload remains dependent on a broker or route that has not been tested |
| Feature activation | Protocol or format changes are approved, observed, and paired with a tested recovery decision | The team cannot state the activation point of no return |
| Closeout | Error rate, lag, leadership, storage access, and alerts are back to the agreed baseline | Operators cannot distinguish upgrade noise from a resulting steady-state failure |
Use the gates in order. If the first broker cannot reopen its data, a wider rollout multiplies the problem. If mixed-version clients fail, enabling the target protocol makes diagnosis harder. If the route fails only after connections drain, the rollback decision belongs to the network path even when broker metrics look clean.
The most useful rollback artifact is a one-page boundary map: last safe binary, held-back compatibility settings, activated formats, controller state, endpoint map, and the person who can authorize a forward fix. For shared storage, add WAL and object-storage access checks. A broker that starts without reading storage has not recovered.
7Make “zero downtime” an evidence claim
A Kafka upgrade is ready when retained data remains readable, mixed-version participants complete real operations, clients follow metadata through the production network, and rollback stops at a known boundary.
Zero downtime is a property of the tested workload and its failure path, not a synonym for a green health check after one broker restart.
If your runbook treats data layout and load-balancer behavior as external details, use the AutoMQ documentation to compare a Kafka-compatible Shared Storage model against your current gates. Start with the hardest retained-data case and least forgiving client route. That is where a version change becomes an architecture decision.
8References
- Apache Kafka upgrade documentation
- Apache Kafka broker configuration: advertised listeners
- Apache Kafka broker configuration: inter-broker protocol version
- Apache Kafka broker configuration: message format version
- Apache Kafka KRaft documentation
- Apache Kafka log directory configuration
- AutoMQ Shared Storage architecture
- AutoMQ Kafka compatibility documentation
- AutoMQ stateless broker documentation
9FAQ
9.1Is a rolling Kafka upgrade the same as zero downtime?
No. A rolling sequence can avoid a planned cluster-wide outage, but clients may still reconnect, retry, or rebalance. Treat zero downtime as a tested workload objective that includes data continuity, protocol compatibility, and client failover through the production network path.
9.2Which data-layout cases belong in a Kafka upgrade test?
Include active and closed segments, index files, retention and compaction boundaries, interrupted recovery, and KRaft metadata recovery. The exact matrix should reflect the topics and features the cluster actually uses.
9.3Do Kafka clients need a load balancer for failover?
Not necessarily. Clients need reachable broker addresses from metadata and a working reconnect path. If a load balancer is used for bootstrap or broker traffic, test advertised listeners, connection draining, idle timeouts, TLS names, and client metadata refresh together.
9.4Does Shared Storage remove Kafka upgrade testing?
No. It can remove or reduce the need to move retained partition data with a broker, but teams still need to test Kafka protocol behavior, KRaft metadata, WAL and object-storage access, client connections, and rollback boundaries.
