Table of Contents
Table of Contents
A blue-green Apache Kafka® migration can have a healthy target cluster and still lose an acknowledged record at the handoff. The failure usually happens between two statements that sound equivalent: “the producer received an acknowledgment” and “the target cluster is ready to serve that record.” They are different facts.
The source cluster may have accepted a record while replication is still in flight. A consumer may have committed progress on the source while the target has a different offset timeline. A downstream database may have applied a side effect before rollback. Changing a bootstrap server cannot settle those differences.
The safe way to run a blue-green migration is to make each boundary explicit: which cluster is authoritative for writes, which record boundary has reached the target, which consumer position is safe to resume, and which failures still permit a clean rollback. The goal is a sequence of proofs that protects every record the source acknowledged before authority changes.
1The endpoint switch is the smallest part
Blue-green migration borrows its useful idea from release engineering: keep the old environment available while the new environment is prepared and tested. For Kafka, that means the source cluster continues to serve production while a target cluster receives data, proves client behavior, and becomes ready for a controlled change of authority.
Migration is more than a copy job. A cutover plan must answer whether producers stopped at the right boundary, whether the target contains it, whether consumers resume from the intended record, and whether external side effects can be repeated safely.
The first design decision is write authority. Keep one authoritative write path during each phase. A replication path can feed the target while the source remains authoritative. An application-level dual writer sends the same logical event to both clusters, but the acknowledgments remain separate outcomes. Kafka producer idempotence and transactions do not turn two cluster writes into one atomic commit.
For every cutover wave, keep an auditable boundary containing the source topic, partition, producer sequence where available, source acknowledgment state, target observation state, and consumer resume position. The fields depend on the bridge and workload, but the question stays the same: can the team show where an acknowledged record is on both sides?
2An acknowledgment is a source boundary, not a migration proof
In Apache Kafka’s producer configuration, acks=all tells the producer to wait for the leader and all in-sync replicas required by the cluster’s replication state. The broker-side min.insync.replicas setting can reject writes when that in-sync set is too small. Those settings matter because they define what a successful source acknowledgment means inside the source cluster.
They do not say that a separate target received the record, that a consumer processed it, or that an external effect is reversible. Treat the acknowledgment as the left-hand boundary, then establish a target boundary before retiring the source.
This distinction changes the cutover question. “Is replication lag zero?” is useful, but it is not sufficient. A lag metric may omit a topic or hide a partition mapping error. The target proof should combine partition-level positions with record-level evidence appropriate to the workload, such as preserved producer sequence information, a stable event identifier, or a verified source-to-target mapping.
The same discipline applies to retries. Kafka’s idempotent producer settings can prevent retry-driven duplicates when their documented constraints are met. They cannot repair a cutover that sends a producer to the target before the last source-authoritative boundary arrives. Retry safety and migration safety remain separate controls.
3Four phases that make a cutover measurable
The four phases below define control gates. The replication mechanism can vary, but the evidence required at each gate should not be left implicit.
-
Dual feed with one write authority. Keep the source authoritative for new records and feed the target through the selected replication or linking path. If the application must write to both clusters, record each result and define partial-success repair. Keep target consumers stopped or isolated until side effects are intentional.
-
Catch up and compare. Measure progress per topic and partition. Compare source and target boundaries, then verify records where identity is observable. Check topic configuration, partition count, retention, ACLs, schemas, and producer errors. A target that has caught up on bytes but differs in configuration is not ready to become authoritative.
-
Freeze the moving parts. Freeze producer deployments, topic administration, retention, schema changes, and migration configuration before capturing the final boundary. Quiesce the source write path long enough to drain in-flight requests and record the final accepted position. A short pause buys a known boundary while the system changes.
-
Cut over in an owned order. Move producer authority to the target, confirm its acknowledgment policy, then hand off consumers using the prepared mapping. Release consumer groups in waves after their side effects are tested. Keep the source available and write-protected until the rollback exit criteria pass.
The order matters. Moving consumers before the producer boundary is stable can expose missing records. Moving producers before the target is proven can create target-only data that the source cannot recover. Changing client addresses does not complete the proof.
4Map offsets by record boundary
Kafka offsets are local to a topic partition. They are not global identifiers that automatically retain their meaning across clusters. If a migration path preserves every record in order for the same topic and partition, source and target offsets may align. That alignment is a property to verify, not an assumption to build into a runbook.
Create an offset map for each consumer group and each affected partition. At minimum, the map should make these relationships explicit:
| Mapping field | What it answers |
|---|---|
| Source topic and partition | Which ordered log produced the source position? |
| Source resume position | From which position would the source consumer read after its final commit? |
| Target topic and partition | Where does the same logical stream live after migration? |
| Target resume position | From which position should the target consumer start? |
| Evidence and owner | Which comparison or tool output proves the mapping, and who approved it? |
Be precise about whether a stored number means the last record processed or the next record to read. Document what happens when a topic is renamed, a partition is filtered, records are transformed, or a connector keeps its own checkpoint. Arithmetic translation may not express application continuity; a replay window, checkpoint export, or workload-specific reconciliation may be safer.
The offset values in the diagram are illustrative. Use the actual positions from the source and target, then verify that the mapped boundary identifies the same ordered records.
Test the handoff with a known boundary, not an empty consumer group. Start a canary consumer at the mapped position, verify its first records, and compare its downstream result with the source-side result. Idempotency may tolerate duplicates; it does not make silent gaps acceptable or make every external system safe to repeat.
5Consumer handoff is an application change
Consumers deserve a separate gate because consumption position and business progress are not always stored in the same place. A Kafka consumer group may commit offsets to Kafka, while a stream processor, connector, or application also persists checkpoints or side-effect state elsewhere.
For a companion readiness pass, see the dual-cluster validation guide and the consumer offset validation checklist.
Before starting the target-side group, decide which state is authoritative. Stop or drain the source-side group unless duplicate processing is an explicit test. Calculate the target resume position, start a controlled member, and verify Kafka progress and the external result before expanding the group.
Use an ownership table during the handoff:
| Owner | Handoff decision |
|---|---|
| Platform team | Target topic configuration, mapped positions, broker and client errors |
| Application team | Idempotency, transaction behavior, external checkpoints, side effects |
| SRE or change owner | Gate approval, freeze state, observation window, rollback call |
| Security or network owner | Credentials, ACLs, listeners, routes, and policy parity |
This division prevents a common gap: the platform team proves that records exist, while the application team assumes that the consumer will resume from the right business point. Both claims need evidence before the target is allowed to own production reads.
6Set rollback gates before switching authority
Rollback is easiest before the target accepts authoritative writes. Once the target has acknowledged new records or target-side consumers have created external effects, a route reversal is incomplete. The plan must say whether to reverse-replicate, replay from a retained boundary, reconcile side effects, or continue forward while fixing the target.
Use explicit gates instead of a single “rollback if unhealthy” sentence:
- Before source freeze: stop the migration path and resume the source while investigating the defect.
- After source freeze, before target authority: reopen the source only after confirming that no target consumer created effects from the frozen window.
- After producer cutover: roll back only if the target-only acknowledgment boundary can be transferred or replayed safely. Otherwise, use a forward-repair decision.
- After consumer handoff: include external effects in rollback state. A database write, notification, billing event, or view update may require reconciliation even when Kafka records are intact.
Define stop conditions before the change starts. A record boundary mismatch, an unexplained target write error, a consumer starting outside its mapped position, a schema or ACL mismatch, or a missing dashboard for a critical gate should stop the wave. Stop the wave whenever the team cannot prove the next irreversible step is safe, even if the cluster does not look unhealthy.
Rehearse failure paths while the source is authoritative: stop replication, restart a consumer at the mapped position, reject a target write, force a producer retry, and simulate a missing topic configuration. Repeat after the freeze gate so the team knows what remains observable before target-only history exists.
7Where a Kafka-Compatible Shared Storage target fits
The migration discipline applies to any target. Architecture changes the staging work around it. A target that couples broker capacity to local data may need more preparation for catch-up, validation reads, and rollback retention. Separating compute from persistent storage can make that preparation easier to adjust, but it does not remove offset proof or application testing.
This is where AutoMQ can enter the evaluation. AutoMQ is a Kafka-compatible cloud-native streaming platform. Its Kafka compatibility documentation describes the client-facing compatibility boundary, while its architecture documentation describes Shared Storage architecture and stateless brokers.
For migration planning, those properties change the target-side question from “how much broker-local data must be moved before testing?” to “how much compute and observation capacity is needed for this wave?” Data still needs to arrive, and client behavior still needs proof. The architecture cannot turn an unverified offset map into a safe handoff.
Teams moving to AutoMQ can read the migration overview and evaluate AutoMQ Linking within the supported boundaries. The test remains: can the team show the source acknowledgment boundary, target record boundary, consumer resume position, and rollback state for one workload?
The first wave should be small enough to observe and important enough to exercise real contracts. Use the same dashboards, ACLs, schemas, producer settings, consumer code, and downstream effects as the larger wave. A rehearsal that omits them is only a green diagram.
8A safer cutover starts with a known boundary
When an incident begins with “the target looked caught up,” the missing detail is usually the definition of caught up. A sound blue-green plan names the source authority, records the last acknowledged boundary, proves the target mapping, transfers consumers from a known position, and preserves a rollback path until the target has earned ownership.
If you are evaluating a Kafka-compatible target, apply this sequence to one production-shaped workload before expanding the move. To test AutoMQ in your own environment, start with AutoMQ and use the migration gates as acceptance criteria rather than treating the endpoint switch as the acceptance test.
9References
- Apache Kafka producer configuration, including acknowledgment and idempotence settings.
- Apache Kafka consumer configuration, including committed position and consumer start behavior.
- Apache Kafka replication design, including in-sync replica behavior.
- Apache Kafka cross-cluster data mirroring, for replication concepts and operational boundaries.
- AutoMQ compatibility with Apache Kafka.
- AutoMQ architecture overview.
- AutoMQ migration overview.
10FAQ
10.1What does blue-green Kafka migration mean?
It means the source and target coexist during preparation and validation. The source remains authoritative until the target passes data, client, consumer, and rollback gates. The switch changes authority in a sequence rather than treating a new bootstrap server as proof of readiness.
10.2Does acks=all guarantee no data loss during migration?
No. acks=all describes the source cluster’s acknowledgment boundary, subject to in-sync replica state and configuration. It does not prove target receipt or reversible downstream effects.
10.3Must source and target Kafka offsets match?
No. They need a verified mapping for each consumer group and partition. Numeric equality is safe only when the migration path preserves topic, partition, and record order. Renames, filters, transformations, and external checkpoints require different proof.
10.4Can consumers run on both clusters during the migration?
They can for a controlled shadow test, but the plan must define duplicate effects and group ownership. Running the same production consumer group on both clusters can process a logical record twice.
10.5Does a Kafka target with Shared Storage remove the need for migration rehearsal?
No. It can change how target compute and persistent data are coupled, but it does not validate acknowledgment boundaries, offset mappings, consumer state, schemas, ACLs, or external effects. Those remain acceptance criteria.
