Schema Evolution on Amazon MSK: Compatibility Rules, Rollouts, and Recovery

A schema change on Amazon MSK is a deployment across time, not a one-line edit to an Apache Kafka event schema written in Avro or Protobuf. The old producer may still be writing while a new consumer is being rolled out. A replay job may read records from last week. A connector may use a different serializer from the service that owns the topic. If those readers and writers do not share a deliberate compatibility contract, a change that passed CI can still fail in production.

The useful question is not “Is this schema valid?” It is “Which versions must read which records, in what order will clients change, and how will the team recover if one consumer is late?” This guide turns that question into a rollout and recovery procedure for Kafka topics running on Amazon MSK.

1Start with contract ownership

Before choosing a compatibility mode, assign ownership for the event contract. The topic owner is responsible for the schema subject, the producer team owns the write path, and each consumer team owns its reader upgrade. Those roles can be held by one team, but they should still be explicit. A registry can reject an unsafe registration; it cannot decide whether an old dashboard, replay worker, or data lake job is still in the compatibility window.

Build an inventory for every topic that will change:

• Writers: service name, deployment cadence, serializer, schema format, and rollback version.
• Readers: consumer group, batch or streaming mode, expected fields, and maximum time an old version can remain deployed.
• Side paths: Kafka Connect or MSK Connect converters, stream processors, replay tools, and dead-letter consumers.
• History: retention period, compacted topics, archived records, and the oldest schema a backfill job may read.
• Authority: the team that can approve a compatibility exception and the person who owns the recovery decision.

This inventory changes the rollout design. A topic with one tightly coupled service can use a short migration window. A topic with dozens of independent consumers needs a compatibility rule that covers the slowest reader and the oldest replay path. For a broader review workflow, see Reducing toil in schema evolution reviews.

2Choose a compatibility rule that matches the rollout

Compatibility modes describe the direction in which schemas must remain readable. The names vary by registry, so verify the actual setting instead of assuming that two similarly named modes behave identically.

With backward compatibility, a new reader can read data written with the previous schema. This is a common fit for an additive producer change when consumers are upgraded before the new field is required. With forward compatibility, an old reader can read data written with the new schema. This helps when producers move first and old consumers remain during a transition. Full compatibility asks for both directions. A transitive variant checks the proposed schema against a longer history instead of only the immediately previous version.

AWS Glue Schema Registry documents modes including BACKWARD, BACKWARD_ALL, FORWARD, FORWARD_ALL, FULL, FULL_ALL, and NONE. Confluent Schema Registry uses names such as BACKWARD_TRANSITIVE and FULL_TRANSITIVE. The labels are useful shorthand, but the rollout obligation is what matters: identify the old and new readers, then test the actual serializer and registry configuration used in production.

For Avro, adding an optional field with a default is usually easier to roll out than removing or renaming a field. A default helps a reader materialize a value when it reads an older record. It does not make a semantic rename safe: a consumer that selects customer_id in SQL still fails if a producer starts emitting only account_id. Protobuf and JSON Schema have their own rules around field numbers, optionality, and unknown fields. Treat the format's rules as implementation details inside a broader contract policy.

A compatibility mode is also a retention decision. If records can be replayed for 30 days, a schema that is safe only against the immediately previous version may be too narrow. Use the transitive or “all history” setting when the replay window and client inventory require it, and record the reason in the topic's contract documentation.

3Use a staged producer and consumer rollout

The safest rollout sequence makes each stage observable and reversible. It avoids asking the registry to solve a deployment-order problem.

3.1Register and test the candidate

Register the candidate schema in a non-production subject or registry. Run compatibility checks against the versions that matter: the current production writer, the oldest supported reader, and any schema used by replay or backfill tooling. A unit test that serializes one record is not enough. Add fixtures from retained production traffic with sensitive fields removed, then decode them with both the old and new readers.

Check more than the schema document. Confirm the subject or schema name, serializer class, registry endpoint, credentials, and wire format. Glue serializers and Confluent serializers can use different envelopes and identifier conventions even when both carry Avro. A record written by one ecosystem should not be assumed readable by the other without an explicit interoperability test.

3.2Make readers tolerant first

Deploy readers that can understand both the current and candidate event shapes. Keep business logic stable: the new field can be ignored, defaulted, or routed to a safe fallback until the producer begins sending it. Watch deserialization failures, validation errors, and downstream projection errors by consumer group.

This is the point where MSK metrics help, but they do not prove schema compatibility. Consumer lag can remain flat while every record is being rejected after deserialization. Add application counters for decode errors, unknown fields, defaulted fields, and dead-letter writes. Alert on those counters alongside lag and throughput.

3.3Roll out writers in a backward-safe mode

Once readers are ready, deploy writers that emit the candidate schema without removing data that existing readers need. For an additive field, this often means writing the old fields plus the new optional field. For a rename, use a period of dual fields or a versioned event instead of silently replacing the old name.

Keep the previous writer artifact available. A rollback is only useful if the old writer can still register or use its schema and if the readers deployed in the previous stage can consume its records. Record the deployment order and the schema ID or version in the release evidence.

3.4Promote the new field deliberately

After the new writer is stable, activate consumers that depend on the new field. Do this by consumer group or service cohort so that one incompatible projection does not stop every reader. For a high-value topic, shadow-read a sample into a validation process before changing the system of record.

Do not remove the old field as soon as the first new consumer succeeds. Wait for the documented compatibility window: the longest rollback period, the slowest consumer deployment, and the replay or retention period that must remain readable.

3.5Close the window

Only after evidence shows that old writers and readers are gone should you register a breaking schema or remove the compatibility exception. Keep an archived copy of the prior schema and the migration decision. “No current consumer uses the field” is different from “no retained record or replay tool needs the field.”

4Verify the side paths that production hides

The most common schema incidents happen outside the main service path. Kafka Connect converters may be configured at worker scope while a connector overrides them. An MSK Connect plugin may bundle an older serializer. A Flink job may restore state from a savepoint created with a previous schema. A batch replay may bypass the service's normal registry client.

Run a compatibility canary for each client class:

Network and identity are part of the test. A service in a private subnet may reach MSK brokers but fail to reach the Glue API or an external registry endpoint. An IAM role that can consume from the topic may not be allowed to register or read schemas. Test from the same subnet, role, and container image used in production.

5Recovery when a rollout fails

A failed schema rollout needs a decision tree, not a generic “roll back the deployment” command. First classify the failure:

1. Registration rejected: the registry compatibility policy blocked the candidate. Keep the old writer, inspect the incompatible change, and either revise the schema or approve a documented exception.
2. Deserialization errors: the schema registered, but one reader cannot interpret the record. Stop promotion for that cohort, route failures to a monitored dead-letter path if safe, and restore the last reader version that can process both shapes.
3. Semantic errors: records decode but produce incorrect projections, defaults, or business decisions. Fence the affected writer or consumer, preserve the input topic, and replay into a validation topic after the contract is corrected.
4. Registry access failure: clients cannot reach or authenticate to the registry. Treat this as a control-plane outage. Do not “fix” it by disabling compatibility checks in production; restore the endpoint path, credential, or role and verify with a canary.

Keep the source topic immutable while investigating. Kafka retention and offset controls give the team a recovery window, but only if consumers are paused or redirected before they advance past records that need reprocessing. That window should be measured as an operating contract, as described in Consumer Replay Windows: Recovery Planning Beyond Lag Metrics. If the consumer has already committed offsets after a semantic error, use a new validation group and a bounded offset range rather than resetting a shared production group without an audit trail.

A recovery test should prove four things: the old reader can still consume old records, the new reader can consume both versions, the writer can return to its previous artifact, and a replay job can reconstruct the expected output. Capture offsets, schema versions, deployment timestamps, and the final consumer-group state as evidence.

6Where AutoMQ fits

Schema evolution is a contract concern, so changing the Kafka-compatible data plane does not remove the need for a registry, serializer, or rollout policy. AutoMQ is a Kafka-compatible streaming storage platform with a shared-storage architecture and stateless brokers. It can be evaluated as a platform option after the event contract, client matrix, and replay procedure are already defined.

The useful test is end to end: run the same producer and consumer binaries, registry credentials, schema versions, retention window, and recovery exercise against the target platform. Kafka protocol compatibility helps only when the surrounding schema and client assumptions continue to hold. Keep the registry decision separate from the broker decision so that a Glue-centered AWS contract or a cross-platform registry remains an explicit architectural choice.

7A reviewable rollout checklist

Before changing a production schema on Amazon MSK, ask:

• Who owns the contract, approves compatibility exceptions, and signs off the rollback window?
• Which writers, readers, connectors, processors, and replay jobs must be compatible?
• Which registry mode and history depth match the retention and replay window?
• Have candidate records been tested with real serializer and registry clients from each network class?
• Are readers tolerant before writers emit the new shape?
• How will the team pause, fence, or redirect a failing cohort without deleting source records?
• What evidence proves that the old schema can still be replayed?

The answer is a production-ready schema rollout only when the team can demonstrate the return path. A compatibility setting prevents some unsafe registrations; it does not replace staged deployment, client observability, or recovery evidence.

7.1FAQ

7.1.1Does Amazon MSK enforce schema compatibility?

No. Amazon MSK provides the Kafka broker data plane. Compatibility is enforced by the schema registry and the producer or consumer libraries that use it. Configure and test the registry separately from MSK authentication, ACLs, retention, and replication.

7.1.2Which Glue Schema Registry mode should I use?

Choose the mode from the reader and replay contract. Backward, forward, full, and transitive or all-history variants impose different obligations. Verify the exact mode supported and configured in your registry, then test it with the serializers and historical versions used by production.

7.1.3Is adding a field safe for MSK Avro consumers?

It can be safe when the field is optional, has an appropriate default, and the reader and writer order matches the selected compatibility rule. Test retained records and downstream projections; an Avro-compatible change can still be a breaking semantic change for SQL, analytics, or alerting logic.

7.1.4Should producers or consumers deploy first?

For an additive change, deploy tolerant readers first, then writers, then consumers that require the new field. A different order may be valid for a forward-compatible contract, but it must be proven with the actual client versions and rollback path.

7.1.5What should I do when a schema rollout fails?

Keep the source topic intact, stop promotion, classify the failure, and restore a reader or writer version that can process the existing records. Re-run a bounded replay with a new validation group after correcting the contract, and record offsets and schema versions before resuming normal consumption.

If you are evaluating a Kafka-compatible platform for the same contract and recovery requirements, review AutoMQ's architecture.

7.2References

• AWS Glue Schema Registry developer guide
• AWS Glue Schema Registry schema evolution and compatibility
• AWS Glue Schema Registry integrations
• Apache Avro specification: schema resolution
• Confluent schema evolution fundamentals
• Streamlined Kafka schema evolution with Amazon MSK and Glue Schema Registry
• Schema evolution with Glue Schema Registry and Amazon MSK