Table of Contents
Table of Contents
An Apache Kafka client can connect to a target broker and publish a record while the migration is still unsafe. The producer may be using an old serializer, the consumer may be looking up a subject under a different name, and a replay job may still depend on schema IDs from the original registry. The broker accepted the bytes, but the application contract did not survive the move.
That is the boundary this runbook makes explicit. Redpanda’s Kafka compatibility is a broker and client question. Schema Registry migration is an application contract and ownership question that sits beside the Kafka data path. Treating the two as one check is how a green connectivity test turns into a decoding failure during cutover.
The migration is ready when the team can answer four questions for each important subject: which bytes are being written, which registry namespace resolves them, who may change the contract, and how a reader returns to the previous path. The answer should be tested with real serializers and retained records, not inferred from a platform label.
1Start with the two contracts
Kafka carries keys, values, headers, timestamps, and metadata. A registry usually governs the schema used to interpret a key or value, but the registry request is made by the producer or consumer application. A broker can be healthy while a producer fails to register a schema, or while a consumer reads a record and cannot resolve the referenced schema. The broker contract and the schema contract therefore need separate acceptance criteria.
| Boundary | What it proves | What it does not prove |
|---|---|---|
| Kafka broker and client | Bootstrap, authentication, produce, fetch, offsets, and the Kafka APIs used by the workload behave as required | Subject naming, schema IDs, compatibility settings, or registry credentials are equivalent |
| Schema Registry API | The application can register, look up, and check schemas through the target registry interface | Existing payloads can be decoded when IDs, subjects, references, or wire formats differ |
| Serialization contract | The exact serializer and deserializer can encode and resolve representative records | Ownership, approval, backup, or rollback responsibilities are assigned |
| Operating model | An owner can rotate credentials, restore metadata, approve changes, and run the cutover | A Kafka client connection alone is enough to migrate the workload |
The distinction matters even when the target exposes a Confluent-compatible registry API. Redpanda documents its Schema Registry surface and Schema Registry API as operational interfaces beside the Kafka data path. API shape is one compatibility layer. Subject naming, compatibility mode, reference handling, schema ID behavior, authentication, and retained data are separate layers that need their own evidence.
2Inventory the registry state before moving a producer
The first migration artifact is an export that another engineer can inspect without access to the running cluster. Capture the schema text and format for every production subject, then record the versions and references that a consumer may encounter in its replay window. Keep the source registry’s subject names, IDs, compatibility settings, and effective configuration scope alongside the export.
The inventory should also identify the code and operational dependencies around each subject:
- Which producers register schemas, and which only reference an existing ID?
- Which consumers use a specific subject naming strategy or record name?
- Which serializer and deserializer versions are deployed in each application?
- Which topics, replay jobs, sinks, and connectors can read the subject’s retained records?
- Which credentials, TLS trust stores, network routes, and rate limits protect registry access?
- Who approves a schema change, who operates the registry, and who owns rollback?
This is more than a documentation exercise. The export tells you whether the target registry has enough information to serve old records, while the ownership map tells you whether someone can make a safe decision when a registration or lookup fails. Keep both under change control and attach a date and source environment to each snapshot.
3Compatibility checks must include the subject, rather than the schema alone
Compatibility mode answers a directional question about writer and reader schemas. Backward compatibility asks whether the proposed reader can read records written with the previous schema. Forward compatibility asks whether an older reader can read records written with the proposed schema. Full compatibility requires both directions. A mode that passes in the source environment can still be the wrong mode for the rollout order or replay window.
Subject identity is the second half of that check. A subject may be derived from the topic, record name, or a custom naming rule. If the target client registers orders-value while an existing consumer requests com.example.Order, the schema definitions can be identical and the migration can still fail. Test the actual serializer configuration, including key subjects, value subjects, references, and any subject-level overrides.
Use a test matrix that exercises both the registry and the Kafka path:
| Test | Evidence to keep | Failure it catches |
|---|---|---|
| Register the current schema under the target subject | HTTP response, subject, version, and effective compatibility mode | Wrong endpoint, credentials, subject name, or policy scope |
| Read a retained record with the target consumer | Raw record fixture, schema lookup, decoded value, and error log | Missing ID mapping, reference, or serializer configuration |
| Register the next schema version | Compatibility response and reviewer approval | Source and target policies disagree, or the rollout direction is unsafe |
| Run source and target readers against old and target records | Four writer/reader combinations and replay result | Mixed-version or rollback window breaks even though the latest record works |
| Repeat with a key, tombstone, and referenced schema | Decoded key/value, null handling, and reference resolution | Key subject and deletion paths were omitted from the happy path |
Do not collapse these checks into “the registry is compatible.” A successful GET proves reachability. It does not prove that the producer registered the subject the consumer expects, that the target preserved the relevant compatibility policy, or that a retained record can be decoded.
4Treat schema IDs as registry-local until proven otherwise
Many Kafka serializers put a registry-specific schema identifier into the encoded value. A schema ID is usually meaningful inside the registry that assigned it; the same integer in a second registry does not automatically identify the same schema. Exporting schema text while ignoring IDs is therefore an incomplete migration plan.
There are three practical ways to handle this boundary, and the right one depends on the serializer and the cutover design:
- Keep the original registry available for old bytes. Producers can register target versions in the target registry, while consumers that replay old records continue to resolve the source IDs until the replay window closes.
- Create a controlled ID mapping. Preload equivalent schemas into the target only when the target’s ID and wire-format behavior are documented and verified. Store the mapping as an auditable artifact; do not assume matching registration order will produce matching IDs.
- Decode and re-encode at a deliberate boundary. Use a reader that can resolve the source registry, then write records with the target serializer and subject rules. Validate ordering, keys, headers, timestamps, and tombstones as part of the bridge.
The migration plan should name which path is used for live traffic and which path is used for historical replay. A target consumer that can read a canary record is not proof that it can read the oldest record still subject to retention. Keep representative encoded fixtures from each writer version and replay them before the source registry is retired.
5Make ownership a release gate
Registry migrations fail operationally when every team assumes another team owns the contract. Assign one owner for the registry service and one owner for each subject family, then write the handoff into the release plan. The service owner operates availability, network access, backups, upgrades, and restore drills. The subject owner approves meaning and compatibility changes. Application owners prove that their serializers, consumers, sinks, and replay jobs use the intended subject and endpoint.
A useful ownership record answers these questions before cutover:
- Who can change global and subject-level compatibility settings?
- Who can register a schema, and are emergency overrides audited?
- Who rotates registry credentials and updates application trust stores?
- Which team restores metadata, and how does a restored registry retain its contract history?
- Who decides whether to pause producers, keep the source registry read-only, or roll back the endpoint?
- Which alert shows a registration failure, a lookup failure, or an unexpected subject?
This division keeps a broker upgrade from becoming an unreviewed data-contract change. It also makes rollback possible: the person who can stop a producer is known, the person who can restore registry metadata is known, and the reader path has a tested source of truth.
For a deeper treatment of the data-contract side, see schema registry compatibility on Kafka-compatible stacks. If a bad version has already reached production, the schema rollback runbook explains why preserving version history is safer than deleting it.
6Roll out with a reversible reader window
The safest cutover sequence starts with readers because a reader that understands both the old and proposed contract can absorb a mixed window. Deploy target-aware readers with the compatibility policy and subject rules already tested. Then register the proposed schema, produce a bounded canary, and compare the decoded result with the source path before moving the full producer fleet.
Keep the source registry available for old records and for the rollback period. If the target producer is stopped after writing a record, the rollback reader must still be able to resolve that record or the team must have a tested decode and re-encode path. Do not delete old subjects or versions as a cleanup step while the retained data, rollback window, or forensic process still depends on them.
Use explicit gates rather than a clock-based cutover:
- Export and review subjects, versions, references, policies, and ownership.
- Validate target API, authentication, subject naming, and compatibility behavior with fixtures.
- Deploy readers that can handle the mixed writer window and verify replay of retained records.
- Register and produce a canary under the target contract; compare business fields, keys, headers, and timestamps.
- Shift producers in a controlled batch while watching registration errors, lookup errors, decode failures, and consumer lag.
- Run the rollback drill before removing source access or changing retention assumptions.
The rollback drill should answer a concrete question: if the target registry or producer release is withdrawn now, can a known reader still decode every record written during the canary and return to the source subject without changing the Kafka topic? If the answer depends on an undocumented ID or subject behavior, the gate is not passed.
7Evaluate the registry separately from the Kafka data plane
Once the registry contract is explicit, the broker migration becomes easier to test. A target Kafka-compatible platform can be evaluated with the same producer and consumer fixtures, while the registry remains an independently owned service or platform component. AutoMQ is one example of a Kafka-compatible data plane that can be tested at this boundary. Its Kafka compatibility is relevant to broker and client behavior; it does not make registry subjects, schema IDs, or ownership move automatically.
Use the same acceptance pack for the target data plane: producer and consumer client versions, transaction or idempotence settings when used, topic configuration, offsets, ACLs, keys, headers, retained records, and the exact serialization fixtures. Then run the registry contract suite separately against the registry implementation and endpoint that the applications will use. This separation keeps a broker result from hiding a registry result.
AutoMQ’s Kafka compatibility documentation describes the Kafka client boundary, while its architecture overview explains the storage layer beneath it. The useful question is narrow: do the Kafka clients, retained records, security controls, and recovery workflow pass your tests when the platform underneath changes? Schema ownership, compatibility policy, and registry rollback remain explicit parts of that evaluation.
8FAQ
8.1Does Redpanda Kafka compatibility include Schema Registry compatibility?
No. Kafka compatibility covers the broker and Kafka client surfaces that the workload uses. Schema Registry adds an HTTP API, subjects, schema versions, compatibility policy, references, credentials, and often a registry-specific ID in the serialized payload. Test those surfaces separately.
8.2Can I copy schemas from Redpanda into another registry and keep producing?
You can export schema definitions and register equivalent versions in a target registry when its format, policy, subject rules, and reference behavior have been verified. Do not assume copied definitions preserve schema IDs or make existing encoded records readable. Test the serializer and retained fixtures first.
8.3Is changing the registry endpoint a broker migration?
It is an application configuration and contract migration. The Kafka broker may remain unchanged while every producer and consumer changes its registry URL, credentials, subject naming, or serializer settings. Track it as a separate release gate so a broker smoke test cannot mask a registry failure.
8.4How long should the source registry stay available?
Keep it available for the longest retained record, replay job, rollback window, and forensic process that still needs its subjects or IDs. Replace that broad rule with a measured window from your topic retention and recovery plan, then prove the oldest required fixture can be decoded before retiring the source.
8.5What should the rollback plan preserve?
Preserve source and target endpoint configuration, subject and schema history, ID mappings if used, serializer versions, credentials, and the reader that can decode the mixed window. A rollback that only changes the broker bootstrap address is incomplete when the registry contract has also changed.
Return to the original migration test: a broker accepting a record is only the first green check. The real gate is a reader that can resolve the right subject, apply the intended compatibility rule, and decode both current and retained bytes under a named owner. When that evidence is ready, start an AutoMQ evaluation with the same Kafka client fixtures and registry contract tests, and let the data plane and schema boundary produce separate, reviewable answers.
