Table of Contents
Table of Contents
A Debezium connector can be running while a CDC pipeline is already losing its recovery options. The database may be retaining WAL for a connector that stopped consuming. The connector may have committed an offset before a downstream consumer finished a batch. An MSK Connect worker may be healthy while its subnets are running out of IP addresses. These failures look unrelated in separate dashboards, but they all come from the same mistake: treating CDC as a connector deployment instead of a chain of stateful contracts.
A production setup has to make five pieces agree: PostgreSQL’s logical replication position, Debezium’s snapshot and event semantics, Kafka Connect’s offset storage, Amazon MSK’s topic and security model, and the consumers that turn events into business state. The useful design question is not “Can Debezium connect to MSK?” It is “If the connector pauses, restarts, or falls behind, which state is authoritative, and how do we prove that no change was silently skipped?”
This guide uses PostgreSQL as the source and Amazon MSK Connect as the managed Kafka Connect runtime. It focuses on the checks that matter after the first successful event: snapshot boundaries, offsets, schema handling, network and IAM paths, backpressure, and recovery evidence.
1CDC semantics come before connector settings
PostgreSQL CDC starts with logical replication. Debezium reads committed row-level changes from PostgreSQL’s write-ahead log through a logical replication slot, then emits Kafka records for inserts, updates, and deletes. The slot is a server-side cursor over change history; it is not a backup. If a connector stops advancing the slot, PostgreSQL can retain WAL that no other consumer needs, so slot health belongs in the database operations runbook.
The first connection has a different contract from steady-state streaming. Debezium’s PostgreSQL connector takes a consistent snapshot by default, records the position associated with that snapshot, and then continues streaming from that point. Changes committed while the snapshot runs are therefore covered by the handoff. A restart before the initial snapshot completes can start the snapshot again, which is why a snapshot interruption should be treated as a planned state transition rather than as an ordinary task retry.
That distinction changes how you test the pipeline. A few rows arriving in a topic proves connectivity, not correctness. A useful test writes a row before the snapshot, updates it during the snapshot, deletes another row after streaming begins, and then verifies both the emitted event order and the final materialized state.
2Build the topology around ownership boundaries
Amazon MSK Connect runs connector logic in managed workers. AWS describes a worker as a Java Virtual Machine process that runs connector logic and tasks; workers coordinate through Kafka Connect consumer groups and can rebalance tasks when workers scale or fail. The connector still needs reachability to the source database, the MSK cluster, and the internal Kafka topics used by Connect and Debezium.
A production topology should make each network hop visible:
- Source path: MSK Connect worker subnets and security groups must reach the PostgreSQL endpoint and replication port. For Amazon RDS or Aurora PostgreSQL, keep the database private and allow only the worker security group to connect.
- Kafka path: The worker execution role, cluster authentication mode, topic ACLs or IAM policies, and TLS settings must agree. An authentication failure can look like a connector failure even when the database path is healthy.
- State path: Kafka Connect stores connector offsets in its offset storage topic. Debezium may also use connector-specific state such as a schema history store, depending on the source connector and configuration. These topics need their own retention, compaction, access, and recovery checks.
- Capacity path: MSK Connect workers consume IP addresses from the customer-provided subnets. Autoscaling a connector can therefore fail because of subnet exhaustion before CPU becomes a problem.
An independent MSK Connect private-subnet troubleshooting guide reaches the same operational boundary from field practice: inspect routes, security groups, DNS, and subnet capacity before treating a connector error as a Debezium bug. Use that third-party walkthrough as a cross-check, then verify each permission and network requirement against the AWS documentation for your account.
MSK Connect custom plugins are copied from the S3 object when the plugin is created. AWS documents that a custom plugin cannot be updated in place; a new plugin version requires replacing the dependent connectors and plugin resource. Treat the Debezium artifact, its dependency set, and the connector configuration as a versioned release bundle. A “small” JAR update can otherwise become a connector recreation during an incident.
For IAM-authenticated MSK clusters, the Kafka client settings need to match the AWS MSK IAM authentication library. The following is the client-side shape; the MSK Connect service execution role and cluster data-plane permissions still have to be configured separately according to the MSK Connect IAM guide and the MSK IAM access-control guide.
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandlerKeep the Debezium connector configuration explicit as well. The following properties show the state-bearing parts; credentials, endpoint names, and topic naming should come from your deployment configuration.
connector.class=io.debezium.connector.postgresql.PostgresConnector
tasks.max=1
database.hostname=<private-rds-endpoint>
database.dbname=<database>
database.user=<replication-user>
plugin.name=pgoutput
topic.prefix=<source-name>
slot.name=<unique-replication-slot>
slot.drop.on.stop=falseThe same settings do not fix a missing route or a blocked security group. Check DNS resolution, subnet route tables, security-group rules, network ACLs, and the MSK bootstrap endpoint from the worker network. A connector log that says “connection refused” is a symptom; the evidence is the path test from the actual worker subnets.
3Keep snapshot, offset, and schema state separate
Snapshot state and streaming state answer different questions. Snapshot state says which tables and rows have been copied. The Kafka Connect offset says where the connector can resume reading the PostgreSQL change stream. The replication slot says how far PostgreSQL must preserve WAL. Recovery is safe only when those three positions are understood together.
Kafka Connect’s distributed runtime persists offsets in a Kafka topic, and Kafka’s documentation requires that offset storage preserve the latest value for each connector partition. Debezium’s state-storage guide also describes compacted Kafka topics for offsets and, for connectors that require it, internal schema history. Do not delete or recreate the Connect internal topics as a cleanup step. Losing offsets can turn a restart into a new snapshot or a replay from an unexpected log position.
PostgreSQL deserves a specific schema-history caveat. Debezium’s storage documentation lists MySQL, Oracle, SQL Server, and Db2 as connectors that require internal schema history to reconstruct database schema changes. The PostgreSQL connector has different semantics because PostgreSQL logical decoding and the connector’s schema handling provide the needed source context. Copying a MySQL schema.history.internal.kafka.topic configuration into a PostgreSQL deployment without checking the connector version can create a false sense of protection. Record the connector version, the exact configuration, and which state stores it actually uses.
Use an ownership table during design reviews:
| State | System of record | Failure question |
|---|---|---|
| Source change position | PostgreSQL logical replication slot | Is WAL being retained because the connector is not advancing? |
| Connector resume position | Kafka Connect offset storage | Which source position will the next task use? |
| Snapshot completion | Debezium snapshot status and offsets | Did the snapshot finish, or will the connector repeat it? |
| Schema interpretation | Connector-specific schema state and source metadata | Can the connector decode the next event after a DDL change? |
| Consumer progress | Kafka consumer group offsets | Which downstream records have been applied? |
The table is more useful than a single “connector healthy” alarm. It tells an on-call engineer which state to inspect before restarting anything. For a wider CDC-to-table view, compare it with the platform engineer’s CDC-to-table pipeline guide.
4Design backpressure as a bounded failure
CDC backpressure starts at the database and moves downstream. A slow sink or consumer increases Kafka topic lag; a slow connector increases the replication-slot lag and WAL retained on PostgreSQL. The pipeline can remain connected while the distance between those positions grows until the database runs out of safe WAL headroom.
For an initial snapshot, measure database read load, snapshot duration, connector task throughput, and the point at which streaming catches up. Debezium’s PostgreSQL documentation describes chunk-based parallel snapshots and incremental snapshots, but those features do not remove database contention or downstream capacity limits. Enable them only after testing table keys, long-running transactions, and the effect on the source database.
For steady-state changes, watch a small set of linked signals instead of tuning one number in isolation:
- PostgreSQL replication-slot lag and retained WAL identify pressure at the source.
- Debezium streaming metrics and task errors identify connector-side stalls.
- Kafka producer request latency and topic traffic identify the write path into MSK.
- Consumer-group lag and processing latency identify downstream backpressure.
- MSK Connect worker capacity, task rebalances, and available subnet IPs identify runtime pressure.
Increasing tasks.max is not a universal fix for a PostgreSQL source connector. The PostgreSQL connector configuration defaults this property to 1, and the source stream is tied to a logical replication slot. Parallelism is usually gained by separating independent source databases or connectors, or by tuning snapshot behavior, not by assuming that more tasks will create more WAL readers.
A good backpressure policy has a stop condition. Pause a rollout when slot lag, consumer lag, or database load crosses the threshold agreed in the service objective. Do not solve a sink outage by dropping the replication slot or deleting the topic that contains the backlog. Those actions remove recovery evidence while leaving the original failure unresolved.
5Recovery checks that produce evidence
A restart is not a recovery test. It is only one action in a recovery sequence. Before restarting, capture the connector configuration version, task state, source slot name, slot lag, Connect offsets, topic partition counts, and consumer-group positions. That snapshot gives you a before-and-after comparison when the task resumes.
During recovery, check these transitions in order:
- Network and identity: the worker can resolve and reach PostgreSQL and MSK, and its execution role still has the required permissions.
- Source position: the replication slot remains present and advances after the connector resumes.
- Connector position: the task resumes from the expected Connect offset instead of silently starting a new snapshot.
- Event continuity: a known update or delete appears once, with the expected key, source timestamp, and operation field.
- Consumer convergence: downstream consumers catch up without an unexplained gap or duplicate side effect.
If the offset is no longer available, the source slot was dropped, or the source database was promoted, stop and choose an explicit recovery mode. A new snapshot may be correct for a rebuild, but it is not the same as resuming an interrupted stream. Document the choice, because downstream consumers need to know whether to rebuild, replay, or deduplicate.
6Verification queries for a PostgreSQL-to-MSK runbook
Verification should use records with known identities, not only aggregate counts. Insert or update a test row in a controlled table, record the transaction timestamp and primary key, and follow it through the Debezium topic and one representative consumer. For deletes, verify the tombstone behavior expected by the consumer and its compaction policy.
The following SQL is a starting point for source-side checks; adapt it to the PostgreSQL version and permissions in your environment.
-- Confirm logical replication is enabled for the instance or cluster.
SHOW wal_level;
-- Inspect the slot used by the Debezium connector.
SELECT slot_name, active, restart_lsn, confirmed_flush_lsn
FROM pg_replication_slots
WHERE slot_name = 'your_debezium_slot';
-- Check active replication clients during a test.
SELECT application_name, client_addr, state, sent_lsn, write_lsn,
flush_lsn, replay_lsn
FROM pg_stat_replication;Compare the source-side LSN movement with the connector’s task metrics and the Kafka consumer offset. A count match alone can hide a missing update followed by a compensating update. If the pipeline has data-quality SLOs, the CDC data-quality SLO guide gives a useful companion checklist. Keep a small verification dataset with inserts, updates, deletes, a schema change, and a restart. That dataset becomes a regression test for Debezium upgrades, plugin replacements, security changes, and MSK Connect worker changes.
7Where AutoMQ fits, and where it does not
The CDC pipeline still has to prove source semantics, connector state, and consumer behavior before its Kafka storage layer becomes the next decision. If long retention and replay-heavy consumers make broker-local storage the limiting resource, a Kafka-compatible shared-storage platform becomes a reasonable architecture option to evaluate. AutoMQ keeps the Kafka protocol while using shared object-storage-backed streaming storage and stateless brokers; that can change how retained CDC history, broker scaling, and recovery data movement are coupled.
That is a boundary, not a promise that AutoMQ fixes a broken connector. PostgreSQL WAL retention, Debezium snapshot semantics, IAM, network paths, and consumer idempotency remain part of the system. Evaluate the storage layer only after the CDC scorecard is passing, then repeat the same snapshot, pause, replay, and consumer-convergence tests against the target platform.
When a CDC pipeline is ready for that comparison, use the AutoMQ architecture documentation and run the workload inside your own AWS account. If you want to test a Kafka-compatible shared-storage design, start with AutoMQ BYOC.
8FAQ
8.1Can Debezium PostgreSQL run on Amazon MSK Connect?
Yes. Package the Debezium PostgreSQL connector as an MSK Connect plugin, provide a worker configuration that can reach both PostgreSQL and MSK, and grant the service execution role the permissions required by the selected MSK authentication mode. Test the plugin and dependency bundle as one versioned artifact.
8.2Does Debezium PostgreSQL need a schema history topic?
Do not assume the MySQL configuration applies. Debezium’s state-storage documentation distinguishes connectors that require internal schema history from PostgreSQL’s logical-decoding path. Check the connector version and document the state stores actually used by your deployment rather than creating an unused topic and calling the schema problem solved.
8.3What happens if the connector stops during the initial snapshot?
Debezium can start the snapshot again when it restarts before the initial snapshot completes. Treat that as an expected recovery branch: verify source load, snapshot progress, Connect offsets, and downstream duplicate handling before restarting repeatedly.
8.4Is tasks.max enough to scale PostgreSQL CDC?
No. The PostgreSQL connector defaults tasks.max to 1, and its stream is tied to a logical replication slot. To increase throughput, measure snapshot behavior, database load, connector serialization, topic partitions, and consumer capacity. Split independent sources when that matches the data ownership model.
8.5How do I know whether lag is in PostgreSQL, MSK Connect, or the consumer?
Compare three positions: the PostgreSQL slot’s confirmed flush position, the connector task and producer metrics, and the consumer-group offset. The gap between those positions identifies which boundary is falling behind; a single connector-health status cannot.
9References
- Debezium PostgreSQL connector documentation
- Debezium state storage and offset topics
- Apache Kafka Connect user guide
- Amazon MSK Connect overview
- Amazon MSK Connect workers
- Amazon MSK Connect custom plugins
- IAM roles and policies for MSK Connect
- Amazon RDS PostgreSQL logical replication
- AWS MSK IAM authentication library
- MSK Connect private-subnet troubleshooting guide (third-party)
