Debezium usually enters the architecture as a connector decision. A team wants change data capture from PostgreSQL, MySQL, SQL Server, Oracle, or another database, and Debezium is the obvious open source option because it speaks the Kafka Connect model and emits database changes as Kafka records. The cost discussion often starts with the connector runtime, but that is only the visible part of the bill.
CDC changes the shape of a Kafka cluster. A conventional event streaming workload might have a small number of high-throughput topics with clear producer ownership. A Debezium deployment often maps many database tables into many topics, keeps long-lived retained logs for replay, writes schema history or schema change metadata, and uses Kafka Connect internal topics for offsets, status, and configuration. When downstream systems need reprocessing or lakehouse loads, retention windows grow and connector workers stay running even when traffic is uneven.
That does not make Debezium a bad choice. It means Debezium cost has to be modeled as a Kafka platform cost, not as a connector line item.
Why CDC is different from simple event streaming
CDC pipelines carry business state, not only business events. That difference sounds philosophical until it lands in Kafka operations. A user event topic can often drop data after a short replay window because the authoritative state lives elsewhere. A CDC topic may become the replayable record of table mutations for analytics, cache rebuilds, search indexing, fraud models, or lakehouse ingestion. The platform team is then asked to keep more data, for more tables, with more predictable recovery semantics.
Debezium also inherits the structure of the source database. The default topic model is table-oriented: Debezium connectors write change events for a captured table to a Kafka topic specific to that table, with connector-specific naming conventions based on the configured topic prefix and database or schema names. That default is useful because it preserves schema boundaries, but it also means source table count becomes a Kafka sizing input.
The main CDC cost drivers usually fall into six buckets:
- Topic and partition count: Every captured table, history topic, DLQ, retry topic, or operational topic adds metadata, file handles, controller work, and partition leadership to manage.
- Retained bytes: CDC logs are often kept long enough for replay, backfills, sink recovery, and audit-style reconstruction.
- Compaction and deletes: Upsert-oriented topics may use compaction, but compaction is not free. Kafka still stores segments, scans keys, and runs log cleaner work.
- Connector runtime: Kafka Connect workers, tasks, JVM memory, plugins, and transforms need capacity even when the source database is quiet.
- Network path: Database-to-Connect, Connect-to-Kafka, Kafka replication, sink egress, PrivateLink, NAT, or cross-zone routing can all show up in cloud bills.
- Operational topics: Connect internal topics, schema history topics, schema change topics, app log topics, DLQs, and monitoring streams are part of the topology.
Most teams notice the first bucket only after the platform is in production. The connector looked small in the pull request. The Kafka footprint grew table by table.
Topic and partition growth
Debezium's default table-to-topic behavior is the right place to start a cost model. If a connector captures many tables, topic count scales with the captured table set. If the team assigns multiple partitions to each table topic for throughput or consumer parallelism, partition count scales again. A low-throughput table with too many partitions can become a fixed metadata and storage overhead problem; a high-throughput table with too few partitions can become a lag and sink bottleneck.
The tricky part is that CDC table count and Kafka partition count do not have the same optimization goal. Database teams think in tables and schemas. Kafka teams think in throughput, ordering keys, consumer parallelism, and partition limits. A table that is small in database size can still be operationally important, and a table that is large can have a low update rate. Treating every table topic the same usually wastes capacity somewhere.
Use this table as a planning lens rather than a universal formula:
| CDC design choice | Kafka resource it creates | Cost question to ask |
|---|---|---|
| Capture every table in a schema | One or more topics per table, plus metadata | Does every table need CDC, or only tables with downstream consumers? |
| Use a fixed partition count for all table topics | Many low-traffic partitions | Can partition count be based on update rate and consumer parallelism? |
| Add DLQ and retry topics | Extra retained topics and connector writes | What error volume and retention policy are expected? |
| Keep schema history or schema change topics | Internal or metadata topics with recovery importance | Are retention and deletion policies aligned with connector recovery needs? |
| Split connectors by database or domain | More workers and more operational topics | Does the isolation justify the runtime and management overhead? |
Schema history deserves special caution. Debezium uses database schema history to interpret changes using the table schema that existed when the database log event was recorded. Losing or truncating the wrong history data can turn a storage optimization into a recovery problem. For cost work, that means schema history topics should be modeled as small but critical infrastructure, not as disposable noise.
Retention, compaction, and storage cost
CDC teams often ask for long retention because replay is the safety valve. If a sink breaks, a warehouse load fails, or a downstream schema needs to be rebuilt, Kafka retention gives the team time to recover without returning to the source database. That safety valve has a direct storage cost, and in traditional Kafka it often multiplies through local disks or cloud block volumes before a byte reaches any archival tier.
Compaction changes the cost profile rather than removing it. A compacted CDC topic can retain the latest value for a key, which is useful for upsert-style consumers, but Kafka still has to write the full change stream first. The log cleaner later scans segments and removes obsolete records according to topic configuration. If a topic uses both compaction and deletion, the team needs to reason about key history, tombstones, segment roll behavior, and recovery windows together.
The storage question is therefore not "Can we compact it?" It is closer to this:
How much replayable CDC state does the business need, and how much broker-local infrastructure are we willing to keep warm for that state?
That question matters because CDC data is rarely uniform. A small reference table might be compacted and retained for a long time with little churn. An orders table might have steady inserts, updates, and deletes. A user profile table might be update-heavy and key-dense. A payment table might require careful retention for audit workflows. One retention policy across all Debezium topics is convenient, but convenience is often what turns CDC into a storage bill.
Connector workers and networking
Kafka Connect adds another cost layer. In distributed mode, Connect stores configuration, offsets, and task status in Kafka topics. Source connectors use offset storage to track progress. Sink connectors can write failed records to a configured dead letter queue topic. Those topics are not the largest cost item in most deployments, but they are part of the operational surface and they matter when many connector clusters exist across environments.
Managed platforms make this easier to run and easier to underestimate. AWS prices MSK Connect by connector workers and MSK Connect Units. Confluent Cloud documents connector billing dimensions such as task-hours and connector data transfer, with dedicated Connect capacity in some cluster modes. Confluent Cloud billing docs also describe Kafka storage, ingress, egress, and CKU or eCKU-style capacity dimensions. None of these models are wrong; they are different ways of exposing the same underlying reality that CDC consumes compute, storage, metadata, and network.
For Debezium specifically, the runtime cost is shaped by more than message volume:
- Snapshot behavior can create bursty reads from the source database and bursty writes into Kafka.
- Task and connector boundaries influence parallelism, isolation, and worker count.
- Single Message Transforms, converters, and schema serialization add CPU and memory pressure.
- DLQ, retry, and app log topics turn bad records into additional writes rather than disappearing them.
- Private networking, cross-zone routing, and sink egress can become visible when CDC spans database, Kafka, and analytics accounts.
This is why a Debezium cost review should look like a topology review. Start with the source databases, then trace every byte through Connect, Kafka storage, replication, internal topics, sinks, and monitoring. A connector-only review will miss the parts of Kafka that CDC quietly expands.
Where AutoMQ can help Kafka-compatible CDC
AutoMQ fits this discussion at the Kafka platform layer. Debezium and Kafka Connect clients expect Kafka APIs, topics, offsets, consumer groups, ACLs, and standard protocol behavior. AutoMQ keeps Kafka compatibility while changing the storage architecture underneath: brokers are stateless, and persistent stream data is stored on object storage instead of depending on broker-local disks as the primary data store.
That matters for CDC because retained logs are often the expensive part. When many Debezium topics need replay windows, the question becomes whether Kafka storage should be tied to long-lived broker disks. AutoMQ's object-storage-based architecture makes retained stream data align more closely with cloud storage economics, while stateless brokers reduce the operational pain of scaling compute separately from storage.
The practical advantage is not that Debezium produces fewer records. It does not. The advantage is that the Kafka-compatible layer can absorb retained CDC streams with a storage model designed for cloud object storage, and Connect-based pipelines can continue to use the Kafka ecosystem rather than forcing a rewrite. AutoMQ also documents a managed Kafka Connect service where connector tasks run in a separate worker cluster, which gives teams a managed integration path without changing the core CDC pattern.
For teams evaluating Kafka replacement or Kafka cost reduction, CDC is a good stress test because it exercises more than peak throughput. It exercises partition metadata, retention, compaction, worker isolation, replay, and cloud networking at the same time. If the replacement platform only optimizes hot throughput but leaves retained CDC state expensive, the bill will come back through a different door.
CDC cost checklist
Before changing platform vendors or rewriting pipelines, build a workload inventory. The goal is not to produce a perfect spreadsheet; the goal is to expose which design choices are fixed requirements and which are inherited defaults.
| Area | What to inventory | Why it affects cost |
|---|---|---|
| Source scope | Databases, schemas, tables, expected change rate | Table count drives topic count; change rate drives throughput and storage growth. |
| Topic policy | Topic naming, partition count, replication, compaction, retention | Defaults can multiply partitions and retained bytes across many low-traffic tables. |
| Connect runtime | Connect clusters, workers, tasks, transforms, converters | Runtime billing and JVM sizing often scale independently from raw CDC volume. |
| Failure handling | DLQ topics, retry topics, app logs, alert streams | Error paths create topics and retained data that teams forget to size. |
| Network path | Database, Connect, Kafka, sink, VPC, PrivateLink, NAT, zones | CDC often crosses more boundaries than application event streams. |
| Recovery requirement | Replay window, snapshot strategy, schema history retention | Cutting storage without preserving recovery semantics creates operational risk. |
Once that inventory exists, platform choices become easier to compare. A traditional Kafka cluster, a managed Kafka service, and AutoMQ can all run Kafka-compatible CDC patterns, but their cost curves differ when retained bytes, partitions, and connector runtime grow. Debezium cost is not one meter; it is the sum of the topology it creates.
Sources
- Debezium documentation on PostgreSQL connector topic behavior and schema history behavior
- Apache Kafka documentation on Kafka Connect worker configuration and Kafka Connect DLQ behavior
- AWS documentation on MSK Connect pricing and MSK Connect internal topics
- Confluent documentation on Confluent Cloud billing dimensions and managed connector pricing
- AutoMQ documentation on Kafka compatibility and object storage architecture, managed connectors, and Table Topic
FAQ
Does Debezium itself make Kafka expensive?
Debezium is not the problem by itself. The cost comes from the Kafka topology that CDC creates: many table topics, partitions, retained logs, schema history, Connect workers, internal topics, DLQs, and sink traffic. A small Debezium deployment can be efficient; a broad database-wide capture with long retention needs a real Kafka cost model.
How many Kafka topics does Debezium create?
By default, Debezium connectors generally write change events for a captured table to a Kafka topic specific to that table, using connector-specific naming conventions. The exact number depends on captured tables, connector type, schema or transaction topics, history topics, DLQs, and any custom topic routing.
Should CDC topics be compacted?
Some CDC topics are good candidates for compaction when consumers need the latest value per key. Compaction does not eliminate storage and compute cost, because Kafka still writes the change stream and later runs log cleaning. Teams should choose compaction, deletion, or both based on replay requirements, tombstone behavior, and downstream recovery needs.
What is the biggest hidden cost in Debezium Kafka pipelines?
The hidden cost is often retained state spread across many low-traffic topics. High-throughput tables are visible because they create lag and alerts. Low-throughput tables are quieter, but hundreds of them can still create partitions, metadata, retention obligations, and operational topics that stay allocated every hour.
Can AutoMQ run Debezium and Kafka Connect workloads?
AutoMQ is Kafka-compatible, so Debezium and Kafka Connect pipelines can use Kafka protocol concepts such as topics, offsets, and consumer groups. AutoMQ also documents a managed Kafka Connect service. Teams should still validate connector plugins, authentication, networking, and operational requirements in their target environment before migration.