Table of Contents
Table of Contents
A team can say it is “using Apache Kafka connectors” and still be operating two very different systems. Kafka Connect is a worker framework that loads connector plugins, coordinates tasks, and stores connector state in Kafka. Redpanda Connect is a pipeline runtime built around inputs, processors, and outputs, with Kafka available as one of its integration points. Both can move data into or out of Kafka. That shared protocol surface does not give them the same ownership model or the same recovery behavior.
The practical question is not which product has more connectors. It is: when a source, worker, network link, or destination fails, which component owns progress, which state is durable, and what can be replayed? A decision that answers those questions is more useful than a feature checklist, especially for change data capture (CDC) and sink workloads where a duplicate or a gap can become a business incident.
1What is being compared
Kafka Connect separates the worker runtime from connector implementations. A distributed Connect deployment runs workers, assigns connector tasks, and persists configuration, status, and offsets in Kafka topics. The framework gives operators a common control surface, but the connector still defines important source and destination behavior. A source connector may expose source offsets; a sink connector may acknowledge a batch only after the destination client accepts it. The framework can coordinate those signals, but it cannot manufacture an end-to-end transaction across an arbitrary database and an arbitrary sink.
Redpanda Connect uses a different boundary. Its configuration describes a pipeline: an input reads messages, processors transform them, and an output sends them to a destination. The input component model and output component model are the units an operator composes. Kafka can be the input or output, but a Kafka component is not the same thing as a Kafka Connect plugin loaded by a Connect worker. Existing Kafka Connect connector JARs therefore require a separate compatibility check; Kafka protocol compatibility alone does not make them interchangeable.
That distinction changes who owns the integration. With Kafka Connect, the platform team usually owns the worker fleet, plugin distribution, worker configuration, internal Kafka topics, and task assignment. With Redpanda Connect, the team owns the pipeline configuration, the runtime process or containers, component versions, and the deployment system that starts and observes those pipelines. If a team treats both as “a connector cluster,” it may miss the state and failure boundary that matters during an incident.
A useful first pass is to map each system to the state it can actually recover:
| Decision area | Kafka Connect | Redpanda Connect | What the team must verify |
|---|---|---|---|
| Work unit | Connector and task managed by a worker | Configured pipeline built from inputs, processors, and outputs | Where concurrency is defined and observed |
| Progress state | Connect offsets and connector-specific source state | Input acknowledgment and component/runtime state | What position is durable before a crash |
| Failure control | Worker/task lifecycle plus connector error handling | Pipeline retry, backpressure, acknowledgment, and process restart behavior | Which errors are retried, skipped, or surfaced |
| Plugin ownership | Plugin artifacts, versions, class loading, and compatibility | Component binaries, config schema, and deployment image | Who tests upgrades and rolls them back |
| Kafka boundary | Worker talks to Kafka and external systems through connector APIs | Kafka input/output components talk to Kafka | Whether the required integration is Kafka Connect-specific |
The table is a starting map, not a delivery guarantee. Delivery is decided at the point where a message is acknowledged and the corresponding progress marker is committed.
2Delivery semantics begin at the acknowledgment boundary
A pipeline can lose or duplicate data at the gap between “the destination accepted the write” and “the source progress moved forward.” That gap exists in both models, but each exposes it differently. For a Kafka Connect source, the source connector emits SourceRecord objects and Connect manages source offsets according to the connector and worker configuration. For a sink, the connector’s put path and the worker’s offset commit form a coordination point, yet the destination protocol may not participate in Kafka’s transaction.
Redpanda Connect exposes the same design problem through its input and output behavior. An input must decide when it has successfully handed a message to the rest of the pipeline; an output must decide what a successful write means and how a failed write is retried. The configuration model makes those choices part of the pipeline definition. If the process stops after the destination write but before the input acknowledgment, replay is possible. If it acknowledges before a durable destination response, loss is possible. The exact result depends on the selected components and settings, so a product label is not enough evidence.
This is why “at-least-once” should be treated as an observed design target, not a universal promise. A Kafka Connect deployment can produce duplicates when a task writes a record and fails before its offset is committed. A Redpanda Connect pipeline can also replay when an input has not acknowledged a message before a restart. Deduplication keys, idempotent destination writes, and a replay runbook are often more useful than a claim that a whole integration stack is exactly once.
Exactly-once requires an end-to-end path. Kafka Connect has framework features for transactional processing in configurations and connectors that support them, but the source system, connector implementation, Kafka producer, and destination all have to meet the required contract. A database change event written to Kafka and then written to an external warehouse is not automatically one atomic transaction. Redpanda Connect should be held to the same standard: an output retry policy or a Kafka producer acknowledgement setting does not prove exactly-once behavior across a source, pipeline, and destination.
A delivery guarantee belongs to a path and a failure test. It does not belong to the word “connector.”
For each critical flow, write down four moments and test them separately:
- Read: when the source makes an event visible to the runtime.
- Acknowledge: when the runtime considers the event handled by the next stage.
- Commit: when a source offset or equivalent progress marker becomes durable.
- Observe: when the destination confirms that the business record is available.
The distance between those moments is where a recovery plan must operate. If a CDC connector reads from a database log, the team needs to know whether a restart resumes from a committed source position, a database cursor, or a connector-specific checkpoint. If a sink writes to an API, the team needs to know whether a timeout means “not written” or “written but not acknowledged.” A retry policy without that distinction can turn a transient timeout into a duplicate storm.
3Retries, dead letters, and the shape of a failure
Retries are not interchangeable. A framework retry around a task, a client retry inside a connector, and a pipeline retry around an output can operate at different layers and see different error information. A task restart may replay a whole batch. An output retry may resend one message. A destination-side idempotency key may make the resend harmless, while a non-idempotent API may apply the operation twice.
Kafka Connect gives operators a standard place to inspect connector and task states, logs, and worker metrics. It also has error-handling settings that can route some record-processing errors to a dead-letter topic. Those settings do not cover every failure inside an arbitrary connector. A connector can fail before the framework sees a record, or it can handle an error internally and expose only a task failure. A dead-letter topic is therefore a diagnostic path, not proof that every failed business operation has been captured.
Redpanda Connect pipelines make failure routing explicit in configuration, but the same boundary applies. A retry can keep a message in flight, block later messages, or eventually send a message to a fallback output, depending on the component. A fallback output can be valuable for forensic data, yet it does not tell the operator whether the original destination applied a partial side effect. The runbook should record the destination response, the retry count, and the message identity that links the original attempt to the fallback record.
A production review should answer these questions for every critical connector or pipeline:
- Which error classes are retried, and at which layer?
- Is backoff bounded, and what happens when the destination remains unavailable?
- Does one poison message block the partition or pipeline?
- Where does the failed record go, and what metadata is preserved?
- Can an operator replay the failed record without replaying the entire source window?
- Which alert fires before lag or source-log retention becomes a data-loss risk?
The answers also determine ownership. In Kafka Connect, a platform team may own worker-level error policy while an application team owns connector-specific retry settings. In Redpanda Connect, one pipeline repository may contain both the retry policy and the destination configuration. Neither arrangement is inherently safer. The safer arrangement is the one that gives the incident responder a single, tested explanation of what happens next.
4Secrets and schemas are operational boundaries
Connector selection is often driven by source and sink coverage, but production incidents frequently start with credentials or data contracts. Kafka Connect workers can load connector configuration and plugin settings through worker and connector properties, with secret handling depending on the deployment and configuration-provider setup. The team must verify how values are injected, masked in logs, rotated, and removed from task snapshots. A password that is safe in a secret store can still leak through a rendered connector configuration or an exception message.
For Redpanda Connect, the deployment team must define where pipeline configuration lives and how secrets are injected. Repository controls and image promotion then become part of the security boundary. Teams should verify which environment variables or secret references are resolved at startup, whether a rotation requires a process restart, and whether the resolved value appears in debug output. The documentation for each input and output component remains the source of truth for its authentication fields.
Schemas create a similar split. Kafka Connect converters, Schema Registry integrations, and connector-specific serializers define how records become SourceRecord or sink payloads. Redpanda Connect processors and outputs may work with structured messages, raw bytes, or component-specific encodings. A Kafka-compatible broker can preserve the record bytes while the integration still changes the schema contract at the edge. Before switching runtimes, test keys, null values, tombstones, headers, timestamps, and schema evolution with real records.
The practical rule is simple: keep the broker contract and the connector contract separate. AutoMQ, for example, documents a Kafka Connect integration in which a Connect Cluster provides worker compute and Connector resources run on that cluster; the AutoMQ Kafka Connect overview describes that boundary. That helps a team evaluate the Kafka data plane and the integration runtime as two related but independently operated layers. It does not make every Redpanda Connect component a Kafka Connect plugin, and it does not remove the need to validate connector prerequisites.
5Scaling means moving a different bottleneck
Kafka Connect scales through workers and tasks. Adding workers gives the coordinator more capacity to place tasks, but a connector may generate fewer tasks than requested, and the external source or sink may impose its own limit. Scaling the worker fleet can therefore leave the real bottleneck unchanged. Measure task assignment, poll or put latency, consumer lag, destination response time, and worker resource pressure together.
Redpanda Connect scales by running more pipeline instances or increasing component concurrency where the selected input and output support it. That can be a clean model for independent pipelines, but it changes the operational unit: each pipeline configuration, process, and deployment replica needs a clear ownership and rollout path. More replicas can also change ordering, destination rate, and duplicate exposure if the source or output does not coordinate them.
The right capacity question is not “How many workers can this product run?” It is “Which state and ordering constraints must remain together while throughput grows?” For a CDC source, partitioning by table or source-log range may be more important than adding generic runtime capacity. For a sink, destination quotas and idempotency may be the first limits. For Kafka itself, broker request capacity and retained data remain separate concerns from connector compute.
A capacity test should include the failure cases that normal throughput tests hide:
- Stop one runtime instance while the destination is healthy.
- Introduce destination timeouts and inspect retry and replay behavior.
- Restart during a batch or transaction boundary.
- Restore the destination and measure backlog drain without violating its rate limit.
- Compare the destination’s observed records with the source’s committed progress.
The test should produce evidence that can be used during an incident: offsets or source positions, message identifiers, retry logs, task or pipeline state changes, and the destination’s own audit trail. A green runtime status is not enough. It means the process is alive; it does not prove that the business records are complete.
6Choosing a boundary for operations
Redpanda Connect is a reasonable fit when the team wants a configurable streaming pipeline and is comfortable owning component-level behavior, deployment images, and pipeline rollout. Kafka Connect is a reasonable fit when the team wants the Kafka Connect plugin model, worker coordination, and a shared operational surface for a fleet of connectors. A team already invested in Kafka Connect plugins should treat migration to Redpanda Connect as an integration rewrite unless compatibility has been demonstrated for the exact component and configuration.
The choice becomes clearer when the incident owner is named. If the platform team owns worker topics, plugin upgrades, Connect REST operations, and task placement, Kafka Connect may align with its existing runbooks. If each application team owns a small pipeline with its own source, processors, destination, and release cycle, Redpanda Connect may make that boundary easier to see. Shared infrastructure can still be used in either model, but the team should avoid splitting one failure path across two owners without an explicit handoff.
The broker underneath the runtime is another independent decision. A Kafka-compatible data plane such as AutoMQ can sit beneath Kafka clients and Kafka Connect workloads while the integration layer remains separately evaluated. Its Shared Storage architecture separates broker compute from durable stream storage, so the team can test connector scaling and broker scaling as distinct variables. That is useful only if the proof of concept measures the real workload: connector lag, replay windows, retention, recovery, and the failure semantics described above.
7FAQ
7.1Is Redpanda Connect compatible with Kafka?
Redpanda Connect can use Kafka through Kafka input and output components, so it can participate in a Kafka-based data path. That is different from running Kafka Connect connector plugins. Confirm the component’s supported Kafka features, authentication, serialization, offset behavior, and failure handling for the exact version you plan to deploy.
7.2Does Kafka Connect guarantee exactly-once delivery?
Kafka Connect provides framework and configuration features that can participate in exactly-once processing when the connector and the full path support the required contract. A generic source-to-external-sink pipeline should not be described as exactly once without a failure test and destination-side evidence. In many systems, idempotent writes and replay-safe consumers remain part of the design.
7.3Should a dead-letter topic be the recovery plan?
A dead-letter topic is a useful diagnostic and isolation mechanism for errors that reach the applicable error handler. It does not prove that a destination did not partially apply a request, and it does not cover every connector-internal failure. Pair it with message identifiers, destination audit data, and a replay procedure.
7.4Which one should a platform team standardize on?
Standardize the runtime that matches the ownership boundary you can support. Compare the plugin or component model, state storage, secret rotation, retry layers, scaling controls, and failure drills. Then run the same CDC and sink workload through both candidates if the decision affects a production migration.
A connector label can hide a large operational decision. The useful comparison starts when the source stops responding, the destination times out, and the runtime must decide whether to retry, replay, or advance. Write down that decision before choosing the runtime, then validate it with the Kafka-compatible data plane and the real systems on both sides. Start an AutoMQ evaluation when you want to test the broker boundary alongside the connector failure plan.
