Pub/Sub Kafka Connector Patterns: Bridging Google Pub/Sub and Kafka on GCP

Many Google Cloud teams do not get to choose between Pub/Sub and Kafka as a clean either-or decision. Pub/Sub might already be the event backbone for Cloud Run services, Cloud Functions triggers, or operational notifications. Kafka might still be the system of record for CDC fan-out, stream processing, or a multi-cloud data platform that depends on Kafka clients and topic replay. The hard part is not installing a connector. The hard part is deciding where the contract lives when two messaging systems have different ideas about ordering, acknowledgment, replay, and failure recovery.

That distinction matters because a Pub/Sub Kafka connector sits on a boundary between two operational worlds. Pub/Sub subscriptions acknowledge messages. Kafka consumers commit offsets. Pub/Sub ordering is based on ordering keys and subscription settings. Kafka ordering is based on partitions. Pub/Sub can retain unacknowledged messages by default and supports replay through seek when retention is configured. Kafka retention and replay are topic-level log behaviors. A bridge that ignores those differences can work in a demo and still fail under retries, backpressure, schema drift, or a partial cutover.

When Pub/Sub and Kafka Need to Coexist

The most common coexistence pattern starts with organizational reality. Application teams on GCP like Pub/Sub because it is managed and integrated across Google Cloud. Platform teams like Kafka because it gives them a durable log abstraction, client portability, partition-level control, and a large connector ecosystem. Both preferences can be reasonable in the same company, especially when the estate includes other clouds or applications that already speak Kafka.

Separate four bridge scenarios before design begins:

Pub/Sub to Kafka: Cloud-native services publish to Pub/Sub, while downstream analytics or stream processing runs on Kafka.
Kafka to Pub/Sub: Kafka remains the upstream event log, but selected events need to trigger GCP-native services.
Bidirectional integration: Events flow in both directions, usually during migration or domain decomposition. This needs loop prevention.
Dual-write during cutover: A producer writes to both systems for a temporary window, moving consistency work into the producer or outbox layer.

These patterns look similar because every one has a source, a bridge, and a sink. They behave differently when the source retries, the sink throttles, or a deployment rolls back. The direction of the bridge decides which system owns replay, lag, and duplicate risk.

Pattern 1: Use the Google Pub/Sub Group Kafka Connector

Google provides the Pub/Sub Group Kafka Connector as a Kafka Connect connector library for moving data between Apache Kafka and Pub/Sub products. The connector includes source and sink connectors: a source connector can read from Pub/Sub and write to Kafka, while a sink connector can read from Kafka and publish to Pub/Sub. That makes it the first option to evaluate when your team already runs Kafka Connect and wants to keep bridge lifecycle management inside the Kafka ecosystem.

Kafka Connect is a good fit when the integration is mostly data movement. You get connector configuration, distributed workers, offset storage, restart behavior, and a familiar operational model for Kafka teams.

The tradeoff is that Kafka Connect does not erase the semantic boundary. A Pub/Sub source connector still has to pull messages, write Kafka records, and coordinate acknowledgments with connector progress. A Kafka sink connector still has to consume Kafka records, publish to Pub/Sub, and handle publish errors. The connector gives you a production-grade mechanism; it does not turn Pub/Sub acknowledgments and Kafka offsets into one atomic transaction.

Use this pattern when:

Kafka Connect is already part of your platform operating model.
The bridge is a long-running integration.
Transformations fit connector configuration and converters.
The team can design downstream deduplication.

When the integration needs enrichment, windowed aggregation, or event-time processing, Dataflow may be a better owner.

Pattern 2: Use Dataflow or Apache Beam for Stateful Bridges

Dataflow becomes attractive when the bridge is more than a pipe. Apache Beam includes I/O connectors for Pub/Sub and Kafka, and Dataflow provides a managed runner for streaming pipelines. That combination is useful when the bridge has to validate schema, enrich events, route by event type, or add stateful deduplication before the sink sees the record.

The architectural shift is subtle but important. With Kafka Connect, the bridge is connector infrastructure. With Dataflow, the bridge is an application pipeline. That gives data engineers more control over transforms and state, but also gives them pipeline versions, worker sizing, checkpoint behavior, and observability to manage.

This pattern fits several GCP-heavy designs:

Bridge need	Why Dataflow helps	What to validate
Pub/Sub events need enrichment before Kafka	Beam transforms can parse and route messages	Sink retries
Kafka topics feed GCP analytics paths	Dataflow can read Kafka and write to GCP services	Network path and lag
Duplicate handling needs state	Stateful processing can remember idempotency keys	State TTL and replay
The bridge branches by event type	A pipeline can route event classes	Schema evolution

Dataflow does not remove the need for a clear source of truth. If Pub/Sub is authoritative, Kafka replay should be a derived copy. If Kafka is authoritative, Pub/Sub should be a delivery layer for GCP-native consumers. Symmetrical diagrams hide asymmetric failure modes.

Pattern 3: Build a Custom Relay When the Contract Is Product-Specific

Some bridges are too domain-specific for a connector and too small for a full stream processing pipeline. A custom relay can be appropriate when the bridge needs product-specific validation, a custom idempotency store, or tight integration with a private control plane. The benefit is control over acknowledgments, batching, and metadata mapping. The cost is that your team now owns connector correctness, including crashes after Kafka produce but before Pub/Sub ack, ambiguous Kafka timeouts, poison messages, and partial redeploys.

The Ack-to-Offset Boundary Is the Real Design Problem

A bridge between Pub/Sub and Kafka crosses two progress models. Pub/Sub tracks delivery through acknowledgments and ack deadlines. Kafka tracks consumption through offsets and offset commits. In production, the bridge must advance source progress only after the sink has durably accepted the record.

This is why at-least-once delivery is the practical baseline for most bridges. If the bridge writes to Kafka and crashes before acking Pub/Sub, the same Pub/Sub message can be delivered again. If the bridge publishes to Pub/Sub and crashes before committing the Kafka offset, the Kafka record can be processed again. Both outcomes are safer than data loss, but both create duplicates.

Exactly-once language needs precision here. Pub/Sub supports exactly-once delivery for pull subscriptions within a cloud region, based on Pub/Sub-defined message IDs. Kafka supports exactly-once processing patterns through idempotent producers, transactions, and read-committed consumers. A bridge that spans Pub/Sub and Kafka still coordinates two systems that do not share a transaction coordinator. The right target is deterministic outcomes under replay.

For Pub/Sub to Kafka, a robust bridge usually needs:

A stable Kafka record key derived from the same business entity as the Pub/Sub ordering key.
An idempotency key copied from the source event or immutable source metadata.
A dead-letter path for messages that fail after bounded retries.
Metrics for message age, produce latency, task failures, and duplicates.

For Kafka to Pub/Sub, preserve Kafka key and headers where the Pub/Sub data model allows it, choose an ordering key deliberately, and treat Pub/Sub publish success as the condition for offset progress. If payloads can approach platform limits, validate size before cutover. Pub/Sub has a 10 MB message data limit, and connector documentation recommends checking Kafka message.max.bytes.

Ordering: Keys Are Not Partitions

Ordering is where many bridge designs drift from correct to lucky. Kafka preserves order within a partition. Pub/Sub ordering uses ordering keys and requires message ordering to be enabled on the subscription. Google Cloud also documents that ordered delivery depends on messages with the same ordering key being published in the same region. That is a different abstraction from Kafka partitions, so the bridge has to make the mapping explicit.

If you bridge Pub/Sub to Kafka, the safest default is to map the Pub/Sub ordering key or business entity ID to the Kafka record key, then let Kafka partitioning place related events consistently. This does not make Pub/Sub ordering keys identical to Kafka partitions, but it gives consumers a stable per-entity order if the source publishes consistently. If the source does not use ordering keys, do not invent an ordering guarantee.

If you bridge Kafka to Pub/Sub, the bridge must choose a Pub/Sub ordering key from the Kafka record. Common candidates include the Kafka key, an entity ID inside the payload, or a composite key such as tenant plus entity. The wrong choice creates hot keys or unnecessary serialization.

Schema and Metadata: Do Not Lose the Contract at the Boundary

Kafka records have keys, values, headers, timestamps, topic names, partitions, and offsets. Pub/Sub messages have data, attributes, message IDs, publish times, and optional ordering keys. The models overlap enough to integrate, but metadata will not survive without design.

This is the minimum mapping table every bridge design should write down:

Concept	Kafka side	Pub/Sub side	Bridge decision
Payload	Record value	Message data	Preserve as bytes
Routing identity	Record key	Ordering key or attribute	Choose one entity key
Metadata	Headers	Attributes	Define mapping
Progress	Offset commit	Ack or seek state	Advance after sink success
Replay	Topic retention and offsets	Retention, snapshots, seek	Document source of truth
Poison message	DLQ or retry topic	Dead-letter topic	Keep payload and reason

This table looks administrative, but it prevents production ambiguity. When a team asks why a header disappeared, why an event was duplicated, or whether replay should start from a Kafka offset or a Pub/Sub timestamp, the bridge contract should already answer.

Choosing the Kafka Side on GCP

The Kafka side of the bridge can be self-managed Kafka on GKE or Compute Engine, Google Cloud Managed Service for Apache Kafka, Confluent Cloud, or another Kafka-compatible platform. The right choice depends on who owns operations, how much Kafka compatibility you need, and whether the cluster is a temporary migration target or a long-term event backbone.

Google Cloud Managed Service for Apache Kafka automates broker provisioning, storage management, scaling, and patching while exposing open source Apache Kafka clusters. Self-managed Kafka gives the most control but leaves the platform team responsible for brokers, storage, rebalancing, upgrades, and incidents. A Kafka-compatible system can be attractive when the main requirement is to keep Kafka APIs while changing the storage and operating model.

This is where AutoMQ fits naturally. AutoMQ is a Kafka-compatible streaming platform that keeps Kafka protocol and ecosystem compatibility while moving Kafka data to shared object storage through a storage-compute separated architecture. In a Pub/Sub-to-Kafka bridge, AutoMQ can act as the Kafka-compatible target for Kafka Connect, Kafka producers, and downstream Kafka consumers. The bridge still has to validate ordering, duplicate handling, and schema mapping, but the Kafka side no longer has to be designed around broker-local disks.

That architecture is especially relevant when the bridge is part of a GCP migration. If the destination Kafka cluster is expected to absorb more topics over time, retention and scaling behavior matter as much as connector configuration.

Bridge Validation Checklist

The most reliable bridge review is concrete. Do not ask, "Does the connector work?" Ask what happens when progress is ambiguous, when a key gets hot, when a schema changes, and when the bridge has to roll back under load.

Before production traffic moves through a Pub/Sub Kafka connector, validate these gates:

Ordering contract: Define whether ordering is per entity, per tenant, per source partition, or not required.
Duplicate policy: Treat duplicates as expected under retry. Store idempotency keys downstream.
Lag and backpressure: Track Pub/Sub message age, Kafka lag, task health, publish latency, and retries.
Schema boundary: Define how payloads, headers, attributes, timestamps, and schema IDs move.
Replay plan: Decide whether replay starts from Pub/Sub seek, Kafka offsets, or a retained source topic.
Rollback path: Keep producers, subscriptions, connector configs, and consumers ready for cutover reversal.

The bridge is ready when those answers are boring. Boring is good here. It means the connector is no longer a mysterious middle box; it is a controlled boundary with known failure behavior.

References

FAQ

Is there an official Pub/Sub Kafka connector?

Yes. Google provides the Pub/Sub Group Kafka Connector, with source and sink connectors for moving data between Pub/Sub products and Kafka through Kafka Connect.

Should I use Kafka Connect or Dataflow for Pub/Sub Kafka integration?

Use Kafka Connect for data movement managed by Kafka platform teams. Use Dataflow or Apache Beam when the bridge needs stateful processing, enrichment, branching, or event-time logic.

Can a Pub/Sub Kafka bridge provide exactly-once processing?

Be precise about the boundary. Pub/Sub and Kafka each have exactly-once capabilities in their own domains, but a bridge needs replay-safe and idempotent design because Pub/Sub acknowledgments and Kafka offset commits are not one shared transaction.

How should Pub/Sub ordering keys map to Kafka partitions?

When Pub/Sub is the source, use the ordering key or a stable entity ID as the Kafka record key. When Kafka is the source, choose the Pub/Sub ordering key from the Kafka key or payload identity.

Where does AutoMQ fit in a Pub/Sub Kafka connector design?

AutoMQ can serve as the Kafka-compatible target or source side when teams want Kafka APIs and ecosystem compatibility with a shared-storage architecture. It does not remove bridge-level semantic work, but it can reduce Kafka-side operational burden.