Table of Contents
Table of Contents
“The Apache Kafka endpoint is private” is not a network design. It tells you how one client reaches one service, but it does not tell you how DNS resolves the name, where connector traffic leaves the VPC, how brokers reach durable storage, or which control path an operator uses during an incident. Those paths can have different identities, routes, failure domains, and audit records.
That distinction matters on Google Cloud because a Kafka deployment often spans a customer VPC, a managed service boundary or customer-owned data plane, connector workers, and a storage service. A private endpoint can protect the first hop while another hop still uses a public route, an unreviewed peering relationship, or a cross-region dependency. The review question is concrete: which path carries each request, and what happens when it fails?
1Start with a traffic inventory
Before choosing VPC peering, Private Service Connect, or another endpoint pattern, list the conversations the platform must support. Treat “Kafka traffic” as a set of flows with different owners and security requirements.
| Flow | Source and destination | What to record |
|---|---|---|
| Client data | Producers and consumers to Kafka listeners | DNS name, port, route, authentication, and region or zone boundary |
| Connector data | Kafka Connect workers to Kafka and a sink or source system | Worker subnet, egress policy, retry path, and sink endpoint |
| Durable data | Brokers or a storage layer to object or block storage | Storage endpoint, service identity, encryption boundary, and read/write path |
| Control traffic | Console, Terraform, or operators to management APIs | Management endpoint, admin identity, audit source, and break-glass route |
The inventory prevents a common mistake: approving a private client listener and assuming the design inherited the same boundary. A connector, customer-owned broker, and operator may use different service paths, subnets, regions, and identities. Write each flow as a short contract and request a documented boundary whenever the answer is “the provider handles it.”
2DNS is part of the security boundary
An endpoint design is only as private as the name resolution that selects it. A producer can use a private IP while a connector resolves the same hostname through a different DNS policy. Split-horizon DNS, forwarding zones, search domains, and cached records can make two workloads appear to use the same hostname while reaching different addresses.
Use Google Cloud DNS documentation to map the zones that answer Kafka names, then test from every workload class. Capture the resolver, returned address, and route; an administrator laptop may not share the VPC’s policy.
The practical decision tree is:
- Does the provider publish a supported private service endpoint? If yes, record the endpoint type and the consumer-side DNS zone it requires.
- Can every producer, consumer, and connector resolve that name from its own runtime network? Test Kubernetes nodes, VM subnets, serverless connectors, and disaster-recovery environments separately.
- Does the resolved address remain inside the intended boundary? Check routes and firewall policy, not only the address family.
- What is the failure behavior? Decide whether a missing endpoint fails closed or uses an approved private failover.
Google Cloud Private Service Connect provides a documented model for consuming published services through private endpoints. It does not decide whether a particular Kafka service exposes the interface you need, nor does it remove the need to validate DNS, IAM, and region support. Treat the product documentation for the Kafka service as the authority for the supported combination.
3Separate endpoint choices from Kafka semantics
Network reviews often use “private” as a proxy for “safe.” Kafka clients still depend on protocol details after the TCP connection succeeds. The bootstrap address must return broker addresses that the client can resolve and reach. Advertised listeners, TLS names, authentication, and retry behavior must line up with the endpoint design.
For each listener, capture four pieces of evidence:
- the address returned at bootstrap and during metadata refresh;
- the certificate name and authentication method presented on the connection;
- the subnet, route, and firewall rule used by the client;
- the behavior when one broker address or endpoint is unavailable.
This is where a private design can fail like an application bug. The bootstrap endpoint may be reachable while a metadata address points to an unroutable subnet, or a firewall blocks another broker range. Capture client logs and flow telemetry together so an operator can connect the Kafka symptom to the network cause.
4Treat connector and storage paths as first-class paths
Kafka Connect is often the first component to expose an incomplete private design. Workers have their own subnets, service accounts, and egress controls. A sink connector may write to a Google Cloud service using a regional endpoint while reading Kafka through a private listener. A source connector can introduce an inbound path that was absent from the original diagram.
For every connector, document worker placement, the Kafka listener, the source or sink endpoint, and the retry path. Test a task restart while the sink is unavailable. Evidence should show whether retries stay inside the approved boundary and whether backlog growth is visible before it affects freshness.
Durable storage deserves the same treatment. Brokers or a storage layer may access object storage through a Google Cloud endpoint, a private route, or a provider-specific service attachment. Confirm the storage endpoint, permissions, encryption settings, and route from the worker network; a private client path does not prove a private broker-to-storage path.
The distinction matters when comparing deployment models. A managed Kafka service may hide broker placement and storage routing behind a provider boundary. A customer-owned data plane exposes more of the route, but also makes DNS, firewall, IAM, and endpoint operations your responsibility. For cross-region dependencies, use the multi-region Kafka recovery framework to extend the same path inventory.
5Design for failure domains, not only normal traffic
A private route can be healthy and still create a large outage domain. Place the endpoint, DNS policy, connector workers, and Kafka clients on a failure-domain map. Mark the region, zone, subnet, and service attachment for each component. Then ask which single failure removes the ability to produce, consume, operate, or clean up data.
| Failure to inject | Expected observation | Evidence to keep |
|---|---|---|
| Private endpoint unavailable | Clients fail over to an approved private path or stop with a clear error | Client error, DNS result, route state, and recovery time |
| DNS forwarding unavailable | Connections fail without silently selecting a public address | Resolver logs, packet path, and application retry behavior |
| One zone or subnet isolated | Clients and connectors in another failure domain keep their contract | Flow logs, broker metadata, and lag or error metrics |
| Storage endpoint degraded | Produce, fetch, or connector backlog follows the documented policy | Storage errors, queue depth, and recovery evidence |
| Control path unavailable | Data traffic stays within its contract and an audited break-glass path remains | Admin attempt, audit event, and change record |
Run these tests before production traffic depends on the design. Recovery targets are workload-specific, so record the expected result as a service-level objective. An “endpoint is up” check cannot prove client failover, metadata reachability, or storage isolation; the diskless Kafka security boundaries framework uses the same evidence-first approach.
6Measure the path you approved
A private networking decision needs measurements that map to the inventory. Keep DNS answers, routes, firewall decisions, flow logs, client connection events, and storage telemetry for the same test window. Correlate timestamps so the evidence is usable during an incident.
Watch for signals that reveal a path change:
- a client resolves an address outside the approved range;
- a broker metadata response contains an unreachable listener;
- connector retries move to a different region or egress route;
- storage errors rise while client traffic remains healthy;
- control-plane actions succeed even though the data plane is isolated.
Google Cloud VPC documentation describes the routing model; the Kafka service documentation defines supported endpoints and listener behavior. Keep both with the design record and recheck them when a service, region, or endpoint model changes. For pricing, use current calculators and measured traffic because endpoint, cross-region, egress, and processing charges depend on product and location.
7Where AutoMQ fits
Once the path inventory is complete, AutoMQ can be evaluated as a Kafka-compatible cloud-native streaming platform with a shared storage layer. Ask whether its client, control, broker, and object-storage paths fit the customer cloud boundary and can be observed with shared evidence.
AutoMQ’s architecture separates Kafka request handling on brokers from durable stream storage in object storage. That gives a network review an explicit storage path to map alongside the Kafka listener path. In an AutoMQ BYOC design, confirm the customer VPC placement, storage endpoint and IAM role, control-plane route, and any connector or Schema Registry traffic. The architecture overview explains the component boundary; deployment-specific endpoint and permission support still needs to be checked against the target GCP region and release.
The benefit is diagnostic clarity. If a consumer cannot fetch, the investigation can distinguish listener reachability, broker metadata, cache or WAL behavior, and object-storage access instead of treating every symptom as “Kafka networking.”
8A pre-production runbook
Use the same sequence for a managed service review and a customer-owned data plane:
- Draw the four flows and name an owner for each.
- Resolve every Kafka hostname from every runtime network, then compare the answers with the approved endpoint map.
- Capture broker metadata and verify that every advertised listener is reachable from the intended clients.
- Run a connector source and sink test, including a task restart and downstream outage.
- Test broker-to-storage access from the actual worker identity and subnet.
- Inject endpoint, DNS, zone, storage, and control-path failures; record the observed behavior.
- Attach flow logs, client logs, Kafka metrics, storage telemetry, and change records to the review.
- Recheck region support, service limits, endpoint behavior, and current pricing before approval.
The result should be a decision record with explicit exceptions. If one path cannot be observed or made private, state the residual risk and its owner. “Private by default” is not evidence; a reproducible path test is.
9FAQ
9.1Is a private Kafka endpoint enough to keep all traffic private?
No. It covers a specific service connection. Producers, consumers, connectors, brokers, storage, and management APIs can use separate routes and identities. Map and test each one.
9.2When should I use Private Service Connect?
Use it when the Kafka service documents a supported Private Service Connect integration and the endpoint, DNS, region, and IAM behavior fit your workload. Validate the actual service contract before committing to the pattern.
9.3Do managed and customer-owned Kafka need the same network review?
They need the same traffic inventory, but the evidence differs. A managed service may hide broker and storage routes behind a provider boundary. A customer-owned data plane exposes those routes and makes their operation part of your runbook.
The private design is ready when an operator can trace a producer record, consumer fetch, connector retry, storage request, and administrative change without guessing which network carried it. Draw the paths, break them on purpose, and keep the evidence with the approval record. To test the same boundary on a Kafka-compatible shared-storage platform, start an AutoMQ evaluation with the path map and failure matrix in hand.
