Table of Contents
Table of Contents
The first question in a GCP Kafka purchase is usually, “Which service should we choose?” That wording skips the decision that will shape every incident, audit, and invoice that follows: who owns the data plane? A managed service can move broker operations to a provider. A customer-owned data plane can keep the workload and storage inside your Google Cloud boundary while leaving more work with your team. Both can work, but they create different evidence requirements.
Procurement becomes safer when “managed” is treated as a responsibility boundary instead of a feature label. Ask which party controls the runtime, storage, identities, network path, upgrades, recovery, and support escalation. Then ask for proof your operators and reviewers can use. The decision should follow that evidence, not the service label.
1Start with the ownership map
Draw the data path before comparing products. A Kafka workload normally has client connections, brokers, metadata and control operations, connector workers, durable storage, observability, and an administrative path. Those paths may cross different projects, VPCs, identities, and support boundaries. A vendor that owns the broker process may still leave your team responsible for private connectivity, topic policy, consumer recovery, and downstream systems.
Use a short ownership map to make the first review concrete:
| Area | Question for the provider | Evidence to request |
|---|---|---|
| Data plane | Who runs brokers, controllers, and the Kafka runtime? | Architecture diagram, service description, and operational responsibility matrix |
| Durable data | Which account and storage service hold records, logs, and backups? | Storage path, encryption boundary, retention controls, and deletion procedure |
| Control plane | Where do provisioning, configuration, and support actions run? | API endpoints, admin identities, audit events, and break-glass process |
| Network path | Which client, connector, and storage paths are private? | DNS names, routes, firewall rules, endpoint behavior, and region scope |
| Recovery | Who detects, declares, and executes recovery? | Runbook, escalation path, recovery test output, and customer action list |
The map should distinguish the provider’s control plane from the Kafka data plane. Google Cloud’s Managed Service for Apache Kafka overview describes the product boundary, but it does not answer every question about your projects, connectors, identities, or recovery obligations. Record those details in procurement.
2Ask what “customer-owned” actually covers
“Customer-owned Kafka” can mean several things: brokers in your VPC, storage in your cloud account, a Kubernetes cluster your team controls, or a contract that gives you more operational access. Ask the supplier to name the resource owner for every persistent component. If an answer says “customer environment” without naming the project, account, region, and identity, keep the item open.
The same discipline applies to access. Request a list of provider and customer roles, the actions each role can perform, and the audit trail for those actions. Check whether support engineers can access metadata, records, logs, or snapshots; whether that access is time-bound; and how your security team can review it. A private endpoint does not by itself define who can administer the cluster or where diagnostic data is copied.
Security reviewers should also ask for lifecycle evidence:
- How are credentials issued, rotated, revoked, and recovered?
- Which keys protect data at rest, and who controls key policy and disablement?
- Which logs cover configuration changes, authentication, topic administration, and support access?
- What happens to records, snapshots, and temporary diagnostics when the service is deleted?
For a practical discussion of the boundary between a managed control plane and customer data, see what stays in your cloud account with BYOC Kafka. Treat it as a design aid, then verify the answer against the specific offer and GCP region under review.
3Compare the operating job, not the brochure
Put managed and customer-owned options into the same scorecard. A managed service usually delegates a defined set of broker and infrastructure tasks. A customer-owned data plane usually gives your team more control over placement, network, storage, and change timing, while requiring a broader runbook. The exact boundary depends on the provider and contract, so the scorecard should record evidence instead of assuming a default.
| Decision area | Managed service question | Customer-owned data plane question |
|---|---|---|
| Provisioning | What can be created through the service API, and which GCP resources remain yours? | Can the team reproduce the full data plane, including network, storage, and identities? |
| Change control | Which upgrades, patches, and maintenance events can the provider schedule? | Who tests, approves, and rolls back broker, storage, and platform changes? |
| Failure handling | What does the provider detect, and which actions still require your operator? | Can your team isolate a failed node, restore data, and prove client recovery? |
| Observability | Which metrics, logs, and traces are exposed, retained, and exportable? | Can operators correlate Kafka, compute, storage, network, and identity signals? |
| Integration | Are connectors, schema services, private access, and client versions supported for your workload? | Who owns the compatibility matrix and the integration lifecycle? |
| Exit | How are records, offsets, topics, ACLs, and configuration exported? | What is the tested migration path if the platform or storage choice changes? |
The scorecard prevents a common procurement error: comparing a managed service’s feature list with a self-managed deployment’s raw infrastructure. Compare the work that remains after purchase. A service can still require a complex application recovery plan. A customer-owned cluster can fit when control is a hard requirement and the team has time to operate it.
4Make security and residency testable
Security answers should describe a path, not a slogan. Trace a producer from its runtime identity to the Kafka endpoint, then trace broker traffic to durable storage and monitoring. Repeat the exercise for a connector, an administrator, and a recovery workflow. Capture the project, region, service account, endpoint, and audit source for each path. This is where a customer-owned data plane can provide a clearer boundary, but the boundary still needs to be configured and observed.
Ask for the controls that matter to your review:
- Identity: Which principals can create clusters, alter topics, read records, or inspect diagnostics? Can access be limited to the customer project?
- Encryption: Which layer encrypts traffic and stored data? Who owns the keys, rotation schedule, and emergency disablement path?
- Residency: In which regions can records, replicas, backups, and support artifacts exist? What happens during a provider incident or support escalation?
- Evidence: Can your team export logs and configuration snapshots for the required retention period, and can it prove who changed what?
Do not turn a region list into a residency guarantee. Storage replicas, snapshots, logs, and support tooling can have separate locations. Require the supplier to identify each location and to explain the behavior during failover, restore, and deletion.
5Test reliability and support before signing
An SLA is one input to a reliability decision. The operational question is what an application sees when a broker, endpoint, region, credential, connector, or storage dependency fails. Ask the provider to map detection, notification, mitigation, customer action, and evidence for each failure class. For a customer-owned data plane, run the same tests yourself with production-like clients and retention settings.
The test plan should cover:
- client reconnect and metadata refresh after a broker or endpoint change;
- producer and consumer behavior while a failure domain is unavailable;
- connector backlog, retry, and replay behavior when a sink is down;
- restore of topics, offsets, ACLs, and configuration from the chosen backup method;
- escalation timing, diagnostic access, and the handoff between your on-call team and the provider.
Recovery claims without a test artifact are assumptions. Keep the client logs, monitoring screenshots, change record, and provider responses with the procurement decision. A managed service may reduce the number of infrastructure actions your team performs, while a customer-owned deployment may expose more levers for a tailored recovery. Neither removes the need to test the application contract.
6Model the full cost boundary
Price comparison should start with the same workload model: ingress and egress, retention, partition and replica choices, client and connector traffic, storage requests, observability, backup, support, and operator time. Use the Google Cloud Managed Kafka pricing page for current managed-service line items, then add the GCP resources and people costs that remain on your side.
For a customer-owned data plane, separate compute, durable storage, network transfer, control-plane dependencies, monitoring, backup, and maintenance work. For a managed service, separate the service charge from application-side costs and any private connectivity or integration resources. Label estimates with region, retention policy, traffic shape, and pricing date. Procurement should be able to explain which assumption changes the total and who can verify it.
7Where a Kafka-compatible shared-storage option fits
The neutral scorecard often exposes a third path. If Kafka compatibility is required, but broker-local storage and data movement are the main operational constraint, evaluate a Kafka-compatible cloud-native streaming platform with shared storage. The question is whether the architecture changes the ownership boundary you care about while preserving the client and offset contract.
That is where AutoMQ can enter the review. AutoMQ is a Kafka-compatible streaming platform that separates Kafka request processing from durable stream storage. In an AutoMQ BYOC design, the data plane runs in the customer cloud environment, so the procurement team can inspect the GCP projects, identities, network paths, and storage boundary directly. The architecture overview explains the compute and storage separation; the GKE deployment guide lists deployment prerequisites that still need to be checked for the target project.
This is not a reason to skip the same review. Confirm the operator and control-plane access model, GCP IAM roles, object-storage endpoint, WAL choice, upgrade procedure, support path, and exit plan. Run client compatibility, failure, recovery, and cost tests using the workload contract from the managed-service comparison. The value of a shared-storage option is a different data-plane boundary, not an exemption from evidence.
8Turn the review into a decision package
Before the purchase meeting, assemble one packet that an architect, security reviewer, operator, and finance owner can read without translating vendor language. Include the ownership map, the scorecard, test results, open assumptions, and an exception register. Mark each item as verified, contract-dependent, or still untested.
Use this sequence to close the gaps:
- Freeze the workload contract: clients, topics, retention, integrations, regions, recovery objectives, and exit requirements.
- Trace data, control, support, and storage paths for every candidate.
- Request evidence for ownership, identity, encryption, residency, support, upgrades, recovery, and export.
- Run the same client, failure, restore, and cost tests in a representative environment.
- Score the remaining work, risk, and commercial assumptions, then record the conditions for approval.
9FAQ
9.1Is a managed Kafka service always the lower-operations choice?
It can delegate broker and infrastructure tasks, but your team still owns clients, topics, connectors, private access, application recovery, and the commercial assumptions. Count the work that remains in your environment.
9.2Does a customer-owned data plane mean the provider has no access?
Not automatically. Confirm support roles, diagnostic paths, temporary access, logs, and revocation in the contract and the deployed configuration.
9.3What should a procurement team ask for first?
Request a resource ownership map and a failure responsibility matrix. They reveal which follow-up questions apply to identity, residency, support, recovery, and cost.
The right GCP Kafka choice is the one whose boundary your organization can explain and operate. If the review points to a customer-owned Kafka-compatible data plane with shared storage, start an AutoMQ evaluation using the same workload contract and evidence packet.
