Table of Contents
Table of Contents
The first IAM question for a GCP Kafka deployment is usually, “Which role should I grant?” That is too narrow for a production review. A Kafka platform has people who manage it, applications that produce and consume records, Connect workers that reach other services, and automation that changes infrastructure. Each identity follows a different path and has a different failure impact.
The private network diagram does not answer those questions. A service account can have no public IP and still hold a project-wide permission that lets a deployment pipeline change unrelated resources. A connector can authenticate to Kafka correctly and still fail because its destination identity cannot write to Cloud Storage or BigQuery. A review that only checks whether a cluster is reachable will miss both problems.
For GCP Kafka, the useful unit of review is the identity-to-resource path. Inventory the principal, the API or Kafka operation, the resource boundary, the expected failure, and the audit evidence for each path. This turns least privilege from a slogan into a worksheet that a platform owner and a security reviewer can sign.
The same path discipline helps with networking. Kafka in Your VPC or Someone Else's shows how to separate client, broker, storage, and support paths before the IAM review starts.
1Start with an identity map, not a role list
Google Cloud IAM controls access to Google Cloud resources, while Kafka authentication and Kafka ACLs govern access inside the Kafka service. They meet at the service boundary, but they are not interchangeable. The Managed Service for Apache Kafka access-control documentation separates Google Cloud access control from Kafka ACL operations. Treating the two as one permission system makes reviews hard to explain and harder to troubleshoot.
Build the inventory around actors and paths. A useful first pass has five rows:
| Principal class | Typical action | Resource boundary | Failure to test |
|---|---|---|---|
| Human platform operator | Create, update, or inspect clusters and network settings | One project or a delegated folder | Operator can change an unrelated project |
| CI or Terraform identity | Apply an approved cluster or policy change | Deployment project and selected shared services | Plan can create resources outside the change set |
| Application workload | Produce or consume records | Kafka cluster, topic, and consumer group | Workload can read or write a neighboring topic |
| Connector worker | Read Kafka and write to a sink or source | Kafka topics plus the named destination | Connector can reach a destination outside its contract |
| Audit or support identity | Inspect logs and configuration evidence | Read-only project or logging scope | Reviewer can mutate production state |
A platform operator may need a broader Google Cloud view than an application identity, while a producer should not inherit the permissions used to create a cluster. Write down the intended owner and the expiration or review cadence for every non-human principal. Service accounts that have no current owner are already an access finding.
2Draw the resource boundary around the real data path
IAM bindings are evaluated through the Google Cloud resource hierarchy. The effective permission can come from the organization, folder, project, or resource level, and a deny policy or custom constraint can change the result. A role name in a Terraform file is therefore not the complete explanation. The review needs the binding location, the principal, the role, the condition, and the resource that the call reaches.
For a GCP Kafka cluster, separate at least four boundaries:
- Management boundary: cluster lifecycle, configuration, networking, quotas, and connector administration. This path usually reaches Google Cloud APIs.
- Kafka data boundary: topic, partition, consumer group, and administrative operations inside Kafka. Use Kafka authentication and ACL evidence for this path.
- Destination boundary: Cloud Storage, BigQuery, databases, or APIs that a connector calls. The destination identity needs its own scoped grant.
- Evidence boundary: Cloud Audit Logs, Kafka audit records where available, and deployment logs. Read access to evidence should not imply write access to the system under review.
That split exposes a common mistake: granting an application identity a project-level role because the application could not publish to one topic. The missing permission may have been a Kafka ACL, a broker authentication mapping, or a destination grant. Expanding Google Cloud IAM can make the symptom disappear while widening the blast radius.
Use a matrix before changing a binding. It should be specific enough that a reviewer can reject one cell without reopening the whole design.
| Principal | Required operation | Narrowest resource | Proof of allow | Proof of deny |
|---|---|---|---|---|
| Producer workload | Write to an approved topic | Topic and producer identity | Produce test with expected principal | Produce to a sibling topic is rejected |
| Consumer workload | Read and commit offsets | Topic plus consumer group | Consume and commit test | Read with a different group is rejected |
| Connect worker | Read source and write named sink | Source topics and destination resource | Connector task writes a test record | Destination outside the allowlist is rejected |
| Deployment automation | Apply reviewed infrastructure | Deployment project and selected resources | Dry-run or plan output matches change | Unrelated resource mutation is rejected |
The “proof of deny” column matters because a passing happy-path test only proves that one request worked. Least privilege is about the boundary around that request. Keep both results with the change record.
3Connector and automation identities need separate scrutiny
Connectors turn one access decision into a chain. A sink worker may consume Kafka records, call a Google Cloud API, write objects, and emit metrics. A source connector reverses the direction. If all of those actions use the same broad service account, a failure in one connector can become an unrelated data-access event.
Give each connector family a named identity or an equivalent isolation boundary. The identity should be able to reach the topics and destination resources that the connector owns, with conditions that reflect the deployment project, environment, or resource name where your platform supports them. Keep credentials out of connector configuration files and logs; use the approved secret or workload-identity mechanism for the runtime.
Automation deserves the same treatment. Terraform, deployment controllers, and break-glass scripts should not share one permanent account. A practical review asks:
- Which identity can create or update a Kafka cluster?
- Which identity can alter topic or ACL state?
- Which identity can change connector configuration or destination grants?
- Which identity can approve or observe those changes?
The answers should describe separate responsibilities even when a small team temporarily combines them. If one account must perform several jobs, record the reason, scope, and compensating audit control. “The pipeline needs admin” is not an explanation; the change set and denied operations are the evidence.
4Make audit evidence part of the access design
An access review is incomplete when it shows only the final policy. You need evidence that the policy was evaluated, the request reached the intended resource, and the denied paths remained denied. Google Cloud documents audit logging for Managed Service for Apache Kafka; use those records alongside deployment logs and Kafka-side authentication or ACL evidence.
For each high-impact operation, retain a small evidence bundle:
- the principal and credential or workload identity used;
- the target project, cluster, topic, group, or destination resource;
- the requested operation and the result;
- the policy or ACL version that was active;
- the time window and change or incident identifier; and
- one negative test showing an out-of-scope request was rejected.
Use a two-stage test for every new environment. First, run the intended operation with the named principal. Then repeat the operation against a resource that is deliberately outside the contract. Capture both results, check the audit entry, and attach the policy version. Repeat the pair after a role, connector, network, or cluster change. Access boundaries drift when the positive test survives but the negative test disappears.
5How a customer-owned deployment changes the review
The required capability is now clear: the team needs a Kafka-compatible data plane, explicit ownership of cloud resources, and separate control and data paths that can be inspected in the customer account. That is a deployment-model decision, not a request for a magic IAM role.
AutoMQ BYOC is one example to evaluate in that frame. AutoMQ's environment documentation describes the control plane and data plane as running in the user's cloud environment, and its Kubernetes deployment documentation covers customer-managed platforms such as GKE. That placement gives the platform team a concrete place to review cloud IAM, network routes, object-storage access, and audit collection. It does not remove the need to define provider support access, service accounts, or Kafka ACLs.
The review should still distinguish the layers. The AutoMQ data plane handles Kafka-compatible client traffic and storage operations; the product control plane manages environment and lifecycle actions. Your GCP policy should identify which identities can operate each layer, which resources they touch, and which logs prove the action. Use the current AutoMQ BYOC environment documentation and Kubernetes deployment overview to validate the exact deployment and identity requirements for your release.
For a procurement or security review that spans network isolation and provider responsibilities, Security and Procurement Questions for Network Isolation Reviews is a useful companion worksheet.
6The approval checklist
Before production, ask the service owner and security reviewer to sign the same six artifacts:
- Identity inventory: every human, workload, connector, automation, service-agent, and support principal has an owner.
- Path map: management, Kafka data, destination, and evidence paths are drawn separately.
- Role and ACL matrix: each grant names its resource, condition, expiry or review date, and proof of deny.
- Test bundle: positive and negative tests cover producer, consumer, connector, deployment, and audit access.
- Incident procedure: break-glass access has an approver, a time limit, and a log review step.
- Change trigger: role, ACL, connector, network, and cluster changes reopen the review where they alter a data path.
If one artifact is missing, record the deployment as pending rather than marking the gap as a future cleanup item. IAM boundaries tend to widen during incidents, and the forgotten exception becomes the permanent path.
7FAQ
7.1Is a Google Cloud IAM role enough to secure Kafka?
No. Google Cloud IAM governs Google Cloud resources and APIs. Kafka authentication and ACLs govern client access inside Kafka. Review the handoff between them and test both allow and deny behavior.
7.2Should every Kafka application get its own service account?
Use separate identities when their topics, consumer groups, destinations, or ownership differ. A shared identity can be acceptable for a tightly bounded workload, but the owner, scope, and negative tests still need to be explicit.
7.3Do connectors use the same identity as producers?
They do not have to. Separating connector identities makes the source-topic and destination-resource boundary visible and limits the impact of a connector failure or configuration mistake.
7.4Does BYOC remove IAM work?
No. A customer-owned deployment can make cloud resource ownership and audit paths clearer, but it still requires scoped identities for lifecycle operations, data-plane access, storage, connectors, and support.
The useful question in a GCP Kafka access review is which identity can reach which resource, through which path, and what proves that the neighboring path is closed. Put those answers in the change record before production. If a customer-owned Kafka-compatible deployment is part of the shortlist, start an AutoMQ BYOC evaluation with the identity map and negative-test evidence ready.
