Table of Contents
Table of Contents
A security review for GCP Kafka often stops at a checkbox: “encryption at rest enabled.” That checkbox answers only one question. It does not show who owns the key, which service account can use it, what happens during rotation, or how the team will recover when Cloud KMS is unavailable.
Google Cloud’s Managed Service for Apache Kafka CMEK documentation describes customer-managed encryption keys (CMEK) for cluster data at rest. The page also describes a specific key hierarchy, regional placement guidance, required IAM, rotation behavior, audit logs, and failure consequences. Those details turn encryption from a product claim into a reviewable operating contract.
The useful standard is simple: approve a GCP Kafka encryption design only when the team can identify every protected asset, prove the key boundary, rehearse rotation, and show recovery evidence for a key-access failure. The review should test the contract around the key, not merely record that a CMEK resource exists.
1Start with the asset inventory, not the key name
Before looking at a key ring, list the data and control paths that the Kafka deployment creates. Google’s Managed Kafka encryption documentation says the service uses a CMEK as a key-encryption key (KEK), which protects a data encryption key (DEK). That DEK is used for data at rest on broker persistent disks and for tiered-storage data in Cloud Storage. A review that covers only broker disks is incomplete if retained segments are stored elsewhere.
Use an inventory that a reviewer can trace to a resource and an owner:
- Broker disks: persistent storage used by brokers for local data and service state.
- Tiered storage: Kafka segment files stored in Cloud Storage under the managed service’s data path.
- Metadata and control resources: cluster configuration, IAM bindings, service agents, and audit records that determine whether data can be read or written.
- Client paths: producer and consumer connections, which need transport and identity controls even though CMEK is about data at rest.
CMEK protects stored data; it does not replace TLS, client authentication, authorization, network controls, or secret management. A design that passes a disk-encryption check can still expose an overly broad client identity or an unprotected administrative path.
2Prove who owns the key and who may use it
A customer-managed key changes the ownership and failure model. The key is created and managed in Cloud Key Management Service, while Managed Kafka uses it to encrypt and decrypt cluster data. Google recommends creating the key in the same region as the regional Kafka resource. That is both a placement decision and a recovery dependency: the cluster’s data path needs access to the regional key version it was configured to use.
The Managed Kafka service agent must have the Cloud KMS roles/cloudkms.cryptoKeyEncrypterDecrypter role on the key. Reviewers should verify the exact principal, resource scope, and change record rather than accepting a project-level role as shorthand. The principal documented by Google follows the managed-service agent pattern, service-${PROJECT_NUMBER}@gcp-sa-managedkafka.iam.gserviceaccount.com; the project number and key resource must be resolved for the actual environment.
Ask for four artifacts:
| Review question | Evidence to retain | Failure if missing |
|---|---|---|
| Which key protects this cluster? | Cluster configuration and fully qualified key resource | A key can be rotated or disabled without knowing its consumers |
| Which identity uses it? | IAM policy showing the Managed Kafka service agent binding | A broad or wrong principal can create access or availability risk |
| Where is the key located? | Key ring, region, project, and owner record | Cross-region assumptions can surface during recovery |
| Who can change it? | Key-admin and cluster-admin role mapping | Rotation or disable actions may bypass change control |
The point is separation of duties. Kafka operators may request a rotation without holding permission to destroy a key. Cloud KMS administrators need a tested handoff with Kafka operators because a key change can affect publishing, delivery, broker restarts, and historical reads.
3Treat rotation as a data-path change
Key rotation is not the same as replacing the key attached to a cluster. Google’s key-rotation guidance says the key association cannot be changed; rotation is performed by creating a new key version and making it primary. That primary version then takes effect on different paths at different times.
For broker disks, the new KEK takes effect after a broker restart. Google describes a rolling restart triggered by a capacity-configuration update as one way to force that restart. For tiered storage, new partition segment files use the new primary key version, with a possible delay of several minutes after the version change. Existing data is not silently re-encrypted as part of the rotation.
That means the change plan needs more than a KMS command. It should name the partitions, brokers, segment files, and clients that must remain available while the new version becomes active. The change window should include a producer test, a consumer read from existing data, a broker restart check, and a Cloud KMS audit-log review.
A practical rotation runbook has these gates:
- Pre-check: confirm the new key version is enabled, in the expected region, and visible to the Managed Kafka service agent.
- Make primary: record the exact timestamp and key version promoted to primary.
- Exercise the paths: produce new records, consume existing records, and observe broker behavior during the planned rolling restart.
- Check evidence: correlate Cloud KMS key-use audit logs with cluster activity and verify that no client path lost authorization.
- Close the change: retain the old version until the retention and recovery policy says it is safe to disable; document who approved that decision.
The old version deserves special attention. Google warns that disabling a non-primary version can make older tiered-storage data unreadable, while disabling the primary version prevents new segment files from being written and can cause broker restarts to fail. A rotation that looks successful because new messages flow may still have broken historical reads.
4Test the outage you are actually accepting
CMEK adds a deliberate dependency on Cloud KMS availability and IAM propagation. Google documents that when Managed Kafka cannot access the key, message publishing and delivery fail. After access is restored, publishing can become available within 12 hours and message delivery can resume within 2 hours. These are documented service behaviors, not an invitation to assume that every workload will recover inside its business RTO.
The review should therefore include a controlled failure test in a non-production environment or an approved game day. Disable the relevant key version or remove the service agent’s permission according to the organization’s change process. Record the first publish and consume errors, the time access is restored, and the time each path recovers. Keep the test separate from a client certificate or network outage so the signal remains attributable to the key dependency. Cloud KMS audit logs provide the control-plane evidence to correlate with those client observations.
The following evidence is more useful than a green dashboard:
- the exact key version and IAM change that caused the failure;
- producer and consumer error samples with timestamps;
- Cloud KMS audit logs showing denied or restored use;
- broker restart and segment-read results after access returns; and
- a runbook owner who can explain whether backlog, retention, and downstream freshness remain within the service objective.
Key deletion is a different class of event. Google states that deleting the key schedules the cluster for shutdown and that the cluster cannot be recovered. A key-retention policy and a deny-by-default deletion workflow belong in the architecture review, even if the organization never plans to delete an active key.
5What the review should ask of a customer-owned data plane
The same questions apply when Kafka runs in a customer-owned deployment. Start with the storage providers and services that actually persist message data, then trace their encryption and key-management controls. Do not infer CMEK support from the presence of a GCP project or a customer-owned bucket; ask for the product’s current integration contract and test it in the target region. The BYOC security boundary checklist is a useful companion for mapping account ownership.
AutoMQ is a Kafka-compatible cloud-native streaming platform with a Shared Storage architecture. In a BYOC deployment, the console and cluster components are deployed in the customer’s cloud environment, and durable stream data uses S3-compatible object storage. That makes ownership and storage boundaries explicit, but it does not prove GCP CMEK support for every path.
AutoMQ’s public data-encryption documentation currently says its transparent at-rest encryption and custom-key (BYOK) support are limited: the documented feature is supported on AWS, and BYOK custom keys are not supported. The GCP installation documentation explains the Google Cloud identity and storage permissions needed to deploy the platform, but it does not establish a GCP CMEK contract. Treat that gap as a verification item. Ask AutoMQ for the current GCP encryption design, supported key controls, and evidence for rotation and key-access failure before making a compliance claim.
This is a useful distinction in procurement: a customer-owned data plane can reduce ambiguity about where data and network resources run, while key ownership and key lifecycle still require product-specific proof. The AutoMQ BYOC overview and GCP installation guide are useful starting points for that conversation.
6A review packet that can survive an audit
A complete packet should let a reviewer reconstruct the encryption decision without relying on a verbal walkthrough. Store the following together:
- Scope statement: which Kafka resources, disks, tiered-storage paths, and regions are in scope.
- Key record: key project, key ring, region, primary version, owners, retention, and deletion controls.
- IAM proof: the exact Managed Kafka service agent binding or the equivalent customer-owned service identity, plus separation-of-duties approval.
- Rotation runbook: pre-checks, primary-version change, broker restart sequence, client tests, and evidence retention.
- Failure drill: key-access outage timeline, producer and consumer behavior, audit logs, restore action, and observed recovery times.
- Product boundary note: any capability that remains unverified for the selected Kafka implementation, including GCP-specific CMEK support.
End the packet with a decision and an owner. “Encryption enabled” is a configuration state; “the team can rotate, recover, and explain the key boundary” is an operational capability. The encryption key ownership readiness criteria can help turn that decision into a release gate.
7FAQ
7.1Does enabling CMEK encrypt Kafka messages in transit?
No. CMEK addresses data at rest. Use the service’s network, TLS, authentication, and authorization controls to review client and administrative paths.
7.2Can a GCP Kafka cluster switch to a different key?
Google’s Managed Kafka documentation says the key association cannot be changed. Rotation uses a new version of the existing key, promoted to primary.
7.3Does rotation re-encrypt all historical data?
No. Google documents that new tiered-storage segment files use the new primary version, while existing data remains tied to the versions that encrypted it. Disabling an older version can make older data unreadable.
7.4What is the safest way to compare managed Kafka and customer-owned Kafka?
Use the same evidence packet for both: asset scope, key ownership, IAM, rotation, outage behavior, recovery, and deletion controls. The deployment model changes the owner of each step; it does not remove the step.
If your team is reviewing GCP Kafka CMEK, start with the key version and the data path that depends on it. Then run the rotation and access-loss tests before the compliance deadline. If a Kafka-compatible shared-storage design is under consideration, bring the same evidence packet to an AutoMQ evaluation, and ask for GCP-specific encryption and key-lifecycle proof before approving the architecture.
