Diskless Kafka Security Boundaries: A Production Framework

An Apache Kafka® cluster can pass an access review and still leave its most important boundary undefined. The broker listener may require TLS, the topic may have an ACL, and the object-storage bucket may be private. Yet a recovery broker, an operator with a diagnostic role, or a second tenant can still reach a path that the original review never named.

Diskless Kafka makes this gap easier to see because a record no longer lives inside a broker's local disk boundary. The record moves through clients, brokers, a cache or WAL (Write-Ahead Log), metadata services, and object storage. Each hop has a different identity, network route, and audit trail. Security is therefore a path review, not a single setting.

The useful production question is precise: which principal can perform which operation on which data path, from which network location, and for how long? Answer that for normal writes, cold reads, recovery, administration, and tenant isolation. The result is a security boundary that can be tested instead of a diagram that only looks private.

1What security boundaries mean in a diskless Kafka design

A security boundary is the point at which an identity, policy, or trust assumption changes. In traditional Kafka, teams often draw the boundary around the broker fleet and its attached volumes. In a diskless design, that drawing is incomplete: durable records and the metadata required to locate them are handled by shared services outside an individual broker process.

Start by naming the assets rather than the products. A useful inventory includes encoded Kafka records, topic and partition metadata, consumer-group state, WAL entries waiting for upload, object manifests or indexes, encryption keys, credentials, logs, and operational snapshots. These assets may have different owners and retention rules even when they belong to one cluster.

Then map the principals. A producer and a consumer need Kafka permissions; a broker needs storage permissions; a controller needs metadata permissions; a connector may need both Kafka and cloud-resource access; and a support or automation role may need a limited diagnostic view. Treating all of them as “the Kafka service account” removes the evidence that an auditor or incident responder needs.

The inventory is more useful when every row has a failure question. What happens if a broker keeps its Kafka identity but loses its object-storage permission? What happens if a replacement broker can read records but cannot list the metadata prefix? What happens if a tenant can guess an object key but has no Kafka permission for the topic? These cases turn a conceptual boundary into a testable contract.

2The mechanism: brokers, cache, metadata, and object storage

Diskless Kafka still presents the Kafka protocol to clients, but the durable path has more participants. A producer sends a record to a broker leader. The broker writes through the configured durability path, acknowledges according to the cluster's contract, and later makes the record available through shared storage and caches. A consumer may read hot data from memory or a local cache, or fetch an older range through object storage after a cache miss.

The identity used at the Kafka listener is not automatically the identity used for storage. A client principal might be mapped to a Kafka user and ACL, while the broker uses a cloud role or access key to call an object-storage API. That separation is valuable: clients do not need bucket credentials. It also creates a boundary that must be documented, because a storage role with broad permissions can bypass the topic-level policy if it is exposed to users or unrelated workloads.

Metadata creates a second path. Partition leadership, offsets, object locations, and compaction state are not the same asset as the record payload. A recovery procedure that restores a bucket but not its metadata may produce a readable set of objects that the Kafka layer cannot locate. Conversely, restoring metadata without the associated object policy can create an apparent topic whose reads fail with authorization errors.

Trace one record through each path and write down the actor at every step:

• Ingress: the client authenticates to a Kafka listener and receives an authorization decision for the topic and operation.
• Durability: the broker writes to the configured WAL or storage endpoint under its service identity; the client never receives that credential.
• Materialization: background work uploads or compacts data into object storage, using only the bucket and object actions required by the deployment.
• Read: the broker serves a hot cache entry or requests a range from object storage, then returns it through the Kafka authorization boundary.
• Recovery: a replacement broker rebuilds serving state and metadata from shared services, with an identity that is independently auditable.

The Kafka Tiered Storage proposal and Diskless Topics proposal are useful references for separating active serving from remote storage. They do not define a provider's IAM policy for you. The security review still needs the actual endpoint, role, bucket, key, and network configuration that will run in production.

3IAM and bucket policies: make the storage role narrow

The storage role should express the broker's job, not the administrator's job. Start with the operations the data path actually performs: writing objects, reading objects, listing the required prefix, completing multipart uploads when used, and deleting objects only when a retention or compaction workflow requires it. A policy that grants an entire account or every bucket is difficult to reason about during an incident.

AutoMQ's object-storage configuration guide makes the same operational point in its provider examples: scope production permissions to the specific bucket. The exact action names differ by provider, but the review questions remain stable:

• Does the role name the data bucket and the required object prefix?
• Can it list only the prefix that contains this cluster's objects?
• Is deletion limited to the lifecycle, compaction, or retention workflow?
• Can the role use only the intended private endpoint or network path?
• Are temporary credentials issued to the workload instead of stored in an image or repository?
• Is there an independent role for operational logs or diagnostics?

An illustrative policy shape is more useful than a provider-neutral promise. Adapt the actions and condition keys to the chosen provider, and validate them with an actual write, read, list, and retention test:

The example intentionally leaves out provider-specific encryption, endpoint, and key conditions. Add them only after confirming how the storage client sends the relevant headers and how the cloud provider evaluates the condition. A condition that looks restrictive but is never present on the request can turn a deployment failure into a rushed policy broadening.

Keep the data bucket separate from an operations bucket when the platform writes logs, metrics, or diagnostic bundles. The people who can inspect an operational bundle may not be allowed to read message payloads. Separate buckets or prefixes make that distinction visible in IAM, object-lock or lifecycle policy, and audit logs.

4Encryption and network boundaries are one review

Encryption at rest answers what happens if storage media or an object snapshot is obtained. It does not answer who can use the key, whether a broker can reach the endpoint, or whether a client connection is protected in transit. Review the controls together so that a green encryption checkbox does not hide an open network route.

For the data path, record the following decisions:

The AWS S3 encryption documentation describes provider options, while the S3 security best-practice guidance covers access controls and monitoring. Use the equivalent documents for the chosen object store. Do not infer that an S3-compatible endpoint provides the same key-management or audit guarantees as a public cloud service; verify the implementation.

Private networking is a boundary, not a replacement for authorization. A broker inside a VPC can still assume an over-privileged role, and a bucket policy can still allow a role that should belong to another cluster. Conversely, a correct IAM policy can fail during a regional endpoint incident if DNS, route tables, or firewall rules send recovery traffic to an unapproved path. Test both the allow case and the deny case from the real broker and recovery environments. For residency and jurisdiction questions, compare the storage and recovery path with the regional data sovereignty review checklist.

5Tenant boundaries: topic ACLs do not equal bucket isolation

Multi-tenancy becomes sharper when durable objects are shared. Kafka ACLs can prevent a tenant from producing to another tenant's topic, but they do not automatically stop a compromised broker role from reading a neighboring object prefix. The storage layer needs its own isolation model.

Choose the model deliberately:

• Cluster or bucket per tenant: the boundary is visible in the storage policy and network path, at the cost of more resources to operate.
• Prefix or namespace per tenant: the model can consolidate infrastructure, but every list, read, write, delete, and recovery operation must preserve the prefix boundary.
• Shared service identity with application filtering: this may be appropriate for a tightly controlled internal platform, but it requires stronger process and audit evidence because the cloud policy cannot distinguish tenants.

A prefix is a naming convention until the storage policy enforces it. Test traversal attempts, list operations outside the assigned prefix, guessed object keys, and recovery jobs that run with a different identity. Also inspect observability: topic names, object keys, payload samples, and exception messages can leak tenant information even when data reads are denied. The secure multi-tenant Kafka isolation patterns article provides a related review frame for shared platform boundaries.

A tenant review should cover more than payload access. Include quotas, cache pressure, request throttling, encryption keys, metrics labels, support sessions, and deletion workflows. A tenant that cannot read another tenant's data but can force its cache eviction or trigger broad compaction still has a meaningful availability path across the boundary.

6Failure, cost, and compatibility checks

A production security boundary is credible when it survives a failure drill. Use a worksheet that binds each scenario to an identity, a network route, a measurable signal, and a stop condition.

Security controls can also change the cost model. A private endpoint may have an associated provider charge. KMS requests, object listing, retries, cross-zone paths, and recovery reads can add costs that do not appear in the broker bill. Keep those lines separate from compute and storage so that a security exception is not justified by an incomplete estimate. Use current provider pricing for the region and endpoint type; do not carry a rate from a different account or date into the decision.

Compatibility checks belong in the same worksheet. A Kafka-compatible client may still use a different TLS setting, serializer, or credential provider in production. Test producers, consumers, Kafka Connect workers, and recovery tooling with the identities and endpoints they will actually use. The Apache Kafka compatibility overview is a useful starting point for the protocol boundary, but it cannot validate your cloud policy or tenant model.

7How AutoMQ changes the operating model

Once the neutral boundary model is explicit, you can evaluate a Kafka-compatible shared-storage platform against the same evidence. AutoMQ keeps the Kafka protocol and ecosystem while replacing broker-local durable log storage with a Shared Storage architecture. That moves the security review toward the identities and endpoints that serve shared storage, while Kafka listener authentication and topic authorization remain first-class controls.

AutoMQ's public architecture documentation describes a storage path that includes a WAL layer, cache, and object storage. Its glossary distinguishes the broker, controller, WAL, bucket, and S3 concepts that should appear in the boundary inventory. The deployment's selected WAL backend and object-storage provider determine the exact identity, network, and encryption controls; verify those values before approving a design.

The operating model changes in ways that are useful for a security review:

• Broker replacement is a boundary test, not only a capacity test. A replacement broker must obtain the same approved storage access without inheriting unrelated permissions from the host or image.
• Shared durable data makes access logging more important. A storage audit event can identify a principal and object path that a Kafka log alone cannot show.
• Compute and storage can be reviewed independently. You can ask whether a tenant, operator, or recovery job needs broker access, bucket access, metadata access, or some combination, instead of granting one role everything.
• Kafka compatibility preserves the client contract. Existing ACL, TLS, and serializer fixtures can be reused while the storage boundary is tested separately.

These are review advantages, not automatic guarantees. In a BYOC or self-managed deployment, the customer still selects the cloud role, endpoint, key policy, bucket layout, and tenant model. Record those choices in the deployment repository, run negative tests from the broker and recovery environments, and retain the audit output with the rollout decision.

8Decision checklist and FAQ

Use this checklist in a design review and attach evidence to each answer:

• Data inventory: Can the team name every durable and temporary asset, including WAL, cache, metadata, snapshots, and diagnostic files?
• Principal inventory: Are client, broker, controller, connector, operator, and recovery identities distinct where their duties differ?
• Storage policy: Does the data role reach only the intended bucket and prefix, with deletion and listing constrained to the required workflow?
• Network route: Do Kafka, metadata, WAL, and object-storage calls stay on approved endpoints during normal operation and recovery?
• Encryption: Are transit, object, WAL, backup, and key-administration controls documented and tested separately?
• Tenant model: Is isolation enforced by cluster, bucket, prefix, or a documented service boundary, with traversal and metadata-leak tests?
• Failure gates: Have role loss, endpoint failure, key revocation, broker replacement, retention deletion, and recovery been exercised?
• Cost evidence: Are endpoint, key, request, transfer, compute, and replay costs measured from current provider data?
• Rollback: Can the team revoke a candidate policy, restore the prior identity, and continue serving or recover data without guessing?

8.1Does diskless Kafka remove Kafka ACLs?

No. Diskless storage changes where durable bytes are served. Client authentication, topic authorization, consumer-group permissions, and administrative controls still apply at the Kafka boundary.

8.2Should clients receive object-storage credentials?

Usually no. Keep object-storage credentials with the broker or storage service identity, and give clients only the Kafka permissions they need. If a client must access objects for a separate export or analytics workflow, use a separately scoped role and audit that path.

8.3Is a private bucket enough for tenant isolation?

No. A private bucket can still be reachable by every workload that shares its role. Enforce the tenant boundary with bucket or prefix policy, network conditions, key authorization, and negative tests for list, read, and delete operations.

8.4Does a WAL make the security review local again?

No. A WAL is part of the durability path and can contain records that have not yet reached the primary object store. Apply the same identity, encryption, backup, and recovery review to the selected WAL backend. Its failure domain may differ from the broker's process or host.

8.5What should a pilot prove?

A pilot should prove the complete path: a client can authenticate and receive the intended Kafka authorization decision; the broker can write, read, and recover through the approved storage role; a replacement broker stays on the approved network route; tenants cannot cross their storage boundary; encryption and key revocation fail as designed; and the audit trail explains each result. Record the evidence before changing the production policy.

When the next security review asks where a Kafka record lives, answer with a path rather than a box: client identity, broker authorization, WAL or cache, metadata, object prefix, encryption key, network endpoint, and audit event. That answer scales from a single cluster to a shared platform because every boundary has an owner and a test. If the worksheet exposes gaps, start an AutoMQ evaluation with the identity matrix, policy tests, and recovery evidence in hand.