Table of Contents
Table of Contents
A storage bill rarely tells a platform team running diskless Kafka on Apache Kafka® which workload created the cost. One topic may keep bytes in object storage for a long retention window; another may drive a high request rate during replay; a third may require broker capacity because its traffic is bursty. If those effects arrive as one cluster charge, the team can report spend but cannot make a good operating decision.
Diskless Kafka makes the accounting boundary more visible. Durable records move through brokers, cache, WAL (Write-Ahead Log) storage, object storage, and network endpoints, while metadata tracks and indexes that path. Each component has a different driver and a different owner. FinOps becomes useful when those drivers are measured together and attributed to a team, topic, workload, or environment.
The practical question is: can an engineer explain why a given workload changed each cost line, then choose an action that changes the line without breaking Kafka semantics? A production framework should answer that question with a ledger, a measurement plan, and rollout gates.
1What FinOps means in a diskless Kafka design
FinOps is a feedback loop between engineering usage and financial ownership. The FinOps Foundation framework describes the loop as making spend visible, allocating it to accountable owners, and using the result to plan and act. Kafka adds a detail that generic cloud reports miss: the same byte can incur separate write, storage, read, and transfer charges as it moves through the system.
Start with an attribution grain that people can act on. A cluster-only view is useful for an invoice check, but it is too coarse for a platform review. A topic or workload view is often better, provided the telemetry can connect broker requests, object-storage operations, and network paths to the same owner. Keep the raw dimensions even when the report rolls them up; otherwise an aggregate hides the reason for a change.
| Cost line | Primary driver | Useful attribution key | Engineering action |
|---|---|---|---|
| Broker compute | Request rate, partition leadership, connection count, and background work | Cluster, broker pool, team, or environment | Change placement, capacity policy, or workload shape |
| Cache and WAL capacity | Hot-data window, write rate, retention before upload, and selected WAL type | Cluster, tier, or workload class | Tune cache policy or select a different storage tier |
| Object-storage bytes | Retained data, object layout, compaction, and lifecycle policy | Bucket, prefix, topic, or retention class | Change retention, compaction, or namespace policy |
| Object-storage requests | Writes, reads, listings, multipart operations, and replay patterns | Topic, consumer group, recovery job, or service | Batch operations, fix replay behavior, or change access pattern |
| Network transfer | Endpoint topology, cross-AZ paths, egress, and recovery traffic | Region, AZ, cluster, or workload | Keep traffic local where allowed and test recovery routes |
| Operations and observability | Metrics, logs, traces, audits, and retention | Environment, team, or service | Set useful retention and sampling policies |
This ledger is deliberately wider than “storage cost.” The platform team should be able to show which observed signal moved a line item and which policy controls that signal. A retention change belongs with the topic owner; a shared broker pool or endpoint belongs with the platform team. That separation keeps a budget conversation tied to an engineering action instead of a generic request to cut spend.
Use two views together. The showback view reports measured usage to the team that owns a topic or consumer group. The chargeback view applies an agreed price model when the organization needs to transfer the cost to a budget. Keep the measurement and the price assumption separate. A provider changes a request or transfer rate; that should update the price model without rewriting the usage history.
2The mechanism: brokers, cache, metadata, and object storage
A diskless Kafka design retains the Kafka client contract while moving durable stream data out of broker-local volumes. The broker still handles protocol requests, partition leadership, quotas, and scheduling. The storage path can include a cache for hot reads, a WAL for durable writes and recovery, and object storage for retained history. FinOps must follow the path instead of assigning every byte to the broker that handled the produce request.
The first useful measurement is a path map. For each workload, record where a record is acknowledged, where it is retained, which component serves a Tailing Read, and which component serves a Catch-up Read. Mark the identity and network endpoint at each boundary. This map makes it possible to ask whether a request is paying for compute, a storage API call, data transfer, or all three.
On the write path, a producer request reaches a broker, the configured WAL (Write-Ahead Log) or durability path records the write, and background work places retained data in object storage. The write rate therefore has at least three signals: broker work, WAL capacity and I/O, and object bytes or write operations. The selected WAL type also changes the failure domain and provider line items. The storage documentation for a shared-storage implementation distinguishes S3 WAL, Regional EBS WAL, and NFS WAL; record the selected type, its location, and the workload it is expected to absorb.
A cache hit can avoid an object-storage read, while a cache miss can produce object requests and network transfer. A replay by one consumer group can therefore increase cost without changing retained bytes. Capture cache hits and misses with the consumer-group and topic labels the platform supports, then compare them with object-store request logs and network telemetry.
Retention and object operations should stay separate in the ledger. A topic can hold a stable volume of bytes while compaction, lifecycle, or replay creates additional writes, reads, listings, or retrieval operations. The Amazon S3 pricing page is an example of a provider price sheet that separates storage, requests, retrieval, and transfer. Other providers expose similar categories with different names and rates. Use the selected region and storage class when converting measured usage into currency; do not copy a rate from a different account or endpoint.
Metadata closes the map. Topic and partition state, object indexes, compaction metadata, lifecycle markers, and recovery jobs create metadata operations and, where configured, management or backup capacity even when they are small in bytes. A recovery test that restores data but rebuilds metadata can produce a request burst that is absent from the retention estimate. Label that work as recovery or platform overhead so the platform team can distinguish shared service cost from a topic owner's usage.
3Build a FinOps worksheet that survives failure
A useful worksheet has one row per measurable driver and one column per owner. At minimum, include the workload identifier, topic or namespace, environment, region, retention class, producer bytes, consumer bytes, request counts, cache hit rate, WAL usage, broker compute time, and network path. The worksheet should be reproducible from exported telemetry rather than a manually edited monthly total.
Treat pricing as a versioned input. Save the provider page, region, storage class, endpoint type, and retrieval assumptions with each conversion. The AWS cost allocation tag guidance shows why tagging needs an activation and reporting step before tags appear in cost data. Tags cannot replace Kafka-level attribution, but they can connect bucket, endpoint, and compute resources to the same team or environment.
Use a simple formula for each line:
usage-based line cost = measured usage × selected unit rate
workload cost = sum of usage-based lines + allocated fixed and shared platform costs
The difficult part is deciding what “measured usage” means and where shared work belongs. If a compaction job serves several topics, allocate it with a documented rule such as bytes processed or object operations. If a broker serves many teams, allocate its compute by request work or a platform capacity pool. Record the rule next to the result so an owner can challenge the assumption without losing the raw signal.
Consider a hypothetical replay of one team's audit topic. The consumer misses the cache, reads an older range from object storage, and crosses an endpoint boundary before the lag recovers. The ledger should connect that event to the consumer group, object-read and network-byte signals, the recovery window, and the team that owns the replay policy. The right action might be a cache or retention change, but the worksheet has to show the latency and recovery effect before the budget owner approves it.
| Review question | Measurement | Decision gate |
|---|---|---|
| Which workload increased object bytes? | Retained bytes by topic or prefix over the billing window | Owner confirms retention and lifecycle policy |
| Which workload increased requests? | Read, write, list, multipart, and retrieval operations by caller or topic | Replay and compaction jobs have named owners |
| Which workload increased network cost? | Bytes by endpoint, AZ, region, and egress path | Cross-domain paths are intentional and documented |
| Which workload consumed compute? | Broker CPU, request work, connection load, and background tasks | Capacity pool has an allocation rule |
| Which workload raised recovery cost? | Rebuild reads, metadata operations, WAL recovery, and duration | Failure drill has a stop condition and owner |
| Which costs are shared? | Control plane, monitoring, backups, and support operations | Allocation rule is approved before chargeback |
Do the worksheet against a normal operating window and a failure window. A broker replacement, object-store retry storm, or long consumer replay can change request and transfer usage without changing retained bytes. If the budget model only covers the quiet path, it will understate the cost of the incident path that the platform must be able to survive.
4Failure, compatibility, and cost checks
FinOps decisions fail when the cost model ignores the behavior that created the bill. A request reduction that increases consumer lag may move cost into recovery work. A private endpoint that improves data-path control may add a provider charge. A longer cache window may reduce object reads while increasing memory or WAL pressure. Write down the trade-off and test it with the application path that matters.
Use the following checks before approving a production change:
- Retention check: change one topic's retention policy in a test environment and verify the object-byte and lifecycle signals that move with it.
- Replay check: rewind a consumer group over a known range and record object reads, cache misses, network bytes, and lag recovery. Treat the range and consumer concurrency as workload inputs, not universal benchmarks.
- Compaction check: run the chosen compaction policy and identify which topic owns the object operations and compute work.
- Endpoint check: route normal and recovery traffic through the intended object-storage endpoint, then verify that audit logs and network telemetry identify the same path.
- Failure check: remove a storage permission or interrupt a dependency in a controlled test. The worksheet should show the failed line and the recovery work that follows.
- Compatibility check: run producers, consumers, Kafka Connect, and administrative tools through the same authentication, serialization, and offset flows that production uses.
A cost dashboard should expose the boundary between measurement and recommendation. “Requests are high” is an observation. “Increase the cache” is a recommendation that needs a latency, memory, and recovery check. “Move this workload to a different storage tier” is a decision that needs a durability and failure-domain review. Keep the evidence beside the action so a later review can tell whether the change worked.
5How AutoMQ changes the operating model
Apply the ledger and tests to each candidate architecture. A Kafka-compatible Shared Storage architecture can then be evaluated on behavior rather than on a storage slogan. AutoMQ is a cloud-native streaming platform that keeps the Kafka protocol and ecosystem while using S3Stream and shared storage for durable stream data. Its architecture overview describes the broker, cache, WAL storage, metadata, and S3 storage paths that should appear in a FinOps map. The related Diskless Kafka architecture guide explains which responsibilities remain in the broker, while the Kafka compatibility guide gives a starting point for the client test matrix.
The architecture changes the questions a FinOps team can ask. Broker capacity and retained bytes are no longer the same resource decision, so compute and storage can be measured and scaled as separate lines. A broker replacement can use shared durable data instead of copying every retained partition replica, but the recovery read, metadata work, and selected WAL backend still belong in the worksheet. S3 storage can reduce the need for pre-provisioned local capacity, while requests, retrieval, endpoints, and cache behavior still determine the workload bill.
AutoMQ's Kafka compatibility documentation gives the client boundary to test. Its WAL storage documentation is the place to verify the selected WAL type and its deployment constraints. The financial model should keep those choices visible: S3 WAL, Regional EBS WAL, and NFS WAL have different capacity, latency, failure-domain, and provider-cost implications.
AutoMQ does not choose an attribution rule for a platform team. In a BYOC (Bring Your Own Cloud) or self-managed deployment, the team still chooses bucket layout, endpoint topology, retention classes, tags, and allocation rules. The value of the architecture is that those paths are explicit enough to measure: a team can connect Kafka workload identity to broker work, storage requests, object bytes, and network routes without pretending that the broker bill is the whole cost.
6Decision checklist and FAQ
The earlier checks validate a production change. Use this shorter checklist to validate evidence and ownership before a design or budget review:
- Attribution grain: Can every chargeable workload be mapped to a topic, namespace, team, or shared platform pool?
- Path map: Can the team identify the broker, cache, WAL, object-storage, metadata, and network steps for a write, Tailing Read, Catch-up Read, and recovery?
- Usage evidence: Are bytes, requests, compute work, cache signals, and transfer paths exported with stable labels?
- Pricing evidence: Does each unit rate name its provider, region, endpoint, storage class, and retrieval assumption?
- Failure window: Has the model been tested during replay, broker replacement, permission loss, and object-store retry behavior?
- Compatibility: Do Kafka clients, Kafka Connect, and administrative tools pass the same protocol and offset tests in the candidate design?
- Decision owner: Does each recommended change have an engineering owner, a budget owner, and a rollback condition?
- Review cadence: Is the worksheet refreshed when retention, workload shape, storage tier, endpoint, or provider pricing changes?
6.1Is diskless Kafka automatically lower cost?
No. The architecture changes which resources are provisioned and where durable bytes live, but total cost still depends on request patterns, retention, cache behavior, network paths, WAL choice, compute shape, and operational policy. Measure the workload and price the selected provider paths before making a savings claim.
6.2Should teams allocate object-storage cost by topic or by team?
Use the smallest grain that the telemetry can support and the owner can act on. Topic-level allocation helps when retention and replay are the main drivers. Team-level allocation may be more stable for shared platform services. Preserve the raw topic and request dimensions so a team-level total can be explained later.
6.3How should shared broker compute be allocated?
Choose an observable rule and publish it. Request work, bytes processed, partition leadership, or a reserved platform pool can all be valid depending on the service contract. Keep the rule separate from the provider price; changing a rate should not change the measured share.
6.4Does object storage eliminate network cost?
No. Reads, recovery, endpoint placement, cross-AZ paths, and egress can all create network charges. A diskless design can change the replication and movement pattern, but the actual route still needs to be measured in the target region and deployment model.
6.5What should a pilot prove before chargeback?
A pilot should reconcile the ledger with provider usage, show that topic or team labels survive normal and failure paths, and demonstrate that a change in retention, replay, cache, or endpoint produces the expected movement in the corresponding cost line. It should also prove Kafka client compatibility and a documented rollback. Chargeback begins after those checks are repeatable, not when the first dashboard appears.
Use the path map, keep usage separate from price, and include failure behavior in the worksheet. If you want to test the model against a Kafka-compatible Shared Storage deployment, start an AutoMQ evaluation with the worksheet, endpoint map, and replay measurements ready.
