A FinOps Checklist for Auditing a Kafka Deployment Before Renewal Season

At 9:12 on a Monday morning, a platform owner forwards a three-year Apache Kafka renewal quote to FinOps. The questions are not: how much capacity is used, which bytes cross Availability Zones (AZs), how much data will be retained, and how long it would take to move if the contract stopped making sense.

Those questions cannot be answered by a feature matrix. A Kafka contract can look stable while its cost is driven by peak capacity, replication paths, retention defaults, or provider-specific operational work. Renewal is a useful audit point, even when the eventual decision is to stay.

Treat the renewal as a review of the system that produces the bill, not a negotiation over the line item. The checklist turns utilization, traffic, storage, and exit readiness into shared evidence.

1Renewal season is decision season

Start with a defined measurement window. Pull invoices and usage exports for a period that includes ordinary traffic, the largest known peak, replay activity, and a maintenance or recovery event when available. Pair that record with broker metrics, partition placement, topic configuration, incident history, and the renewal terms. Label each number as measured, calculated, or assumed, and separate provider charges from adjacent costs such as connector workers, private networking, observability, and operations.

Use four owners for the evidence:

1. FinOps owns invoices, rate cards, commitment discounts, and allocation rules.
2. Platform engineering owns topology, broker capacity, partition distribution, and recovery behavior.
3. Application teams own throughput shape, replay, retention requirements, and data criticality.
4. Procurement owns renewal dates, minimum commitments, price protections, and termination conditions.

The important question is not whether the bill can be reduced in isolation. It is which assumptions the contract would lock in.

2Utilization: find the capacity reserved for a short peak

Begin with broker-hours and the resources that force them to exist. Review CPU, network throughput, disk occupancy, partition count, connection count, and request latency by broker and by hour. Report median, p95, and peak values over the same window. Percentiles show whether a cluster is continuously busy or carrying a large reserve for a narrow event.

That reserve may be justified. A service objective can require headroom for failover, a partition-skew pattern can leave some brokers full while others are quiet, and a recovery plan can require capacity that normal traffic does not consume. The audit is not asking the team to remove that safety margin. It is asking the team to name the reason, the owner, and the test that proves the reason still applies.

Use this basic model to make the conversation concrete:

reserved capacity = provisioned capacity - measured demand
unexplained reserve = reserved capacity - documented service and recovery headroom

“Capacity” could be broker throughput, CPU, disk, or a provider unit. If the unexplained reserve is large, ask what prevents a smaller footprint during ordinary periods and what the scale-up path costs when the peak arrives.

Do not treat an average as a target capacity. A low average with a high p95 may call for elastic capacity, traffic shaping, or a different storage model. A high average with low variance may support a longer commitment, provided the growth assumption is documented. The commercial choice follows the workload shape; it should not be inferred from one utilization graph.

3Traffic: trace the bytes before accepting the network bill

Kafka throughput is a logical measure. A provider bill is a record of physical movement and the billing rules attached to it. Draw the paths for producer ingress, broker replication, consumer reads, connector traffic, replay, recovery, and cross-region copies. Mark the AZ, region, account, and network service at each boundary.

The useful calculation is not “messages per second multiplied by a price.” It is:

monthly network cost = billable GiB × provider rate per GiB
billable GiB = sum of measured paths × provider-specific billing directions

For a traditional Kafka deployment, replication factor, broker placement, and follower traffic can make physical movement larger than producer ingress. A consumer in another AZ can add a separate path, and a recovery event can create a temporary spike that never appears in a monthly average. The coefficient depends on placement, compression, client behavior, and provider data-transfer rules, so use the formula as a measurement plan rather than a universal multiplier.

Pull network usage from the provider’s billing export and compare it with broker metrics, flow logs, and service-specific usage. AWS Cost Explorer can help isolate data-transfer dimensions, while the AWS EC2 pricing page documents applicable conditions. Repeat the review with the relevant page for Azure or Google Cloud when needed. Rates and exemptions vary by service, direction, region, and network path, so record the pricing page and retrieval date.

Traffic ownership matters as much as traffic volume. Platform engineering can change broker placement or replication policy, but an application team may control a replay job, and security may control an inspection path. Assign every material path to one owner and write down whether it is required, avoidable, or not yet understood. That classification gives procurement a better question than “can you lower the network price?” It asks whether the renewal would preserve a topology that creates billable movement.

Before modeling savings, check the Kafka controls that alter traffic and durability: replication factor, min.insync.replicas, rack or AZ awareness, producer compression, consumer locality, quotas, and throttling. These controls protect different objectives. Lowering replication or throttling a recovery may change resilience or recovery time, not only cost. A network line item is not an invitation to remove a safety control without updating the service objective.

4Storage: retention determines the long tail

Storage audits fail when they look only at today’s used bytes. Rank topics by retained byte-months, then separate hot reads from data kept for replay, audit, backfill, or regulatory reasons. Check retention.ms, retention.bytes, log compaction, segment behavior, and tiering policy. Kafka’s topic configuration documentation and KIP-405 are useful references, but a feature name is not a cost model: measure local storage, object storage, requests, retrieval, and egress for the implementation you run.

For a first estimate, use:

retained data = average ingest rate × retention window × effective copy factor
storage cost = retained GiB × storage rate per GiB-month
              + requests + retrieval + egress + attached-cache cost

The effective copy factor is not always the configured replication factor. Compaction can remove older records, compression changes physical bytes, and a tiered design may keep only a hot set on broker storage. Replay or rehydration can create request and retrieval charges that a byte-month calculation misses. State each assumption before comparing options.

Ask storage questions in time order:

• What data must be available immediately, and what data is acceptable to retrieve from a colder tier?
• Which topics are retained for business use, and which exist because the default was never revisited?
• Does a retention increase add object storage only, or does it force more broker disk and therefore more compute?
• During broker replacement or scale-out, does data move between brokers, or can new capacity read shared durable data?

Long retention is where storage architecture starts to shape renewal risk. If durable data is bound to broker-local disks, a retention decision can also become a broker-sizing decision. If the provider offers a remote tier, confirm its read latency, minimum retention, request pricing, retrieval behavior, and deletion semantics. If the provider offers a shared-storage architecture, confirm the WAL (Write-Ahead Log) medium, cache policy, object-store access pattern, and failure boundary. The label “object storage” hides several cost and recovery choices.

Do not reduce retention to meet a commitment until the data owner signs off. A lower bill that removes the recovery history needed by an incident response or backfill process is a transfer of cost, not a saving. Record the owner and the business purpose for each long-lived topic before changing its policy.

5Exit options: preserve choice before you sign

An exit audit is not a migration announcement. It tests how much of the platform is portable and how much of the operating model is provider-specific. List the assets that would have to move: Topics and configurations, offsets, schemas, connectors, security identities, client bootstrap settings, monitoring, runbooks, and replayable data.

Then classify each asset:

The decision framework should produce one of three outcomes. Renew when utilization is understood, network and storage behavior are predictable, the operating boundary earns its cost, and an exit test has no unresolved blocker. Renegotiate when the platform fits but the commitment, minimum spend, rate protection, or support terms do not match the workload. Evaluate a platform change when broker-local storage, cross-AZ replication, retention growth, or rebalance work drives a cost curve the current model cannot separate.

Make the exit path executable. Export a representative slice of data, run shadow consumers or mirrored traffic where appropriate, validate client and connector behavior, measure catch-up time, and document rollback. The test does not need to prove a complete migration. It needs to expose the work hidden behind the phrase “Kafka-compatible.” Compatibility at the protocol layer does not automatically cover schemas, offsets, security, connector semantics, or operational ownership.

This is also the point to derive architecture requirements. If the audit shows that compute is idle outside peaks, storage growth is driving broker expansion, and cross-AZ replication is a recurring charge, a candidate platform should be tested against four requirements: Kafka protocol compatibility, storage that is shared rather than broker-bound, brokers that can be replaced without moving durable history, and a deployment boundary the customer can operate and audit. Those requirements come from the bill and the recovery plan, not from a product brochure.

6When a shared-storage review is warranted

AutoMQ is relevant when those requirements are present. It is a Kafka-compatible streaming platform built around a Shared Storage architecture. Its architecture overview describes Kafka request processing on brokers with S3Stream, WAL storage, data caching, and S3-compatible object storage beneath it. The architectural question is whether durable stream data remains independent of broker ownership, not whether object storage is automatically less expensive.

That distinction maps directly to the audit. Stateless brokers can change the capacity question because adding or removing compute does not mean copying the full retained history between broker-local disks. Shared storage can change the traffic question because durable data is available across nodes without relying solely on broker-to-broker replication. The result still has to be measured: object-storage requests, retrieval, cache, WAL, and client traffic remain part of the bill, and a WAL choice can change latency, failure domains, and storage charges.

For a customer-owned cloud account, review AutoMQ BYOC environment documentation alongside the compatibility test. The relevant questions are where the control plane and data plane run, who owns the object-storage account, how identities and network paths are audited, and which operational tasks remain with the platform team. The Kafka compatibility documentation is a starting point, not a substitute for testing the application estate, connectors, schemas, and recovery runbooks.

7The printable renewal checklist

Bring these questions to the renewal meeting and require an evidence source for each answer:

• What are the p50, p95, and peak broker, storage, and network demands?
• Which reserve is required by a service objective, and which reserve is unexplained?
• Which network paths are cross-AZ, cross-region, or provider egress, and who owns each one?
• What is the effective retained data set after compression, compaction, and tiering?
• Which charges come from storage bytes, requests, retrieval, egress, cache, and WAL, and does retention growth require more broker compute?
• Which client, offset, schema, connector, identity, and runbook assets have been tested outside the current provider boundary?
• Is the decision to renew, renegotiate, or evaluate a platform change supported by measured evidence?
• What must be tested before the next contract anniversary?

Return to the renewal email that started the review. The useful response is no longer “the quote looks familiar.” It is a documented answer to what the contract pays for, which assumptions could change, and how the team would respond. For a workload that points toward shared storage and Kafka compatibility, request a workload-specific AutoMQ review using the same evidence sheet rather than a separate product scorecard.

8References

• Apache Kafka design documentation
• Apache Kafka topic configuration: retention.ms
• KIP-405: Kafka Tiered Storage
• AWS Cost Explorer
• AWS EC2 On-Demand pricing and data transfer
• AWS S3 pricing
• AWS EBS pricing
• Amazon MSK pricing
• AutoMQ Shared Storage architecture
• AutoMQ compatibility with Apache Kafka

9FAQ

9.1When should a Kafka FinOps audit start before renewal?

Start when there is enough time to collect a representative billing window, capture a known peak or replay event, and run one controlled exit or architecture test. Waiting until the quote is final removes useful evidence.

9.2Should FinOps own Kafka architecture decisions?

FinOps should own cost evidence, allocation rules, and commercial questions. Platform engineering, application, security, and procurement teams should own the technical and contractual inputs. A shared record prevents topology costs from being treated as pricing problems or retention requirements as infrastructure preferences.

9.3Is Tiered Storage the same as Shared Storage?

No. Tiered Storage can move older data to a remote tier while a broker still owns local state and serves the hot path. Shared Storage makes durable stream data available independently of a broker’s local ownership. The right choice depends on read patterns, recovery objectives, latency, cost, and operational boundaries, so compare the actual implementation rather than the label.