Kafka for Telecom & 5G: Diskless Streaming Cost

Telecom Kafka clusters rarely start as a cost problem. They start as a visibility problem: a team needs to stream RAN telemetry, core-network events, call detail records, device logs, edge application metrics, or customer-experience signals into systems that can react while the data is still useful. Kafka fits that shape well because the same event can feed operations, billing, fraud detection, observability, and analytics without each team building a separate ingestion path.

The bill shows up later. Once the platform becomes the shared memory of the network, retention stretches from hours to days, consumer groups multiply, replay becomes normal, and peak-hour traffic determines how much infrastructure must sit idle during quiet periods. Traditional Kafka can handle the workload, but its disk-attached broker model can make telecom-scale event data expensive in a cloud environment.

The better question is not whether Kafka belongs in telecom. It already does in many network-data architectures. The question is which parts of telecom streaming need disk-bound brokers, and which parts are better served by a Kafka-compatible architecture that moves durable storage to object storage and treats brokers as elastic compute.

Telecom Data Streaming Is Becoming Operational Memory

5G and Open RAN change the role of streaming data. In older monitoring architectures, network events were often collected after the fact, aggregated into reporting systems, and used by operations teams when something broke. That pattern still exists, but it is no longer enough for networks that rely on software-defined functions, edge applications, programmable RAN control, and AI-assisted optimization.

Open RAN makes this visible. The O-RAN architecture separates radio-side functions such as Near-RT RIC, O-CU, O-DU, and O-RU from management-side functions such as SMO and Non-RT RIC. The Near-RT RIC is defined around near-real-time control and optimization using fine-grained data collection over the E2 interface, while the Non-RT RIC supports longer-horizon optimization, AI/ML workflows, model updates, and policy guidance.

5G core analytics has a similar shape. 3GPP TS 23.288 defines Network Data Analytics Function (NWDAF) services around analytics subscription, notification, bulk data related to analytics, historical data, and analytics context transfer. The language is telecom-specific, but the infrastructure pattern is familiar to data-platform teams: many producers, multiple consumers, historical replay, and different latency budgets for different use cases.

Telecom teams usually end up with several streaming domains:

Hot operational streams, such as alarms, health signals, and fast remediation events. These need predictable latency and clear ownership because they influence incident response.
High-volume observability streams, such as logs, metrics, traces, CDR-like records, and RAN telemetry. These are often consumed by multiple systems and retained for investigation.
Analytics and AI feature streams, such as subscriber-experience aggregates, cell-level trends, anomaly features, and model-training data. These are replay-heavy and cost-sensitive.
Compliance and audit streams, where retention and data control matter as much as throughput.

The mistake is treating those domains as one undifferentiated bus. A hard real-time control loop and a three-day replayable network log are both "streaming," but they should not force the same storage architecture.

Why Traditional Kafka Gets Expensive in 5G and Open RAN

Kafka's core durability model is log replication. Apache Kafka documents that the log for a topic partition is replicated across a configurable number of servers; under normal operation, each partition has one leader and zero or more followers. This is a proven design.

The trade-off appears when telecom workloads run in a multi-zone cloud deployment. With a replication factor of 3, every retained byte is stored multiple times at the Kafka layer. If followers sit in different availability zones, replica traffic can also become inter-zone network traffic. If consumers are spread across zones and fetch from remote leaders, reads add another source of movement. None of this is exotic. It is what a resilient Kafka deployment is supposed to do.

The cost problem is that telecom workloads hit every multiplier at once.

Cost driver	Why it matters in telecom	What changes in a diskless model
Retention	Network teams often need days of logs for incident review, billing reconciliation, and model features.	Object storage becomes the durable log, so retention is priced closer to cloud storage economics.
Replication	Traditional Kafka stores replicas at the broker layer for availability.	Durability shifts toward cloud storage, reducing duplicated broker storage.
Cross-zone traffic	Multi-AZ brokers can generate replication and consumer traffic between zones.	Zone-aware routing and object-storage-backed persistence can sharply reduce broker-to-broker transfer.
Peak sizing	Busy-hour traffic pushes teams to provision for peaks.	Stateless brokers are easier to scale with changing load.
Replay	Analytics, troubleshooting, and AI pipelines turn historical data into active load.	Separating compute from storage makes replay less tied to broker disk capacity.

This is why "Kafka cost optimization" advice often feels too small for telecom. Compression, retention tuning, better partitioning, and consumer placement all help. But they do not change the fact that a disk-attached broker cluster carries storage, compute, and network movement as a coupled unit.

Diskless Kafka Is Not Tiered Storage With a New Label

Tiered storage and diskless Kafka both involve object storage, but they solve different problems. Tiered storage usually keeps local broker disks in the hot path and offloads older segments to remote storage. That helps with longer retention, yet the broker still owns local storage sizing, local disk performance, and much of the operational behavior of a stateful Kafka cluster.

Diskless Kafka changes the center of gravity. Durable data lives in object storage, while brokers become serving and coordination compute. The practical effect is not that disks disappear from the universe; cloud infrastructure still has physical disks somewhere. The point is that Kafka operators no longer have to size every broker around long-lived local log segments.

For telecom, that distinction matters because the highest-cost data is often not the most latency-sensitive data. Network observability, CDR processing, customer-experience analytics, and AI feature pipelines can be large, replayable, and retained. They need reliable streaming semantics and Kafka compatibility, but they may not need every byte to sit on broker-attached volumes.

This creates a more useful architecture boundary:

Keep the tightest control loops close to the systems that need deterministic latency.
Use Kafka-compatible streaming for events that must fan out across operations, analytics, billing, and AI.
Put high-volume retained logs on an architecture where storage capacity and broker compute scale separately.
Use object storage as the economical durable layer for replayable network data.

That boundary is more honest than saying "use diskless Kafka for all telecom workloads." Some workloads deserve conservative placement near the network function. Others are excellent candidates for diskless architecture because their cost is dominated by retention, replay, and bursty load rather than microsecond-level latency.

A Reference Architecture for Telecom Kafka on Object Storage

A practical telecom streaming architecture starts with domain separation. RAN telemetry, 5G core events, edge logs, CDR records, and observability data should not share one undifferentiated topic namespace. Each stream should carry an explicit owner, retention class, replay expectation, and data-control requirement.

The backbone can still be Kafka-compatible. Producers publish through standard Kafka clients or existing collectors. Consumers include network analytics systems, observability tools, billing pipelines, fraud systems, data lakes, and AI feature services. The change is behind the Kafka API: instead of tying durable storage to broker disks, the streaming platform writes durable data to object storage and scales brokers as stateless compute.

For cloud telecom teams, this gives three architectural benefits:

Retention becomes less punitive. Keeping several days of high-volume logs does not require multiplying hot broker disks by the replication factor.
Elasticity becomes operationally realistic. Brokers can be added, removed, and updated with less data movement because they are not carrying long-lived partitions as local state.
Data control stays compatible with regulated environments. A BYOC deployment keeps data inside the operator's cloud account or VPC, while automation can still feel closer to a managed service.

The cost case should be calculated workload by workload. AutoMQ's published benchmark compares a 1 GiB/s, three-day-retention, three-AZ workload on AWS and reports monthly TCO of $12,899 for AutoMQ versus $226,671 for Apache Kafka in that benchmark setup. That is more than an 80% reduction under those assumptions, driven heavily by storage and cross-availability-zone traffic differences. It is a useful signal, not a universal promise. A telecom operator with lower retention, fewer consumers, a different cloud, or stricter latency constraints will get a different number.

Still, the mechanism is exactly where telecom pain tends to live. If your largest streams are retained network logs and replayable analytics events, the savings do not come from a smaller instance type alone. They come from removing the structural need to treat every retained byte as broker-local replicated disk.

Where AutoMQ Fits Without Turning This Into a Product Brochure

AutoMQ is a Kafka-compatible streaming platform that reimplements Kafka's storage layer on cloud storage while preserving the Kafka protocol surface. That matters because telecom data platforms rarely have the luxury of rebuilding every integration. Fluentd, OpenSearch, Sumo Logic, Kafka clients, connectors, operational scripts, and security processes are already part of the estate.

The most relevant public telecom proof point is LG U+. AutoMQ's case study says LG U+ processes more than 2.2 billion log messages daily in a hybrid public AWS and private cloud environment. The migration preserved Kafka protocol compatibility for existing Fluentd, Sumo Logic, and OpenSearch integrations, moved persistent storage to Amazon S3, and enabled stateless broker operations on AWS ECS with Terraform-managed infrastructure. That is much closer to a real telecom operating model than a synthetic "hello world" streaming demo.

AutoMQ also fits the governance side of telecom architecture. Its BYOC model is designed to run the data plane in the customer's cloud account or VPC; the product page describes data staying in the VPC and infrastructure being managed through metadata rather than requiring data to leave the environment. For operators that care about data sovereignty, private networking, auditability, and cloud-account control, this deployment model is often as important as raw throughput.

The open-source posture matters too. AutoMQ's public repository is licensed under Apache License 2.0. In a telecom architecture, license clarity is not a footnote. It affects procurement, long-term exit options, internal security review, and whether engineering teams can inspect the system that sits in the network-data path.

AutoMQ is not the answer to every telecom streaming problem. If the requirement is an ultra-tight RAN control action with a strict latency budget, that decision belongs close to the RIC, xApp, or network function design. But for high-volume Kafka-compatible ingestion, observability, log retention, analytics fan-out, and replay-heavy AI workloads, AutoMQ is a serious architectural option because it changes the cost model without asking teams to abandon Kafka.

Workload-Fit Checklist for Telecom Teams

The fastest way to misuse diskless Kafka is to evaluate it as a generic replacement for every message path. A better approach is to score each workload against five questions.

Question	Strong diskless fit	Be more careful
How much data is retained?	Hours to days of high-volume logs or events.	Tiny streams where storage is not material.
How often is data replayed?	Frequent replay for incidents, analytics, ML, or backfills.	Pure pass-through streams with little historical value.
How many consumers exist?	Multiple teams and systems read the same event stream.	One producer and one consumer with a narrow purpose.
Is traffic elastic?	Busy-hour peaks, event bursts, and changing subscriber behavior.	Steady load with simple capacity planning.
What is the latency budget?	Millisecond-level streaming is acceptable and cost matters.	Hard real-time control loops require local deterministic paths.

This checklist often separates telecom data into two broad groups. The first group is operational control, where latency, determinism, and failure-domain design dominate. The second group is network intelligence, where the value comes from collecting, retaining, replaying, joining, and analyzing massive event streams. Diskless Kafka is usually more compelling in the second group.

The decision also depends on organizational maturity. A platform team that already operates Kafka, Kubernetes or ECS, object storage, Terraform, and observability pipelines can evaluate diskless Kafka as an architecture upgrade. A team still standardizing topic ownership, schema discipline, and consumer isolation should fix those foundations too. Storage architecture will not rescue a chaotic data contract.

How to Estimate Telecom Kafka Cost Before Migrating

Before replacing a Kafka cluster, build a cost model that makes the invisible visible. Telecom teams often know total cluster spend but not which mechanism produces it. The important split is compute, storage, cross-zone network transfer, object storage API calls, observability overhead, and operations time.

Start with one representative workload rather than the whole estate:

Pick a high-volume topic family, such as network logs, CDR enrichment, RAN telemetry, or observability events.
Measure ingress, egress, retained bytes, consumer fan-out, retention period, partition count, and peak-to-average traffic ratio.
Identify where replicas and consumers cross availability zones.
Estimate the cost of broker-attached storage at the current replication factor.
Compare that against an object-storage-backed design with the same durability and retention requirement.

Do not hide the assumptions. If the model uses AWS, cite the region, storage class, instance family, retention window, and transfer rules. If the model assumes three availability zones, say so. If the savings depend on large retained volumes, say that too. Scenario-based numbers are credible because readers can change the inputs; universal savings claims are not.

One useful pattern is to keep the first migration narrow. Move a high-volume replayable stream, keep clients Kafka-compatible, mirror data during validation, and compare operating behavior under real traffic. The goal is not a dramatic cutover. It is to prove that the architecture can preserve client compatibility, reduce storage and network pressure, and make scaling or rolling updates less painful.

What Success Looks Like

A successful telecom Kafka modernization is not measured only by a lower bill. Cost matters because telecom event volume is enormous, but the platform also has to be boring in production. It should preserve existing clients, survive maintenance, scale with busy-hour traffic, keep data inside the required control boundary, and make replay practical enough that teams actually use historical data.

That is the deeper reason diskless Kafka is interesting for 5G and Open RAN. It does not claim that streaming is new to telecom or that every network function should be rebuilt around one bus. It says the expensive parts of telecom streaming are now storage, elasticity, replay, and cloud network movement, and those are exactly the parts traditional Kafka was not designed to optimize for cloud object storage.

If your Kafka estate is mostly small control messages, the architecture may not pay back. If it is carrying billions of logs, telemetry records, billing events, and analytics features every day, the storage model is no longer an implementation detail. It is the economic foundation of the network-data platform.

FAQ

Is Kafka a good fit for telecom and 5G data streaming?

Kafka is a strong fit for telecom workloads that need durable event ingestion, fan-out to multiple consumers, replay, and integration with analytics or observability systems. It is less appropriate as the primary mechanism for hard real-time control loops where deterministic local latency dominates every other requirement.

What is diskless Kafka?

Diskless Kafka is a Kafka-compatible architecture where durable log storage is moved away from broker-local disks and into cloud storage, while brokers act more like elastic compute. The term does not mean storage hardware disappears. It means Kafka operators no longer size each broker around long-lived local log segments.

How can diskless Kafka reduce telecom Kafka cost by more than 80%?

The reduction is scenario-based. In AutoMQ's published 1 GiB/s, three-day-retention, three-AZ AWS benchmark, the reported monthly TCO difference is more than 80% compared with Apache Kafka. The main drivers are lower duplicated broker storage and sharply lower cross-availability-zone traffic. Your result depends on traffic, retention, cloud region, consumer placement, and latency requirements.

Does diskless Kafka replace Open RAN Near-RT RIC or NWDAF?

No. Near-RT RIC, Non-RT RIC, and NWDAF are telecom architecture functions. Kafka-compatible streaming is infrastructure that can move and retain events used by analytics, operations, and applications. It should support these domains rather than pretend to replace them.

Can telecom teams keep existing Kafka clients?

That is one of the main reasons to consider Kafka-compatible diskless platforms. AutoMQ's LG U+ case study says existing Fluentd, Sumo Logic, and OpenSearch integrations continued without client-code or configuration changes because AutoMQ preserved Kafka protocol compatibility.

When should a telecom operator avoid diskless Kafka?

Be cautious with workloads that have very tight deterministic latency requirements, little retained data, no replay value, or a single narrow consumer path. Diskless Kafka is most compelling for high-volume, replayable, multi-consumer data where storage, elasticity, and cloud network cost are material.