AutoMQ utilizes a shared storage architecture based on WAL and object storage, completely cloud-native. Data persistence is entirely managed by cloud storage services, eliminating dependence on Apache Kafka's ISR multi-replica mechanism.
When writing data, there is no need to replicate partition data as in Apache Kafka, making the entire computing layer's Broker completely stateless. This stateless Broker, enabled by underlying architectural innovation, allows AutoMQ to utilize cost-effective Spot instances and automatic scaling capabilities to reduce computing layer costs.
Moreover, eliminating the need to replicate partition data like Apache Kafka significantly reduces computing instance network usage and disk I/O.
The S3 protocol is the de facto standard for cloud object storage. Building modern data infrastructure on mature S3 is now an industry consensus. Compared to traditional local disks or cloud block storage services, S3 offers unlimited capacity, high durability, and a low-cost advantage.
Cost-wise, S3's per GB storage price is just 10% of EBS. With unlimited capacity, AutoMQ doesn't need to pre-reserve local storage capacity like Kafka. This enables AutoMQ to achieve significant storage-level cost reductions compared to Apache Kafka.
AutoMQ is based on an innovative shared storage architecture, where the persistence of data is ensured by the multi-replica mechanism within the dependent cloud storage, rather than relying on Kafka's ISR multi-replica mechanism.
Since the cluster does not need to replicate partition replicas among multiple Brokers like Apache Kafka, it consumes less network bandwidth than Apache Kafka. This means that, with the same network bandwidth of the computing instance, AutoMQ will have a higher throughput limit.
Cloud providers like AWS and GCP charge $0.02 USD/GB for cross-AZ network costs. Utilizing S3's shared storage architecture, AutoMQ avoids all cross-AZ network traffic costs while maintaining full compatibility with Apache Kafka. Within the AutoMQ cluster, partition data migration is eliminated, preventing cross-AZ network traffic costs.
For consumers, AutoMQ's AZ Aware mechanism ensures consumption within the same AZ. For producers, S3's shared storage feature eliminates cross-AZ network costs during data transmission.
This not only poses significant operational and maintenance challenges for Apache Kafka but also exacerbates over-provisioning issues. Many cost problems among AutoMQ service customers stem from resource over-provisioning.
Apache Kafka's architecture, which is based on local disk storage and computation integration, makes it difficult to scale up and down. At the same time, every time a new computation node is added, the capacity of computation and storage must also be increased. For instance, in the gaming industry, Kafka clusters must be scaled up in advance for new version releases to handle peak throughput. Throughout testing and production, substantial costs arise from significant resource waste.
AutoMQ features a powerful built-in Self-Balancing capability that identifies and manages data hotspots in real-time, ensuring balanced write traffic and QPS across brokers.
This not only eliminates stability risks from Apache Kafka's partition hotspots but also significantly enhances cluster resource utilization, helping customers save resources wasted by low utilization.
Real-world price comparisons with Confluent and Apache Kafka underscore AutoMQ's notable cost reduction benefits.
.png)
