Uber pioneered Tiered Storage for cloud
AutoMQ released fully Kafka-compatible Diskless architecture
Why S3? Scale, Standard, and Savings
Scale brings Elasticity. Elasticity brings Efficiency. S3 is not just storage; It is the Inevitable Destination for All Data.
Unmatched Scale
500 Trillion+ Objects
> 1 PB/s Peak Traffic
1/4th the Cost of EBS
EBS gp3: $0.08 per GiB
S3 Standard: $0.023 per GiB
The Universal Standard
Available in EVERY Cloud and Region, accessible via simple HTTP.
Engineered for S3: A True Diskless Architecture
Legacy Kafka treats S3 as backup. We treat S3 as the Primary Store. Through intelligent Write-Ahead Logging and Memory Cache Layer, we deliver S3-native high-performance real-time streaming.
Constant S3 API Cost: The O(1) Write Model
The Problem
S3 API calls are expensive. Traditional one-file-per-partition mapping leads to O(N) cost growth. More partitions = higher bills.
The Solution
Aggregate writes from all partitions into a single WAL Object. Write frequency stays constant whether you run 10 or 10,000 partitions.
The Result
Batch first, split later. O(1) API costs regardless of partition count. Your S3 bill stays flat as you scale.
Millisecond Latency: The Abstracted WAL Layer
The Problem
Direct S3 latency (hundreds of ms) kills real-time streaming. Object storage alone can't deliver low-latency pipelines.
The Solution
Abstract WAL with flexible backends—Regional EBS, NFS/FSx, or S3. All options support Multi-AZ durability with zero RPO. Choose based on your latency requirements.
The Result
Writes hit WAL at sub-10ms. Async flush to S3 in background. Tiny WAL window delivers premium latency at negligible cost.
Extreme Read Efficiency: The Dual-Cache Engine
The Problem
S3 is slow. OS page cache suffers from noisy-neighbor drops. Traditional caching can't isolate hot and cold streams.
The Solution
Bypass OS with Direct Memory Dual-Cache. Hot reads (tailing) from WAL Cache (FIFO). Cold reads (catch-up) from LRU Block Cache.
The Result
5x faster cold reads with guaranteed isolation. Tailing gets zero S3 latency—served directly from memory. No cross-contamination.
Zero Cross-AZ Costs: The Shared Storage Relay
The Problem
Cross-AZ transfer is cloud Kafka's hidden tax ($0.02/GB). Replication + remote clients >3x network bill.
The Solution
"Stay Local, Store Regional" architecture. No broker-to-broker replication. No cross-AZ client-broker traffic. All sharing via Regional Storage.
The Result
Delete the network bill. Multi-AZ availability at Single-AZ cost. Clients never leave their AZ. Zero replication traffic.
True Elasticity: Instant Scaling and Balancing
The Problem
Traditional Kafka scaling moves terabytes, takes hours, forces over-provisioning. Rebalancing destabilizes clusters.
The Solution
Stateless brokers via shared storage. Partition reassignment = metadata update. Auto-Balancer eliminates hotspots instantly.
The Result
Scale 3→30 brokers in seconds, back to 3 when quiet. Zero data movement. Stop paying for idle capacity. Waste goes to zero.
All on S3. Low-Latency WAL When You Need It.
S3 for everything is the default—simplest, most elastic, lowest cost. For real-time workloads, switch to a low-latency WAL without changing your architecture.
All on S3
Typical latency: ~500msLow-Latency WAL
Typical latency: <10msRecommended Low-Latency WAL by Cloud
Not Sure Which Configuration Fits?
Our team can help you choose the right storage setup based on your latency requirements and budget.
Talk to Our ExpertsWhat Industry Leaders Say
"AutoMQ offers Kafka compatibility while cutting operational burden, cross-AZ costs, and painful rebalances. The shared-durability model (object storage) lets you treat brokers as elastic compute—scaling becomes a capacity decision. For ultra-low-latency needs, NFS-backed deployment provides a dedicated option alongside cost-efficient object storage. Being open-source, it's inspectable, reduces lock-in, and lets teams validate behavior under their own workloads."
FAQs
Don't see an answer to your question? Check our docs, or contact us.
Available on Cloud Marketplaces
Subscribe to AutoMQ directly from your preferred cloud platform
