In the stream storage repository S3Stream, WAL is one of the core components. WAL storage primarily addresses issues such as high latency when using object storage as the main data store and low efficiency in writing to multiple partitions. It has two primary responsibilities:
  • Providing low-latency, high-performance data persistence writes. Once the data is successfully written to WAL, confirmation is returned to the client.
  • During a Broker node failure that requires failover, it recovers data from WAL that has not been timely uploaded to S3.

​
WAL Storage Implementation

WAL is implemented as a fixed-size, cyclically written storage space. Various storage media can be selected, but the implementation mainly considers the following points:
  • Centralized writes, unlike Apache Kafka®, AutoMQ does not need to write separate log files for each partition. By mixing the data from all partitions into the WAL, it supports efficient writing in scenarios with a large number of partitions.
  • Sequential writes and group commits, data is written sequentially into the WAL, combined with the group commit mechanism, requiring only a small amount of IOPS to achieve high-throughput writes.
  • Direct IO Writing: Data is directly written through to the storage medium to fully utilize the durability of cloud storage. Write operations are considered successful only upon confirmation, unaffected by the operating system’s Page Cache dirty page collection.
  • Support for Raw Device Writing[1]: AutoMQ needs to write only one file. If EBS is used as the storage medium, it can be used directly as a raw device for writing, without the need to mount a file system, thereby avoiding additional overhead brought by the file system. This ensures optimal performance and latency.
Through the aforementioned technology, WAL storage significantly reduces the dependency of stream storage systems on IOPS, addressing the issues of low write efficiency and high API call costs associated with object storage, thus making S3Stream widely applicable. AutoMQ supports both latency-sensitive and high-throughput systems.

​
WAL Storage Medium Selection

Public Cloud providers generally offer three types of storage services, namely:
  • Block storage, such as AWS EBS, Azure Zone-redundant Disk, GCP Regional Persistent Disk, and Alibaba Cloud Regional ESSD. Among these services, EBS adopts a single AZ multi-copy architecture, while the other block storages use a multi-AZ multi-copy architecture, collectively referred to as Regional EBS.
  • Object storage is the most standard storage service provided by cloud providers, with virtually all cloud providers supporting the standard S3 protocol.
  • File storage, primarily NFS protocol-based file storage services, is also widely used in big data and other business scenarios, such as AWS EFS and AWS FSx series.
The aforementioned storage mediums can be used as WAL storage mediums, with specific differences as shown in the table below.
-
EBS & Regional EBS WAL
S3 WAL
NFS WAL
Multi AZ
  • EBS has a single AZ multi-replica architecture
  • Regional EBS has a multi AZ multi-replica architecture
S3 offers both single AZ and multi AZ product options
NFS offers both single AZ and multi AZ product options
Durability
Between 5 nines and 9 nines
Around 11 nines
Around 11 nines
Latency
Sub-millisecond
Hundred-millisecond
Millisecond-level
Cost
Low
Low
Moderate
Applicable Scenarios
Regional EBS is the best option, suitable for all Kafka use cases
Suitable for most latency-insensitive scenarios, such as logging and monitoring
Low-latency solutions on AWS, such as core transaction matching scenarios
In scenarios where AutoMQ chooses S3 WAL, S3 simultaneously offers both WAL storage and primary storage services. The WAL storage is optimized for writes, requiring fewer S3 API calls. Especially in environments with numerous partitions, S3 WAL provides efficient data ingestion capabilities, while the primary storage component supports efficient data consumption through various compaction mechanisms. Notably, for the open-source version of AutoMQ, only S3-compatible storage services are supported as WAL storage options. This indicates that the open-source version of AutoMQ is a cloud-native Kafka implementation that relies solely on object storage, with a minimal deployment architecture. For the commercial version of AutoMQ, different WAL options are available on various cloud providers to support more workloads.

​
References

[1]. How AutoMQ Achieves High-Performance WAL Based on Raw Devices: https://www.automq.com/blog/principle-analysis-how-automq-implements-high-performance-wal-based-on-raw-devices [2]. Azure Regional EBS: https://learn.microsoft.com/en-us/azure/virtual-machines/disks-redundancy#zone-redundant-storage-for-managed-disks [3]. GCP Regional EBS: https://cloud.google.com/compute/docs/disks/regional-persistent-disk [4]. Alibaba Cloud Regional EBS: https://developer.aliyun.com/special/live/regionalessd_bdrc [5]. AWS EFS: https://aws.amazon.com/efs/ [6]. AWS FSx: https://aws.amazon.com/fsx/netapp-ontap/