Skip to main content

​
S3Stream: Shared Streaming Storage

S3Stream is a core stream storage component in AutoMQ, adhering to AutoMQ’s concept of decoupling storage and computation. It offloads Apache Kafka’s built-in ISR-based log storage layer to object storage and provides accelerated WAL storage capabilities. S3Stream is a stream storage library rather than a distributed storage service. AutoMQ innovatively implements a set of core stream storage APIs on top of object storage, including position management, Append, Fetch, and Trim data. The following code snippet shows several core interfaces of these APIs. 
public interface Stream {
    /**
     * Get stream id
     */
    long streamId();

    /**
     * Get stream start offset.
     */
    long startOffset();

    /**
     * Get stream next append record offset.
     */
    long nextOffset();

    /**
     * Append RecordBatch to stream.
     */
    CompletableFuture<AppendResult> append(RecordBatch recordBatch);

    /**
     * Fetch RecordBatch list from a stream. 
     */
    CompletableFuture<FetchResult> fetch(long startOffset, long endOffset, int maxBytesHint);

    /**
     * Trim stream.
     */
    CompletableFuture<Void> trim(long newStartOffset);
}

​
Core Characteristics of Stream Storage

All data on the internet is generated in a streaming manner, then stored and computed in streams to extract the business value of real-time data. This also implies that stream data imposes at least the following requirements on storage:
  • Low Latency: The greatest value of streaming data lies in its freshness. For example, businesses related to ad recommendations have very high real-time requirements. The faster data is stored and computed, the more value it can provide.
  • High Throughput: Since all data is generated in a streaming manner, stream storage must support extremely high throughput. Many businesses require at least GiB/s bandwidth.
  • Low Cost: Massive amounts of streaming data mean high storage costs. Additionally, many businesses need data playback and re-computation capabilities, making daily storage of stream data a business norm.
Object storage offers significant advantages in terms of scalability and cost for big data, particularly within data lake ecosystems. However, inherent characteristics of object storage, such as 100ms-level latency per API call and billing by API usage, limit its application in low-latency and high-IOPS storage scenarios, which are critical requirements for stream storage. To overcome this limitation and make object storage suitable for stream storage scenarios, AutoMQ introduces a WAL (Write-Ahead Log) storage module within S3Stream. This module leverages centralized storage to mix data from all partitions of a node into a single WAL file or object, achieving high write efficiency and low IOPS consumption. AutoMQ innovatively combines WAL storage and object storage, addressing the deficiencies of object storage in stream storage scenarios by providing low latency, high throughput, low cost, and nearly unlimited capacity for stream storage.

​
S3Stream Architecture

In the core architecture of S3Stream, data is first durably written to the WAL and then almost in real-time uploaded to S3 storage. Additionally, to efficiently support both Tailing Read and Catch-up Read models, S3Stream incorporates a data caching component to accelerate read operations.
  • WAL Storage: Select different storage media based on your business’s latency and durability requirements; options include block storage, object storage, or file storage.
  • S3 Storage: The largest object storage service offered by cloud providers is selected to provide high-throughput and cost-effective primary data storage services.
  • Data Caching: Both hot data and prefetched cold data are stored in the cache to accelerate reads. At the same time, an efficient eviction mechanism based on consumer focus ensures optimal memory utilization.
Notably, for the open-source version of AutoMQ, only S3-compatible storage services are supported as a WAL storage option. This indicates that the open-source version of AutoMQ is a cloud-native Kafka implementation dependent solely on object storage, offering a highly streamlined deployment architecture. For the commercial version of AutoMQ, various WAL storage options are provided across different cloud providers to support a broader range of workloads.