Skip to Main Content

Biweekly #2: Cold Read and Hot Access Isolation

List of community contributors​

Selected updates from AutoMQ Kafka​

https://github.com/AutoMQ/automq-for-rocketmq/pull/728

https://github.com/AutoMQ/automq-for-rocketmq/pull/729

https://github.com/AutoMQ/automq-for-rocketmq/pull/743

Originally, all writes and callbacks were processed in a single-threaded thread pool to ensure data safety, posing issues with thread context switching communication and queuing in single-thread processing. This optimization has restructured the data architecture in the writing process into a thread-safe concurrent mode, allowing concurrent writing across different streams, reducing the average write latency of the AutoMQ Kafka client by 0.3ms.

Fast and Slow Read Isolation​

https://github.com/AutoMQ/automq-for-kafka/pull/472

Isolate fast reads from the cache and slow reads from S3 to prevent slow reads from monopolizing the thread pool and affecting fast read performance.

Spot Instance Forced Recapture Disaster Recovery​

https://github.com/AutoMQ/automq-for-kafka/issues/447

Spot instances, often priced up to 90% less than on-demand instances, can be forcibly reclaimed without notice. This feature supports disaster recovery in seconds by remounting data volumes to surviving machines in the event of Spot instance forced recapture.

AutoMQ RocketMQ Feature Highlights​

Stream Module Performance Optimization​

Cold Read Optimization for Historical Data​

https://github.com/AutoMQ/automq-for-rocketmq/pull/766

Historical Data Catch-up Read Optimization: Fetch requests (50MB & 50 stream) cold reads penetrating to S3 have been optimized from 4s per Fetch to 100ms. Even with full penetration cold reads, the S3 read throughput is exceptionally high.

Optimization of read and write latency in LogCache​

https://github.com/AutoMQ/automq-for-rocketmq/pull/731

Added a record of the last cache read position to avoid binary search for locating data in each LogCache read, reducing the time for 100,000 messages and 100,000 queries from 71s to 86ms.

https://github.com/AutoMQ/automq-for-rocketmq/pull/701

By implementing read-write locks, upgraded LogCache to a thread-safe data structure, enhancing the concurrent read efficiency of LogCache.

Released version v0.0.3-alpha​

https://github.com/AutoMQ/automq-for-rocketmq/releases/tag/v0.0.3-alpha

This version includes the following features and optimizations

  • Stability and performance enhancement: Fixed potential OOM issues and improved performance of the stream module, see [Changelog]

  • Engineering improvements: Introduced [Nightly build] and [E2E test CI]

  • Enhanced observability: Introduced Metrics for the stream module; introduced Trace for the Proxy and Store modules.

Releasing Helm Chart​

Helm Charts now enable rapid deployment of AutoMQ RocketMQ clusters within Kubernetes


$ helm repo add automq https://charts.automq.com
$ helm search repo automq                                                                                                                                                            
NAME                            CHART VERSION   APP VERSION     DESCRIPTION                                                                                                           
automq/automq-for-rocketmq      0.0.4           v0.0.3-alpha    A Helm chart for automq-for-rocketmq

Deploying this Chart will set up an AutoMQ RocketMQ Broker along with necessary MySQL and Minio components. Optional observability dependencies will be progressively added.

Launching Documentation Site​

Introduces basic usage of AutoMQ RocketMQ, including: local builds, deployment using docker compose, deployment on Kubernetes. It also covers cluster management using CLI, and managing resources like Topics.

#Link to documentation site to be added after deployment in Beihai

More Things​