Skip to Main Content

Biweekly #7: Index Caching and Grafana Dashboards Replica

Community Contributors List

This week's new community contributor: @lifepuzzlefun

https://github.com/AutoMQ/automq-for-rocketmq/pull/903

This PR optimizes the additional List copy during the binary search in LogCache

AutoMQ mainline dynamics

Transaction index and timestamp index file caching

https://github.com/AutoMQ/automq-for-kafka/pull/740

https://github.com/AutoMQ/automq-for-kafka/pull/743

https://github.com/AutoMQ/automq-for-kafka/pull/745

Designed file caching that is more suitable for small data, repeated reading, and binary search with skip reading, based on the querying characteristics of transaction and timestamp indices:

  • Cache space limited to 200MB, does not scale linearly with the number of partitions;

  • While persisting indices, data is simultaneously written to cache, allowing the cache space to support the merging of small Objects into large Objects in S3Stream, thus maintaining high efficiency in index query performance for AutoMQ Kafka;

Accelerated WAL recovery

https://github.com/AutoMQ/automq-for-rocketmq/pull/910

By implementing a buffer reading mechanism, the recovery time of WAL has been accelerated from 92.7s to 15.3s

Grafana User & Developer Dashboards

https://github.com/AutoMQ/automq-for-kafka/pull/778

New dashboards have been added for Grafana users and developers:

  • In the user view, users can overview the entire cluster's performance and drill down to individual Broker dashboards, Topic dashboards, and Group dashboards as needed;
  • In the developer view, developers can monitor details such as Broker CPU, JVM, request throughput & latency, and the internal operations of S3Stream;

More Things