Skip to Main Content

Biweekly #3: Metrics Integration and GC Optimization

AutoMQ's latest release enhances scalability with OpenTelemetry metrics integration and GC optimizations for Kafka, offering robust cloud-native performance and cost efficiency.

Biweekly #3: Metrics Integration and GC Optimization

List of community contributors

New community contributors this week: Luo Tianxiang from Shanghai Telecom

https://github.com/AutoMQ/automq-for-kafka/pull/517

This PR simplifies the network configuration options for AutoMQ Kafka.

AutoMQ for RocketMQ Development Updates

Release Updates

We have launched the first Beta version, v0.1.0-beta, officially marking the entry of AutoMQ for RocketMQ into the Beta phase: [Release v0.1.0-beta]

In this release, we primarily focused on introducing new features and performance enhancements, along with augmenting our code quality through continuous unit testing

Support for Transactional Messages

In this version, we have implemented support for Apache RocketMQ's transactional message type, ensuring full compatibility with all three advanced message types of Apache RocketMQ (sequential messages, timed messages, transactional messages)

Moving forward, we will continue to address the pain points associated with message duplication in Apache RocketMQ, with message idempotence now added to our Roadmap. We invite developers to keep an eye on our ongoing updates

https://github.com/AutoMQ/automq-for-rocketmq/pull/790

https://github.com/AutoMQ/automq-for-rocketmq/pull/796

Added Operational Tools

To facilitate developer testing and use, we have added several MQAdmin tool commands, including:

  • Resetting consumer offsets

  • Querying consumer progress

  • Querying Topic positions and other metadata

  • Querying client connections

  • ...

https://github.com/AutoMQ/automq-for-rocketmq/pull/800

https://github.com/AutoMQ/automq-for-rocketmq/pull/802

https://github.com/AutoMQ/automq-for-rocketmq/pull/807

Supporting queue reassignment

Leveraging Stream based on object storage for shared storage capabilities, we have implemented partition reassignment in a few seconds to another node, also providing a CLI tool for easy operation and maintenance. Developers can easily test this feature using [Docker compose to deploy clusters]

https://github.com/AutoMQ/automq-for-rocketmq/pull/818

Additionally, through message forwarding, we have achieved an almost imperceptible migration experience for clients: using two senders to send 4k messages at a fixed 500tps to one queue, after stabilizing the traffic, the queue is reassigned to another node, with the reassignment completed in seconds and traffic quickly restored. Detailed test data can be viewed in PR #821

https://github.com/AutoMQ/automq-for-rocketmq/pull/821

Enhanced cold read performance

This version has enhanced cold read performance by optimizing metadata interfaces and eliminating unnecessary long polling in cold read scenarios. Additionally, since cold read data is retrieved from object storage, it benefits from ultra-high throughput, independent of disk IOPS and bandwidth constraints.

https://github.com/AutoMQ/automq-for-rocketmq/pull/827

https://github.com/AutoMQ/automq-for-rocketmq/pull/828

AutoMQ for Kafka trunk update

Release of version 1.0.0-rc0

https://github.com/AutoMQ/automq-for-kafka/releases/tag/1.0.0-rc0

Enhanced observability: Metrics integrated with OpenTelemetry

https://github.com/AutoMQ/automq-for-kafka/pull/548

Kraft ObjectDelta GC optimization

https://github.com/AutoMQ/automq-for-kafka/pull/547

Added DeltaMap data structure to reduce the amount of data copied from ObjectDelta to ObjectImage.

Cold read GC optimization

https://github.com/AutoMQ/automq-for-kafka/issues/543

Broker consumption throughout the entire chain utilizes PooledByteBuf to reduce heap memory allocation, eliminating Allocation Stall under zgc.