Skip to Main Content

AutoMQ Benchmark

AutoMQ, utilizing a compute-storage separation architecture based on S3 shared storage, enhances the Apache Kafka experience by providing benefits like rapid elasticity, cost efficiency, and high performance, all while maintaining complete compatibility. Compared to Apache Kafka, AutoMQ offers better throughput performance during cold reads and supports higher maximum throughput. This document details how to conduct performance testing on an AutoMQ cluster.

Prerequisites​

Before conducting cluster performance testing, ensure the following criteria are met:

  1. Complete the installation and deployment of the AutoMQ cluster. You can refer to the following methods to install and deploy AutoMQ:
  1. Prepare the necessary load generator resources. It's recommended to create a sufficient number of load generators within the same VPC network to ensure that network throughput and CPU of the load generators do not become bottlenecks in the test scenarios.

Test Scenario​

Scenario 1: Tail Read​

Tail Read, also known as "tailing read" or "hot read," involves testing the scenario where the offset gap between Producer and Consumer is minimal. The focus for hot write and hot read scenarios is primarily on the following metrics:

  • Read/Write Throughput: Real-time read/write scenarios are typical for Kafka. With the same hardware resources, higher throughput indicates better performance.

  • Write and E2E Latency: Under the same network throughput conditions, lower message write latency is preferable, and lower E2E latency (the time taken from production to consumption of a message) is ideal.

In the Tail Read scenario, messages sent by the Producer are immediately consumed by the Consumer once they're written into the Broker. At this point, the messages consumed by the Consumer come directly from the AutoMQ Log Cache, eliminating the need to read from object storage, thereby resulting in lower resource consumption.

Scenario 2: Catch-Up Read​

Catch-Up Read, also known as "cold read," tests situations where the Consumer's consumption offset is significantly behind the Producer's offset. In this scenario, the Consumer is initially paused, allowing a certain amount of messages to accumulate before resuming consumption. At that time, the messages consumed by the Consumer are read from object storage, pre-read, and cached by the Block Cache.

In this test scenario, focus primarily on the following two metrics:

  • Is the catch-up read speed sufficient? Observe whether the consumption speed of each Consumer Group exceeds the Producer's writing speed. Only if the consumption speed exceeds the writing speed can the Consumer catch up with the Producer.

  • Does write throughput get impacted during data catch-up? Monitor whether the throughput of messages sent by the Producer decreases during the catch-up process and whether the send latency increases.

Test Using Kafka CLI​

Users can conduct performance tests by using the Kafka CLI tools, executing kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh.

If the previous AutoMQ cluster has been deployed by following the reference Deploy Multi-Nodes Cluster on Linux▸, then the cluster Bootstrap addresses will be similar to "192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092".

Tip

Please replace the bootstrap-server address below with the actual address of the deployed cluster based on the configuration.

Creating Topic​


docker run automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server 192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092"

Publishing Messages​


docker run automqinc/automq:latest /bin/bash -c "/opt/kafka/kafka/bin/kafka-producer-perf-test.sh --topic test-topic --num-records=1024000 --throughput 5120 --record-size 1024 --producer-props bootstrap.servers=192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092"

Consuming Messages​


docker run automqinc/automq:1.0.4 /bin/bash -c "/opt/kafka/kafka/bin/kafka-consumer-perf-test.sh --topic test-topic --show-detailed-stats --timeout 300000 --messages=1024000 --reporting-interval 1000 --bootstrap-server=192.168.0.1:9092,192.168.0.2:9092,192.168.0.3:9092"

Testing with AutoMQ Performance Tool​

The AutoMQ team has enhanced the Kafka performance testing CLI tools developed by the community, releasing a simplified stress testing tool. This performance testing tool is built on the OpenMessaging Benchmark framework and provides the following benefits:

  • Support for Multiple Clients in a Single Process, Improving Testing Density and Efficiency: Compared to the native Apache Kafka scripts, kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh, the automq-perf-test.sh allows you to launch multiple Producers and Consumers simultaneously within a single process and send and receive messages across multiple Topics. This capability is more representative of real-world scenarios and offers greater ease of use.

  • One-Click Execution Without Distributed Deployment: Unlike the OpenMessaging Benchmark framework, the automq-perf-test.sh does not require distributed deployment of multiple Workers. Tests can be executed with a single click on a single machine, making it more convenient for scenarios that do not involve particularly large-scale testing.

  • Provides Catch-Up Read Testing Scenarios: Additionally, the automq-perf-test.sh script supports more complex cold read scenarios by enabling the launch of multiple Consumer Groups, with each Group starting consumption from different offsets. This approach prevents cache reuse during cold reads, enabling performance testing under more extreme conditions.

  • Open and Neutral: This testing script relies solely on the Apache Kafka Client, supporting performance testing on stream systems that are compatible with the Kafka protocol, such as Apache Kafka and MSK.

Performance Tuning and Technical Support​

The performance of AutoMQ clusters is influenced by several factors, including the specifications of computing resources and kernel parameter tuning. Performance tuning and baseline calibration for production environments can be quite complex.

You can reach out to the AutoMQ team via this form to receive performance test reports and best practices for performance tuning in production environments.