Prometheus/VictoriaMetrics

Introduction

In modern enterprises, as the demand for data processing continues to grow, AutoMQ [1] emerges as a key component for real-time data processing due to its efficient and cost-effective stream processing capabilities. However, with the expansion of cluster scale and increased business complexities, ensuring the stability, high availability, and performance optimization of AutoMQ clusters becomes critically important. Thus, integrating a robust and comprehensive monitoring system is essential for maintaining the healthy operation of AutoMQ clusters. Prometheus[2], with its powerful data collection and querying capabilities, has become the ideal choice for monitoring AutoMQ clusters within enterprises. To address Prometheus's limitations in large-scale data storage, VictoriaMetrics[3] serves as an efficient time-series database storage backend that provides enhanced storage and query performance. By integrating Prometheus and VictoriaMetrics, enterprises can achieve comprehensive monitoring of AutoMQ clusters, ensuring system high availability and performance optimization.

AutoMQ Overview

AutoMQ is a cloud-reimagined stream processing system that maintains 100% compatibility with Apache Kafka while significantly enhancing cost efficiency and elasticity capabilities by offloading storage to object storage. Specifically, AutoMQ constructs a stream storage repository, S3Stream, built on S3, offloading storage to the shared cloud storage EBS and S3 provided by cloud vendors. This offers low-cost, low-latency, high-availability, high-reliability, and infinite capacity stream storage capabilities. Compared to traditional Shared Nothing architectures, AutoMQ adopts a Shared Storage architecture, significantly reducing storage and operational complexity while enhancing system elasticity and reliability.

The design philosophy and technical advantages of AutoMQ make it an ideal choice for replacing existing Kafka clusters in enterprises. By adopting AutoMQ, enterprises can significantly reduce storage costs, simplify operations, and achieve automatic scaling and traffic self-balancing of clusters, thus efficiently responding to changes in business demands. Moreover, AutoMQ's architecture supports efficient cold read operations and zero service interruption, ensuring stable operation under high load and sudden traffic conditions.

Prometheus Overview

Prometheus is an open-source system monitoring and alerting toolkit initially developed by SoundCloud and released as open source in 2012. It offers robust data collection, storage, and querying capabilities, enabling real-time monitoring of system and application performance metrics. Prometheus utilizes a multidimensional data model that uses labels to tag and query monitoring data, and it supports the flexible query language PromQL. The primary advantages of Prometheus include efficient data collection, strong query functionality, and a rich alerting mechanism, making it widely used in the cloud-native ecosystem.

However, Prometheus has certain limitations in large-scale data storage, particularly when it comes to long-term storage and high-concurrency querying. Its storage performance and query efficiency may be affected. Consequently, integrating an efficient storage backend becomes essential to address these challenges.

VictoriaMetrics Overview

VictoriaMetrics is a high-performance, open-source time-series database designed for high-concurrency data writing and querying. It features efficient data compression and storage capabilities, offering high throughput for data writes and queries with minimal hardware resource consumption. VictoriaMetrics supports Prometheus' remote storage interface, allowing seamless integration into existing Prometheus monitoring systems. Below is an architecture diagram of VictoriaMetrics:

By utilizing VictoriaMetrics as the storage backend for Prometheus, organizations can significantly improve the storage and query performance of their monitoring systems, fulfilling the requirements for large-scale data storage and high-concurrency queries. The primary advantages of VictoriaMetrics include efficient data compression, rapid data querying, and low resource consumption, making it an excellent option as a storage backend for Prometheus. Below, I will proceed with the deployment of AutoMQ, Prometheus, and VictoriaMetrics.

Prerequisites

Deploy a functional AutoMQ node/cluster and open the Metrics collection port
Deploy VictoriaMetrics as the storage backend for Prometheus
Deploy Prometheus to retrieve Metrics data

Deploying AutoMQ, VictoriaMetrics, Prometheus

Deploying AutoMQ

Refer to the AutoMQ documentation: Deploy Multi-Nodes Cluster on Linux▸. Before starting deployment, add the following configuration parameters to enable Prometheus's pull interface. Once the AutoMQ cluster is launched with these parameters, each node will additionally open an HTTP interface for retrieving AutoMQ monitoring metrics. These metrics comply with the Prometheus Metrics format.

bin/kafka-server-start.sh ...\
--override  s3.telemetry.metrics.exporter.type=prometheus \
--override  s3.metrics.exporter.prom.host=0.0.0.0 \
--override  s3.metrics.exporter.prom.port=8890 \
....

With AutoMQ monitoring metrics enabled, you can access Prometheus format monitoring metrics from any node via HTTP at the address: http://{node_ip}:8890, with the response example as follows:

....
kafka_request_time_mean_milliseconds{otel_scope_name="io.opentelemetry.jmx",type="DescribeDelegationToken"} 0.0 1720520709290
kafka_request_time_mean_milliseconds{otel_scope_name="io.opentelemetry.jmx",type="CreatePartitions"} 0.0 1720520709290
...

For a detailed introduction to the metrics, refer to the AutoMQ official documentation: Prometheus Metrics▸.

Deploying VictoriaMetrics

Reference documentation: VictoriaMetrics [7]. VictoriaMetrics supports deployments via binary versions, Docker images, and source code. Here, we choose to deploy via Docker. Execute the following command to start the VictoriaMetrics container and map the data files to the folder /home/VictoriaMetrics/data for easy monitoring of data import changes:

# Create a New Data Folder
sudo mkdir -p /home/VictoriaMetrics/data
# Docker Start Command, Mount the Data Folder
docker run -d --name victoria-metrics \
  -p 8428:8428 \
  -v /home/VictoriaMetrics/data:/victoria-metrics-data \
  victoriametrics/victoria-metrics

Verify a successful installation by accessing: http://{client_ip}:8428 in a browser.

Deploying Prometheus

Prometheus can be deployed by downloading the binary package or using Docker. Below is an introduction to both deployment methods.

Binary Deployment

For ease of use, you can create a new script and modify the Prometheus download version as needed. After creating the script, simply execute it to complete the deployment. First, create a new script:

cd /home
vim install_prometheus.sh
# !!! Paste the Script Content Below, Save, and Exit
# Grant Permissions
chmod +x install_prometheus.sh
# Execute Script
./install_prometheus.sh

The content of the script is as follows:

version=2.45.3
filename=prometheus-${version}.linux-amd64
mkdir -p /opt/prometheus
wget https://github.com/prometheus/prometheus/releases/download/v${version}/${filename}.tar.gz
tar xf ${filename}.tar.gz
cp -far ${filename}/*  /opt/prometheus/

# Config as a Service
cat <<EOF >/etc/systemd/system/prometheus.service
[Unit]
Description="prometheus"
Documentation=https://prometheus.io/
After=network.target

[Service]
Type=simple

ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --storage.tsdb.path=/opt/prometheus/data --web.enable-lifecycle --web.enable-remote-write-receiver

Restart=on-failure
SuccessExitStatus=0
LimitNOFILE=65536
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=prometheus


[Install]
WantedBy=multi-user.target
EOF

systemctl enable prometheus
systemctl restart prometheus
systemctl status prometheus

Next, modify Prometheus's configuration file, add the task to collect AutoMQ observable data, and restart Prometheus. Execute the command:

# Fill in the Following for the Configuration File Content:
vim /opt/prometheus/prometheus.yml
# Restart Prometheus
systemctl restart prometheus

The configuration file reference is as follows: Please change the client_ip to the address exposed by AutoMQ for observable data.

# My Global Config
global:
  scrape_interval: 15s # Set the Scrape Interval to Every 15 Seconds. Default Is Every 1 Minute.
  evaluation_interval: 15s # Evaluate Rules Every 15 Seconds. the Default Is Every 1 Minute.

scrape_configs:
  # The Job Name Is Added as a Label `job=<job_name>` to Any Timeseries Scraped from This Config.
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
      
  - job_name: "automq"
    static_configs:
      - targets: ["{client_ip}:8890"]

# Config VictoriaMetrics IO Port
remote_write:
  - url: "http://{client_ip}:8428/api/v1/write"

remote_read:
  - url: "http://{client_ip}:8428/api/v1/read"

After deployment, you can access Prometheus via a browser to verify if the Metrics data from AutoMQ has been successfully collected. Visit http://{client_ip}:9090/targets.

Docker Deployment

If you currently have a running Prometheus Docker container, execute the command to remove the container first:

docker stop prometheus
docker rm prometheus

Create a new configuration file and attach it when launching Docker:

mkdir -p /opt/prometheus
vim /opt/prometheus/prometheus.yml
# Use the Configuration Details Provided in the Previous "Binary Deployment" Section.

Launch the Docker container:

docker run -d \
  --name=prometheus \
  -p 9090:9090 \
  -v /opt/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
  -m 500m \
  prom/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --enable-feature=otlp-write-receiver \
  --web.enable-remote-write-receiver

This sets up a Prometheus service to collect AutoMQ Metrics. For further details on integrating AutoMQ Metrics with Prometheus, please visit: Integrating Metrics into Prometheus | AutoMQ [8].

Check the Integration Results.

VictoriaMetrics Data Verification

VictoriaMetrics offers a Dashboard accessible via http://{client_ip}:8428/vmui, where the initial display is:

Data file verification. Existing data files are identified in the data directory:

root@debian: ls /home/VictoriaMetrics/data/small/2024_07/
17E5088F4DC9C068  17E5088F4DC9C073  17E5088F4DC9C0B3  17E509EE0189D399  17E509EE0189D39C  17E509EE0189D39F  17E509EE0189D3A2  17E509EE0189D3A5
17E5088F4DC9C06A  17E5088F4DC9C075  17E509EE0189D38D  17E509EE0189D39A  17E509EE0189D39D  17E509EE0189D3A0  17E509EE0189D3A3  17E509EE0189D3A6
17E5088F4DC9C071  17E5088F4DC9C07B  17E509EE0189D398  17E509EE0189D39B  17E509EE0189D39E  17E509EE0189D3A1  17E509EE0189D3A4  parts.json

By accessing the status at http://{client_ip}:8428/api/v1/status/tsdb, you can obtain a JSON file:

 "status": "success",
    "data": {
        "totalSeries": 3125,
        "totalLabelValuePairs": 15564,
        "seriesCountByMetricName": [
            {
                "name": "prometheus_http_request_duration_seconds_bucket",
                "value": 150
            },
            {
                "name": "prometheus_http_response_size_bytes_bucket",
                "value": 135
            },
            {
                "name": "kafka_request_queue_time_50p_milliseconds",
                "value": 90
            },
...............

Grafana Dashboard (Optional)

Since we have used VictoriaMetrics as Prometheus storage, we can configure it as a data source for Grafana at http://{client_ip}:8428, to access the status information of the AutoMQ cluster. For Grafana template files, you can refer to the official AutoMQ templates: grafana [9]. The final demonstration output is as follows:

Cluster Overview: Provides cluster-level monitoring information, including metrics like the number of nodes, data size, and cluster traffic. Additionally, it offers an overview of dimensions such as Topic, Group, and Broker, with support for drill-down functionality to examine detailed monitoring information.

With this, our integration process concludes; further features to explore can be found in the VictoriaMetrics official documentation [10].

Summary

In this article, we have thoroughly examined how to integrate AutoMQ, Prometheus, and VictoriaMetrics to enable comprehensive monitoring of AutoMQ clusters. We began by introducing the functions and advantages of each component, explaining why Prometheus was chosen as the monitoring system and why VictoriaMetrics is essential as the storage backend for Prometheus. Next, we offered step-by-step guidance on deploying and configuring AutoMQ, VictoriaMetrics, and Prometheus, and confirmed the effectiveness of this integration.

With this integration approach, enterprises can gain the following benefits:

Efficient data collection and storage: Prometheus handles efficient data collection, while VictoriaMetrics provides efficient data storage and compression capabilities, ensuring that the system can store large-scale time-series data with minimal hardware resource consumption.
Fast data querying: VictoriaMetrics offers rapid data query performance to meet high concurrent query demands, ensuring efficient queries in scenarios involving large-scale data storage.
Seamless Integration: VictoriaMetrics supports Prometheus' remote storage interface, enabling smooth integration with existing Prometheus monitoring systems and enhancing the overall performance and stability of such systems.

Through this integration, enterprises can achieve comprehensive monitoring of AutoMQ clusters, ensuring high availability and performance optimization, promptly detecting and addressing potential issues, optimizing system performance, and ensuring business continuity and stability. Looking ahead, as business demands evolve and technology advances, we can further explore and refine monitoring solutions to address more complex application scenarios and higher performance requirements efficiently.