Skip to Main Content

Integrating Metrics with Prometheus

Prometheus is a widely adopted service for monitoring, alerting, and time series data storage. It facilitates the collection, storage, and querying of metrics. This article outlines how to integrate AutoMQ with Prometheus to achieve seamless metrics data collection and querying.

Metrics Integration​

AutoMQ OpenSource utilizes the OpenTelemetry SDK to implement metrics collection and export, enabling the exposure of Apache Kafka business metrics and underlying storage-related metrics. Both types of metrics are unified and transformed using the OTLP format for exposure.

Based on the diagram above, first choose an appropriate metrics export solution to export metrics data to Prometheus. There are three ways to export metrics data:

For the above figure, first select an appropriate metrics export solution to export metrics data to Prometheus. There are three methods to export metrics data:

  • Using OpenTelemetry Collector

  • Using Prometheus OTLP Receiver

  • Directly Pulling Metrics from the Server

Prometheus Metrics Definition​

After integrating the metrics into Prometheus through either the OT Collector or Prometheus OLTP Receiver, you can refer to Prometheus Metrics▸ to view the metrics definitions.

Option 1: Using OpenTelemetry Collector​

  1. Start the OpenTelemetry Collector and configure the Prometheus Exporter to collect and export metrics data.

exporters:
  prometheus:
    endpoint: $endpoint_for_promehteus

  1. When starting the Prometheus service, make sure to add the following configuration.

scrape_configs:
  - job_name: $job_name
    scrape_interval: 5s
    honor_labels: true # DO NOT CHANGE THIS
    static_configs:
      - targets: ["$endpoint_for_prometheus"]

  1. Configure AutoMQ to push metrics data to the Collector.

s3.telemetry.metrics.exporter.type=otlp
s3.telemetry.exporter.otlp.protocol=grpc
s3.telemetry.exporter.otlp.endpoint=http://${endpoint}:4317

# Or with Http Protocol
# S3.telemetry.exporter.otlp.protocol=http
# S3.telemetry.exporter.otlp.endpoint=http://${endpoint}:4318/v1/metrics

Option 2: Using Prometheus OTLP Receiver​

Integration into Self-Managed Open Source Prometheus​

According to Prometheus's version requirements, the user's Prometheus service needs to meet the following conditions:

  • Prometheus version must be at least 2.47.

  • Prometheus needs to enable feature flags: --enable-feature=otlp-write-receiver. Refer to the command below for enabling it.


./prometheus --config.file=prometheus.yml --enable-feature=otlp-write-receiver

If your current self-hosted Prometheus service is based on the VictoriaMetrics Enterprise edition, ensure that the version of VictoriaMetrics is at least 1.92.0. Configure the access point according to the relevant documentation on Sending Data via OpenTelemetry.

Integrate into Cloud Provider's Prometheus Service​

If users utilize managed services provided by public cloud vendors, it is advisable to consult the technical staff from the cloud vendor. Taking Alibaba Cloud as an example, Alibaba Cloud's Prometheus service provides ready-to-use support for the OT protocol. Refer to the OT Protocol Data Reporting Documentation.

After starting the Prometheus service, configure AutoMQ to push metrics to Prometheus:


s3.telemetry.metrics.exporter.type=otlp
s3.telemetry.exporter.otlp.protocol=http
s3.telemetry.exporter.otlp.endpoint=http://${endpoint}:9090/api/v1/otlp/v1/metrics

# Or Use with Aliyun Prometheus OpenTelemetry Endpoint
# S3.telemetry.exporter.otlp.endpoint=https://${endpoint}/api/v1/metrics

Solution Three: Pull Metrics Directly from the Server​

In this mode, AutoMQ acts as a Prometheus HTTP Server, and users can configure scrape tasks on the Prometheus backend service to fetch Prometheus metrics from specific endpoints. The configuration method is as follows:


s3.telemetry.metrics.exporter.type=prometheus
s3.metrics.exporter.prom.host=127.0.0.1
s3.metrics.exporter.prom.port=9090