You can reference the relevant code from AutoMQ Labs, which includes integration examples of AutoMQ and Redpanda Console here. Launch with a single click to experience it.

​
Manage Kafka/AutoMQ Clusters More Conveniently Using the Kafka Web UI.

With the rapid advancement of big data technologies, Kafka has become a cornerstone for real-time data processing in enterprises as a high-throughput, low-latency distributed messaging system. However, managing and monitoring Kafka clusters is not an easy task. Traditional command-line tools and scripts, while powerful, can be complex and unintuitive for developers and operations personnel. To address these challenges, the Kafka Web UI has emerged, providing users with a more convenient and efficient way to manage Kafka clusters. After over a decade of development, Apache Kafka has amassed a wealth of ecosystem resources within the industry. AutoMQ, as a successor to Apache Kafka, can leverage products within its ecosystem due to its full compatibility with Kafka. The enterprise edition of AutoMQ already offers substantial control capabilities. If you are using AutoMQ, you can also utilize products like Kafdrop, Redpanda Console [1], and similar tools to manage AutoMQ clusters. Today’s content covers how to monitor the status of AutoMQ clusters using Redpanda Console [1] to enhance system maintainability and stability.

​
Integration of AutoMQ with Redpanda Console

Redpanda Console is a Kafka Web UI provided by Redpanda, designed for monitoring and managing Redpanda or Kafka clusters. It offers an intuitive user interface through which users can easily view cluster status, monitor performance metrics, and manage topics and partitions. This console is aimed at simplifying the day-to-day operations of data streaming systems, enabling users to maintain and monitor their clusters more effectively. Thanks to AutoMQ being fully compatible with Kafka, it seamlessly integrates with Redpanda Console. By leveraging Redpanda Console, AutoMQ users can also benefit from an intuitive user interface to monitor the real-time status of AutoMQ clusters, including vital metadata such as topics, partitions, consumer groups, and their offsets. This monitoring capability not only enhances the efficiency of problem diagnosis but also helps optimize cluster performance and resource usage. This tutorial will guide you on how to launch the Redpanda Console service and use it alongside AutoMQ clusters to effectively monitor and manage cluster status. Redpanda Console

​
Prerequisites

  • Deploy AutoMQ Cluster
  • Prepare Redpanda Console Environment

​
Deploy AutoMQ Cluster

Please refer to the official AutoMQ documentation: Deploy Multi-Nodes Cluster on Linux▸ [3].

​
Deploy Redpanda Console

There are two ways to deploy the Redpanda Console: Docker deployment and distribution deployment. Docker deployment is simpler, and if you want a quick and easy experience integrating AutoMQ with the Redpanda Console, it is recommended to choose Docker for deployment. If you have specific requirements, such as login authentication, SASL authentication, TLS configuration, and log level settings, you should opt for distribution deployment. Below, I’ll introduce the configuration methods for both.

​
Docker Deployment

Redpanda Console can be deployed via Docker, referring to Quick Start [4]. In the process described above, after setting up the AutoMQ cluster, you will know the addresses and ports that all Broker nodes listen to. Thus, you can establish an association between the Redpanda Console and the AutoMQ cluster by specifying the KAFKA_BROKERS parameter in the Docker startup command. The Docker container startup command is as follows: 
docker run -p 8080:8080 -e KAFKA_BROKERS=192.168.0.4:9092,192.168.0.5:9092,192.168.0.6:9092 docker.redpanda.com/redpandadata/console:latest
  • -p 8080:8080 : Specifies the service access mapping port for the Redpanda Console.
  • KAFKA_BROKERS : Must be specified as the Broker address of your AutoMQ cluster.

​
Deployment of Distribution

You need to choose an appropriate version to download and extract it into a specified directory, such as /opt, from Redpanda Console’s GitHub Releases page: Release Redpanda Console [5]. The command is as follows: 
# Ubuntu Linux
cd /opt
sudo curl -L -o redpanda_console.tar.gz https://github.com/redpanda-data/console/releases/download/v2.6.0/redpanda_console_2.6.0_linux_amd64.tar.gz
# Unzip, Get Redpanda_console
sudo tar -xzf redpanda_console.tar.gz
# Config Set
sudo mkdir -p /etc/redpanda
# Write Config
sudo vim /etc/redpanda/redpanda-console-config.yaml
An example of the contents for the redpanda-console-config.yaml configuration file is shown below: 
kafka:
  #Brokers is a list of bootstrap servers with
  #port (for example "localhost:9092").
  brokers:
    - broker-0.mycompany.com:19092
    - broker-1.mycompany.com:19092
    - broker-2.mycompany.com:19092
Note: Please ensure that the server where you are installing Redpanda Console has access to the server where the Broker node mentioned in the configuration file is located.
For more detailed settings, refer to the official documentation: Redpanda Console Configuration [6]. After completing the configuration, you need to set environment variables so that the Redpanda Console executable can access the configuration file information and start the Redpanda Console: 
# Set Env
export CONFIG_FILEPATH="/etc/redpanda/redpanda-console-config.yaml"
# /opt/ Run Console
./redpanda-console
You will get the following results: 
./redpanda-console
{"level":"info","ts":"2024-07-10T09:52:52.958+0800","msg":"started Redpanda Console","version":"2.6.0","built_at":"1717083695"}
{"level":"info","ts":"2024-07-10T09:52:52.963+0800","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata"}
{"level":"info","ts":"2024-07-10T09:52:54.780+0800","msg":"successfully connected to kafka cluster","advertised_broker_count":1,"topic_count":2,"controller_id":0,"kafka_version":"at least v3.6"}
{"level":"info","ts":"2024-07-10T09:53:05.620+0800","msg":"Server listening on address","address":"[::]:8080","port":8080}

​
Access the Console Page

After completing the above deployment operations, you can access the console service by entering the address in your browser (e.g., http://console_ip:8080). The display will appear as follows:

​
Cluster Overview

The Cluster Overview page provides users with a broad perspective, displaying the core information of the AutoMQ cluster, including but not limited to:
  • Cluster Operational Status: Displays the current health of the cluster, helping users quickly identify any issues.
  • Storage Utilization: Indicates data storage usage within the cluster, aiding users in managing and planning storage effectively.
  • Version Information: Shows the version of the running AutoMQ cluster, simplifying the tracking and upgrading process.
  • Number of Online Brokers: Displays the real-time count of online brokers, an essential metric.
  • Number of Topics and Replicas: Provides information on the number of topics and replicas, assisting users in understanding the cluster’s scale and data replication status.
Monitoring the state of a cluster is essential for maintaining the stability and performance of messaging queue systems. By observing real-time metrics, such as the cluster’s operational status, storage utilization, version information, the number of online brokers, and the count of topics and replicas, operations personnel can swiftly detect and address potential issues, preventing system failures from affecting business operations. Furthermore, these metrics support capacity planning and resource management, ensuring the system is prepared to manage future data growth. Having access to cluster version information also enables users to upgrade software promptly to benefit from the latest features and security patches, thereby enhancing overall system reliability and efficiency.

​
Topic Overview

On the Topic list page, users can see an overview of all Topics in the current AutoMQ cluster, including essential details for each Topic, such as the number of partitions and replica strategy. This page allows users to efficiently browse and manage Topics.

​
Topic Details

By clicking on a specific Topic, users navigate to a detailed page for that Topic, where they can further explore understanding and operational functionalities:
  • Message List: Browsing and searching messages within a Topic is highly beneficial for tracking and debugging purposes.
  • Consumer Information: Displays information on consumers and consumer groups currently subscribed to the Topic, facilitating the monitoring of consumption status.
  • Partition Status: Provides detailed information for each partition, including key metrics like Leader and ISR.
  • Configuration Information: Lists the configuration settings of the Topic, with support for modifications to optimize performance or behavior.
  • ACL (Access Control List): Manages access permissions for the Topic to ensure data security.
Furthermore, Redpanda Console also supports users in manually creating and publishing messages, which is highly valuable for message injection during testing or specific scenarios. Monitoring topic details allows us to gain insight into the operational status of the message queue by navigating through the message list for tracking and debugging, checking consumer information to monitor consumption status, understanding partition state to ensure data distribution and high availability, managing configuration information to optimize performance, and setting access controls to safeguard data security. These features aid in the timely identification and resolution of issues, enhancing the overall efficiency and reliability of the system.

​
Summary

This article explains the integration process of Redpanda Console with AutoMQ, demonstrating how this powerful tool can simplify and enhance the management of AutoMQ clusters. Hopefully, this article can serve as a practical reference for users aiming to improve the efficiency and functionality of message queue management.

​
References

[1] Redpanda Console: https://redpanda.com/redpanda-console-kafka-ui [2] Redpanda: https://redpanda.com/ [3] Cluster Deployment for AutoMQ: https://www.automq.com/docs/automq/getting-started/deploy-multi-nodes-test-cluster-on-docker [4] Quick Start: https://github.com/redpanda-data/console?tab=readme-ov-file#quick-start [5] Release Redpanda Console: https://github.com/redpanda-data/console/releases/tag/v2.6.0 [6] Redpanda Console Configuration: https://docs.redpanda.com/current/reference/console/config/#example-redpanda-console-configuration-file [7] Kafdrop Github: https://github.com/obsidiandynamics/kafdrop