Skip to Main Content

Table Topic Quick Start

Table Topic is a core feature of AutoMQ, designed for modern data lake architectures. Its primary purpose is to create seamless integration between streaming data and static data lakes through architectural innovation, addressing traditional issues like stream-batch separation, complex ETL processes, and high costs. This document is aimed at new users experiencing AutoMQ for the first time, focusing on how to deploy an AutoMQ cluster with Table Topic support in a local development environment (by deploying MinIO to provide object storage services), enabling users to quickly experience the features of Table Topic.

This document is intended only for testing and learning purposes. For production environment usage of the Table Topic feature, please refer to deployment documentation such as Deploy Multi-Nodes Cluster on Linux▸.

Prerequisites​

  • Linux/Mac/Windows Subsystem for Linux

  • Docker

  • Docker Compose version > 2.22.0

  • At least 8GB of free memory

Install and Start AutoMQ Cluster​

Run the following command to download the Docker Compose configuration file for a single-node deployment and start the AutoMQ cluster using Docker Compose.


curl -O https://raw.githubusercontent.com/AutoMQ/automq/refs/tags/1.5.0/docker/table_topic/docker-compose.yml && docker-compose up -d

Once started successfully, we will send Table Topic messages and perform queries using Spark SQL to experience automatic table creation, table querying and updating, schema changes, data partitioning, and CDC capabilities. Users can open http://localhost:8888 in their browser and interactively execute commands using Jupyter Notebook.

Create Table​

The first three steps in the Notebook complete the creation of a Table Topic, write a message, and trigger the generation of an Iceberg table.

Query Data​

Note:

There may be a delay between writing the message and the data being entered into the table. If an error occurs during the query, we recommend retrying after a short wait.

Update Records​

When querying updated results, the data ops will change from I to U.

Delete Data​

After querying the deletion results, records are empty.

Delete Table​

Stop and Uninstall the AutoMQ Cluster​

Once you have completed the functional tests, run the following commands to stop and uninstall the current AutoMQ cluster.


curl -O https://raw.githubusercontent.com/AutoMQ/automq/refs/tags/1.5.0/docker/table_topic/docker-compose.yml && docker-compose down -v

For detailed parameter configuration, please refer to the Table Topic Configuration▸ section.