RisingWave is a distributed streaming database that provides a standard SQL interface, fully compatible with the PostgreSQL ecosystem, and integrates seamlessly without requiring any code changes. RisingWave treats streams as tables, enabling users to perform complex queries on both streaming and historical data smoothly. With RisingWave, users can concentrate on query analysis logic without having to learn Java or the specific underlying APIs of different systems.
This article will detail the process of importing data from AutoMQ into the RisingWave database using RisingWave Cloud.
Prepare AutoMQ and test data
Follow the Stand-alone Deployment guide to deploy AutoMQ, ensuring network connectivity between AutoMQ and RisingWave.
Swiftly create a topic named example_topic in AutoMQ and write a test JSON message by following these steps.
Create Topic
Utilize the Apache Kafka command line tool to create the topic, making sure you have access to a Kafka environment and that the Kafka service is operational. Here is an example command to create a topic:
./kafka-topics.sh --create --topic exampleto_topic --bootstrap-server 10.0.96.4:9092 --partitions 1 --replication-factor 1Once the topic has been created, use the following command to confirm its successful creation.
./kafka-topics.sh --describe example_topic --bootstrap-server 10.0.96.4:9092Generating test data
Generate JSON formatted test data that corresponds to the previously mentioned table.
{
"id": 1,
"name": "testuser",
"timestamp": "2023-11-10T12:00:00",
"status": "active"
}Writing test data
Test data can be written to a topic named "example_topic" using Kafka's command-line tools or programmatically. Here's an example using command-line tools:
echo '{"id": 1, "name": "testuser", "timestamp": "2023-11-10T12:00:00", "status": "active"}' | sh kafka-console-producer.sh --broker-list 10.0.96.4:9092 --topic example_topicTo view the data recently written to the topic, use the following command:
sh kafka-console-consumer.sh --bootstrap-server 10.0.96.4:9092 --topic example_topic --from-beginningCreate an AutoMQ source on RisingWave Cloud
-
Navigate to Clusters on RisingWave Cloud to create a cluster.
-
Go to Source on RisingWave Cloud to create a source.
-
Specify the cluster and database, and log into the database.
-
AutoMQ is fully compatible with Apache Kafka®, so just click on Create source and choose Kafka.
-
Follow the guide on RisingWave Cloud to configure the connector, set up source information, and define the schema.
-
Review the generated SQL statement, click Confirm to finalize the source creation.
Query data
-
Go to RisingWave Cloud Console and log into the cluster.
-
Run the following SQL statement to access the imported data, replacing the variable `your_source_name` with the custom name specified when creating the source.
SELECT * from {your_source_name} limit 1;