Skip to Main Content

Timeplus

Timeplus is a data analytics platform dedicated to stream processing, utilizing the open-source streaming database Proton. It offers comprehensive functionalities that empower teams to quickly and intuitively manage both streaming and historical data. It's suitable for organizations across various sizes and industries, enabling data engineers and platform engineers to fully leverage the value of streaming data through SQL.

This article guides you on how to import AutoMQ data into Timeplus using the Timeplus console. Since AutoMQ is fully compatible with Apache Kafka, you can also create Kafka external streams to analyze AutoMQ data without the need to relocate the data.

Prepare AutoMQ and Test Data​

Refer to Deploy Multi-Nodes Cluster on Linux▸ to deploy AutoMQ, ensuring network connectivity between AutoMQ and Timeplus.

Info

If you need to maintain an IP whitelist, be sure to include the static IP of the Timeplus service in the whitelist.

52.83.159.13 for cloud.timeplus.com.cn

Quickly create a topic named example_topic in AutoMQ and write a sample JSON data into it by following these steps.

Create Topic​

Use the Apache Kafka command-line tool to create the topic. Ensure that you have access to the Kafka environment and that the Kafka service is running. Below is an example command for creating a topic:


./kafka-topics.sh --create --topic exampleto_topic --bootstrap-server 10.0.96.4:9092  --partitions 1 --replication-factor 1

Tip

When executing the command, replace the topic and bootstrap-server with the actual Kafka server address you are using.

After creating the topic, you can use the following command to verify if the topic was successfully created.


./kafka-topics.sh --describe example_topic --bootstrap-server 10.0.96.4:9092

Generate Test Data​

Generate test data in JSON format corresponding to the table mentioned earlier.


{
  "id": 1,
  "name": "Test User"
  "timestamp": "2023-11-10T12:00:00",
  "status": "active"
}

Write Test Data​

You can write test data into a topic named example_topic using Kafka command-line tools or programmatically. Below is an example utilizing a command-line tool:


`echo '{"id": 1, "name": "Test User", "timestamp": "2023-11-10T12:00:00", "status": "active"}' | sh kafka-console-producer.sh --broker-list 10.0.96.4:9092 --topic example_topic`

You can view the data that was just written into the topic with the following command:


sh kafka-console-consumer.sh --bootstrap-server 10.0.96.4:9092 --topic example_topic --from-beginning

Tip

When executing the command, you'll need to replace the topic and bootstrap-server with the actual Kafka server address you're using.

AutoMQ Data Source​

  1. In the left navigation menu, click on "Data Ingestion," then press the "Add Data" button at the top right.

  2. In the pop-up window, you can view the connectable data sources and other methods for adding data. Since AutoMQ is fully compatible with Apache Kafka, directly select Apache Kafka.

  3. Enter the broker URL, and disable TLS and authentication.

  4. Input the AutoMQ topic name and choose the data format for "Read As," supporting JSON, AVRO, and text formats.

    1. It's recommended to choose Text to save the entire JSON document as a string, which makes it easier to handle schema changes.

    2. When choosing AVRO, you can enable the "Auto Extract" option to store top-level properties as separate columns. You'll need to provide the schema registry address, API key, and secret.

  5. In the "Preview" step, display at least one event. By default, a new stream is created in Timeplus for the new data source.

  6. Name the stream and confirm the column information. You can set the event time column; if not set, the system will default to the ingestion time. Alternatively, you can select an existing stream.

  7. After previewing the data, name the source, add a description, and review the configuration. Click "Finish," and the stream data will be instantly available in the specified stream.

AutoMQ Source Instructions​

When using AutoMQ data sources, the following constraints must be followed:

  1. Currently, AutoMQ Kafka topics support messages only in JSON and AVRO formats.

  2. JSON properties at the topic level are converted into stream columns. For nested properties, elements are preserved as String columns and can be queried using one of the JSON functions.

  3. Numeric or Boolean types in JSON messages are converted to their corresponding types in the stream.

  4. DateTime or timestamp values are stored as string columns. They can be converted back to DateTime using the to_time function.