Runs on CubeFS

AutoMQ utilizes S3 storage, and CubeFS supports the S3 access protocol, making it an appropriate storage backend for AutoMQ. Below is a guide for deploying AutoMQ with CubeFS.

Prerequisites

Install CubeFS. Please refer to: https://cubefs.io/docs/master/deploy/env.html
Install the CubeFS Object Gateway. Please refer to: https://cubefs.io/docs/master/user-guide/objectnode.html

Create Users and Buckets

Creating CubeFS User

curl -H "Content-Type:application/json" -X POST --data '{"id":"automq","pwd":"12345","type":3}' "http://172.16.1.101:17010/user/create"

By default, the newly created user has all the necessary permissions for AutoMQ. For minimal permission configuration, please consult the official CubeFS documentation for custom settings. The output after executing the above commands is as follows:

{
    "code":0,
    "msg":"success",
    "data":{
        "user_id":"automq",
        "access_key":"AEv7EVirKDJtfyK5",
        "secret_key":"fIW2OvamdKnP1XQcY0dwKzKFzNNXv5r6",
        "policy":
            {
                "own_vols":[],
                "authorized_vols":{}
            },
        "user_type":3,
        "create_time":"2024-05-16 16:56:13",
        "description":"",
        "EMPTY":false
    }
}

Creating CubeFS Bucket

You can configure AWS CLI's Access Key and Secret Key using environment variables.

export AWS_ACCESS_KEY_ID=AEv7EVirKDJtfyK5
export AWS_SECRET_ACCESS_KEY=fIW2OvamdKnP1XQcY0dwKzKFzNNXv5r6

Use AWS CLI to create an S3 bucket.

aws s3api create-bucket --bucket automq-data --endpoint=http://10.1.0.240:17410

Retrieving Configuration Information

Once the user and Bucket have been created, you can gather the necessary configuration information required to deploy AutoMQ.

Parameter Name	Default Value in This Example	Description
KAFKA_S3_ACCESS_KEY	AEv7EVirKDJtfyK5	Replace with actual values after creating CubeFS user.
KAFKA_S3_SECRET_KEY	fIW2OvamdKnP1XQcY0dwKzKFzNNXv5r6	Replace with actual values after creating CubeFS user.
region	us-west-2	This parameter is invalid in CubeFS and can be set to any value, such as us-west-2.
endpoint	http://10.1.0.240:17410	This parameter is the address for the CubeFS object gateway service. If there are multiple machines, it is recommended to use a load balancer (SLB) to consolidate them into a single IP address.
data-bucket	automq-data	Sets the bucket for message storage.
ops-bucket	automq-ops	Sets the bucket for operational data storage.

Deploy AutoMQ

Please refer to the following documentation for deploying AutoMQ:

Prerequisites​

Create Users and Buckets​

Creating CubeFS User​

Creating CubeFS Bucket​

Retrieving Configuration Information​

Deploy AutoMQ​