Runs on CEPH

AutoMQ utilizes S3 storage, and since CEPH supports the S3 access protocol, it can efficiently serve as the storage backend for AutoMQ. Below is a guide for deploying AutoMQ using CEPH.

Prerequisites

Install CEPH; please refer to: https://docs.ceph.com/en/latest/install/
Install CEPH's S3-compatible component, RGW; please refer to: https://docs.ceph.com/en/latest/cephadm/services/rgw/

Create Users and Buckets.

Creating a CEPH User

radosgw-admin user create --uid="automq" --display-name="automq"

By default, the newly created user is granted all the permissions required by AutoMQ. For configuring minimal permissions, refer to the official CEPH documentation for custom settings. The result after executing the above commands is as follows:

{
    "user_id": "automq",
    "display_name": "automq",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "automq",
            "access_key": "X1J0E1EC3KZMQUZCVHED",
            "secret_key": "Hihmu8nIDN1F7wshByig0dwQ235a0WAeUvAEiWSD"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

Creating a CEPH Bucket

Set the environment variables to configure the Access Key and Secret Key needed by the AWS CLI.

export AWS_ACCESS_KEY_ID=X1J0E1EC3KZMQUZCVHED
export AWS_SECRET_ACCESS_KEY=Hihmu8nIDN1F7wshByig0dwQ235a0WAeUvAEiWSD

Use AWS CLI to create an S3 bucket.

aws s3api create-bucket --bucket automq-data --endpoint=http://127.0.0.1:80

Retrieving Configuration Information

Once users and Buckets have been created, you can gather the necessary configuration details for deploying AutoMQ.

Parameter Name	Default Value in This Example	Description
KAFKA_S3_ACCESS_KEY	X1J0E1EC3KZMQUZCVHED	Replace with actual details after creating the Ceph user.
KAFKA_S3_SECRET_KEY	Hihmu8nIDN1F7wshByig0dwQ235a0WAeUvAEiWSD	Replace with actual details after creating the Ceph user.
region	us-west-2	This parameter is invalid in Ceph and can be set to any value, such as us-west-2.
endpoint	http://127.0.0.1:80	This parameter represents the address of Ceph's S3-compatible RGW component for external services. If there are multiple machines, it is recommended to use a load balancer (SLB) to aggregate into a single IP address.
data-bucket	automq-data	Set the bucket for message storage.
ops-bucket	automq-ops	Set the bucket for storing operational data.

Deploy AutoMQ

Please refer to the following document to deploy AutoMQ:

Prerequisites​

Create Users and Buckets.​

Creating a CEPH User​

Creating a CEPH Bucket​

Retrieving Configuration Information​

Deploy AutoMQ​