Object Storage Configuration

AutoMQ is a cloud-native, redesigned distribution of Kafka that fully utilizes cloud object storage services. It can build the storage layer using object storage from major cloud providers or storage software.

Object Storage URI Construction

The URI is a unique identifier used by AutoMQ to represent writes to object storage, adhering to established standards to reconcile differences among various object storage service providers. The URI components are as follows: s3 specifies the protocol; xxx_bucket indicates the specific bucket; region and endpoint are additional parameters used by different vendors to locate the bucket.

Ultimately, replace the variables in the configuration templates global.config and global.envs from Deploy Multi-Nodes Cluster on Linux▸ to generate the final configuration files.

Provider	Type	URI Example
AWS	Cloud Provider	s3://xxx_bucket?region=us-east-1
AWS-CN	Cloud Provider	s3://xxx_bucket?region=cn-northwest-1&endpoint=https://s3.amazonaws.com.cn
ALIYUN	Cloud Provider	s3://xxx_bucket?region=oss-cn-shanghai&endpoint=https://oss-cn-shanghai.aliyuncs.com
OCI	Cloud Provider	s3://xxx_bucket?region=us-ashburn-1&endpoint=https://xxx_namespace.compat.objectstorage.us-ashburn-1.oraclecloud.com&pathStyle=true
CEPH	Storage Software	s3://xxx_bucket?region=us-west-2&endpoint=http://127.0.0.1:80
CubeFS	Storage Software	s3://xxx_bucket?region=us-west-2&endpoint=http://10.1.0.240:17410
MinIO	Storage Software	s3://xxx_bucket?region=us-west-2&pathStyle=true&endpoint=http://10.1.0.240:9000

If the s3.wal.path is set, you can also include the batchInterval and maxBytesInBatch parameters in the URI to control the submit interval and the maximum size of bytes per batch, respectively. The default settings are batchInterval=250 milliseconds and maxBytesInBatch=8388608 bytes.

Cloud Object Storage Configuration

AWS

It is recommended to refer to the Policy and Endpoint & Region links in the AWS documentation to get detailed configuration instructions for IAM and S3.

In a production environment, it is advisable to scope IAM Policy permissions to specific buckets to avoid unintended authorizations.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Alibaba Cloud

We recommend reviewing the Policy and Endpoint & Region links to refer to Alibaba Cloud documentation for more detailed configuration instructions on RAM and OSS.

In a production environment, it is advisable to narrow the RAM Policy authorization scope down to specific buckets to avoid unintended authorizations.

{
    "Version": "1",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "oss:PutObject",
                "oss:AbortMultipartUpload",
                "oss:GetObject",
                "oss:DeleteObject"
            ],
            "Resource": [
                "acs:oss:*:*:automq-data",
                "acs:oss:*:*:automq-ops"
            ]
        }
    ]
}

GCP

Since the AWS S3 SDK cannot directly access GCP Cloud Storage, the open-source version of AutoMQ cannot run directly on Google Cloud Storage. If you need to run AutoMQ on GCP, it is recommended to contact the AutoMQ team for enterprise support, as the enterprise version of AutoMQ is compatible with GCP Cloud Storage.

Azure

Azure Blob Storage is not compatible with the AWS S3 network protocol; therefore, the open-source version of AutoMQ cannot run directly on Azure Blob Storage. Users can address this as follows:

• Set up your own S3 Proxy to access Blob Storage.

• Contact the AutoMQ team for enterprise support. The AutoMQ Enterprise Edition is compatible with Azure Blob Storage.

OCI

We recommend checking the official website for more detailed information.

It is advisable to consult the Policy, Detailed Policy, and Endpoint & Region links, and review OCI documentation for more detailed configuration instructions on IAM and Object Storage.

Allow group 'AutoMQ PoC' to manage objects in tenancy where any {target.bucket.name='automq-data', target.bucket.name='automq-ops'}

Other Cloud Providers

To deploy AutoMQ in other public cloud environments, make sure that the cloud provider's object storage service is compatible with the S3 protocol. You can contact the AutoMQ team via this form to obtain the necessary assistance and best practices.

Storage Software Configuration

If you wish to deploy AutoMQ in a private data center, you can also use storage software like MinIO, Ceph, or CubeFS that supports the S3 protocol to provide object storage services.