AutoMQ is a cloud-native, redesigned distribution of Kafka that fully utilizes cloud object storage services. It can build the storage layer using object storage from major cloud providers or storage software.
Object Storage URI Construction
The URI is a unique identifier used by AutoMQ to represent writes to object storage, adhering to established standards to reconcile differences among various object storage service providers. The URI components are as follows: s3 specifies the protocol; xxx_bucket indicates the specific bucket; region and endpoint are additional parameters used by different vendors to locate the bucket.
Ultimately, replace the variables in the configuration templates global.config and global.envs from Deploy Multi-Nodes Cluster on Linux▸ to generate the final configuration files.
| Provider | Type | URI Example |
|---|
| AWS | Cloud Provider | s3://xxx_bucket?region=us-east-1 |
| AWS-CN | Cloud Provider | s3://xxx_bucket?region=cn-northwest-1&endpoint=https://s3.amazonaws.com.cn |
| ALIYUN | Cloud Provider | s3://xxx_bucket?region=oss-cn-shanghai&endpoint=https://oss-cn-shanghai.aliyuncs.com |
OCI
| Cloud Provider
| s3://xxx_bucket?region=us-ashburn-1&endpoint=https://xxx_namespace.compat.objectstorage.us-ashburn-1.oraclecloud.com&pathStyle=true
|
CEPH
| Storage Software
| s3://xxx_bucket?region=us-west-2&endpoint=http://127.0.0.1:80
|
CubeFS
| Storage Software
| s3://xxx_bucket?region=us-west-2&endpoint=http://10.1.0.240:17410
|
MinIO
| Storage Software
| s3://xxx_bucket?region=us-west-2&pathStyle=true&endpoint=http://10.1.0.240:9000
|
If the s3.wal.path is set, you can also include the batchInterval and maxBytesInBatch parameters in the URI to control the submit interval and the maximum size of bytes per batch, respectively. The default settings are batchInterval=250 milliseconds and maxBytesInBatch=8388608 bytes.
Cloud Object Storage Configuration
AWS
Aliyun
GCP
Azure
OCI
Other Cloud Providers
It is recommended to refer to the Policy and Endpoint & Region links in the AWS documentation to get detailed configuration instructions for IAM and S3.In a production environment, it is advisable to scope IAM Policy permissions to specific buckets to avoid unintended authorizations.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:AbortMultipartUpload",
"s3:DeleteObject",
"s3:ListBucket"
],
"Resource": [
"*"
]
}
]
}
It is recommended to refer to the official Alibaba Cloud documentation on RAM Permission Policy Language Overview and OSS Endpoints and Regions for detailed instructions on configuring RAM permissions and OSS.In a production environment, it is advisable to scope RAM policy permissions to specific buckets (OSS storage spaces) to avoid overly broad authorization and reduce security risks.
{
"Version": "1",
"Statement": [
{
"Effect": "Allow",
"Action": [
"oss:PutObject",
"oss:AbortMultipartUpload",
"oss:GetObject",
"oss:DeleteObject",
"oss:PutObjectTagging"
],
"Resource": [
"acs:oss:*:*:examplebucket"
]
}
]
}
Since the AWS S3 SDK cannot directly access GCP Cloud Storage, the open-source version of AutoMQ cannot run directly on Google Cloud Storage. If you need to run AutoMQ on GCP, it is recommended to contact the AutoMQ team for enterprise support, as the enterprise version of AutoMQ is compatible with GCP Cloud Storage. Azure Blob Storage is not compatible with the AWS S3 network protocol; therefore, the open-source version of AutoMQ cannot run directly on Azure Blob Storage. Users can address this as follows:
-
Set up your own S3 Proxy to access Blob Storage.
-
Contact the AutoMQ team for enterprise support. The AutoMQ Enterprise Edition is compatible with Azure Blob Storage.
We recommend checking the official website for more detailed information.It is advisable to consult the Policy, Detailed Policy, and Endpoint & Region links, and review OCI documentation for more detailed configuration instructions on IAM and Object Storage.
Allow group 'AutoMQ PoC' to manage objects in tenancy where any {target.bucket.name='automq-data', target.bucket.name='automq-ops'}
To deploy AutoMQ in other public cloud environments, make sure that the cloud provider’s object storage service is compatible with the S3 protocol. You can contact the AutoMQ team via this form to obtain the necessary assistance and best practices. Storage Software Configuration
If you wish to deploy AutoMQ in a private data center, you can also use storage software like MinIO, Ceph, or CubeFS that supports the S3 protocol to provide object storage services. 
