Skip to main content
All components of the AutoMQ Cloud BYOC environment are deployed within your AWS account to ensure data privacy and security. Before installing AutoMQ, you need to prepare various AWS resources in advance. This article describes how to prepare the necessary cloud resources for AutoMQ on the AWS public cloud.
In this document, “AutoMQ,” “AutoMQ service provider,” and “the AutoMQ service” all refer to AutoMQ HK Limited and its affiliates.

​
Prepare VPC

For POC testing scenarios, it is recommended to create a new VPC and deploy AutoMQ and test applications within it. For production scenarios, you will likely use an existing VPC. In this case, you should check if the VPC meets AutoMQ’s installation requirements and make any necessary adjustments. To install the AutoMQ BYOC environment, you need to provide a VPC on AWS that meets the following conditions:
VPC Checklist
Description
Public Subnet: 1
  • The public subnet is used to deploy the AutoMQ BYOC environment console. The environment console provides a WebUI, which users need to access via the public network.
Private Subnet: 1 or 3
  • The private subnet is used to deploy the AutoMQ data plane cluster.
  • AutoMQ supports single-availability-zone and three-availability-zone deployments, with each availability zone requiring one subnet.
S3 Endpoint: 1
  • AutoMQ components access S3 via the S3 endpoint using the internal network, avoiding public network transmission.
  • The S3 endpoint is created at the VPC level, and the endpoint route table must cover all private subnets.
EC2 Endpoint: 1
  • AutoMQ components access EC2 and ASG APIs via the EC2 endpoint using the internal network.
  • The EC2 endpoint is created at the VPC level, and its scope must cover all private subnets.
DNS Hostnames and Resolution: Enabled
  • AutoMQ uses Route 53 to assign private DNS resolution within the VPC. Therefore, the VPC needs to have DNS hostnames and DNS resolution enabled.
Public NAT Gateway: 1
  • If you plan to deploy the AutoMQ data plane cluster on EKS, you need to ensure that the private subnet where EKS is located is configured with a public NAT gateway and has the correct outbound routes.

Note: If a public NAT gateway is not configured, you will not be able to install the AutoMQ cluster on EKS.

​
Scenario 1: Create a New VPC

If you do not have an available VPC resource or want to use a new VPC for a POC test scenario, you can refer to the documentation below to create a new VPC.

​
Step 1: Create a VPC and Enable the S3 Endpoint

Log in to the VPC console with your AWS cloud account. Taking the Singapore region as an example, visit the VPC console and click “Create VPC.” When using the quick create VPC wizard, you can choose to create the VPC and other resources at the same time. Confirm the following options and click “Create”:
  1. VPC Name: Set a custom name for the VPC.
  2. CIDR: Set a reasonable CIDR block.
  3. Availability Zones: It is recommended to select 1-3 zones based on your business deployment.
  4. Public Configuration: If you need to access the AutoMQ environment console from the public network, you need to configure this.
  5. VPC Endpoints: Be sure to open the configuration and create an S3 gateway.
  6. DNS Options: Ensure that both “Enable DNS hostnames” and “Enable DNS resolution” are enabled. The Kafka cluster access point will rely on the DNS service for domain name resolution.
  7. NAT Gateway: If you are deploying AutoMQ on EKS, you need to enable the NAT Gateway.

​
Step 2: Create an EC2 Endpoint

Create an EC2 Endpoint within the current VPC to allow subsequent AutoMQ data nodes to access the EC2 API via the VPC private network. When creating an EC2 access point, pay attention to the following configuration items:
  1. Service Type: Select “AWS services.”
  2. VPC: Select the VPC where you need to deploy AutoMQ.
  3. Availability Zones and Subnets: Select all the availability zones and subnets where you need to deploy AutoMQ.
  4. Security Group: It is recommended to create a new security group and ensure that an inbound rule is set to allow access to port 443 from 0.0.0.0/0.
The AutoMQ cluster needs to access the EC2 API via the VPC internal network. Therefore, when creating an EC2 Endpoint, you must ensure that the availability zone subnet configuration and security group configuration are complete, otherwise the AutoMQ cluster will fail to start.You must cover all subnets and availability zones where you plan to deploy the AutoMQ cluster, and no possible subnets should be omitted.The security group rules must ensure that port 443 and the access source are unobstructed. It is recommended to allow access from 0.0.0.0/0.

​
Scenario 2: Configure an Existing VPC

If you already have an available VPC network in your current production environment and your application is already deployed in it, you will need to install the AutoMQ environment in the existing VPC network. In this case, you still need to check whether the current VPC configuration meets the constraints.

​
Step 1: Check and Enable DNS Configuration

Access the existing VPC network, click on “Details,” and view the VPC configuration. Confirm that DNS is already configured and enabled for the current VPC network.

​
Step 2: Check and Create S3 Endpoints

Confirm whether an S3 gateway endpoint has been created for the current VPC. If there is no S3 endpoint, you need to create one. Create an S3 endpoint (if not configured in the previous step).
It is important to note that when creating an S3 endpoint, you must cover all possible subnets where the Kafka cluster might be deployed when adding route tables. Otherwise, the subsequent Kafka cluster will not be able to access the S3 service via the internal network.

​
Step 3: Check and Create EC2 Endpoints

Check and create an EC2 Endpoint in the current VPC to allow subsequent data nodes to access the EC2 API via the private network. When creating an EC2 access point, pay attention to the following configuration items:
  1. Service Type: Select “AWS services.”
  2. VPC: Select the VPC where you need to deploy AutoMQ.
  3. Availability Zones and Subnets: Select all the availability zones and subnets where you need to deploy AutoMQ.
  4. Security Group: It is recommended to create a new security group and ensure that an inbound rule is set to allow access to port 443 from 0.0.0.0/0.

​
Prepare EC2 for Deploying AutoMQ Console

The AutoMQ console is the core component for managing the AutoMQ Cloud BYOC environment, and it is deployed on a separate EC2 instance. Therefore, you need to prepare this EC2 instance before installation.

​
Basic Requirements

To ensure the stable operation of the console, the EC2 instance you prepare needs to meet the following basic requirements:
  1. Instance Specifications:
    • Configuration: At least 2-core CPU and 8 GB of memory.
    • System Volume: At least 20 GiB of storage space.
    • Data Volume: An additional data volume must be mounted to store metadata, with a space of no less than 20 GiB.
Important Note: To prevent metadata loss from the console due to EC2 instance migration or failure, ensure that an additional data volume is mounted to the EC2 instance for the AutoMQ console, and store metadata on this data volume.
  1. Network Location:
    • This EC2 instance must be located in the same VPC as the subsequently deployed AutoMQ cluster.
    • Typically, this VPC should also be consistent with the VPC where your application services are located to ensure network connectivity.
  2. Public Network Access:
    • This EC2 instance must be able to access the Internet to download installation packages and synchronize necessary metadata. You can achieve this by associating an EIP (Elastic Public IP) or configuring a public NAT gateway.

​
Creation Steps

  1. Log in to the AWS EC2 console and click Launch instance.
  2. Operating System (AMI): Choose a Linux distribution you are familiar with, such as Amazon Linux 2 or Ubuntu.
  3. Instance Type: Select an instance type that meets the above specification requirements (e.g., t3.large or higher).
  4. Network Settings:
    • Select the target VPC and a public subnet (if using an EIP) or a private subnet (if using a NAT gateway).
    • Ensure that the security group opens the necessary ports, such as the SSH (22) port for you to log in and manage, and the console service’s service port (default is 8080). The access source can be set to 0.0.0.0/0 (allowing all IP access, suitable for test environments) or restricted to a specific IP range according to your needs.
  5. Storage Configuration:
    • Configure the system volume (root volume) size to be at least 20 GiB.
    • Click Add new volume to add an additional data volume and ensure its size is not less than 20 GiB.
  6. After completing other configurations, launch the instance.
  7. Mount the Data Volume:
    • After the instance starts, log in to the EC2 instance.
    • Mount the additional data volume you added in the Storage Configuration to a specified directory in the operating system (e.g., /data).
    • This path will be used as the data mount path when starting the Docker container later.
    • Please ensure that the mount point has the correct permissions.
  8. Install and Start the Docker Service:
    • Install Docker on the EC2 instance. For specific installation steps, please refer to the official Docker documentation.
    • After installation, start the Docker service and configure it to start on boot.

​
Prepare Private DNS

The AutoMQ cluster exposes service addresses to applications through Private DNS, which relies on private domain name resolution within the VPC. Therefore, a Private DNS must be configured to provide reliable domain name resolution services. Generally, Private DNS is created at the same time as the VPC. If your VPC does not have a Private DNS, or if you need to create a new one, you can refer to the steps below for configuration.

​
Private DNS Prerequisites

When configuring Private DNS, the following two key conditions must be met:
Condition
Description
Importance
Associated VPC
The Private DNS must be associated with the VPC where AutoMQ is deployed.Required
Domain Name and TLS Consistency
If you need to use the TLS feature later, the domain name of the Private DNS must be consistent with the domain name in the TLS certificate.As needed

​
Private DNS Configuration Steps

​
Step 1: Check Existing Private DNS

  1. Log in to the AWS console and visit the Route53 console.
  2. In the left navigation bar, select Hosted zones.
  3. Check if a Private DNS associated with the target VPC already exists.

​
Step 2: Create or Configure Private DNS(Optional)

If a Private DNS already exists:
  1. Select the target Private DNS and click Details.
  2. Confirm that the Type is Private hosted zone.
  3. Check the VPCs section to ensure it includes the VPC where AutoMQ is deployed.
  4. Record the domain name information for subsequent TLS feature configuration.
If you need to create a new Private DNS:
  1. Click Create hosted zone.
  2. Configure the basic information:
    • Domain name: Enter a private domain name (e.g., internal.example.com).
    • Type: Select Private hosted zone.
    • VPC: Select the VPC where AutoMQ is deployed and the corresponding region.
  3. Click Create hosted zone.

​
Step 3: Verify the Configuration

  1. On the Private DNS details page, confirm the VPC association status.
  2. Check if the domain name configuration meets the planning requirements for the TLS feature.
  3. Record the ID and domain name of the Private DNS, which will be provided to the AutoMQ components during subsequent installation.