Skip to main content
Refer to Overview▸, AutoMQ supports deployment on Kubernetes. This article outlines the installation process for deploying AutoMQ on the AWS EKS platform.
Throughout the product’s full lifecycle, AutoMQ will maintain deep and continuous integration with Amazon Elastic Kubernetes Service (EKS) and EKS Anywhere (EKS-A).To uphold this commitment, we ensure:Continuous version compatibility: We guarantee that each new version of AutoMQ will be compatible with both current and future versions of EKS and EKS-A. Additionally, we proactively adapt to updates on the AWS platform, offering users a seamless, forward-compatible experience.Timely response and resolution: We have established a clear response mechanism to quickly identify and resolve any configuration, security, and non-security issues reported by customers or AWS.

​
Install the Environment Console

Refer to Overview▸, AutoMQ supports deployment on EKS clusters. In EKS deployment mode, you must first install the AutoMQ console and then use the console interface to operate EKS, deploying the cluster onto EKS. For instructions on installing the console, please refer to Prepare AWS BYOC Environment▸.

​
Prepare EKS Cluster and Necessary Node Pools and Other Resources

If you expect to run the AutoMQ cluster on Kubernetes, you need to prepare an EKS cluster for AutoMQ to use. If you expect to deploy the AutoMQ cluster on EC2, you do not need to prepare an EKS cluster.

​
Step 1: Create the Required IAM Roles

Before creating the EKS cluster and node groups, you need to create a dedicated IAM Role. This Role is used for the EKS cluster’s own service authorization.
  1. Access the IAM console and click Create Role.
  2. Trusted entity type: Select AWS Service.
  3. Service Use case: Select EKS-Cluster.
  4. Click Next, enter a custom role name, and create the IAM Role.
After creating the IAM role for the EKS cluster, you also need to create an IAM Role for the EKS node pool. Refer to the steps in the AWS node pool documentation “Amazon EKS node IAM role” to create the node pool IAM Role.

​
Step 2: Create EKS Cluster

  1. Log in to the AWS EKS console, click Create cluster, and select the custom creation mode.
  2. Configure basic cluster information:
    • Bind the EKS cluster IAM Role created in the previous step.
    • Select the Kubernetes version, choose version 1.32.
The AutoMQ cluster needs to access the EC2 Node’s Metadata service, which by default requires an AL2 (Amazon Linux 2) version node pool to work properly. Starting from EKS version 1.33, AWS no longer supports AL2 node pools by default, so it is recommended to choose an EKS cluster of version 1.32.
  1. Configure VPC network:
    • Select the VPC you have prepared for AutoMQ; this VPC should be consistent with the VPC where the application is located.
    • Select the private subnets for deploying the cluster.
    • It is recommended to select the default security group for the VPC.
  2. Keep other default configurations and create the EKS cluster.
  3. Authorize console access. To allow the AutoMQ console to manage the EKS cluster, you need to authorize it.
    1. Go to the Access menu of the EKS cluster and click Create Access Entry.
    2. Select the IAM Role used by the AutoMQ console (usually created when installing the console).
    3. Set Type to Standard.
    4. Click Next, and on the Add access policy page, select the AmazonEKSClusterAdminPolicy policy.
    5. Select Cluster for the Scope and click Add.
    6. Complete the creation.
  4. Set security group rules to allow access to Pods and the EKS API Server service from outside the EKS cluster.
    The AutoMQ console component needs to call the EKS cluster’s API Server, so you need to ensure that the security group where the console is located can access the EKS cluster’s API Server port (port 443). In addition, the AutoMQ cluster’s Broker needs to expose service ports. If components outside the cluster (including the console and Producer/Consumer applications) need to access AutoMQ, you need to add inbound rules to the EKS cluster’s security group.
    • Ports: 443, 9090, 9092, 9102, 9122, 9112, 8083.
    • Source: Set according to the CIDR range where your application is located. At a minimum, you need to ensure that the EC2 where the AutoMQ console is located can access it.

​
Step 3: Create EKS Node Groups

You need to create at least two types of node groups for the EKS cluster: a public node group for running system components, and a dedicated node group for running AutoMQ workloads.

​
1. Create a Public Node Group

  1. Go to the EKS cluster details page, click the Compute menu, and Create node group.
  2. IAM Role: Select the node group IAM Role you created in Step 1.
  3. AMI Type: Select Amazon Linux 2.
  4. Instance Configuration: It is recommended to select t3.medium or equivalent instances, with a quantity of 2-3.
  5. Subnet: Select all private subnets required for EKS deployment.
  6. Complete the node group creation.

​
2. Create a Dedicated AutoMQ Node Group

  1. Go to the Compute menu again and Create node group.
  2. IAM Role: Select the node group IAM Role you created in Step 1.
  3. Taints: Configure a taint for this node group to ensure that only AutoMQ’s Pods can be scheduled on it. Please use the following configuration:
    • Key: dedicated.
    • Value: automq.
    • Effect: NO_SCHEDULE.
  4. Instance Configuration: Please refer to AutoMQ’s instance recommendations to select the instance type.
  5. AMI Type: Select Amazon Linux 2.
  6. Subnet: Depending on your AutoMQ cluster plan, select private subnets in one or three availability zones.
  7. Quantity: It is recommended to start with 3 nodes and reasonably evaluate the maximum number of nodes based on the business scale.
  8. Complete the node group creation.
If the AWS console cannot create an AL2 system, you can create it using CLI tools or Terraform. Here is a reference for the CLI command:
aws eks create-nodegroup \
  --region {regionId} \
  --cluster-name {EKS Cluster Name} \
  --nodegroup-name {EKS Node Group Name} \
  --subnets {Subnet list} \
  --node-role {Node Group Role ARN} \
  --instance-types {EC2 VM Type} \
  --ami-type AL2_x86_64 \
  --disk-size 20 \
  --scaling-config {minSize=3,maxSize=3,desiredSize=3}

​
3. Create Kafka Connector Node Group (Optional)

If you plan to use the managed Kafka Connect service provided by AutoMQ, you also need to prepare node resources for running the Connector. It is recommended to create a node pool by referring to the method for creating a public node pool, or directly scale up the public node pool to ensure sufficient computing resources.

​
Step 4: Initialize Local Tools and Configure EKS Plugins

The newly created EKS cluster needs to have some necessary system plugins installed to work with AutoMQ.

​
1. Initialize Local Tools

  1. Install and configure AWS CLI, kubectl, and Helm in your local environment.
  2. Execute the following command to generate the KubeConfig configuration file so that kubectl can connect to your EKS cluster. 
    # Replace <region> and <cluster-name> with your actual values
aws eks update-kubeconfig --region <region> --name <cluster-name>

​
2. Configure EKS AutoScaler (Optional)

To enable on-demand auto-scaling of EKS node groups, you need to configure the Cluster Autoscaler. If you do not need to use auto-scaling, you can skip this step.
  1. Download the AutoScaler configuration file from the download link.
  2. Modify the configuration file, replacing <YOUR CLUSTER NAME> with your EKS cluster name.
  3. Execute the installation command: 
    kubectl apply -f cluster-autoscaler-autodiscover.yaml

​
3. Create IAM OIDC Provider

The AutoMQ data plane Pod needs to access cloud resources such as S3 and needs to assume an IAM Role to obtain temporary access authorization. In EKS, for a Pod to be able to assume an IAM Role, the EKS cluster must first be registered as a trusted identity provider with IAM. In addition, installing the EBS CSI plugin and the Load Balancer Controller plugin also requires an OIDC Provider. You can refer to the official AWS documentation “Create an IAM OIDC provider for your cluster” to create an OIDC Identity Provider for your EKS cluster in the IAM console.

​
4. Configure Amazon EBS CSI Driver

The EKS cluster needs this plugin to manage storage volumes.
  1. Refer to the official AWS documentation “Amazon EBS CSI driver” to create a dedicated IAM Role for the CSI Driver after creating the OIDC Provider above.
  2. After completing the IAM Role and Trust Policy configuration, go to the Add-ons tab of the EKS cluster console, add the Amazon EBS CSI Driver, and select the IAM Role created in the previous step.

​
5. Install AWS Load Balancer Controller

This plugin is used to manage Load Balancers and provide an external access entry for AutoMQ to the EKS cluster. To install the Load Balancer Controller, you can refer to the official documentation.
  1. Execute the following command to generate the IAM Policy. 
    curl -o iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.10.0/docs/install/iam_policy.json
aws iam create-policy --policy-name AWSLoadBalancerControllerIAMPolicy --policy-document file://iam-policy.json
  2. Create an IAM Role and attach the Policy created in the previous step to this Role. 
    eksctl create iamserviceaccount \
  --cluster=<cluster-name> \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --attach-policy-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy \
  --override-existing-serviceaccounts \
  --region <region-code> \
  --approve
  3. Add the Helm repository and install the CRD: 
    helm repo add eks https://aws.github.io/eks-charts
kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller/crds?ref=master"
  4. Execute the installation command, paying attention to replace <eks-cluster-id> with your cluster name, <Your RegionID> with your region ID, and <Your VPCID> with your VPC ID: 
    helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=<eks-cluster-id> --set region=<Your RegionID> --set vpcId=<Your VPCID>

​
Create an AutoMQ Instance, Select Deploy to Kubernetes

Log in to the AutoMQ console, create an instance, select Kubernetes as the deployment type, and configure the following information as required.
  1. Deployment Type: Select Kubernetes.
  2. Kubernetes Cluster: Fill in the cluster name of the EKS cluster.
  3. Node Pool: Fill in the name of the node pool used to deploy AutoMQ. Note that the instance type specifications and quantity of the node pool must meet the environmental requirements▸.
  4. Namespace (only required when managing IAM permissions yourself): Fill in the namespace used to deploy AutoMQ.
  5. ServiceAccount (only required when managing IAM permissions yourself): Fill in the ServiceAccount used to deploy AutoMQ. Note that this ServiceAccount must be associated with the corresponding IAM Role (via EKS OIDC Provider or AWS EKS IRSA).
  6. Preview the configuration information and complete the creation.