Skip to Main Content

Deploy to Google Cloud GKE

Refer to Overview▸, AutoMQ supports deployment on Kubernetes. This article explains the installation process of deploying an AutoMQ cluster to the Google Cloud GKE platform.

In this article, references to AutoMQ product service provider, AutoMQ service provider, or AutoMQ specifically refer to AutoMQ HK Limited and its affiliates.

Operation Procedure​

Step 1: Install Environment Console​

Refer to Overview▸, AutoMQ supports deployment on GKE cluster. In the GKE deployment mode, you first need to install the AutoMQ console, then use the console interface to operate GKE and deploy the cluster on GKE.

On Google Cloud, both Marketplace and Terraform are supported for installing the environment console.

After the AutoMQ console installation is complete, you need to obtain the console address, initial username, and password from the console interface or Terraform output menu.

Step 2: Create an IAM Service Account​

The AutoMQ data plane cluster is deployed using a dedicated node pool, so it is necessary to bind a dedicated IAM service account to the AutoMQ node pool to access GCP cloud services. Follow these steps to create an IAM service account:

  1. Navigate to IAM Console, click Service Accounts, then Create Service Account. Fill in the custom service account name.

  1. Grant the service account the necessary system roles. The list of system roles is as follows:

    • roles/resourcemanager.tagUser

    • roles/artifactregistry.reader

    • roles/autoscaling.metricsWriter

    • roles/monitoring.metricWriter

    • roles/stackdriver.resourceMetadata.writer

    • roles/monitoring.viewer

    • roles/logging.logWriter

  1. Complete the service account creation, and record the service account's name and ID, which are needed for the subsequent creation of the node pool.

Step 3: Create a GKE Cluster​

Refer to Overview▸. Users need to create an independent GKE cluster allocated for AutoMQ. Access the Google Cloud GKE product console and follow the steps below.

Configure GKE Cluster Type and Region​

  1. Log in to the Google Cloud GKE Console. Click Create Cluster.
  1. Select the cluster type as Standard: You manage your cluster.

  1. Fill in the cluster name and region information:

    • Location Type: Choose Regional so you can create an AutoMQ instance with three availability zones later.

    • Target Release Channel: It is recommended to choose the stable version.

Configure GKE Cluster Network​

  1. Switch to Networking Configuration and select the Target VPC and subnet.

Note:

The default Firewall policy between multiple subnets in Google Cloud does not allow intercommunication by default. Therefore, it is recommended to install both GKE and the AutoMQ environment console in the same subnet.

If the GKE, AutoMQ environment console, and Kafka client applications are not in the same subnet, you need to go to VPC Firewall Policies.

If you need to manually add firewall policies, it is recommended to allow all IP ranges of the subnet where the AutoMQ environment console resides. The port list includes: 9090, 9092-9093, 9102-9103, 22.

  1. Modify the network configuration. Ensure the following options are set:

    1. Enable VPC-native traffic routing.

    2. Enable Dataplane V2.

Set up the GKE Cluster Public Node Pool​

Switch to the NODE POOLS configuration item, configure the default node pool for deploying GKE system components. Adjust the number of nodes to 1 per available zone, ensuring the total number of nodes is at least 3. Apart from the configuration shown in the figure, other options for the GKE system components node pool can use the default configuration.

Add an AutoMQ Dedicated Node Pool​

Refer to Overview▸. Users need to create a dedicated node pool for AutoMQ to apply for machines in subsequent deployment instances. Therefore, when creating a GKE cluster, you need to create 2 node pools: one public node pool for deploying GKE system components and one dedicated node pool for AutoMQ to deploy the AutoMQ cluster.

  1. Click the Add Node Pool button above to configure the AutoMQ dedicated node pool.
  1. Enable Cluster AutoScaler and configure it to Balanced type, with the size limit set to Total limits. Set an appropriate upper limit (recommended upper limit is 50-100).
  1. In the Nodes tab, refer to Overview▸ to configure the machine type list that meets the requirements of AutoMQ.
  1. In the Security tab, bind the service account created for the AutoMQ node pool in Step 2.

The node security configuration is as follows:

  1. In the Metadata tab, set the taint for AutoMQ exclusive use. The key of the taint is dedicated, the value is automq, and the effect is NO_SCHEDULE.

Click to Create the GKE Cluster​

  1. After verifying all configurations are correct, click the Create button below to create the GKE cluster.

  2. Once the GKE cluster is created, copy the cluster Name for use in step 3 configuration.

  1. Locate the GKE cluster in the cluster list and add a tag to it. This tag will be used to manage permissions in the subsequent AutoMQ console.

AutoMQ Console only supports managing GKE clusters that include the automqAssigned: automq tag. Therefore, you need to add the aforementioned tag to your GKE cluster.

Step 4: Access the Environment Console and Create Deployment Configuration​

When you first enter the AutoMQ BYOC console, you need to create a deployment configuration to set up Kubernetes cluster information, GCS Bucket information, and other settings before you can create instances.

Note:

If the AutoMQ environment console (Kafka client application) and GKE cluster are deployed in different VPC subnets, users need to go to the VPC firewall page to add traffic rules. Otherwise, network issues may impact the creation and usage of instances.

It is recommended to allow all IP sources from the subnet where the environment console is located in the added firewall policy. The port list is: 9090, 9092-9093, 9102-9103, 22.

  1. Copy the cluster name of the GKE cluster created in step 3.

  2. Log in to the console, enter the cluster name and other configurations, and click Next.

    1. Deployment Type: Select Kubernetes.

    2. Kubernetes Cluster: Enter the name of the GKE cluster.

    3. DNS ZoneId: Enter the ZoneId of the Private DNS used for deploying AutoMQ.

    4. Bucket Name: Enter the name of the data Bucket used for storing messages in AutoMQ. Multiple GCS Buckets are supported.

  1. After filling in the cloud resource information, generate the necessary permissions for the data plane GKE node pool. Refer to the console guidance to create a custom authorization policy. Then bind the authorization policy to the AutoMQ IAM delegate created in Step 2, and enter the name of the node pool delegate, click Next to preview.
  1. Preview the deployment configuration information, complete the creation. You can then go to the instance management page to create an instance.