Skip to main content
In the Kafka Connect framework, a Connector defines and manages a set of Tasks for performing specific data integration tasks (ETL). In the AutoMQ managed Kafka Connect service, the multiple Tasks running under each Connector are scheduled in a separate Worker cluster, ensuring resource isolation and stable operation between tasks.

​
Prerequisites

Before creating and deploying a Connector, you need to ensure that the following dependent resources are ready:
  • A running AutoMQ instance and ACL user credentials for data access.
  • Kafka Topics and Consumer Groups for the data integration source or target.
  • A correctly configured Amazon EKS (Elastic Kubernetes Service) cluster, with a Service Account and Namespace prepared for the Connector to run in the EKS cluster.
  • An AWS IAM Role with specific permissions to authorize the Connector to access cloud resources, and this IAM Role must be bound to the Service Account.

​
Step 1: Create an AutoMQ Instance

Data integration tasks require an AutoMQ instance as a metadata service and as a Sink or Source for data integration. If you have not yet created an instance, please refer to the following documentation to complete the creation.

​
Step 2: Create ACL User Credentials and Complete ACL Authorization

The Connector needs to use the Kafka client protocol to access the AutoMQ cluster, so it must be provided with a valid set of ACL user credentials.
Important Constraint: When creating an ACL user, you must ensure that it is granted read/write permissions for all Topics that the Connector is expected to access. If your Connector is a Sink Connector (data export), you must also grant it read permissions for the relevant Consumer Group. Otherwise, the Connector task will fail to start due to insufficient permissions.

​
Step 3: Prepare the EKS Environment and IAM Role

The Worker of AutoMQ managed Kafka Connect runs in your own EKS cluster to ensure data security and network isolation. At the same time, it needs to obtain permissions to access other cloud resources (such as S3) through a bound IAM Role. When preparing the EKS environment, please pay special attention to the following points:
  • Binding of Service Account and IAM Role: You need to create a Kubernetes Service Account and bind it to an AWS IAM Role (usually through IRSA - IAM Roles for Service Accounts).
  • IAM Role Permission Policy: You must attach a permission policy to the above IAM Role, granting it at least read and write permissions to the current AutoMQ environment’s operational storage bucket (S3 Bucket).
Here is an example of an IAM Policy (please replace YOUR_BUCKET_NAME with the actual bucket name): 
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetObjectVersion",
                "s3:DeleteObjectVersion",
                "s3:ListBucketVersions"
            ],
            "Resource": [
                "arn:aws:s3:::YOUR_BUCKET_NAME/*",
                "arn:aws:s3:::YOUR_BUCKET_NAME"
            ]
        }
    ]
}
Important Reminder: If the current Connector needs to access other cloud resources besides the AutoMQ storage bucket (such as RDS, DynamoDB, etc.), please ensure that the corresponding access permissions are added to the IAM Role’s permission policy.
After completing the preparation of all the above cloud resources and related configurations, you can create a Kafka Connector.

​
Create a Connector

The process of creating a Connector is mainly divided into steps such as deployment information, plugin parameters, and Metrics integration.

​
Step 1: Fill in the Deployment Configuration

  1. Log in to the AutoMQ console and select Kafka Connect in the left navigation bar.
  2. Click the Connector tab, and then click the Create Connector button.
  3. On the creation page, you need to fill in the following basic parameters:
    • Select Plugin: Select a plugin and its version that you want to use from the drop-down list. This is a required option.
    • Select Kafka Cluster: Configure the information required for the Connector to access the AutoMQ cluster here.
      • AutoMQ Instance: Select the AutoMQ instance you have prepared.
      • ACL Username/Password: Fill in the ACL user credentials for data access.
      • Topic / Consumer Group: Depending on whether the plugin is a Source or Sink type, fill in the Topic or consumer group to be read from or written to.
    • Select EKS Cluster: Configure the underlying computing resources for the Connector to run.
      • EKS Cluster: Select the EKS cluster you have prepared.
      • Namespace: Fill in the Kubernetes namespace prepared for the Connector.
      • Service Account: Fill in the service account that has been bound to the IAM Role.
    • Fill in Capacity Configuration: Allocate resources for the Connector based on your data load.
      • Max Tasks: Specify the maximum number of Tasks that the Connector can run in parallel.
      • Number of Workers: Specify the number of Worker processes to run these Tasks.
      • Worker Specifications: Select the appropriate CPU and memory specifications for each Worker. You can choose different sizes of specifications based on the resource consumption of the plugin.
You need to provide the necessary Kubernetes resource quotas (CPU, memory, etc.) for the Connector to run. Please ensure that the node pool in the EKS cluster has sufficient resources to meet the Connector’s running requirements.

​
Step 2: Fill in the Plugin Configuration

After completing the basic configuration, click Next to enter the plugin parameter configuration page. AutoMQ supports two modes for filling in the custom configuration required by the plugin:
  • Form Mode: AutoMQ will automatically parse the plugin’s configuration definition (Schema) and render it into a structured form. You can directly fill in according to the description and prompts of each parameter, which is intuitive and convenient.
  • Custom Mode: If you are more familiar with the plugin’s parameters or need to fill in advanced parameters not covered by the form mode, you can choose this mode. In this mode, you can completely freely specify all configurations in the form of key-value pairs.

​
Step 3: Configure Metrics Export (Optional)

To achieve custom monitoring and alarming, AutoMQ supports exporting the Connector’s runtime Metrics data to your own Prometheus system. In this step, you can choose to configure the remote_write address and access credentials (such as Basic Auth) of the external Prometheus. After configuration, all relevant Metrics data will be forwarded in real time.

​
Step 4: Preview and Create

After completing all configurations, you can check all parameters on the preview page. After confirming that there are no errors, click the Create button, and AutoMQ will start deploying and starting your Connector task.

​
Manage a Connector

After the Connector is created, you can perform comprehensive management and operation and maintenance operations on its details page.

​
Modify Configuration

You can enter the Connector details page at any time to modify its configuration.
  • Modify Capacity Configuration: According to changes in business load, you can dynamically adjust the Max Tasks, Number of Workers, and Worker Specifications to achieve elastic scaling of resources and meet the performance requirements of different scenarios.
  • Modify Runtime Parameters: You can modify the custom configuration of the plugin to adjust its running behavior.

​
Manage Task Lifecycle

On the Connector details page, you can perform lifecycle management operations on the underlying Tasks, which is convenient for debugging or maintenance.
  • Pause: Temporarily stop all data processing activities of all Tasks.
  • Resume: Resume the operation of all Tasks from the paused state.
  • Restart: Batch restart all Tasks, usually used to load new configurations or recover from an error state.

​
Delete a Connector

When you no longer need to run a certain Connector, you can delete it at any time. Procedure:
  1. On the Connector list page, select the target Connector.
  2. Click the Delete button.
  3. After confirming the operation in the secondary confirmation dialog box, the Connector will be permanently deleted.
Note: The delete operation is irreversible. Deleting a Connector will also remove all its running Tasks, as well as the consumption offsets and progress information related to these Tasks.

​
Production Environment Monitoring and Alarming

For Connectors running in a production environment, complete monitoring and alarming are the key to ensuring their stability. AutoMQ provides observability capabilities in two dimensions: Metrics and logs.
  • Metrics Integration: When creating or editing a Connector, you can connect the core indicators of the Connector’s operation (such as throughput, latency, error rate, etc.) to your own Prometheus monitoring system by configuring Metrics export, and configure alarm rules based on this data.
  • Log Query: When you need to troubleshoot problems, you can go to the Connector’s details page at any time to query its running logs and quickly locate the root cause of the problem.