Table of Contents
Table of Contents
An Amazon MSK cluster often lives in a platform account while its Kafka clients live in application, analytics, or security accounts. The first connection may be easy to prove. The harder question is what happens after teams need access, one account changes its CIDR range, or the cluster owner must revoke one consumer without disturbing the others.
VPC peering, AWS Transit Gateway, and Amazon MSK multi-VPC private connectivity can all keep traffic private. They create different ownership and failure boundaries, though. Peering joins two networks, Transit Gateway centralizes routing between many networks, and MSK multi-VPC connectivity exposes a managed PrivateLink path to a specific cluster. Treating them as interchangeable usually produces a design that is difficult to govern.
The useful decision starts with the consumer boundary. Should each account receive broad network reachability or a narrow connection to one streaming service? Do CIDR blocks overlap? Who owns route tables and attachments? Those answers are more useful than a generic “PrivateLink is more secure” claim.
1The short decision
Use this as a starting point, then test the path with real client subnets, authentication, and account policies.
| Topology | Best fit | Boundary it creates | Main constraint |
|---|---|---|---|
| VPC peering | One or a few stable consumer VPCs that need private IP reachability | A direct relationship between two VPCs | Non-transitive routing and non-overlapping CIDRs |
| Transit Gateway | Many VPCs that need a centrally managed routing domain | A shared network hub with route-table segmentation | The hub, attachments, and route policies become shared dependencies |
| MSK multi-VPC private connectivity | Cross-account clients that need access to one MSK Provisioned cluster in one Region | A managed, cluster-scoped PrivateLink connection | Same Region, supported cluster/client prerequisites, and one auth scheme per managed connection |
This table describes network shape, not authorization. A reachable broker does not grant Kafka topic access. IAM policies, Kafka ACLs, security groups, and client credentials still decide whether a producer or consumer can use the cluster.
For a small number of accounts with non-overlapping address space and a need to access several private services, peering can be the simplest operating model. For a growing AWS organization with a network team and multiple shared services, Transit Gateway usually makes routing ownership clearer. For an MSK cluster that must be consumed by separate accounts without exposing a broad routed network, the managed MSK PrivateLink path is often the narrower option—provided its documented requirements match the cluster.
2Start with the account and consumer boundary
Write down who owns each side before choosing a connection:
- Cluster owner: the account that operates MSK, its broker configuration, authentication settings, and cluster policy.
- Client owner: the account that runs producers, consumers, connectors, or stream processors.
- Network owner: the team that approves CIDRs, routes, attachments, endpoints, and flow-log access.
- Authorization owner: the team that grants IAM actions or Kafka ACLs to topics and consumer groups.
Those owners may be one team or several. A peering request is a direct agreement between VPC owners. Transit Gateway adds a central attachment and route-table decision. A managed MSK connection requires the cluster owner to enable it and the client account to create its side.
If the requirement is regional failover rather than same-Region client access, start with the Amazon MSK Replicator and MirrorMaker 2 comparison instead of treating a VPC connection as a replication design.
The boundary affects revocation: removing a shared route can affect several services, while removing one managed MSK connection can isolate one client VPC.
3VPC peering: a direct path for a small, stable graph
VPC peering creates a private network connection between two VPCs. AWS describes it as a one-to-one relationship; peering connections do not provide transitive routing. If a client VPC reaches a shared-services VPC through a third VPC, the client does not automatically gain a path to an MSK VPC through that chain. The route tables on both sides must explicitly send the relevant destination CIDR through the peering connection.
For MSK, the platform and client teams must coordinate:
- Confirm that VPC CIDR blocks do not overlap.
- Accept the request and add routes for the broker destinations.
- Permit the client subnets in both sides' security groups.
- Verify bootstrap DNS resolution and private reachability.
- Configure Kafka authentication and authorization separately.
The direct graph is easy to understand when one application account connects to one cluster. It becomes a coordination problem as the graph grows. Each additional VPC can add another peering relationship and another set of route and security-group changes. The peering path also does not solve overlapping CIDRs, which is common after acquisitions or when multiple teams created VPCs independently.
Peering fits a small consumer set with controlled CIDRs and access to several services in the cluster VPC. It is a poor fit when an account should reach MSK without inheriting a general route to platform subnets.
4Transit Gateway: a routed hub with explicit segmentation
AWS Transit Gateway acts as a regional network transit hub. VPCs attach to the hub, and Transit Gateway route tables determine which attachments can reach one another. This makes it possible to centralize routing policy and avoid a full mesh of peering connections.
The benefit is organizational as much as technical. A platform team can create a route domain for approved Kafka consumers, associate application attachments with it, and keep unrelated VPCs isolated. Transit Gateway peering can connect hubs across Regions, but that is separate from same-Region MSK managed PrivateLink.
The trade-off is a larger shared control plane. An attachment, route-table association, propagation rule, or inspection change can affect multiple applications. Monitor attachment state and route health as part of the MSK service path; flow logs and route-table evidence belong in the incident runbook.
Use Transit Gateway when the question is “Which VPCs may communicate within this routed network?” It is a good fit for organizations that already operate a hub-and-spoke network and need MSK to participate in that network. It is less attractive when each consumer should receive a narrow, separately revocable service connection or when the accounts have overlapping CIDRs that the routing domain cannot reconcile.
5MSK multi-VPC private connectivity: a managed PrivateLink boundary
Amazon MSK provides multi-VPC private connectivity powered by AWS PrivateLink. The feature is designed for Kafka clients in different VPCs and AWS accounts to connect to an MSK cluster without the cluster owner operating peering or Transit Gateway routes for each client. The cluster owner enables selected authentication schemes and attaches a cluster policy. The client account creates a managed VPC connection, selects its VPC and subnets, and receives bootstrap broker strings for that connection.
The current AWS documentation defines important boundaries:
- The feature is for Amazon MSK Provisioned clusters and supports Apache Kafka 2.7.1 or later.
- The cluster and client VPCs must be in the same AWS Region; this feature is not a cross-Region replication path.
- Supported client authentication schemes are IAM, TLS, and SASL/SCRAM. Unauthenticated clusters and plaintext access are not supported for this path.
- Connecting VPCs may use overlapping IP ranges, which avoids one of the main peering and routed-network constraints.
- The number of client subnets must match the number of cluster subnets, and their Availability Zone IDs must match.
- Each managed VPC connection uses one authentication scheme.
- The feature does not provide access to ZooKeeper nodes.
These are design gates, not footnotes. If the cluster is MSK Serverless, if the application needs a cross-Region route, or if the client relies on an unsupported authentication integration, this topology should leave the shortlist before implementation work begins.
PrivateLink also changes how ownership is expressed. The MSK cluster policy can authorize another AWS account to create a connection, while the client account needs permissions such as kafka:CreateVpcConnection, ec2:CreateTags, and ec2:CreateVPCEndpoint for its workflow. IAM authentication can use the cluster policy for granular Kafka data-plane permissions. For TLS or SASL/SCRAM, Kafka ACLs still need to be configured, and AWS requires allow.everyone.if.no.acl.found=false when using the managed multi-VPC path.
The connection is cluster-scoped; it does not make every service in the cluster VPC reachable. Choose it when the platform team wants service-level exposure.
6Identity, DNS, and client configuration
A network connection is complete only when the client can resolve and authenticate to the brokers. Keep three test layers separate:
- Network: subnet routes or endpoint state, security-group rules, and reachability to the broker listener.
- Name resolution: the bootstrap broker names resolve from the client VPC to addresses that are reachable through the selected topology.
- Kafka authorization: the client identity can authenticate and perform the intended topic and group actions.
Peering and Transit Gateway usually preserve a routed path to the MSK VPC, so the client configuration must use broker endpoints that resolve correctly across the VPC boundary. PrivateLink creates a managed connection and a connection-specific set of bootstrap broker strings; do not reuse a bootstrap value from the cluster VPC without checking the managed connection details.
Authentication should follow the identity owner: IAM for AWS runtime identities, SASL/SCRAM for Kafka users and Secrets Manager credentials, and TLS for client certificates and Kafka ACLs. The Amazon MSK authentication comparison covers client and rotation implications in more detail. The topology does not remove secret rotation, certificate renewal, or authorization tests.
A useful acceptance test produces evidence for each layer:
- DNS lookup output for every bootstrap name.
- A TCP connection test from each client subnet.
- Broker or client logs showing the selected authentication mode.
- A produce-and-consume test limited to the intended topic and consumer group.
- A negative test proving that an unapproved topic or account is denied.
7Failure domains and operational ownership
Ask what can fail together.
With peering, the direct connection, route entries, and security-group rules are the main shared dependencies. With Transit Gateway, a hub or route-table mistake can affect many attachments, so the network team must identify which route table and association carried the traffic.
With managed MSK PrivateLink, the evidence chain is the cluster connectivity setting, cluster policy, managed VPC connection, endpoint security group, and connection-specific bootstrap brokers. A single connection can be deleted or rejected without changing every consumer path.
Define ownership in the runbook:
| Evidence | Peering | Transit Gateway | MSK managed PrivateLink |
|---|---|---|---|
| Route decision | Two VPC route tables | TGW route table and attachment association | Managed connection and endpoint state |
| Access policy | Security groups plus Kafka auth | TGW routes, security groups, and Kafka auth | Cluster policy, endpoint security group, and Kafka auth |
| Common outage scope | One peering pair | Multiple attached VPCs | One connection or the MSK connectivity setting |
| First responder | VPC/application owners | Central network team | MSK owner plus client account owner |
8Cost and governance
Avoid choosing by a single “data transfer cost” line. Model the full path:
- recurring hub or endpoint charges for the selected AWS networking service,
- data processing or transfer charges that apply to the path and Region,
- the number of client VPCs and Availability Zones,
- the engineering time required to approve routes, policies, and endpoint connections,
- the operational cost of observing and troubleshooting the shared path.
AWS pricing changes by service, Region, and traffic direction, so use the current Amazon VPC pricing and AWS Transit Gateway pricing pages for a workload-specific estimate. Keep the estimate separate from MSK broker and storage charges.
Governance should be expressed as policy. Tag connections and attachments with the owning team and review date. Record which account may connect, which topics it may access, and who can revoke it. Flow logs and Kafka audit records should identify the account, subnet, and listener without reconstructing the path during an incident.
9Recommendation by topology
Choose VPC peering when a small, known set of VPCs needs broad private reachability, CIDRs do not overlap, and the teams can manage a direct relationship.
Choose Transit Gateway when the organization runs a hub-and-spoke network, needs centralized segmentation for many VPCs, and accepts the hub as a shared failure boundary.
Choose MSK multi-VPC private connectivity when the requirement is narrower: clients in separate accounts need private access to one MSK Provisioned cluster in one Region, CIDRs may overlap, and the cluster and clients meet the authentication, subnet, and Kafka-version prerequisites.
If none of those boundaries fit, change the problem statement before changing the service. A cross-Region requirement points to replication or a separate regional cluster. A need to expose several private services points to a routed network. A need for a single-cluster service boundary points back to PrivateLink.
10Where AutoMQ fits
After the connectivity decision, some teams revisit data-plane ownership. A Kafka-compatible platform such as AutoMQ can separate broker compute from durable stream storage through a shared-storage architecture. Its architecture documentation is relevant for teams evaluating a BYOC data plane while keeping Kafka client compatibility. That is separate from choosing a network path for an existing MSK cluster; validate it against one representative workload.
11FAQ
11.1Is Amazon MSK multi-VPC private connectivity the same as VPC peering?
No. It is a managed AWS PrivateLink path to an MSK cluster, not general routing. It supports overlapping VPC IP ranges; peering requires a direct routed relationship and non-overlapping ranges.
11.2Can MSK multi-VPC private connectivity connect Regions?
No. AWS documents this feature for AWS accounts within the same Region. Use a replication or multi-Region architecture when the requirement is regional failover or data movement.
11.3Should I use Transit Gateway for every cross-account MSK client?
Use it when clients belong in a shared routed network. For one cluster and a separately revocable boundary, consider a managed MSK VPC connection when its prerequisites are satisfied.
11.4Does PrivateLink replace Kafka ACLs?
No. IAM, TLS, and SASL/SCRAM establish the supported client identity path, but authorization still needs to be configured. AWS requires Kafka ACL handling for the SASL/SCRAM and mutual TLS modes on the managed multi-VPC path.
11.5What should I test before approving the topology?
Test DNS, TCP reachability, authentication, topic and group authorization, failure isolation, and revocation. Save the outputs for later changes.
Cross-account MSK access is a topology decision with an ownership consequence. Pick the path that makes the desired boundary visible: direct VPC reachability for a small graph, a centrally governed routed network for many VPCs, or a managed cluster-scoped connection for separate accounts. If you are evaluating Kafka-compatible streaming with the same level of boundary detail, start with AutoMQ and test one workload end to end.
