Table of Contents
Table of Contents
The client does not know whether a Redpanda broker runs in a Kubernetes Pod or on a virtual machine. It still sends Kafka requests to a bootstrap address, waits for an acknowledgment, and reads records by offset. The failure domain underneath that protocol is different, though. A Pod can be recreated after a node event, while a virtual machine may keep its identity and attached volume through the same event. Both behaviors can be correct. They create different recovery work for the Kafka operator.
That difference is often missed when a deployment review starts with a preference for Kubernetes or VMs. The useful question is narrower: which layer owns the state, which layer can move it, and which team is responsible when the move is incomplete? A Redpanda Kubernetes deployment should be compared with a VM deployment by those boundaries, not by the number of YAML files or the number of servers in a diagram.
1Start with the failure domain, not the substrate
A Kafka failure domain is the smallest boundary in which a fault can remove service, delay recovery, or force data movement. For a Redpanda cluster, that boundary can include the broker process, its local data path, the node or VM hosting it, the network route to clients, the scheduler that replaces it, and the storage system that keeps the log. A zone or region may be a larger boundary, but it is one of several boundaries that matter during an incident.
The first review artifact should be a map that names each boundary and an owner. Kubernetes adds a scheduler, controllers, node pools, PersistentVolume (PV) and PersistentVolumeClaim (PVC) behavior, and cluster networking to the path. A VM deployment exposes more of the host and disk lifecycle directly to the team, while automation may add its own replacement behavior. Neither model removes the need to explain where a Partition's durable bytes live during a broker restart.
flowchart LR
C[Kafka clients] --> N[Network and listener path]
N --> B[Redpanda broker process]
B --> S[Partition log and metadata]
S --> D[Persistent storage]
B --> O[Scheduler or VM automation]
O --> H[Node or VM failure boundary]
H --> NUse the map to ask the same questions of both deployment models:
- Process: What restarts the broker, and how does the platform decide it is healthy?
- Identity: Which hostname, listener, volume, and broker identity must survive replacement?
- Storage: Can the replacement attach the same durable state, and what is the attach or recovery window?
- Network: Can Producers, Consumers, and administrative clients reach the replacement through the same intended path?
- Ownership: Which team approves a disruptive action, and which team can prove that the cluster returned to the required Kafka state?
Redpanda's Kubernetes deployment overview and production workflow describe the deployment concerns that belong in this map. They are useful starting points, but the exact Redpanda release, chart or operator path, storage class, and cloud integration still need to be recorded for the cluster under review.
2Kubernetes changes storage ownership and scheduling
Kubernetes can recreate a Pod, but recreating a Pod is not the same as recovering a Kafka broker. A stateful Redpanda Pod normally depends on a stable storage identity, a volume attachment path, listener configuration, and enough time to validate its log before serving traffic. If any one of those steps is delayed, the scheduler's fast response does not produce a fast Kafka recovery.
The storage class is therefore part of the failure-domain decision. A PVC may bind to a zonal volume, a regional volume, or another provider-specific implementation. The binding rules can constrain where a Pod is allowed to run, and a node failure can turn a scheduling decision into a storage-attachment decision. Kubernetes documents the lifecycle and access behavior of Persistent Volumes and StorageClasses. Read those rules together with the Redpanda storage requirements for the chosen release.
This is where a deployment can look highly automated while retaining a manual recovery step. The Pod is rescheduled, but the replacement cannot mount the volume in the selected zone. Or the volume mounts, but the network identity used by a client or peer is different from the one in the original topology. The incident is then a coordination problem across scheduler, storage, and networking, not a Pod restart problem.
Kubernetes gives operators useful controls for this work, including StatefulSet identity and Pod Disruption Budgets (PDBs). A PDB can limit voluntary disruption, but it does not make an involuntary node or storage failure safe. A StatefulSet can preserve naming and ordinal identity, but it does not make a failed storage path healthy. Treat each control as one part of a recovery contract, then test the contract under the failures that matter.
The Redpanda persistent storage guidance is a useful reference for the Kubernetes side of that contract. It should not be read as proof that every cloud storage class or topology has the same attach, expansion, or recovery behavior. Those details belong in the environment manifest and the failure drill.
3VM deployment makes the host and disk path explicit
A VM deployment puts more of the failure path in the operator's hands. The team chooses the VM image, the attached block device or local disk, the placement boundary, the startup mechanism, and the way a replacement receives broker identity and configuration. This directness can make an incident easier to reason about because the host and storage relationships are visible. It can also make the operating burden larger when those relationships are maintained by scripts or runbooks rather than a controller.
The important comparison is not “Kubernetes is automated” versus “VMs are manual.” A VM fleet may have mature instance replacement and configuration automation, while a Kubernetes cluster may use a storage class whose failure behavior is poorly understood by the Kafka team. Compare the actual control loops:
| Failure or change | Kubernetes questions | VM questions |
|---|---|---|
| Node or host loss | Can a replacement Pod schedule in a location that can attach its PV? Who observes attach and readiness failures? | Is the VM recreated with the same disk, identity, and listener placement? Who validates the broker before traffic returns? |
| Storage interruption | Does the CSI (Container Storage Interface) path report the failure clearly? What remains on the volume after detach? | Does the block or local disk remain usable, and what is the documented replacement path? |
| Network change | Will service, advertised listeners, DNS, and firewall rules converge together? | Will the replacement's addresses, routes, security groups, and DNS records converge together? |
| Capacity change | What moves when an additional Pod or node is added, and what stays on existing volumes? | What data movement and rebalance work follows an additional VM or a larger disk? |
The table exposes an ownership detail that a platform diagram often hides. Kubernetes has a control loop that can keep trying, but someone still has to decide when a retry has become a failed recovery. VM automation can make the same decision through a health check, but someone still has to verify that the check represents Kafka readiness rather than process liveness.
Storage layout also affects the blast radius of a mistake. A local disk can provide a fast path and a clear host boundary, but it may turn host loss into replica recovery or reassignment work. A network block device can survive a host replacement, but its own availability and attachment semantics become part of the broker's recovery objective. A PVC-backed Pod and a VM with an attached volume are both storage designs, not opposite categories of reliability.
4Networking is a failure domain in both models
Kafka connectivity has at least two paths: the client path to the advertised listener and the broker path used for replication or internal coordination. Kubernetes adds Service, DNS, kube-proxy or eBPF datapath behavior, ingress or load-balancing choices, and network policies to the first path. VM deployments expose load balancers, routes, security groups, and host interfaces more directly. In both cases, a broker can be alive while a client cannot reach the address it was given.
For a Redpanda Kubernetes review, write down which address a Producer receives, which address a Consumer receives, and which address the broker advertises to peer traffic. Then test a node drain, a Pod replacement, and a client reconnect from each relevant network location. Do the same for the VM deployment after a host replacement or IP change. A green process probe is not a green Kafka listener.
Network placement also changes the data movement created by recovery. If a replacement broker is placed in another zone, replica catch-up and Consumer fetches may use a different path from the steady state. A cross-zone route may be acceptable for the recovery objective, but the cost and latency belong in the design record. Avoid a generic “multi-zone” label. Name the zones, the storage placement, the client placement, and the route used by the recovery test.
The Redpanda rack-awareness documentation is relevant when the Kubernetes scheduler and storage layout are being used to express physical separation. The equivalent VM review should document the placement mechanism used by that environment. Rack or zone labels do not create isolation by themselves. The failure drill must prove that replicas and storage are actually separated in the way the team expects.
5Upgrade plans are recovery plans in disguise
An upgrade changes more than the broker binary. It changes the order in which processes stop, the time they are absent, the data they must load before serving, and the network and storage work created by the transition. The safest deployment model is the one whose upgrade runbook states those steps and gives the operator a way to stop before the next disruption.
For Kubernetes, test a controlled node drain, a voluntary Pod eviction, a storage detach and reattach, and a failed readiness transition. Check the PDB, Pod termination grace period, listener convergence, and broker membership at every step. For VMs, test the automation that replaces one host, the time required to restore its disk and identity, and the operator action when the health check is green but Kafka metadata is still catching up.
Use Kafka-facing evidence during every drill. Record the number of unavailable Partitions, leader changes, Consumer lag, Producer error rates, and the time from the fault to a stable cluster. These measurements are more useful than a deployment controller's “rollout complete” event because they describe the client-visible contract.
Redpanda's production deployment documentation and scale guidance can help identify release-specific procedures. They do not replace a test against the actual storage class, node pool, VM family, and listener topology. Mark every setting that comes from a vendor document, and mark every result that comes from your own drill.
6Choose the model by the work you are willing to own
The deployment decision becomes clearer when the team writes an ownership matrix before choosing a substrate. Kubernetes may fit a platform team that already operates storage classes, node pools, DNS, and policy controllers. VMs may fit a team that needs direct disk and host control, has a mature image pipeline, or has an existing replacement runbook. The deciding factor is the evidence and operational authority available for the Kafka service, not a general preference for containers.
Score each candidate against the same acceptance questions:
- State recovery: Can the replacement broker attach or reconstruct the correct Partition state within the RTO (Recovery Time Objective)?
- Failure isolation: Do node, VM, storage, and zone boundaries match the replication and durability objective?
- Client continuity: Do advertised listeners and authentication paths remain correct through replacement?
- Capacity movement: Does adding or removing compute create bounded data movement, and who approves it?
- Upgrade control: Can the team pause, observe, and roll back a change before Kafka availability degrades?
- Evidence: Are logs, metrics, and event timelines available to the team that owns the incident?
Keep the answers dated and tied to a named Redpanda version and deployment path. Redpanda's current documentation may change its recommended Kubernetes workflow, chart or operator packaging, and cloud-specific support. A review that does not record those inputs will eventually compare a remembered platform to a different one.
7When the storage boundary is the real decision
The Kubernetes-versus-VM debate can hide a deeper constraint: durable Kafka data is bound to the lifecycle of each broker. If every broker owns a local log, replacing or resizing a broker also creates a data placement and recovery problem. Kubernetes can automate the Pod lifecycle around that constraint, and VM automation can automate the host lifecycle around it, but neither substrate removes the storage coupling.
That is the point at which a different architecture becomes worth evaluating. AutoMQ is a Kafka-compatible cloud-native streaming platform with a Shared Storage architecture. It separates broker compute from durable stream storage through S3Stream, a WAL (Write-Ahead Log) layer, and S3 storage. The architecture overview explains the layers; the useful comparison is whether your recovery and capacity tests move less retained history with each broker change.
The storage boundary still needs a deployment-specific review. AutoMQ Open Source uses S3 WAL, as described in the WAL storage documentation, while AutoMQ BYOC (Bring Your Own Cloud) and AutoMQ Software use the storage options supported by the selected deployment. In an AutoMQ BYOC environment, the control plane and data plane run in the customer's cloud account and VPC, as described in the BYOC environment overview, so the cloud storage, network paths, and failure boundaries remain part of the customer's test. The architecture changes which layer owns durable data; it does not remove the need to test storage availability, listener reachability, cache behavior, and recovery evidence.
The related Kafka compute and storage separation comparison and Kubernetes Kafka stability checklist apply the same boundary questions to Kafka operations. They are context for the review, not proof that a particular Redpanda or AutoMQ deployment will meet your RTO.
8FAQ
8.1Is Redpanda on Kubernetes more resilient than Redpanda on VMs?
Neither substrate has a universal resilience result. Kubernetes provides scheduling and storage abstractions that can reduce manual work when the team understands their failure behavior. VM deployment can provide a more direct host and disk path when the replacement automation is well tested. Compare the measured recovery path, client continuity, and ownership matrix for the exact topology.
8.2Does a StatefulSet guarantee that a Redpanda broker returns to the same state?
No. StatefulSet identity can preserve a stable name and ordinal, but storage attachment, broker readiness, network listeners, and log recovery still need to succeed. Treat the controller's identity guarantee as one input to the Kafka recovery test.
8.3Should I use local disks or network volumes for Redpanda Kubernetes?
That is a storage and failure-domain decision. Local disks may have a direct performance path but can make host loss more dependent on replica recovery. Network volumes can change host replacement behavior but introduce attachment, network, and provider availability assumptions. Evaluate both against retention, replication, recovery, and cost requirements.
8.4When should I evaluate a shared-storage Kafka architecture?
Start when broker replacement, retention growth, or capacity changes repeatedly create data movement and recovery work. Write down the current broker-local storage boundary, then run the same client and failure tests against a shared-storage candidate. Keep the selected WAL type, object-storage boundary, network placement, and recovery evidence in the result.
The next time someone asks whether Redpanda should run on Kubernetes or a VM, bring the failure-domain map to the discussion. Name the storage owner, listener owner, scheduler or replacement path, and the evidence that proves a broker has recovered. If the map shows that broker-local storage is the recurring constraint, start an AutoMQ evaluation with the same Redpanda workload, failure drills, and RTO requirements.
