ThisCalculator

Everyday

Protect VMs and Containers in Kubernetes

Updated · Standard formula, shown below

This guide shows how to secure a Kubernetes cluster that runs both virtual machines (VMs) and containers together, using policy, network controls, and runtime protection. You’ll set up admission policies, isolate workloads, restrict privileges, and monitor activity so that a compromise in one workload type doesn’t spread to the other.

What You’ll Need

  • A Kubernetes cluster (v1.26 or later for most admission policies) with admin access.
  • kubectl configured to your cluster.
  • If you plan to run VMs, a VM operator like KubeVirt installed and working.
  • Helm (optional, for installing policy engines like Kyverno or OPA Gatekeeper).
  • Basic understanding of Kubernetes RBAC, NetworkPolicies, and Pod Security Standards.

Step-by-Step

  1. Separate workloads by namespace. Create distinct namespaces for VM workloads and container workloads, e.g., vms and apps. This gives you a clean boundary for applying different policies and makes it easier to audit what runs where.
  2. Enforce Pod Security Standards (PSS) on container namespaces. Use the built-in admission controller (since Kubernetes v1.25) to enforce the restricted profile on the apps namespace. Create a label: kubectl label ns apps pod-security.kubernetes.io/enforce=restricted. This blocks privileged containers, hostPath volumes, and other risky settings.
  3. Apply a separate security policy for VM workloads. VMs need more privileges (like device access) than typical containers. If using KubeVirt, it requires the privileged PSS profile. Set that on the vms namespace: kubectl label ns vms pod-security.kubernetes.io/enforce=privileged. Then, use a policy engine (Kyverno or OPA Gatekeeper) to add extra restrictions specifically for VM objects, such as disallowing host networking or limiting which storage classes can be used.
  4. Install a policy engine (if not already present). For fine-grained control, install Kyverno via Helm: helm install kyverno kyverno/kyverno -n kyverno --create-namespace. Then create a ClusterPolicy that blocks VM objects (e.g., VirtualMachineInstances) from using privileged service accounts or mounting host paths unless explicitly allowed.
  5. Isolate network traffic. Create NetworkPolicies in both namespaces. For the apps namespace, deny all ingress/egress by default, then allow only needed traffic. For the vms namespace, allow only required ports (e.g., SSH) and block all other ingress. Use a default-deny policy first: kubectl create -f - <<EOF
    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
    name: default-deny
    spec:
    podSelector: {}
    policyTypes: ["Ingress", "Egress"]
    EOF     Then add allow rules as needed.
  6. Restrict egress to prevent data exfiltration. For both namespaces, create a NetworkPolicy that allows egress only to DNS (port 53) and to specific internal services. For VMs, you may need to allow egress to a proxy or specific API endpoints. This limits the blast radius if a workload is compromised.
  7. Use RBAC to limit who can deploy VMs vs containers. Create separate Roles and RoleBindings for each namespace. For example, a role that can only create VirtualMachine objects in vms, and another that can only create Deployments in apps. This prevents a user from accidentally or maliciously deploying a privileged VM in a restricted namespace.
  8. Disable privileged containers and root for containers. In the apps namespace, the restricted PSS already does this. For VM workloads, ensure the VM spec does not run as root inside the guest OS; set runAsNonRoot: true in the VM’s securityContext if the guest OS supports it. Also, avoid giving VMs any host device access unless absolutely necessary.
  9. Enable runtime security monitoring. Install a runtime security tool like Falco. Falco monitors syscalls and can detect suspicious behavior in both containers and VMs (if the VM operator exposes the host kernel). Create rules to alert on shell access inside a container, or on VMs attempting to mount unexpected host paths.
  10. Regularly scan images and VM templates. Use a container image scanner (e.g., Trivy) on all container images. For VMs, scan the VM disk images for known vulnerabilities before deploying. Integrate scanning into your CI/CD pipeline so that only clean images are allowed.
  11. Audit and log everything. Enable Kubernetes audit logging and send logs to a central system. Create an audit policy that logs all requests to the vms and apps namespaces. This helps you detect unauthorized attempts and provides a trail for incident response.
  12. Test your policies. Try to deploy a privileged container in the apps namespace and confirm it is rejected. Try to create a VM with a hostPath volume and confirm the policy engine blocks it. Regular testing ensures your controls work.

Common Mistakes

  • Applying the same PSS profile to both VMs and containers. VMs often need more privileges, so using restricted on VM namespaces will break them. Conversely, using privileged on container namespaces defeats security.
  • Forgetting to set default-deny NetworkPolicies. If you only add allow rules without a default-deny, all other traffic is allowed, making the policy ineffective.
  • Not updating RBAC when adding new VM operators. The operator may need extra permissions; if you lock down too much, the operator fails, and you might be tempted to grant broad permissions that weaken security.
  • Ignoring egress rules. Many attacks involve data exfiltration; without egress restrictions, a compromised workload can send data anywhere.
  • Not testing policies after changes. A small change to a policy can break legitimate workloads, so always test in a staging environment.

Tips

  • Use namespaces as the primary isolation boundary. Combine that with NetworkPolicies to enforce micro-segmentation.
  • For VM workloads, use the KubeVirt specific security features like virtualMachineInstanceSpec.domain.devices.disks to control disk access, and avoid using hostDevices unless necessary.
  • Leverage Kubernetes PodDisruptionBudgets to ensure availability during updates, but that’s not security—just don’t forget it.
  • Implement a policy-as-code repository (GitOps) so that changes to security policies are reviewed and versioned.
  • Use Open Policy Agent (OPA) or Kyverno to enforce that all images come from an approved registry.
  • Regularly update your VM operator and Kubernetes to get the latest security patches.

Troubleshooting

  • Workloads cannot start after applying NetworkPolicies. Check if DNS is blocked. Add an egress rule to allow UDP/TCP port 53 to kube-dns. Also, check if the NetworkPolicy selector matches the pod labels correctly.
  • VM fails to start due to PSS. If the VM namespace is set to restricted, change it to privileged and rely on a policy engine for finer control. Also, ensure the VM operator’s namespace is not restricted.
  • Policy engine (Kyverno) blocks everything. Check the policy logs: kubectl logs -n kyverno deployment/kyverno. Often, the policy is too broad. Use kubectl get policyreport to see violations.
  • Containers can’t reach the internet. Your egress NetworkPolicy might be too strict. Add a rule to allow egress to your proxy or to specific external IPs. If you don’t need internet, that’s fine, but document it.
  • RBAC denies access to VM operator. Check the operator’s ClusterRole and RoleBindings. You may need to add permissions for the VM operator’s service account in the vms namespace.
  • Falco produces too many alerts. Tune the rules to reduce noise. Start with the default rules, then disable or adjust ones that are irrelevant to your environment.

Frequently Asked Questions

Can I use the same security policy for VMs and containers?

No. VMs require more privileges (e.g., device access, kernel modules) than typical containers. Use a privileged profile for VMs but compensate with a policy engine to restrict specific risky features.

Do NetworkPolicies apply to VM workloads?

Yes, if the VM operator uses standard Kubernetes networking (e.g., bridge or masquerade). NetworkPolicies apply to pods, and VMs in KubeVirt run as pods, so they are covered.

What is the best way to isolate VMs from containers?

Use separate namespaces, then enforce NetworkPolicies that deny all cross-namespace traffic by default. Only allow specific communication through explicit policies.

Is it safe to run privileged containers if I use a policy engine?

No. A policy engine can reduce risk, but privileged containers still have a larger attack surface. Avoid them unless absolutely necessary, and always run them in a dedicated namespace with strict egress rules.

How do I monitor VM activity if I can’t see inside the guest OS?

Use runtime security tools like Falco that monitor host syscalls. For deeper visibility, install an agent inside the VM (if you control the image) that sends logs to a central SIEM.

What should I do if a VM is compromised?

Immediately isolate the VM by applying a NetworkPolicy that denies all traffic to/from it. Then snapshot the VM for analysis, shut it down, and investigate the incident. Restore from a known-good backup after patching the vulnerability.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.