Deploy an LLM Inference Server on Kubernetes

Updated 8 November 2026 · Standard formula, shown below

Deploying an LLM inference server on Kubernetes gives you a repeatable way to serve model requests, scale replicas, and manage availability alongside other workloads. This guide uses a GPU-backed Kubernetes cluster and a vLLM container with an OpenAI-compatible API; you will need a model access token if the model repository requires one.

What You’ll Need

A Kubernetes cluster with kubectl configured and permission to create workloads, Secrets, and Services.
A Linux worker node with a compatible NVIDIA GPU, drivers, and Kubernetes device plugin installed. The node must advertise an allocatable resource such as nvidia.com/gpu.
A container runtime that can pull the selected inference image, plus network access to the model repository or a preloaded model.
A model identifier that fits the GPU’s available memory. Check its license and access requirements before deployment.
Basic familiarity with YAML, namespaces, and Kubernetes logs.

Step-by-Step

Check cluster and GPU readiness. Confirm the active cluster and available nodes:
```
kubectl config current-context
kubectl get nodes
kubectl describe nodes
```
Look under node capacity and allocatable resources for the GPU resource name. If no GPU is listed, install and configure the GPU vendor’s device plugin and node drivers before deploying. Do not assume that specifying a GPU in a pod installs drivers or makes a GPU available.
Create a namespace and store any model token safely. A namespace keeps this deployment separate from other workloads:
```
kubectl create namespace llm
```
If the model requires authentication, create a Kubernetes Secret from a token stored in an environment variable. Avoid putting the token directly in a manifest committed to source control:
```
kubectl -n llm create secret generic model-access \
  --from-literal=HF_TOKEN="$HF_TOKEN"
```
Restrict Secret access using your cluster’s RBAC, and use your normal secret-management process for production.
Choose an image and model. Select a maintained inference-server image compatible with your GPU, CUDA environment, and Kubernetes node architecture. The example below uses a vLLM image and its OpenAI-compatible server entry point. Pin a tested image version or digest rather than using latest. Replace the example model with a model you can access and that fits in the GPU memory, including runtime overhead. Model names and server arguments can change; verify them against the selected image’s documentation.

Save a Deployment manifest. Create llm-deployment.yaml, replacing the image tag and model identifier as appropriate. This example requests one GPU and exposes the API only inside the cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference
  namespace: llm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: inference
  template:
    metadata:
      labels:
        app: inference
    spec:
      containers:
        - name: server
          image: vllm/vllm-openai:PINNED_VERSION
          args: ["--model", "MODEL_ID"]
          ports:
            - name: http
              containerPort: 8000
          env:
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef:
                  name: model-access
                  key: HF_TOKEN
          resources:
            requests:
              nvidia.com/gpu: "1"
            limits:
              nvidia.com/gpu: "1"
          readinessProbe:
            httpGet:
              path: /health
              port: http
            initialDelaySeconds: 30
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /health
              port: http
            initialDelaySeconds: 60
            periodSeconds: 20

For a public model without a token, remove the env entry. Confirm that the image supports the probe path; adjust it to the server’s documented health endpoint if needed. Large models can take several minutes to download and load, so set startup or probe delays accordingly rather than restarting the container during normal initialization.

Apply the Deployment and watch startup.
```
kubectl apply -f llm-deployment.yaml
kubectl -n llm rollout status deployment/inference --timeout=10m
kubectl -n llm get pods -o wide
```
If the rollout stalls, inspect pod events and logs before changing settings. The initial model download can consume substantial disk space and network bandwidth.
Create an internal Service. Save this as llm-service.yaml and apply it:
```
apiVersion: v1
kind: Service
metadata:
  name: inference
  namespace: llm
spec:
  selector:
    app: inference
  ports:
    - name: http
      port: 8000
      targetPort: http
  type: ClusterIP
```
A ClusterIP Service makes the API reachable from within the cluster, not directly from the public internet. For external access, add an approved ingress or gateway, TLS, authentication, and suitable traffic controls instead of exposing an unauthenticated model endpoint.
Test the API from inside the cluster. Start a temporary curl pod and make a small request. Use the model identifier configured in the Deployment:
```
kubectl -n llm run api-check --rm -i --restart=Never \
  --image=curlimages/curl:PINNED_VERSION -- \
  curl -sS http://inference:8000/v1/models
```
Then test a chat completion using the API’s documented request format. If authentication is enabled at the server or gateway, include the required credential securely. Avoid sending sensitive prompts to an endpoint until its access and logging policies are understood.
Prepare for real workloads. Add monitoring for request latency, errors, GPU memory and utilization, and pod restarts. Set CPU, memory, ephemeral-storage, and GPU requests based on observed behavior. Add persistent or cached model storage if repeated downloads are too costly, and configure node placement if only particular nodes have compatible GPUs. Do not simply increase replicas: each replica may need its own GPU and model memory.

Common Mistakes

GPU resource missing: A GPU on the machine is not enough; the drivers and device plugin must expose it to Kubernetes.
Model too large: Model weights, context length, batch size, and cache use all affect GPU memory. A pod can be scheduled but still fail while loading.
Wrong image or arguments: Server entry points, model flags, and supported GPU architectures vary between images. Follow documentation for the exact pinned image.
Probe starts too early: Download and initialization take time. A liveness probe that fails during startup can repeatedly kill a healthy but not-yet-ready server.
Accidental public exposure: A Service alone is not a complete access-control plan. Use authentication and encrypted ingress when serving outside a trusted cluster network.

Tips

Begin with one replica and a small test model; measure before increasing context length, concurrency, or replica count.
Pin image versions and record the model revision so deployments can be reproduced and rolled back.
Use Kubernetes resource requests and limits deliberately, and check GPU memory as well as ordinary pod memory.
Consider a model cache or persistent volume for large downloads, while checking the server’s cache paths and storage requirements.
Keep tokens out of manifests and logs. Rotate credentials and limit which service accounts can read Secrets.

Troubleshooting

Pod remains Pending: Run kubectl -n llm describe pod POD_NAME. Check events for unavailable GPUs, insufficient CPU or memory, node selectors, or storage constraints. Confirm that a node has the requested GPU resource.
ImagePullBackOff: Check the image name and pinned tag, registry connectivity, and any required image-pull Secret. Use kubectl -n llm describe pod POD_NAME to see the pull error.
Container exits during model loading: Read kubectl -n llm logs deployment/inference. Look for access denial, an incorrect model identifier, unsupported model files, disk exhaustion, or GPU out-of-memory errors. Verify token access and reduce model size or memory-demanding settings.
Service returns connection errors: Check that the pod is Ready, the Service selector matches its labels, and its target port matches the server port. Inspect endpoints with kubectl -n llm get endpoints inference.
Requests are slow or time out: Check server logs, GPU utilization, model loading status, and client or gateway timeouts. Test a short prompt first; review context length and concurrent request settings before scaling.

Frequently Asked Questions

Can I run an inference server without a GPU?
Yes, if the chosen server and model support CPU inference, but performance may be much lower. Remove the GPU resource request and choose suitable CPU and memory resources.

How many GPUs should I request?
Request the number supported by your server configuration and required by the model. Start with the model’s documented memory and parallelism guidance; Kubernetes schedules GPUs as discrete resources.

Does a ClusterIP Service make the API public?
No. It is reachable from inside the cluster. Use a secured ingress or gateway for external clients.

Why does the pod restart while loading the model?
It may be failing a probe too soon, running out of memory or disk, or encountering a download or access error. Check events and logs, then adjust the underlying cause and startup timing.

Can I scale this Deployment to several replicas?
Yes, if enough compatible GPU capacity is available and the server can handle the traffic pattern. Each replica generally loads its own model, so account for GPU memory and cost before scaling.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.