Why Kubernetes Cluster Slows Down With Finished Jobs Piled Up

A Kubernetes cluster can slow down even when all pods are healthy and finished jobs are merely sitting in the cluster because completed pods and their associated objects (Jobs, ReplicaSets, ConfigMaps, Secrets) consume API server memory, etcd storage, and controller-manager processing time. The pile-up of finished jobs is not a cosmetic issue; it directly degrades API responsiveness, increases etcd latency, and can exhaust control-plane resources, which in turn slows down scheduling, health checks, and even kubectl commands.

What Counts as a “Finished Job” in Kubernetes

In Kubernetes, a Job creates one or more pods and ensures that a specified number of them successfully terminate. When a Job completes, its pods enter the Succeeded or Failed phase. By default, Kubernetes does not automatically delete completed Jobs or their pods. The Job controller retains them so you can inspect logs, results, and error messages. Similarly, a CronJob creates Jobs on a schedule, and each run leaves behind a finished Job unless a history limit is set.

These finished pods and Jobs are stored in etcd, the cluster’s key-value store. Every object—whether active or not—occupies space in etcd and is served by the API server. The more completed objects you accumulate, the larger the etcd database grows, and the more work the API server does when listing, watching, or garbage-collecting them.

How Finished Jobs Affect Cluster Performance

The slowdown is not caused by the pods themselves consuming CPU or memory (they are terminated), but by the control plane’s overhead. Here are the main mechanisms:

• etcd size and latency: etcd is a distributed key-value store that keeps the entire cluster state. A large number of objects increases the database size, which can exceed the default compaction and defragmentation thresholds. Large etcd databases lead to slower reads and writes, and can cause the etcd server to run out of memory or disk space.
• API server memory and CPU: Every object is cached in the API server’s memory (in the watch cache). Thousands of finished pods and Jobs inflate this cache, increasing memory usage and making list/watch operations slower. When you run kubectl get pods, the API server must filter through all those completed objects.
• Controller manager workload: The Job controller continuously reconciles Jobs. Even for finished Jobs, it performs periodic status updates and garbage collection checks. A large backlog of completed Jobs increases the number of reconcile loops, consuming CPU and potentially delaying reconciliation of active workloads.
• Garbage collection churn: Kubernetes uses garbage collection to delete objects whose owners are gone. When many Jobs finish at once, the garbage collector may be overwhelmed trying to delete their dependent pods, leading to a backlog of deletion requests that further load the API server.

Why “Healthy Pods” Don’t Prevent the Slowdown

Pod health checks (liveness and readiness probes) only tell you whether a pod is running correctly. They do not reflect the state of the control plane. A cluster can have all pods in Running and Ready, yet the API server and etcd may be struggling under the weight of thousands of completed objects. The slowdown manifests as:

• Increased latency for kubectl commands
• Delayed pod scheduling (the scheduler waits for API responses)
• Slow or failed health checks on the kubelet (which communicates with the API server)
• Expired TLS certificates or leader election timeouts due to slow API responses

How to Detect and Diagnose the Problem

Before fixing, you need to confirm that finished Jobs are indeed the culprit. Look for these signs:

1. Run kubectl get jobs --all-namespaces and count how many are in Completed or Failed state. A large number (hundreds or thousands) is a red flag.
2. Check etcd database size: etcdctl endpoint status --write-out=table (if you have access) or use the etcd metrics endpoint. A size above a few hundred MB is unusual for a small cluster.
3. Monitor API server metrics: look at apiserver_request_duration_seconds and etcd_request_duration_seconds in Prometheus. High latency for LIST requests often correlates with many objects.
4. Check the Job controller’s metrics: job_controller_jobs_finished or similar (if exposed). A high count of finished but not deleted Jobs is a direct indicator.

How to Clean Up Finished Jobs Safely

The standard solution is to delete completed Jobs and their pods. You can do this manually or automatically.

• Manual deletion: kubectl delete job --all --field-selector status.successful=1 (or use status.phase=Succeeded for pods). This removes the Job and its pods.
• Automatic cleanup with TTL: Kubernetes supports a ttlSecondsAfterFinished field on Jobs. Set it to, for example, 3600 to delete the Job one hour after it completes. This is the cleanest method for recurring workloads.
• CronJob history limits: For CronJobs, set successfulJobsHistoryLimit and failedJobsHistoryLimit to small numbers (e.g., 3) to keep only the most recent runs.
• Use a cleanup CronJob: If you have many Jobs from different sources, create a CronJob that periodically deletes Jobs older than a certain age using kubectl or the Kubernetes API.

Be careful when deleting Jobs: if you delete a Job, its associated pods are also deleted, and you lose the logs. If you need logs, export them before deletion.

Preventing the Problem in the Long Term

Cleaning up once is not enough. To avoid recurrence, adopt these practices:

• Set ttlSecondsAfterFinished on every Job you create, even in development.
• For CronJobs, always set history limits. The defaults are 3 successful and 1 failed, but many users override them to 0 or 1.
• Use namespaces to isolate workloads and apply ResourceQuotas to limit the number of objects per namespace.
• Enable etcd compaction and defragmentation. Most managed Kubernetes providers handle this, but if you run your own, configure --auto-compaction-retention (e.g., 1h) and run defragmentation periodically.
• Monitor the number of objects in the cluster and set alerts for when the count of completed Jobs exceeds a threshold.

Frequently Asked Questions

Can finished Jobs cause a node to run out of disk space?

Yes, indirectly. Finished pods are removed from nodes, but their container images and logs may remain. More importantly, the etcd database on the control plane can grow large enough to fill the disk, which can cause the entire cluster to become read-only. Regular cleanup of Jobs and etcd compaction helps avoid this.

Does deleting finished Jobs affect running workloads?

No, deleting a completed Job only removes the Job object and its terminated pods. It does not affect other Jobs or Deployments. However, if you delete a Job that is still running (not finished), it will terminate the pods, so only delete Jobs that are in Completed or Failed state.

What is the difference between ttlSecondsAfterFinished and history limits?

ttlSecondsAfterFinished is a field on a Job that tells the controller to delete the Job after a certain number of seconds from completion. History limits are properties of a CronJob that control how many successful and failed Jobs are kept. They serve similar purposes but at different levels: one for individual Jobs, the other for CronJob-managed Jobs.

Why does `kubectl get pods` become slow even though I only have a few running pods?

Because kubectl get pods lists all pods in the namespace or cluster, including the thousands of completed ones. The API server must read and filter all of them. Using --field-selector=status.phase=Running can speed up the command, but the underlying load remains.

Is it safe to delete all Jobs with `kubectl delete job –all`?

It is safe if you do not need the logs or results from those Jobs. However, it will delete all Jobs in the current namespace, including any that might still be running (if you didn’t filter). Always use a field selector to target only completed or failed Jobs.

Can I set a default TTL for all Jobs in a namespace?

Kubernetes does not have a built-in default TTL for Jobs. You must set ttlSecondsAfterFinished on each Job or use a mutating admission webhook to inject it automatically. Some organizations use policies or templates to enforce this.

Related

• Find Venmo Pending Split & Scheduled Payments
• Arc Raiders Grapple vs Snap Hook & Zipline
• Delete Files Off External Hard Drive Mac: Full Guide