Sign inBook a Demo

How To

A DIY Kubernetes Cost Optimization Audit with Native Tools

A hands-on Kubernetes cost optimization audit using only native and free tooling — no third-party platforms. Part two of our Kubernetes cost optimization series walks through five tools with exact commands: kubectl top and metrics-server to expose the requests-vs-usage gap, kube-state-metrics PromQL for a 7-day per-namespace view, VPA in recommendation mode for safe right-size targets, cluster-autoscaler logs to find idle nodes and scale-down blockers, and the EKS, GKE, and AKS console cost views that convert cores into dollars. Closes with the honest limits of DIY: point-in-time findings, recurring engineer-hours, and manual remediation.

ST
·
kubernetescostoptimizationfinopskubectlautoscalingdevops
Cover Image for A DIY Kubernetes Cost Optimization Audit with Native Tools

A DIY Kubernetes Cost Optimization Audit with Native Tools

Block out one afternoon, pick your most expensive cluster, and you can walk out with a findings document that names every over-requested deployment, every half-empty node, and every workload blocking scale-down — using nothing but tooling that ships with Kubernetes or comes free from your cloud provider. That's the whole promise of this article: a hands-on Kubernetes cost optimization audit with kubectl and metrics-server, kube-state-metrics, the Vertical Pod Autoscaler in recommendation mode, cluster-autoscaler logs, and the cost views built into the EKS, GKE, and AKS consoles.

In part one of this series we mapped where cluster spend leaks: overprovisioned requests and limits, idle and underpacked nodes, orphaned volumes and load balancers, forgotten namespaces, and missing autoscaling. This is part two — the audit itself, grouped by tool, because each tool answers a different question. Budget 3–4 hours for a single production cluster.

Tool 1: kubectl top + metrics-server — the requests-vs-usage gap

The single biggest Kubernetes waste source is the gap between what pods request and what they use. Your cluster autoscaler and scheduler bill you for requests, not usage — so this gap is priced into every node you run.

First, confirm metrics-server is installed (EKS requires you to install it; GKE and AKS ship it by default):

kubectl get deployment metrics-server -n kube-system

Then look at the node picture:

kubectl top nodes

How to read it. kubectl top nodes shows actual CPU and memory usage per node. Now compare it with what's reserved:

kubectl describe nodes | grep -A 8 "Allocated resources"

For each node you get two very different numbers: allocated (sum of pod requests, what the scheduler thinks is used) and the live usage from top. A node showing 78% CPU requested but 11% CPU used is the classic signature — the node can't be scaled away, yet it's mostly air. On a first audit of a mid-size cluster, cluster-wide CPU utilization of actual-vs-requested under 30% is common; well-tuned clusters run 50–70%.

Find the worst offenders at the pod level:

kubectl top pods -A --sort-by=cpu
kubectl top pods -A --sort-by=memory

And dump every workload's Kubernetes resource requests for comparison:

kubectl get pods -A -o custom-columns=\
'NS:.metadata.namespace,POD:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory'

Cross-reference the two lists. A pod requesting 2 CPU while top shows 40m is a 50x overask. Also note every pod whose request columns come back empty — unset requests are the opposite failure mode (they schedule greedily and evict unpredictably), and they'll pollute every number in the rest of this audit.

One caveat before you file tickets: kubectl top is a point-in-time sample. A batch job that spikes nightly will look idle at 2 p.m. That's what the next two tools are for.

Tool 2: kube-state-metrics + Prometheus — requests at scale, over time

kube-state-metrics exports the declared state of the cluster — including every container's requests and limits — as Prometheus metrics. If you run any Prometheus-compatible stack (kube-prometheus-stack installs both), you can turn the spot check above into a 7-day view.

Requested CPU per namespace:

sum(kube_pod_container_resource_requests{resource="cpu"}) by (namespace)

Actually used CPU per namespace:

sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (namespace)

And the number that drives the whole audit — utilization of requests, per namespace:

sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (namespace)
/
sum(kube_pod_container_resource_requests{resource="cpu"}) by (namespace)

Repeat both with resource="memory" and container_memory_working_set_bytes for the memory side.

How to read it. Run the ratio query over the last 7 days (in Grafana, look at the max, not just the average). A namespace that never exceeds 25% of its requested CPU across a full week — including its busiest hour — is carrying at least 2–3x more request than it needs. Rank namespaces by absolute requested cores, not by ratio: a 40-core namespace at 30% utilization is worth more than a 2-core namespace at 5%. That ranking is your remediation priority list, and it usually surfaces the same three or four services that dominate the bill.

Tool 3: VPA in recommendation mode — the right-size numbers

You now know which workloads are over-requested. The Vertical Pod Autoscaler tells you what the requests should be — and with updateMode: "Off" it only recommends, never touches a running pod. GKE has VPA built in (enable it per cluster); AKS offers it as a managed add-on; on EKS you install the upstream version:

git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler
./hack/vpa-up.sh

Create one VPA object per suspect workload from your Tool 2 ranking:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-api-recommender
  namespace: prod
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  updatePolicy:
    updateMode: "Off"

Wait a day or two for it to observe real traffic, then read the recommendation:

kubectl describe vpa checkout-api-recommender -n prod

How to read it. The Status.Recommendation block gives you four numbers per container. Target is the request VPA would set — that's your right-size figure. Lower Bound is the floor below which the pod likely gets throttled or OOM-killed; Upper Bound is the ceiling worth considering for spiky workloads; Uncapped Target shows what VPA wanted before any policy constraints. A deployment requesting 2000m CPU with a target of 312m is your finding, with the safe new number attached. Let VPA watch through at least one weekly traffic cycle before trusting the target for anything customer-facing, and keep limits at 1.5–2x the new request rather than removing them.

Tool 4: cluster-autoscaler logs — idle nodes and scale-down blockers

Overprovisioned requests waste money inside nodes; the autoscaler tells you about waste at the node level — nodes it knows are unneeded but cannot remove. On EKS with the self-managed cluster autoscaler, start with the status ConfigMap:

kubectl -n kube-system get configmap cluster-autoscaler-status -o yaml

Then grep the logs for scale-down decisions:

kubectl -n kube-system logs deploy/cluster-autoscaler --tail=2000 \
  | grep -iE "scale.?down|unneeded|no.?scale"

On GKE and AKS the autoscaler is managed and its pod isn't visible; read the equivalent events in Cloud Logging (GKE: filter logName for container.googleapis.com cluster-autoscaler entries, or check the ScaleDown events in the console) and in AKS via kubectl get events -A --field-selector source=cluster-autoscaler.

How to read it. Two findings live here. First, nodes marked unneeded for long stretches that never terminate — each one is a full instance of pure cluster autoscaler cost. Second, and more actionable, the reasons scale-down is blocked, which the logs state explicitly: pods without a controller, pods using local storage, pods with restrictive PodDisruptionBudgets, or the "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" annotation. A single kube-system pod with no PDB can pin an entire node group. Every blocker you fix compounds — it doesn't just remove one node today, it lets the autoscaler keep packing nodes every day after.

While you're at node level, check bin-packing directly: if kubectl describe nodes shows most nodes at 50–65% requested, your pods' request shapes don't fit your instance shape. A different node size (or a second node group for large pods) often recovers 10–20% of node spend with zero workload changes.

Tool 5: the cloud console cost views — dollars, not cores

Everything so far is in cores and GiB. Your cloud's billing console converts it to dollars per namespace or workload — which is what turns an audit into an approved ticket.

  • EKS: enable split cost allocation data for EKS under Billing and Cost Management → Cost Management Preferences. After ~24 hours, Cost Explorer can group EKS spend by cluster, namespace, and workload.
  • GKE: enable GKE cost allocation (gcloud container clusters update CLUSTER_NAME --enable-cost-allocation), then group billing reports by namespace or cluster label. The GKE console's Clusters → Cost optimization tab also charts requested-vs-used per cluster with no setup.
  • AKS: enable the AKS cost analysis add-on (az aks update -g RG -n CLUSTER --enable-cost-analysis, Standard or Premium tier), then open Cost Management → Cost analysis and pick the Kubernetes views for namespace- and asset-level breakdowns.

How to read it. Multiply each namespace's monthly dollar figure by the over-request ratio you computed in Tool 2. That product — "namespace X costs $4,200/month and uses 22% of what it requests" — is the sentence that gets a right-sizing ticket prioritized.

Where the DIY audit stops

Run this audit; it works. But know its limits before you make it your strategy:

It's a snapshot. Requests drift the day after you tune them — every deploy resets the game. The VPA targets you collected this week describe this week's traffic.

It's engineer-hours, recurring. One cluster, one afternoon. Ten clusters across three clouds, quarterly? That's real headcount, and it's the first thing dropped in a busy sprint.

The tools don't talk to each other. You manually joined top output, PromQL, VPA objects, autoscaler logs, and a billing console. Nothing correlates a VPA target with the node-level saving it unlocks.

Nothing here remediates. Every finding becomes a PR to someone else's Helm values, and unowned tickets are where the savings quietly expire.

What comes next

Every check in this guide can run continuously instead of quarterly. CloudThinker's CostOps agent connects to your clusters read-only, watches requests-vs-usage, idle nodes, and orphaned resources around the clock, and gates every fix behind your approval — that's part three of this series. Or point it at your own cluster today: try CloudThinker free — 100 premium credits, no card required.