Automating Kubernetes Cost Optimization with an AI Agent

Every Kubernetes cost optimization effort eventually hits the same wall: the fixes don't stick. This closes out our three-part series — part one mapped the six places cluster spend leaks (overprovisioned requests, idle and underpacked nodes, orphaned volumes and load balancers, missing autoscaling), and part two turned those leaks into a DIY audit built on kubectl top, kube-state-metrics, VPA in recommendation mode, and the EKS/GKE/AKS cost views.

Suppose you ran that audit and shipped a rightsizing PR. It's already going stale. The service whose requests you trimmed got a new feature and a new memory profile. Someone bumped an HPA's minReplicas during an incident and never reverted it. Two new deployments shipped this week with copy-pasted requests from a template written in 2023. Kubernetes cost optimization isn't a project you finish — it's a control loop, and a quarterly audit is a control loop with a three-month delay.

Closing that loop is what this article is about: how the CloudThinker CostOps agent keeps the part-two audit running against your clusters every day, what a first scan of a mid-market environment tends to surface, and the boundaries it operates under — starting with everything it refuses to do until you approve it.

Why point-in-time rightsizing loses

The math is simple. A cluster with 30 active deployments and a weekly release cadence sees hundreds of changes between audits. Each one can shift the requests-versus-usage gap: a dependency upgrade changes memory behavior, a traffic shift strands a node pool, a deleted namespace leaves its PersistentVolumes and cloud load balancers behind. The waste sources from part one don't stay fixed — they regenerate at deploy speed, while your audit runs at calendar speed.

Native tools don't close this gap either. VPA recommends but doesn't reconcile recommendations across your HPA settings and PodDisruptionBudgets. The cluster autoscaler scales nodes but won't tell you that your requests are inflated 3x, so it faithfully bin-packs the wrong numbers. What's missing is the layer that watches all of it, every day, and turns findings into approval-gated fixes.

How the CostOps agent works on Kubernetes

Connecting: read-only RBAC, about five minutes

The agent connects to your cluster the way your security team would insist on anyway: a dedicated ServiceAccount bound to a read-only ClusterRole — get/list/watch on workloads, nodes, and metrics, no write verbs, no secrets access. On EKS, GKE, or AKS you can scope it further with your existing admission and network policies; the agent needs API server reachability, nothing on your nodes. Cloud billing context (node pricing, load balancer and volume costs) comes from your cloud connection, also read-only.

You apply a provided manifest, paste the connection details, and the first scan starts. The whole flow takes about five minutes — the Kubernetes connection guide has the exact RBAC manifest so your security review can see precisely what is, and isn't, granted.

What it scans continuously

Once connected, the agent runs the same checks you did manually in part two — continuously, across every namespace and node pool:

1. Requests versus actual usage per container, over rolling 7- and 30-day windows — the P95-usage-to-request ratio that dominates most cluster bills
2. Node utilization and bin-packing — nodes running under 30% requested, pools where pod requests and instance shapes don't fit cleanly
3. Idle and stranded nodes — nodes kept alive by a single system pod or a PodDisruptionBudget that blocks consolidation
4. Orphaned resources — Released/Available PersistentVolumes, load balancer Services with no healthy endpoints, unattached cloud disks left behind by deleted StatefulSets
5. Autoscaling gaps — deployments with no HPA, HPAs pinned by stale minReplicas, pools where Karpenter or cluster autoscaler consolidation is off or blocked
6. Non-production schedules — dev and staging namespaces running full replica counts at 3 a.m. on Sunday

The difference from your manual audit isn't the checks — it's the cadence. A deployment that ships Tuesday with 4 CPU requested and 200m used is a finding on Wednesday, not a line in next quarter's spreadsheet. Every finding carries the evidence you'd have gathered yourself: the metrics window, the current monthly cost, the projected saving as a range.

Graduated autonomy: you decide what the agent may touch

Every remediation type has an autonomy level you set per cluster and per environment:

• Notify — the agent reports the finding. Nothing else. (The default for everything.)
• Suggest — the agent proposes the specific change: new requests/limits values, the node pool consolidation plan, the PV deletion list, with rollback notes.
• Approve — the agent prepares the change and executes only after a named human approves it.
• Autonomous — the agent executes and reports. Most teams reserve this for provably safe, reversible actions in non-prod — scaling a staging namespace to zero on schedule, deleting a Released PV that's been snapshotted — and only after weeks of watching the agent be right at the Approve level.

Every action at every level lands in an audit trail: what was found, what was proposed, who approved, what changed, when. Your part-two audit skills transfer directly — you're reviewing the same evidence, minus the hours spent collecting it.

What the first scan typically returns

Numbers below are illustrative — a composite of what a first scan tends to surface across two or three mid-market clusters spending around $35K/month on compute. Your clusters will differ.

That's roughly $16,000/month identified on day one — and the shape matches what part one predicted: requests-versus-usage and non-prod scheduling dominate, with a long tail of orphaned resources. The agent's value isn't exotic findings; it's that this table refreshes daily and each row arrives with a proposed, approval-gated fix instead of a ticket you still have to write.

Prompts to try in your first session

CostOps is conversational — you ask Alex, CloudThinker's cost engineering agent, in plain language:

"@alex show me the 10 deployments with the biggest gap between CPU requests and actual P95 usage over the last 30 days"

"@alex which nodes could be consolidated right now, and what's blocking the autoscaler from doing it?"

"@alex what would scaling the staging namespace to zero outside business hours save us?"

Each answer cites the underlying data — the same metrics-server and kube-state-metrics evidence you used in part two — so you can verify rather than trust.

What the agent does not do

Worth stating plainly, because "AI agent with cluster access" should make you ask:

• It is read-only by default. The RBAC grant has no write verbs. Remediation requires you to explicitly grant scoped write access and raise the autonomy level above Notify for that action class.
• It does not edit requests, drain nodes, or delete volumes without approval unless you have deliberately set that specific action to Autonomous for that specific cluster.
• It does not bypass your Kubernetes guardrails. Changes go through the API server like any other client, subject to your admission controllers, PDBs, and quotas.
• It does not make architectural calls. It will tell you a PodDisruptionBudget is blocking $2,400/month of consolidation; whether that PDB is load-bearing is an engineering decision, and it stays yours.

If you did the part-two audit manually, the right mental model is: the agent is that audit, running daily, with a remediation queue attached — not an autopilot you hand cluster-admin on day one.

From quarterly audit to control loop

You know where Kubernetes spend leaks. You know how to audit it with native tools. The remaining gap is cadence — and cadence is an automation problem, not an effort problem. Teams that move from quarterly rightsizing PRs to continuous, approval-gated optimization typically cut cluster compute spend by 30–50% over the following months, with the biggest wins from requests rightsizing and non-prod scheduling — the two rows that were stale in your last PR before it merged.

Try CloudThinker free — 100 premium credits, no card required — and follow the Kubernetes connection guide to see your own cluster's first findings table within the hour.