The Kubernetes Agentic Operations Revolution: From Manual Management to Autonomous Intelligence with CloudThinker

The cluster was supposed to be self-healing. At 3 a.m. on Monday, the only thing healing anything was a very tired platform engineer named Marcus.

Marcus runs twelve Kubernetes clusters for a logistics company that processes 2 million shipment events a day: production, staging and regional deployments across three cloud providers. On paper, Kubernetes handles the complexity with autoscaling, self-healing pods and declarative config. In practice, Marcus handles it. This post follows one week of his job, and what changed once an agent started doing the investigation alongside him. (Marcus and his company are a composite, and the numbers in his week are illustrative. The failure modes are ones every platform team will recognize.)

Why "self-healing" still needs a person

Kubernetes heals exactly what it was told to heal. Everything outside that contract lands on a human, for structural reasons:

Resource limits are written once. Requests and limits get set at first deploy and rarely revisited. Workloads change shape (bigger payloads, new features, more tenants) and the limit becomes a trap. Kubernetes enforces it faithfully, which is the problem.

Restarts aren't fixes. A pod that's OOMKilled gets replaced by an identical pod with the same limit. The controller has done its job. The loop continues until someone reads the usage history and changes the manifest.

Autoscalers are tuned by scar tissue. HPA parameters reflect the last incident: too aggressive once and a weekend of over-provisioning got expensive, so now they're conservative, and the next surge outruns them.

Audits are snapshots. A CIS benchmark run is a point-in-time export. Findings get triaged by hand against an exception list, tickets get filed, feature work wins, and next quarter the same findings come back.

Context lives in people. Which limits are stale, which exceptions are approved, which namespaces are fragile: that knowledge sits in Marcus's head, across twelve clusters.

01Stage 1

Manual

kubectl, hand scaling, reactive troubleshooting, patching by hand.

02Stage 2

Basic automation

CI/CD pipelines, basic monitoring, Helm charts, simple autoscaling.

03Stage 3

Advanced tooling

GitOps, service mesh, deep monitoring, policy engines.

04Stage 4

Agentic operations

An agent that connects the signals, investigates and acts within policy.

Most platform teams sit at stage two or three: plenty of signals, and a person still connecting them.

Marcus has monitoring, alerts and runbooks. That's a solid stage-two-to-three setup. What he lacks is something that connects rising memory usage, a stale limit and an eviction loop, and acts on it before the 3 a.m. page.

How CloudThinker connects

01Your clusters

Kubernetes

API serverPods, events, HPA, PDBs, RBAC
MetricsPrometheus, metrics-server
DashboardsGrafana panels and alerts
Desired stateManifests and Helm values in Git

EKS, GKE, AKS and self-managed

02Investigation

CloudThinker

TriageLimits, capacity, deploy or dependency
CorrelateUsage history, events, recent changes
ProposeManifest diff, HPA change, audit finding
VerifyPods stable, queue drained

Read-only by default

03Where the team works

Chat and Git

SlackFindings, approvals, summaries
Pull requestsResource and HPA changes as diffs
TicketsAudit findings with evidence

Every write goes through Auto Mode

CloudThinker reads the cluster through the same APIs your team uses and works where the team already decides.

CloudThinker reads the cluster through the Kubernetes API and your metrics stack, with a read-only service account scoped to the namespaces you choose. It sees pods, events, resource limits, HPA status and RBAC, plus the usage history in Prometheus. Changes come out as proposals (a manifest diff, an HPA change, a ticket) and run only when someone approves them or a policy you wrote pre-approves them.

Monday: the eviction loop

01Signal

OOMKilled loop

Three order-processing pods restart, hit the same ceiling, restart again.

02Readings

Leak, load or limit

Each predicts a different memory curve over the last 30 days.

03Evidence

Stale limit

Usage grew with payload size; the limit is eight months old.

04Action

Limit diff

New limit from P99 usage, proposed as a manifest change.

05Verify

Queue drains

Restarts stop and the 47,000-event backlog clears.

Monday's eviction loop, investigated from hypothesis to verified fix.

It started with OOMKilled pods in the order-processing namespace. Three worker pods hit their memory limit and were killed. The replacements came up, hit the same ceiling and were killed again: a restart loop at 3 a.m., with 47,000 unprocessed shipment events queuing up behind it.

The old way, Marcus got into the cluster, checked kubectl top pods, read the memory graphs in Grafana, calculated new limits from 30 days of P99 usage, updated the manifests, applied them and watched the pods settle. Two hours and fourteen minutes, most of it analysis that didn't need him. By the time he finished, three Slack threads were asking about delayed shipment notifications.

Here's the same incident with CloudThinker investigating.

1. Name the readings. An OOMKilled loop has three common explanations, and each predicts different data:

A memory leak in a new release. Usage climbs steadily after a deploy, on the new version only.
A load spike. Usage jumps with traffic and falls back when it passes.
A stale limit. Usage has crept up for weeks with payload size, and the limit never moved.

2. Check the pod's own story.

kubectl get pods -n order-processing \
  -o custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount,LAST:.status.containerStatuses[0].lastState.terminated.reason

kubectl get events -n order-processing --field-selector reason=BackOff --sort-by=.lastTimestamp

Every restart says OOMKilled, and there's no deploy in the namespace for nineteen days. That weakens the leak reading.

3. Read the history, not the moment.

# P99 working set per container over 30 days
quantile_over_time(0.99,
  container_memory_working_set_bytes{namespace="order-processing", container="worker"}[30d])

# The limit it's running against (kube-state-metrics)
kube_pod_container_resource_limits{namespace="order-processing", container="worker", resource="memory"}

Usage has climbed about 2% a week for two months, tracking average payload size, with no traffic spike tonight. The limit was last changed eight months ago. The stale-limit reading survives; the other two are ruled out with the graphs attached.

4. Propose a diff, not a command. The fix goes out as a change to the manifest in Git, with the evidence in the description:

resources:
  requests:
    memory: 1Gi # was 512Mi
  limits:
    memory: 1536Mi # was 768Mi; P99 over 30 days is 1.1Gi

5. Verify. After the change rolls out, CloudThinker watches the restarts stop and the backlog drain, and posts the outcome in the thread. Marcus approves one pull request instead of spending two hours on analysis.

Wednesday: the security audit

Marcus had barely recovered from Monday when the security team dropped a compliance request on his desk: a quarterly CIS Kubernetes Benchmark assessment across all twelve clusters, required for SOC 2.

The last one took Marcus and a colleague three full days. They ran kube-bench against each cluster, exported the results, cross-referenced failures against the exception list, wrote tickets for the genuine findings and assembled a report. Half the findings were the same as the quarter before, because the tickets had lost to feature work. This time the deadline was Friday.

The checks themselves are standard. On each cluster, kube-bench runs as a Job:

kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl logs job/kube-bench > kube-bench-$(kubectl config current-context).txt

The slow part was never running it. It was everything after: matching each [FAIL] against approved exceptions, separating new gaps from recurring ones, and writing it up for auditors. That's the part CloudThinker took. It ran the benchmark across all twelve clusters in parallel, matched findings against the exception list, marked which were new and which had recurred, and drafted the report in the format the SOC 2 auditors expected. Forty minutes instead of three days. Marcus reviewed it, approved two of the recommended remediations, and sent the package to compliance before lunch.

Friday: the traffic surge

Friday, 4:47 p.m. A major retail partner pushed a flash-sale notification to 3 million customers, and the shipment-tracking service became, from Marcus's point of view, the most popular page on the internet.

Request rates tripled in twelve minutes. The Horizontal Pod Autoscaler kicked in, but with conservative settings, because the last aggressive tuning had cost $14,000 in idle compute over a weekend. In the old world, Marcus would have been adjusting HPA thresholds by hand, watching node capacity and hoping the cluster autoscaler kept up.

This Friday, Marcus had already given CloudThinker the band he was comfortable with weeks earlier: for shipment-tracking in production, scaling between 4 and 40 replicas. CloudThinker recognized the pattern (a sudden spike with sustained throughput) within two minutes, and tightened the HPA's scale-up behavior inside that band:

behavior:
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
      - type: Percent
        value: 100
        periodSeconds: 15

It also checked that the PodDisruptionBudgets wouldn't block the scale-up and that the node pool had headroom for the projected replica count, then posted in the ops channel: "Traffic surge on shipment-tracking. Scale-up behavior adjusted within policy. Capacity headroom 340%. No action needed."

Marcus read it on his phone walking to his car. He kept walking.

What changes in day-to-day operations

The investigation pattern above extends across the four areas that eat most platform-team time:

Cost. Pod rightsizing from actual usage, node pool and spot mix, autoscaler efficiency, persistent volume cleanup, and namespace-level cost allocation across EKS, GKE and AKS.
Security. Pod Security Standards compliance, network policy gaps, over-privileged service accounts in RBAC, image vulnerabilities and admission controller coverage, each finding with its evidence.
Incidents. Log and event correlation across clusters, rollback candidates when a release regresses, and a post-incident note with the prevention change as a diff.
Performance. CPU throttling, HPA and VPA recommendations, placement through affinity and anti-affinity, and storage latency on the workloads that need it.

Why this beats the traditional workflow

	Traditional Kubernetes operations	With CloudThinker
OOMKilled loop	Someone wakes up and reads graphs	Readings tested against 30 days of history, limit diff proposed
Resource limits	Set at first deploy, revisited after an incident	Compared to real usage continuously
CIS audits	Three days of exports and spreadsheets	Run in parallel, matched to exceptions, report drafted
Traffic surges	Hand-tuned HPA under pressure	Adjusted within the band you set, then reported
Recurring findings	Re-flagged every quarter	Tracked as recurring, with the ticket history
Knowledge	In one engineer's head	Stored and reused on the next matching incident

The deeper difference is where judgment sits. Kubernetes controllers reconcile toward the state you declared, and they're very good at it. Deciding whether that declared state is still right (whether a limit, an HPA setting or an exception still matches reality) is investigation, and that's the work CloudThinker takes on.

Gated by design

CloudThinker acts within Auto Mode, so you decide what runs on its own and what waits for approval, whether it's a limit, an autoscaler or an RBAC binding. Every action carries the evidence and the reasoning behind it.

The week after

The following Monday, Marcus pulled up the numbers. Three incidents that would once have needed his hands (the eviction loop, the audit and the surge) took under 90 minutes of his time combined, mostly reading findings and approving changes. The previous quarter's average for similar work was 18 hours a week.

He spent Monday morning on the cluster migration that had been stuck in "when I have time" since October. For the first time since taking over twelve clusters, he had time.

"Kubernetes was supposed to abstract away infrastructure complexity," he wrote in the team retro. "It didn't. It moved the complexity from servers to clusters. This is the abstraction layer we were promised."

Getting started

Start read-only, on one cluster. Ask from chat:

"Find every pod in production that was OOMKilled in the last 7 days. For each, compare its memory limit to P99 usage over 30 days and propose new values. Propose only, change nothing."

"Run the CIS Kubernetes Benchmark on the staging cluster, match the failures against our exception list, and tell me which findings are new since last quarter."

"For shipment-tracking in production, keep HPA scale-up between 4 and 40 replicas during traffic spikes, and check with me before going outside that band."