Human Expert Guidance Meets Agentic AI: The Architecture for Scalable Autonomous Operations
How organizations build, test and share reusable automation assets (agents, skills, runbooks and approval policies) so AI handles the 80% of operational tasks that are common and low-risk, while humans stay in control of the 20% that need judgment.
From "AI that alerts" to "AI that acts"
A senior SRE fixes the same Redis connection-pool exhaustion for the fourth time this quarter. She knows the signs, the three commands that confirm it and the one change that fixes it. None of that knowledge leaves her head. Next month someone else is on call, and they spend an hour rediscovering it.
That's the paradox of the last decade in IT operations. We built monitoring that detects problems faster than ever, but people still investigate, decide and act. The result is alert fatigue at industrial scale: hundreds of alerts a day, most of them noise, and the same fixes rediscovered by whoever happens to be awake.
The industry moved through three stages:
- Manual monitoring (2010 to 2018). Threshold alerts, static dashboards, human-driven investigation and response. Detect only.
- AIOps 1.0 (2018 to 2023). ML anomaly detection, event correlation and noise reduction, but still executed by humans. Detect and recommend.
- Agentic AIOps (2023 onward). AI agents investigate, decide and act within governed boundaries. Detect and act.
Most platforms get one thing wrong: they treat automation as a binary switch. Either a human does it, or the machine does. The useful version is to encode human expertise into shareable, testable, governed automation assets that agents can run across an entire organization.
The VibeOps principle. The goal isn't zero-touch operations. The goal is autonomous execution of the 80% of tasks that are common, well understood and low-risk, while escalating the 20% that need human judgment with full context already assembled.
This is the architecture of VibeOps: closed-loop, self-healing operations where human experts guide agentic AI through shareable automation primitives, and every resolution makes the system smarter.
Why scripts and static runbooks don't scale
The traditional answer to repeated work is a runbook or a script. Both hit the same ceiling, for structural reasons.
Runbooks are read, not run. A wiki runbook depends on a person finding it, trusting it and executing it correctly at 3 a.m. It can't tell when it's out of date.
Scripts are rigid. A script runs the same steps every time its trigger fires. It can't notice that this alert is a symptom of something upstream, or that the fix it encodes doesn't apply tonight.
Knowledge stays local. The team that wrote the fix keeps it in their repo or their Notion page. The team next door rebuilds it from scratch.
Trust is all or nothing. Either a script runs automatically or it doesn't. There's no middle state where a human sees the proposal, approves it, and gradually lets proven actions run on their own.
Scaling autonomous operations means fixing all four: assets that reason over context, that anyone can reuse, and that earn autonomy step by step.
Six building blocks of scalable autonomous operations
Scalable automation isn't about writing more scripts. It's about composable, shareable, governed primitives that any team can develop in a workspace, test in isolation and deploy across the organization. Each one encodes a different part of operational expertise:
01Agents
/create-agent
Specialists with defined skills, permissions and sandbox boundaries.
02Knowledge
/create-knowledge
RAG-indexed knowledge bases from Confluence, Notion, Slack and past incidents.
03Skills
/create-skill
Capability definitions with triggers, prompts, guardrails and output schemas.
04Topology
/create-topo
Live maps discovered from infrastructure: dependency graphs and blast radius.
05Tasks
/create-task
Scheduled operations: cron-based, event-triggered, or chained through skills.
06Incidents
/create-incident
Incident orchestration: severity classification, RCA and war-room setup.
Each primitive is a unit of shareable expertise. When a senior SRE creates a skill for diagnosing Redis connection-pool exhaustion, that skill doesn't stay in her head or her team's Notion page. It becomes an organization-wide capability that any agent can invoke and any team can benefit from, and it improves with use.
The anatomy of a skill
Skills are the atomic unit of agent behavior. A traditional runbook is a static document humans read and execute. A script is rigid code that breaks when context changes. A skill is a contextual, self-evaluating capability definition that an agent reads and applies. Here's an illustrative SKILL.md for SSL certificate expiry checks:
---
name: ssl-certificate-expiration-check
version: 2.4
triggers: [ssl expiring, certificate renewal, TLS check, cert audit]
tools: [aws-cli, acm-api, route53, slack-webhook]
connections: [aws-prod, aws-staging, cloudflare]
---
# SSL certificate expiration check
You are a certificate auditor. Use the topology and past incidents as context.
Assess every certificate in scope, then act within the guardrails below.
## Guardrails
- Scans are read-only.
- Alerts go to Slack.
- Auto-renewal waits for approval.
- Never delete a certificate.
## Output
domain (string), expiry_date (datetime), days_remaining (int),
risk_level (enum), issuer (string), action_taken (string)
## Chain
- days_remaining < 14: create a Jira ticket
- days_remaining < 7: propose renewal and notify the owner
- renewal failed: escalate to a human
Each run is scored, feedback is recorded, what was learned goes into the knowledge base, and the score history decides whether the skill's actions can graduate. One team develops and tests it; once approved through governance, it's shared across the organization. Every team gets SSL monitoring without writing code.
Develop, test, share, automate
The path from human expertise to autonomous execution is a governed, four-stage lifecycle. Every automation asset is validated before it touches production and monitored after it ships.
01Workspace
Develop
Describe the procedure in chat; the platform drafts the SKILL.md.
02Sandbox
Test
Run against mirrored infrastructure in an ephemeral sandbox, scored for accuracy.
03Governance
Share
Security, compliance and team leads review before it reaches other teams.
04Auto Mode
Automate
You decide what runs on its own and what waits for approval.
Stage 1: Develop in workspaces
Inside each organization, teams create Workspaces as isolated development environments, split by project, department or environment. Each Workspace keeps its own knowledge base, skill configurations, agent permissions, connection credentials and scheduled tasks. A DevOps team's Workspace is separate from the Security team's: different knowledge, different connections, different permission boundaries.
Development is conversational. You describe the procedure to the orchestrator in plain language:
"Create a skill that checks database backup status across all production RDS instances every morning, and flags any instance whose latest snapshot is older than 26 hours."
The platform drafts the full SKILL.md with trigger patterns, required tools, prompt template, guardrails, output schema and chain triggers. Work that used to take weeks of engineering takes a conversation.
Stage 2: Test in sandboxes
Every agent operation runs in CloudThinker's Sandbox, created on demand and destroyed right after execution. No data persists, no cross-sandbox access is possible, and every operation leaves a full audit trail.
That makes testing a risk-free production simulation. Skills run against real or mirrored infrastructure with their actual tool chains, inside boundaries that contain the blast radius. The evaluation pipeline scores each run with LLM-as-judge accuracy checks plus human feedback, and those confidence scores feed the approval workflow.
Stage 3: Share through governed approval
Once a skill, agent or runbook passes testing, it enters the governance layer: RBAC policies, approval workflows, credential scoping, audit trails and metering. This is where "human-guided" matters most. A senior engineer doesn't build a skill and push it to production. They submit it for review: security checks credential scope, compliance checks guardrails, and team leads check the business logic.
Sharing works at three tiers: built-in skills maintained by the platform (Code Review, Incident Response, HelpDesk, Cloud Ops, Security, FinOps), partner skills built by managed service providers and shared across their clients, and enterprise custom skills built in-house for domain needs like banking compliance or healthcare regulation.
Stage 4: Automate under Auto Mode
Shared skills don't start with full autonomy. They act within Auto Mode, so you decide what runs on its own and what waits for approval. The 80% target maps to the work that ends up running on its own: password resets, SSL renewals, disk space alerts, cost anomaly notifications, routine backup validation. These tasks eat most operations time and carry little risk when automated properly. The other 20% (novel incidents, architecture decisions, security breaches, compliance investigations) goes to humans with full context already assembled, which shortens even the human-handled work.
Three-tier isolation: organizations, workspaces, sandboxes
Letting agents run operations across your infrastructure only works if the isolation model holds. The architecture uses a three-tier hierarchy that separates tenants, teams and individual executions, which matters for banking, healthcare and any enterprise with compliance obligations.
- Organization: SSO, RBAC, billing, audit, compliance and user management per tenant. No data crosses organization boundaries.
- Workspace: knowledge base, skill config, agent permissions and credentials per team, for example Production Ops and IT Support side by side.
- Sandbox: one ephemeral sandbox per execution. No data persists, and no sandbox can reach another.
Every execution leaves an immutable audit trail. Isolated sandboxes, per-tenant VPCs and encryption keys at rest give the defense in depth that SOC 2 Type II, banking regulation and data residency controls ask for.
This is what makes "share across the organization" safe. When a DevOps team shares a skill with the Security team, only the skill definition is shared. The credentials, execution context and data access stay scoped to each team's Workspace. The Security team's copy connects to their tools with their permissions. Same blueprint, isolated execution.
From expert knowledge to autonomous resolution
Here's the full path for one piece of expertise, from a senior SRE's head to an organization-wide capability. The scenario and its numbers are illustrative.
1. Spot the pattern. Redis connection-pool exhaustion has caused 12 incidents in the past quarter. Each time, someone spent the first half hour ruling out the same three explanations.
2. Encode the diagnosis, hypothesis first. The SRE describes what she checks:
"Create a skill for Redis connection-pool exhaustion. When the API's Redis errors spike, check three things: is Redis itself saturated (CPU, memory, evictions), did a deploy change the client pool size, or did traffic outgrow the current pool. Show the evidence for each and propose the matching fix."
The platform generates the SKILL.md with triggers, tools (redis-cli, CloudWatch, the Kubernetes API) and guardrails. Each hypothesis maps to a concrete check: redis-cli INFO clients and INFO stats for connected clients and rejected connections on the server side, the deploy diff for the client pool setting, and the request-rate graph against the pool ceiling.
3. Test in a sandbox. She runs the skill against a mirrored staging environment: 98% accuracy across 50 simulated scenarios. The two misses both involved a network partition, so she adds a fourth check for that.
4. Submit for approval. Security reviews tool access. The DevOps lead checks the remediation logic. Compliance confirms the audit trail.
5. Share across the organization. The approved skill is published to the Skills Library with RBAC tags.
6. Graduate deliberately. For its first two weeks, every fix the skill proposes waits for approval. After 15 successful, approved resolutions, the team lets pool resizes in staging run on their own. After 90 days of clean history, the team decides whether production follows.
7. Keep learning. Every run feeds the Knowledge Graph. The next occurrence is caught at the first sign of pool pressure, before it becomes an incident.
One senior SRE's expertise, twelve incidents a quarter that stop needing her, and a capability every team now has.
One conversation, many capabilities
The orchestration layer makes these primitives work as one system. A single orchestrator is the entry point: you talk to it in natural language, and it classifies intent, picks the right skills, and hands work to specialist agents when needed.
Every request (a plain-language question, a skill call, or a scheduled task trigger) flows through the same pipeline: intent detection, Guard-In, skill selection, context build, LLM routing, sandbox boot, tool execution, chaining to sub-tasks, Guard-Out, event log, evaluation and tracing, then memory write and delivery. That keeps every operation secure (Guard-In and Guard-Out), auditable (event log), evaluated (LLM-as-judge tracing) and improving (memory write). There are no shortcuts and no unlogged executions.
The Guardrails Engine runs as an independent safety agent. It doesn't answer to the orchestrator or to the executing agent. It performs PII detection, schema enforcement, injection defense and output validation on every run. The agent that executes an action is never the agent that validates it.
Why every automation makes the system smarter
The strongest part of this architecture isn't any single component. It's the flywheel that forms when the primitives share one Knowledge Graph: more skills mean more capabilities, which open more use cases, which bring more teams, whose work makes the AI smarter, which makes the next skill more valuable.
Over time, every resolved incident adds a Knowledge Graph entry. Every skill run produces evaluation data that improves the next run. Every team that brings in its runbooks through /create-knowledge enriches the context for all agents. Every topology mapped through /create-topo deepens the system's view of your infrastructure. The platform doesn't only execute; it accumulates operational knowledge.
Why this beats traditional AIOps
| Traditional AIOps | Agentic VibeOps | |
|---|---|---|
| Response model | Alert, human investigates, manual fix | Detect, agent investigates, resolves within policy |
| Knowledge | Static runbooks read by humans | Knowledge Graph and RAG, invoked by agents |
| Learning | Manual rule updates | Improves with every execution |
| Cross-domain work | Siloed per tool | Coordinated through one orchestrator |
| Trust | Automation on or off | Gated by Auto Mode, with a full audit trail |
| Scale | Linear with headcount | Grows with the skill library |
The deeper difference is what happens to expertise. In traditional AIOps, the tools get better at telling people something is wrong, and the knowledge of what to do stays with the people. Here, the knowledge becomes an asset that runs, gets reviewed, and improves, while people decide how much of it runs alone.
The 80% target
The 80% target isn't about replacing people. It's a categorization of operational work by risk and repeatability, with the same approval gate applied systematically across domains.
Candidates to run on their own: password resets, VPN troubleshooting, software installation, access provisioning, SSL renewals, disk space management, cost anomaly alerts, routine backup validation, deployment rollbacks for known failure patterns, certificate rotation, DNS changes, and security group changes within approved templates.
Stays human-guided: novel incident types, architecture decisions, security breach investigation, compliance audit interpretation, budget allocation, vendor selection, cross-team escalation policy, infrastructure migration planning and regulatory change assessment.
The key design choice is that the human-handled 20% benefits from AI too. When an incident escalates, the agent has already assembled the diagnostic context: the topology showing blast radius, Knowledge Graph entries for similar past incidents, the timeline of related changes, and a proposed remediation with a confidence score. The human starts from a complete briefing, not from zero.
From reactive chaos to self-healing operations
Reaching 80% autonomous resolution doesn't happen on day one. A typical rollout is a 12-month, four-phase journey that builds trust step by step.
01Months 1 to 3
Foundation
Unified data layer, event correlation and the Knowledge Graph.
02Months 3 to 6
Agentic core
Incident commander, automated RCA, every action approved by a person.
03Months 6 to 9
Scale
More teams and domains; proven actions run on their own.
04Months 9 to 12
Self-healing
Closed-loop remediation and runbooks that improve with every run.
Each phase widens both the number of teams on the platform and the share of work that runs on its own. The pattern is the same every time: start new domains with every write approved, prove accuracy, build confidence, then let proven actions run alone. The organization never takes more risk than its data supports.
The human expert doesn't disappear; they scale
The common misconception about autonomous operations is that it replaces human expertise. It does the opposite: it scales expertise to the whole organization. One senior SRE's diagnostic method stops helping only when she's on call. It helps every team, on every shift, and improves as it runs.
Shareable primitives, workspace-isolated development, sandbox testing, RBAC-governed sharing, Auto Mode, three-tier isolation and closed-loop learning aren't a theoretical framework. They're the mechanics of how human knowledge becomes autonomous capability at enterprise scale. Organizations that get this right build an advantage that compounds: every incident resolved, every skill created and every knowledge base enriched makes the system measurably better. Teams still on traditional AIOps are improving their alerts; these teams are improving their answers.
Getting started
Start with one recurring task your team already knows how to fix:
"Look at our incident history for the last quarter and list the five issues we fixed most often, with the steps we took each time."
Then turn the top one into a skill, proposing only:
"Create a skill from the Redis pool exhaustion fixes. Run it against staging, show me the accuracy, and propose fixes only. Don't change anything."
Start your free trial or book a demo to see human-guided agentic AI in action.