CircleCI Automation with AI Agents: Build Triage That Ends the Rerun Reflex

The nightly workflow goes red at 2 a.m. By the time someone sees it, the ritual is automatic: open the failed job, scroll a few thousand lines of log, squint at the last stack trace, and click Rerun workflow from failed. If it goes green, nobody looks again. If it stays red, the same person scrolls the same log a second time. That loop — scroll, squint, rerun — is how most teams "do" CircleCI automation, and it quietly burns credits every time a flaky test gets a free retry.

This is part three of our CI/CD reliability series. Part one mapped the failure and waste patterns that dominate real CircleCI usage — flaky tests auto-rerun into a credit sink, resource classes oversized for the work, workflows with no caching that rebuild dependencies every run, fan-out that stalls on concurrency limits, and failed workflows nobody re-examines once the rerun goes green. Part two covered the DIY toolkit: the Insights dashboard and API, test splitting and parallelism, cache keys that actually hit, approval jobs as manual gates, rerun-with-SSH, and the v2 API with curl for custom reporting.

If you worked through part two, you reached its honest ceiling: Insights will name the flaky test, but deciding whether to fix it, quarantine it, or delete it is still a human loop — and that loop runs again with every red workflow. This article closes that gap. It covers how CloudThinker agents watch your failed workflows continuously, classify each one, track deployment jobs across environments, and propose fixes under graduated autonomy — with your approval jobs left exactly where you put them.

Connecting CircleCI: a read-only API token, about five minutes

CloudThinker connects to CircleCI the way you would want any external system to connect to your pipeline: with a read-only API token — a personal or project API token — and nothing installed in your config.yml. No orb to add, no context secrets shared, no write access at connection time.

You paste the token and select the org; the first scan runs against the same v2 API endpoints you called with curl in part two — GET /project/{project-slug}/pipeline, /pipeline/{id}/workflow, /workflow/{id}/job, and the Insights routes under /insights/{project-slug}/workflows. The CircleCI connection guide lists the exact token scope so your security review has the minimal grant on record. The whole flow takes about five minutes.

What the agents do with every failed workflow

Read the job logs like a senior engineer

The core skill of build triage is classification, and it is the one skill Insights cannot do for you: it counts failures, but it does not tell you what a given failure is. When a workflow fails, the agents pull the job's step output and its test results and answer the question the red badge didn't:

• Infrastructure failure. A docker pull timing out against the registry, checkout failing on a network blip, an out-of-memory kill from a resource class that is too small for the step, a cache restore that half-succeeded. The commit is innocent; the finding is the container or resource condition, not the code.
• Flake — and how flaky. A test that has passed for months and failed once is not the test that has failed 30 times this month and passed on every rerun. The agents keep per-test failure history across the project, which is what turns "flaky builds" from a gut feeling into a ranked list with a credit cost attached to each entry.
• Real regression. Compile errors, assertion failures that appear at exactly one commit and reproduce on rerun. This is the case where the author should be pinged immediately with the evidence — not after two hopeful reruns quietly spent 15 minutes of a large resource class.

Each classification arrives with the evidence: the relevant log excerpt, the test's failure history, the commit, and the resource class and duration of the job that failed.

Correlate with the commit and recent config.yml changes

A failure judged in isolation is half an answer. The agents correlate each red workflow with the commit that triggered it and with recent changes to .circleci/config.yml. That is how the sneakiest part-one pattern gets caught: the wave of "test failures" that started the morning someone bumped a resource class down, changed a cache key, or reordered a requires in the workflow graph. Instead of blaming whichever commit had the bad luck to run next, the agents point at the config diff that actually moved the behavior.

Track deployment jobs across environments

Beyond failures, the agents follow your delivery workflows job by job: which commit reached staging, which reached production, where the last deploy stalled and for how long, and whether a workflow is parked on an approval job that everyone forgot to click. For teams whose CircleCI pipeline is the deployment system of record, "what is actually in prod right now" becomes a question you ask in chat instead of a walk back through the pipelines list.

Graduated autonomy: proposals before actions, approval jobs untouched

Triage produces findings; findings imply fixes. Every fix class has an autonomy level you set:

• Notify — report the finding and its classification. Nothing else. The default for everything.
• Suggest — propose the specific change with evidence: quarantine this flaky test (a pull request tagging it for exclusion until it is fixed), correct the cache key so restore_cache stops missing on every run, right-size the resource class for a job that runs at 8% of its memory, split an oversized test job that is serializing your slowest feedback.
• Approve — the agents prepare the change and open it only after a named human approves.
• Autonomous — execute and report. Teams reserve this for reversible, low-blast-radius actions, and only after weeks of watching the suggestions be right.

One line matters more than the rest here: approval jobs stay human. The agents observe your workflow's type: approval gates and report their state — they never click one. A deploy waiting on a human is a deploy the agents will surface and explain, never advance. Every finding, proposal, approval, and action lands in an audit trail. If your team has been arguing about test-quarantine policy or auto-rerun in the abstract, this is where the argument gets settled with credit numbers attached instead of opinions.

What a first pass typically finds

Illustrative numbers for a mid-market setup — roughly 40 projects, a few hundred workflows a day, credit spend nobody attributes per project. Yours will differ.

Note the shape: almost none of this is exotic. It is the part-one inventory, surfaced automatically, each row arriving with a classification and a proposed, approval-gated fix instead of a dashboard you have to remember to open.

[SCREENSHOT: CloudThinker findings table for a CircleCI org — flaky tests, oversized resource classes, and broken cache keys, each row showing estimated credits and an autonomy badge]

Prompts to try in your first session

You direct the agents in plain language from chat:

"Why did last night's nightly workflow on api-service fail? Tell me whether it was infrastructure, a known flaky test, or the commit — and show me the log and test evidence."

"Rank the flakiest tests across all projects over the last 30 days by credits burned on auto-reruns. Which would you quarantine first?"

"Which jobs are running on a resource class larger than they need, and what would right-sizing them save in credits per month?"

"Walk me through the deploy workflow: which commit is in staging versus production, and is anything parked on an approval job right now?"

Each answer cites the underlying CircleCI data — the same pipeline, workflow, job, and Insights responses you pulled by hand in part two — so you verify rather than trust.

[SCREENSHOT: chat thread where an agent classifies a failed workflow as a known flaky test, links the job log excerpt and the test's rerun history, and proposes a quarantine PR under the Approve autonomy level]

What the agents will not do

Because "AI with access to CI" should raise exactly these questions:

• Read-only by default. The API token grants no write permission. Quarantining a test, editing a cache key, or changing a resource class requires you to grant scoped write access and set that action class above Notify.
• No auto-rerun of real failures. The agents will not paper over a regression with reruns to turn the board green. A rerun is proposed only where the failure class is provably infrastructural — the opposite of the credit-burning reflex part one warned about.
• Approval jobs stay yours. The agents read the state of every type: approval gate and never advance one. A human deploy gate remains a human deploy gate.
• No invisible changes. Every proposal and execution is in the audit trail with its evidence and approver. If it is not in the trail, it did not happen.

[SCREENSHOT: audit trail entry — a resource-class right-sizing proposal with the before/after config, the memory evidence, the approver, and the timestamp]

From red workflows to answers

You know the CircleCI failure and waste patterns. You know how far the Insights API and native tooling take you. What is left is the triage itself — the per-workflow classification and the credit-waste attribution that no dashboard does for you — and that is an automation problem, not an effort problem. Teams that move from scroll-squint-rerun to continuous, approval-gated triage typically cut time-to-answer on a red workflow from tens of minutes to under one, and reclaim the credits that the auto-rerun reflex was quietly spending every night.

Try CloudThinker free — 100 premium credits, no card required — and follow the CircleCI connection guide to see your own first findings table within the hour.