Sign inBook a Demo

How To

CircleCI Automation: Build Triage with Native Tools (Insights, Config, and the v2 API)

A hands-on CircleCI automation guide to triaging build failures with only native tooling — no third-party agents. Part two of our CI/CD reliability series covers the Insights dashboard and API (duration, success rate, flaky-test detection), timing-based test splitting and parallelism, cache keys that actually hit, approval jobs as human gates, rerun-with-SSH for in-place debugging, and the v2 API with curl (pipelines, workflows, jobs) for org-wide reporting. Exact config.yml and curl examples throughout, plus the honest ceiling: Insights names the flaky test, but deciding whether to fix, quarantine, or delete it is still a human loop.

ST
·
circlecicicddevopsflakytestsbuildautomationcontinuousintegration
Cover Image for CircleCI Automation: Build Triage with Native Tools (Insights, Config, and the v2 API)

CircleCI Automation: Build Triage with Native Tools (Insights, Config, and the v2 API)

Everyone on your team has seen the flaky-tests tab. Fewer teams have a quarantine process — the actual decision of what to do with the test the tab names. That gap is where CircleCI automation lives or dies: the platform is remarkably good at telling you which test fails intermittently, and completely silent on whether you should fix it, quarantine it, or delete it.

This is part two of our CI/CD reliability series. Part one mapped the waste patterns that dominate CircleCI usage — flaky tests burning credits on auto-rerun, oversized resource classes, missing caches, fan-out queuing on concurrency limits. This article is the hands-on version: how to triage build failures and tune workflows using only the tools CircleCI ships — the Insights dashboard and API, config.yml primitives, rerun-with-SSH, and the v2 API with curl. No third-party tooling. Budget an afternoon for a first pass; expect the findings to pay for themselves in recovered credits within a billing cycle.

We'll go tool by tool, because that's how the work actually happens.

Tool 1: Insights — the dashboard and the API

CircleCI Insights is the map. It answers "which workflows are slow, which are unreliable, and which tests flake" without you instrumenting anything. Open it at your project → Insights in the CircleCI web app (docs).

The three views that matter

  • Workflow duration and success rate. Sort workflows by median duration and by success rate. A workflow under 90% success that isn't gated on external flakiness is a triage target — someone is re-running it to green and moving on.
  • Per-job credit and duration. The job-level breakdown shows which jobs consume the most compute. A job whose p95 duration is far above its median is either resource-starved or racing a flaky dependency.
  • Flaky tests. The flaky-tests view surfaces tests that passed and failed on the same commit — the cleanest possible flake signal. This requires uploading test results (covered below); without it the tab is empty.

Pull the same data from the Insights API

The dashboard is fine for eyeballing; for reporting you want the API. The Insights endpoints live under the v2 API. Get a workflow's summary metrics over the last 90 days:

curl -s -H "Circle-Token: $CIRCLE_TOKEN" \
  "https://circleci.com/api/v2/insights/gh/your-org/your-repo/workflows/build-and-test?reporting-window=last-90-days" \
  | jq '.items[] | {date: .window_start, success: .metrics.success_rate, p95: .metrics.duration_metrics.p95, credits: .metrics.total_credits_used}'

For flaky tests specifically, there's a dedicated endpoint:

curl -s -H "Circle-Token: $CIRCLE_TOKEN" \
  "https://circleci.com/api/v2/insights/gh/your-org/your-repo/flaky-tests" \
  | jq '.flaky_tests[] | {test: .test_name, file: .file, job: .job_name, times: .times_flaked}'

How to read it. Sort flaky tests by times_flaked. The top few are almost always responsible for the majority of your auto-reruns and a disproportionate share of wasted credits. That ranked list is your triage backlog — but note that the API hands you the names, not the verdicts. Deciding what happens to each one is the manual loop we return to at the end.

Tool 2: Test results and test splitting — feed the flake detector

Insights can only name flaky tests if you upload results. Emit JUnit XML and store it:

jobs:
  test:
    docker:
      - image: cimg/node:20.11
    steps:
      - checkout
      - run: npm ci
      - run:
          command: npm test -- --reporters=jest-junit
          environment:
            JEST_JUNIT_OUTPUT_DIR: ./test-results
      - store_test_results:
          path: ./test-results

store_test_results is what powers both the flaky-tests view and timing-based test splitting (docs).

Split by timing, not by count

The common mistake is splitting tests across parallel containers by filename count, which leaves one container carrying all the slow suites. Split by historical timing instead — CircleCI stores it from your uploaded results:

jobs:
  test:
    parallelism: 4
    docker:
      - image: cimg/node:20.11
    steps:
      - checkout
      - run: npm ci
      - run: |
          TESTFILES=$(circleci tests glob "test/**/*.test.js" | \
            circleci tests split --split-by=timings)
          npm test -- $TESTFILES --reporters=jest-junit
      - store_test_results:
          path: ./test-results

How to read it. Set parallelism to the point where median job duration stops falling — past that you're paying for containers that finish early and idle. The Insights per-job duration view tells you exactly where that knee is (test-splitting docs).

Tool 3: Caching keys that actually hit

A cache that never hits is worse than no cache — you pay to save it and pay again to rebuild. The failure mode is almost always a key that changes every run. The fix is a key derived from a lockfile checksum:

    steps:
      - checkout
      - restore_cache:
          keys:
            - deps-v1-{{ checksum "package-lock.json" }}
            - deps-v1-
      - run: npm ci
      - save_cache:
          key: deps-v1-{{ checksum "package-lock.json" }}
          paths:
            - ~/.npm

How to read it. The primary key includes the lockfile checksum so it invalidates only when dependencies actually change. The second, partial key (deps-v1-) is the fallback — a near-miss restore beats a cold install. Confirm hits by reading a job's restore_cache step output: "Found a cache" versus "No cache found" is the whole story. Bump the v1 prefix to force a clean rebuild when a cache goes bad (caching docs).

Tool 4: Approval jobs — the human gate you keep

Not every gate should be automated, and CircleCI's approval job is the primitive for the ones that shouldn't. It pauses a workflow until a human clicks approve in the UI:

workflows:
  deploy:
    jobs:
      - build
      - test:
          requires:
            - build
      - hold:
          type: approval
          requires:
            - test
      - deploy-prod:
          requires:
            - hold

How to read it. The hold job has no steps — its only job is to stop the workflow and wait (approval-job docs). Everything downstream of it is human-gated by design. This matters for the rest of this series: when you later automate triage, the approval job stays a human decision. Automation should make the flaky-vs-regression call obvious; it should not click your production approve button.

Tool 5: Rerun with SSH — debugging the failure in place

When a job fails in a way you can't reproduce locally, don't add echo statements and push five times. Re-run the failed job with SSH from the job's page (Rerun → Rerun job with SSH), then connect to the exact container:

ssh -p 54782 12.34.56.78
# then, inside the container, inspect the real environment:
cat ~/project/test-results/*.xml
env | grep CIRCLE

How to read it. The SSH session lands you in the same image, same working directory, same environment variables the job saw (SSH-debug docs). This is the fastest way to tell an infrastructure failure (missing binary, wrong Node version, exhausted memory) apart from a genuine test regression. The session stays open for about two hours or until you disconnect — it holds a container, so close it when you're done.

Tool 6: The v2 API — custom reporting with curl

The dashboard shows you one project at a time. For org-wide reporting you script the v2 API. The three endpoints you'll use most are pipelines, workflows, and jobs (API reference).

List recent pipelines for a project, then walk down to failed jobs:

# 1. Recent pipelines
curl -s -H "Circle-Token: $CIRCLE_TOKEN" \
  "https://circleci.com/api/v2/project/gh/your-org/your-repo/pipeline" \
  | jq -r '.items[].id' | head -20

# 2. Workflows for one pipeline
curl -s -H "Circle-Token: $CIRCLE_TOKEN" \
  "https://circleci.com/api/v2/pipeline/PIPELINE_ID/workflow" \
  | jq '.items[] | {name, status, id}'

# 3. Failed jobs in a workflow
curl -s -H "Circle-Token: $CIRCLE_TOKEN" \
  "https://circleci.com/api/v2/workflow/WORKFLOW_ID/job" \
  | jq '.items[] | select(.status=="failed") | {name, job_number, type}'

How to read it. Chaining these three gives you a failed-job feed across every project, which you can dump to CSV and sort by frequency — the org-level version of the flaky-tests list. You can also POST to .../workflow/WORKFLOW_ID/rerun with {"from_failed": true} to re-run only the failed jobs, though be deliberate: auto-rerunning flaky failures is exactly the credit leak part one warned about. Generate a token under User Settings → Personal API Tokens (read-only for reporting) or a project token for CI-scoped access.

The honest part: where native triage stops

You should run everything above. You should also know its limits before treating it as a strategy.

It's point-in-time. The Insights window you read describes last week. This sprint's new tests and config changes start regenerating flakes and cache misses immediately — every pattern here is a process, not a one-time cleanup.

It costs engineer-hours, on repeat. The first pass is an afternoon. Every pass competes with feature work, so "weekly triage" quietly becomes "when the build's been red for a day."

The verdict is still human. This is the real ceiling. The flaky-tests endpoint names the test. It cannot tell you whether that test guards a real race condition worth fixing, a genuinely non-deterministic assertion worth deleting, or something you should quarantine while you investigate. Insights ends at the name; fix-versus-quarantine-versus-delete is a judgment call about your codebase — and every day that call sits unmade, the test keeps auto-rerunning and burning credits.

What comes next

Every check in this guide can run continuously with the verdict attached. CloudThinker agents watch failed workflows through a read-only CircleCI token, read the job logs and test results, classify infra-vs-flake-vs-regression, correlate the failure with the commit and recent config.yml changes, and propose the fix — cache-key correction, resource-class rightsizing, test quarantine — under autonomy you control, with your approval jobs left firmly human. That's part three of this series.

Or see your own project's flaky-test and credit picture directly: Try CloudThinker free — 100 premium credits, no card required.