Sign inBook a Demo

How To

Jenkins Build Failure Analysis Using Only Native Tools

A hands-on Jenkins build failure analysis workflow using only native tooling: Script Console Groovy for queue depth, stuck builds, and disk pressure; Timestamper and stage views for readable pipeline logs; the JUnit plugin's Test Result Trend for spotting flaky builds; build discarders and cleanWs as waste control; the Jenkins REST API with curl; and retry/timeout wrappers in your Jenkinsfile done right. Part two of our three-part Jenkins CI/CD reliability series — with an honest look at where DIY triage stops and root-causing a red build stays human work.

ST
·
jenkinscicddevopsflakytestsbuildfailurescontinuousintegration
Cover Image for Jenkins Build Failure Analysis Using Only Native Tools

Jenkins Build Failure Analysis Using Only Native Tools

Paste these three lines into your Script Console (Manage Jenkins → Script Console, admin required) before you read anything else:

// Minutes the oldest queued item has been waiting
println(Jenkins.instance.queue.items.collect { ((System.currentTimeMillis() - it.inQueueSince) / 60000) as int }.max() ?: 0)

// Builds that have been running for more than an hour, right now
println(Jenkins.instance.getAllItems(Job).count { it.isBuilding() && System.currentTimeMillis() - it.lastBuild.startTimeInMillis > 3600000 })

// Jobs with no build discarder — unbounded disk growth
println(Jenkins.instance.getAllItems(Job).count { it.buildDiscarder == null })

If any of those numbers surprised you, this guide is for you. Effective Jenkins build failure analysis doesn't require buying anything — the controller already knows why your queue is deep, which builds are stuck, and where the disk went. It just doesn't volunteer the information. This is a hands-on tour of extracting it with native tooling only: the Script Console, the log-reading UIs, the JUnit plugin, build discarders, the REST API, and Jenkinsfile wrappers done properly.

This is part two of a three-part series. Part one mapped the failure and waste patterns that dominate Jenkins installations — flaky tests, executor starvation, error-swallowing pipelines, plugin drift. Part three covers running this triage continuously with an AI agent.

Budget 2–4 hours for a first pass on a controller with 100+ jobs.

Tool 1: the Script Console — fleet-wide truth in Groovy

Everything in the Jenkins UI is scoped to one job or one node. The Script Console is the only native surface that sees the whole fleet at once, which makes it your first stop, not your last resort. (It runs with full controller privileges — read-only queries only, and never paste code you haven't read.)

Queue analysis. A deep queue is either a capacity problem or a label problem, and why tells you which:

Jenkins.instance.queue.items.each { i ->
  def mins = ((System.currentTimeMillis() - i.inQueueSince) / 60000) as int
  println "${i.task.fullDisplayName} — ${mins} min — ${i.why}"
}

"Waiting for next available executor" across the board means you're short on agents. "There are no nodes with the label X" means someone renamed a label or a Jenkinsfile demands an agent that no longer exists — a build that will queue forever while looking like a capacity issue.

Stuck and marathon builds. Builds holding an executor for hours are usually waiting on something that will never answer (a dead input step, a hung network call with no timeout):

Jenkins.instance.getAllItems(Job).each { job ->
  def b = job.lastBuild
  if (b?.isBuilding()) {
    def mins = ((System.currentTimeMillis() - b.startTimeInMillis) / 60000) as int
    if (mins > 60) println "${job.fullName} #${b.number} — running ${mins} min"
  }
}

Disk pressure per node. Agent workspaces fill quietly until builds start failing with errors that look nothing like "disk full":

import hudson.node_monitors.DiskSpaceMonitor

Jenkins.instance.computers.each { c ->
  def ds = DiskSpaceMonitor.DESCRIPTOR.get(c)
  println "${c.name ?: 'built-in'} — ${ds?.gbLeft ?: '?'} GB free"
}

Failure rate over the last 24 hours. One number that tells you whether today is a normal day:

def since = System.currentTimeMillis() - 86400000
def total = 0, failed = 0
Jenkins.instance.getAllItems(Job).each { job ->
  job.builds.byTimestamp(since, System.currentTimeMillis()).each { b ->
    total++
    if (b.result == Result.FAILURE) failed++
  }
}
println "Last 24h: ${failed}/${total} builds failed"

Run these weekly and paste the output into a doc. That's a health baseline most teams never have.

Tool 2: reading pipeline logs without drowning

The classic console log for a parallel pipeline interleaves every branch into one stream — genuinely hard to read past a few thousand lines. Two native fixes:

Use a stage-oriented view. Blue Ocean renders each stage and parallel branch as its own log pane, so you click the red stage and read only its output. Note that Blue Ocean is in maintenance mode; on newer controllers, install its designated successor, the Pipeline Graph View plugin, which does the same job and is actively developed. Either way, the workflow is the same: never start from the full console log — start from the failed stage.

Timestamp everything. The Timestamper plugin prefixes every log line with a clock. In a declarative Jenkinsfile:

pipeline {
  agent any
  options {
    timestamps()
  }
  ...
}

Timestamps turn "the build is slow" into "the npm install step took 14 minutes at 03:12" — which is the difference between a hunch and a diagnosis. Gaps between consecutive timestamps are where your build time actually goes; a 10-minute silence usually means a network wait or a lock, not compute.

When you must grep the raw log, pull it as text rather than scrolling the UI: BUILD_URL/consoleText (more on the API below), then search from the bottom up — the first error printed is often noise, and the line that killed the build is near the end.

Tool 3: the JUnit plugin — trends and flaky-test spotting

If your pipelines don't publish test results, you're triaging blind. The JUnit plugin ingests XML reports from effectively every framework (Surefire, pytest, Jest via reporters, go test via converters):

post {
  always {
    junit testResults: '**/build/test-results/**/*.xml', skipPublishingChecks: true
  }
}

Put it in post { always } — not as a build step — so results are captured even when the stage fails, which is exactly when you need them.

Once results flow, the job page grows a Test Result Trend chart. Read it like this:

  • Flat red line at a constant count — a permanently failing test that everyone has learned to ignore. Delete it or fix it; it's training your team to skip red builds.
  • Red count that oscillates between 0 and a small number with no correlated code change — flaky tests. Click into a failing test and open its History view: a pass/fail record across builds. A test that fails every 5th–15th run on unchanged code is flaky, full stop.
  • Sudden jump in total test duration — often the earliest visible sign of a resource-starved agent, before builds actually fail.

Flaky tests deserve a quarantine list, not a retry loop — part one covers why retry-until-green is the most expensive non-fix in CI.

Tool 4: build discarders and workspace cleanup — waste control

Jenkins keeps every build's logs and artifacts forever unless told otherwise, and a controller that's low on disk gets slow, then flaky, then down. Two settings fix 90% of it.

Build discarders. In a declarative pipeline:

options {
  buildDiscarder(logRotator(numToKeepStr: '50', artifactNumToKeepStr: '10'))
}

For the jobs that lack one, the Script Console can audit — and fix — the whole fleet:

import jenkins.model.BuildDiscarderProperty
import hudson.tasks.LogRotator

Jenkins.instance.getAllItems(Job).findAll { it.buildDiscarder == null }.each { job ->
  job.addProperty(new BuildDiscarderProperty(new LogRotator(-1, 50, -1, 10)))
  println "Discarder set: ${job.fullName}"
}

(Pipelines that define options { buildDiscarder(...) } in the Jenkinsfile will override this on their next run — which is fine; the Jenkinsfile is the right home for it.)

Workspace cleanup. Stale workspaces on agents are the other disk sink. The Workspace Cleanup plugin gives you cleanWs():

post {
  always {
    cleanWs(deleteDirs: true, notFailBuild: true)
  }
}

Reserve pre-build cleanup (cleanWs() as the first step) for jobs with proven cache-corruption problems — wiping caches on every run trades disk for minutes of rebuild time.

Tool 5: the REST API — build metadata with curl

Every page in Jenkins has a machine-readable twin at /api/json (remote access API docs). With a user API token (your profile → Security → API Token), you can script the questions you keep asking the UI. The tree parameter keeps responses small:

JENKINS=https://jenkins.example.com
AUTH='you:11your-api-token'

# Every job whose last build failed
curl -s -u "$AUTH" "$JENKINS/api/json?tree=jobs[name,lastBuild[number,result,duration]]" \
  | jq -r '.jobs[] | select(.lastBuild.result == "FAILURE") | "\(.name) #\(.lastBuild.number)"'

# One build's metadata
curl -s -u "$AUTH" "$JENKINS/job/my-app/lastBuild/api/json?tree=number,result,duration,timestamp"

# Per-stage timing for a pipeline build (Pipeline Stage View plugin)
curl -s -u "$AUTH" "$JENKINS/job/my-app/123/wfapi/describe" \
  | jq '.stages[] | {name, status, durationMillis}'

# The last 200 lines of a failing build's log
curl -s -u "$AUTH" "$JENKINS/job/my-app/123/consoleText" | tail -n 200

Ten lines of shell around these endpoints gets you a daily "what failed overnight" digest in Slack — the single highest-leverage script most Jenkins teams never write.

Tool 6: retry and timeout wrappers, done right

retry and timeout are the most misused steps in the pipeline vocabulary. Two rules make them useful instead of harmful.

Rule 1: nesting order is a decision, not a detail.

// Per-attempt budget: each of 2 attempts gets its own 20 minutes
retry(2) {
  timeout(time: 20, unit: 'MINUTES') {
    sh './deploy-to-staging.sh'
  }
}

// Total budget: both attempts must finish inside 20 minutes combined
timeout(time: 20, unit: 'MINUTES') {
  retry(2) {
    sh './deploy-to-staging.sh'
  }
}

Per-attempt is usually what you want for infrastructure steps; total-budget is what you want when the stage must not delay the pipeline past an SLA.

Rule 2: retry infrastructure, never tests. Wrapping a test stage in retry(3) converts every flaky test into three times the compute and a green checkmark that means nothing. Modern declarative pipelines can express this precisely — retry only when the agent died, not when the code failed:

stage('Integration tests') {
  options {
    retry(count: 2, conditions: [agent(), nonresumable()])
  }
  steps {
    sh './run-integration-tests.sh'
  }
}

That retries on agent disconnection (a spot instance reclaimed mid-build) and refuses to retry a genuine test failure — the correct default for every stage that runs on ephemeral agents.

Where DIY triage stops

Run all of this and you'll have something rare: a Jenkins installation you can actually see into. Be honest about what you still won't have.

The dashboards show symptoms, not causes. The trend chart says a test is flaky; it doesn't say the test shares a port with another parallel branch. The queue query says builds wait 40 minutes; it doesn't say which label to add capacity to.

Root-causing a red build is still human work. Someone still opens the log, distinguishes an infra failure from a test failure from a real regression, correlates it with the commit and the agent's state, and decides. On a busy controller that's a rotating tax of several engineer-hours a week, paid by your most senior people.

The scripts are point-in-time. The Groovy snippets describe this morning. Nobody re-runs them at 2 a.m. when the queue actually melts down.

What comes next

Everything above — reading the failing log, classifying the failure, correlating it with the commit and the node, tracking deployments through stages — is exactly the loop CloudThinker agents run continuously against Jenkins over a read-only API token, with proposed fixes (quarantine the flaky test, resize the agent pool) gated behind your approval. That's part three of this series.

Or see it on your own controller first: Try CloudThinker free — 100 premium credits, no card required.