New Relic Automation with AI Agents: From Alert to Root Cause

The alert says the error rate on checkout-service doubled. That is one sentence of information. Between that sentence and actually knowing why sit roughly twenty NRQL queries: errors faceted by transaction, then by error class, latency percentiles for the hour around the spike, throughput compared against last week, external service durations, and finally the question everyone should have asked first — did anything deploy? An experienced SRE runs that sequence in twenty to forty minutes. The alert condition that fired took milliseconds. Everything expensive happens after it.

This is the final part of our New Relic automation series, and it is about closing exactly that gap. Part one mapped the problem: alert condition sprawl, static thresholds on dynamic services, and the APM signals that actually predict incidents. Part two built everything New Relic's native tooling offers — NRQL conditions, policies, workflows, webhooks with custom payloads, muting rules, and NerdGraph-driven condition management — and ended at the honest ceiling: workflows route incidents; they do not read the APM data for you.

CloudThinker agents are the layer that does. Your alert conditions, dashboards, and instrumentation stay exactly where they are — this is not a New Relic replacement. Agents connect read-only, pick up violations as they open, run the investigation sequence an SRE would run, and propose (or, with explicit permission, apply) the fix.

Connecting New Relic: a user API key, about five minutes

The connection needs three things: a user API key, your account ID, and your data center region (US or EU). Create the key under your user menu → API keys in New Relic — or, better, create a dedicated user assigned a read-only role and generate the key from that user, so the key can never do more than its owner can. New Relic's API key documentation covers the key types; the user key is the one that works with NerdGraph, which is how the agents query everything.

Paste the key and account ID into CloudThinker, and the first scan starts. No agent installs, no changes to your instrumentation, no write access at connection time. The connections guide has the step-by-step, including the recommended read-only role setup for the security review.

The investigation, run the way an SRE would run it

When an alert condition opens an incident, the agent starts with the same NRQL an on-call engineer would type — except it starts within seconds, and it types faster.

First, break the errors down. Not "errors are up" but which transactions, which error classes:

SELECT count(*) FROM TransactionError
WHERE appName = 'checkout-service'
FACET transactionName, error.class
SINCE 30 minutes ago

If 94% of the new errors are Net::ReadTimeout on a single transaction that calls the payment provider, the investigation is already half over. If errors are spread across every transaction evenly, it points at the runtime or the infrastructure underneath — a different path entirely.

Second, frame the incident window with latency percentiles. Averages hide bimodal failures; percentiles around the incident window show whether everything got slow or a subset fell off a cliff:

SELECT percentile(duration, 50, 95, 99) FROM Transaction
WHERE appName = 'checkout-service'
TIMESERIES 5 minutes SINCE 2 hours ago

The agent also compares throughput against the same window a week earlier (COMPARE WITH 1 week ago), because "error rate doubled" during a traffic drop means something very different from during a traffic spike.

Third, check for changes. Most production incidents trace back to a deployment or a configuration change, and it is the check humans run last because it lives in a different tab. The agent queries New Relic's change tracking data through NerdGraph for markers on the affected entity and everything related to it in the relevant window — the part-one point about entity relationships nobody uses during triage is exactly where this pays off. A deploy on an upstream service four minutes before your error spike is the likely cause, and the entity graph is how the agent finds it without anyone remembering the service topology.

Then it writes down the answer. Not a forwarded alert — a finding: what fired, the error breakdown, the latency shape, the correlated change, the most likely cause with the supporting query results attached, and a proposed remediation with a rollback note. It lands in the same Slack channel your part-two workflow already posts to, arriving as a diagnosis instead of a link.

The difference from the workflows you built in part two is branching. A workflow executes the steps you predicted in advance. The agent chooses the next query based on what the last one returned — timeout errors lead to external service checks; uniform errors lead to infrastructure checks — the same way an engineer reasons, with the reasoning logged.

Graduated autonomy: you set what agents may touch

Every action class carries an autonomy level, configured per environment:

• Notify — investigate and report. Nothing else. The default for everything, and where every team should start.
• Suggest — propose the specific remediation with expected impact and rollback steps, then wait.
• Approve — stage the action; execute only after a named human approves in chat.
• Autonomous — execute and report. Reserved, in practice, for reversible actions in non-production — retiring a confirmed-flappy condition in staging, say — after weeks of watching the agent be right at the Approve level.

Escalation stays intact at every level. If the evidence says real outage rather than noise, the agent pages through your existing path — with the investigation already attached to the page. Everything — every query run, finding produced, approval given, change applied — lands in an audit trail.

What a first scan typically finds

Alongside live incident response, the first connection runs a hygiene pass over your alerting estate — the review from parts one and two, compressed into minutes. The numbers below are illustrative: a composite for a mid-market environment with 30–60 instrumented services and a few hundred alert conditions. Yours will differ.

Note the shape: almost none of these rows say "add more monitoring." They say your existing alerting is lying to you in specific, fixable ways — and each row arrives with a proposed, approval-gated fix, most of them one NerdGraph mutation away.

Prompts to try in your first session

CloudThinker is conversational. You ask in plain language, and the agent answers with the NRQL and results cited, so you verify rather than trust:

"The error rate on checkout-service doubled around 14:30 UTC. Break the errors down by transaction and error class, pull latency percentiles for the two hours around it, and check for deployments on this service and anything it depends on. Notify only."

"Which alert conditions opened more than ten incidents last week that nobody acted on, and what threshold or aggregation-window changes would quiet them without losing real signal?"

"List every service reporting APM data that has no alert condition on error rate or latency, ranked by throughput."

Run the first one during your next real incident and compare it against what on-call assembled by hand. That comparison — not this article — is what should decide whether agents earn a place in your escalation path.

What the agents will not do

An AI agent connected to production telemetry should raise questions. The answers:

• Read-only by default. The connection key reads; it does not write. Changing anything in New Relic — thresholds, muting rules, workflow routing — requires write permission and an autonomy level above Notify for that action class.
• No silent changes. Below Autonomous, nothing changes without a named human approving in chat, and Autonomous is set per action class, per environment — never globally.
• No skipped escalations. The agent never decides an incident is not worth waking someone for. Suppression is yours to configure, explicit, and logged.
• No unauditable actions. Every query, finding, proposal, and change is in the audit trail with its evidence.

If you built the part-two automation, keep it. Webhooks and workflows are the right tool for deterministic routing. Agents take the part that was never automatable before: reading the data and deciding what it means.

The distance that remains

Part one showed which APM signals matter and why static alerting drowns them in noise. Part two took native automation to its ceiling: incidents routed, enriched, correlated — and still uninvestigated. The twenty NRQL queries between "error rate doubled" and "the 14:28 deploy to the payments gateway caused it" were, until now, the irreducibly human part of on-call. Teams that put an agent layer on that part typically cut time-to-diagnosis from half an hour to a few minutes, and clean up the alert hygiene debt from part one as a side effect.

Try CloudThinker free — 100 premium credits, no card required — and follow the connections guide to watch your next incident arrive already diagnosed.