New Relic Alert Automation with NRQL, Workflows, and NerdGraph

New Relic's native alerting stack will take you surprisingly far: a well-written NRQL condition catches the error-rate spike within a minute or two, a workflow routes it to the right Slack channel with context attached, and Applied Intelligence folds the three related incidents into one issue. That's real New Relic automation, built entirely from features you're already paying for. And then the notification lands, and a human opens six tabs — the APM summary, the errors inbox, distributed traces, the deployments page, the infrastructure host, and the dashboard someone built two years ago — because the pipeline that detected and routed the problem cannot investigate it.

This guide builds that pipeline properly, tool by tool, using nothing outside the platform. In part one of this series we covered which APM signals deserve alerts in the first place — error rate, latency percentiles, throughput shifts, external service degradation — and the NRQL behind each. Here we turn those queries into a production alerting system: conditions, policies, workflows, webhooks, muting rules, and managing the whole thing as code through NerdGraph. Budget half a day for a first serious pass on one team's services.

Step 1: NRQL alert conditions — get the signal math right

Every alert worth having on New Relic is a NRQL alert condition: a query evaluated on a rolling window, compared against a threshold. Console path: Alerts → Alert conditions → New alert condition → Write your own query.

Start from queries like the ones from part one:

-- Error rate for one service
SELECT percentage(count(*), WHERE error IS true)
FROM Transaction
WHERE appName = 'checkout-api'

-- p95 latency, seconds
SELECT percentile(duration, 95)
FROM Transaction
WHERE appName = 'checkout-api'

The query is the easy half. Three settings decide whether the condition is trustworthy:

Aggregation window and method. The window (default 60 seconds) is the bucket each data point is computed over. The method decides when a bucket is considered complete. EVENT_FLOW (the default) closes a bucket when newer data arrives — right for steady, high-throughput telemetry like APM transactions. EVENT_TIMER closes a bucket a fixed time after its last data point — use it for sparse or batchy data such as cloud integration metrics or infrequent log events, where event flow can evaluate a bucket before late data lands and fire falsely. If your condition on a low-traffic queue keeps flapping, this setting is usually why.

Threshold duration. "Above 5% at least once in 5 minutes" pages you for a single bad bucket; "above 5% for at least 5 minutes" waits out blips. For paging conditions, prefer for at least, sized to your real tolerance. Keep a warning threshold below the critical one so trends surface before they page.

Loss of signal. A crashed service sends no data, and a condition on missing data evaluates nothing — it stays silently green. Set the signal-loss expiration (e.g. 10 minutes) and choose open a new incident on signal loss for anything whose disappearance matters. This is the difference between "the error rate is fine" and "the service is gone."

For dynamic services where any static threshold is wrong half the day, use an anomaly condition instead of static: New Relic builds a baseline from recent behavior and you alert on deviation. Good for throughput and latency on traffic with strong daily shape; still wrong on the day of a marketing launch — which is what muting rules are for (step 4).

Step 2: Policies and incident preference — control the blast radius

Conditions live in alert policies, and each policy carries one deceptively important setting: incident preference, which controls how incidents roll up into issues (the things that actually notify).

• One issue per policy (default): everything in the policy, one issue at a time. Quiet, but a new failure gets swallowed by an issue that's already open for something else.
• One issue per condition: each condition gets its own issue. The sane default for most teams.
• One issue per condition and signal: every entity/facet gets its own issue. Maximum granularity — and maximum noise when 40 hosts degrade together.

Organize policies by ownership, not by technology: a checkout-team policy that pages the checkout on-call beats a global latency policy that pages everyone. Alert-condition sprawl — hundreds of conditions across dozens of policies nobody remembers creating — starts with policies organized any other way.

Step 3: Workflows and destinations — routing with context

Notification routing is two objects: destinations (a configured Slack workspace, PagerDuty service, email, Jira project, ServiceNow instance, or generic webhook) and workflows, which filter issues and send them to destinations. Console: Alerts → Workflows → Add a workflow.

Two workflow features do real work:

Enrichments. A workflow can run a NRQL query when an issue fires and attach the result to the notification — for example, the current error breakdown for the affected service:

SELECT count(*) FROM TransactionError
WHERE appName = 'checkout-api'
FACET `error.class` SINCE 30 minutes ago

This is the closest native New Relic gets to automated investigation: one predefined query, attached to the page. Use it — an on-call engineer who sees the top error class before opening the laptop is minutes ahead.

Webhook destinations with custom payloads. For anything downstream — a ticketing system, an automation runbook, your own service — send a webhook and shape the JSON with a custom payload using message templates (Handlebars-style variables):

{
  "issueId": {{ json issueId }},
  "title": {{ json annotations.title.[0] }},
  "priority": {{ json priority }},
  "state": {{ json state }},
  "policies": {{ json accumulations.policyName }},
  "conditions": {{ json accumulations.conditionName }},
  "entities": {{ json entitiesData.names }},
  "issueUrl": {{ json issuePageUrl }}
}

The state field is CREATED, ACTIVATED, or CLOSED — handle all three in your receiver, or you'll open tickets you never close. This webhook is also the standard integration point for anything acting on New Relic alerts from outside, which matters in part three.

Step 4: Muting rules — schedule the silence

Every deploy window and maintenance job that pages someone teaches on-call to ignore pages. Muting rules (Alerts → Muting rules) suppress notifications for incidents matching your criteria — a specific policy, condition, entity, or tag — on a one-off or recurring schedule. Two habits keep them safe: scope tightly (mute the batch-etl entity during its window, not the whole policy), and never create an unscheduled, open-ended rule — that's how a real outage goes unnoticed for six hours. Incidents still open while muted; only the notification is suppressed, so the history stays intact.

Step 5: Applied Intelligence — let correlation shrink the noise

New Relic's incident correlation ("decisions") merges related incidents into one issue based on shared entities, timing, and learned patterns — a database failure and the four service-latency incidents it caused become one notification instead of five. It's on by default with built-in global decisions, and you can define your own correlation logic under Alerts → Correlate → Decisions. Correlation typically cuts notification volume meaningfully, but understand what it does and doesn't do: it groups symptoms that belong together. It does not tell you which of the five correlated incidents is the cause and which are downstream.

Step 6: NerdGraph — manage conditions as code

Once you're past a handful of conditions, click-ops stops scaling: thresholds drift, naming diverges, nobody knows which conditions are load-bearing. The NerdGraph API manages the whole alerting surface programmatically. Creating a static NRQL condition:

mutation {
  alertsNrqlConditionStaticCreate(
    accountId: 1234567
    policyId: "987654"
    condition: {
      name: "checkout-api p95 latency"
      enabled: true
      nrql: {
        query: "SELECT percentile(duration, 95) FROM Transaction WHERE appName = 'checkout-api'"
      }
      terms: [{
        threshold: 1.5
        thresholdDuration: 300
        thresholdOccurrences: ALL
        operator: ABOVE
        priority: CRITICAL
      }]
      signal: {
        aggregationWindow: 60
        aggregationMethod: EVENT_FLOW
        aggregationDelay: 120
      }
      expiration: {
        expirationDuration: 600
        openViolationOnExpiration: true
      }
    }
  ) {
    id
    name
  }
}

Run it in the NerdGraph explorer at https://api.newrelic.com/graphiql with a user API key. The matching alertsNrqlConditionsSearch query lets you export every existing condition — do that first and diff it against what you think you have; the gap is usually educational. If you'd rather not hand-write GraphQL, the New Relic Terraform provider wraps the same API, and either way you get review, versioning, and consistent thresholds across services.

The ceiling: workflows route incidents, they don't read the trace

Build all six steps and you have a genuinely good alerting system: trustworthy conditions, sane grouping, context-bearing notifications, scheduled silence, correlated issues, and configuration in git. Be honest about what you don't have.

Routing is not investigation. When the p95 condition fires, the workflow delivers the issue — it doesn't query the error breakdown by transaction, pull latency percentiles around the incident window, check for a deployment marker ten minutes earlier, or walk entity relationships to see that the payment service's upstream dependency degraded first. Enrichments attach one predefined query; real triage is a sequence of queries where each answer decides the next. That sequence still runs in a human's head, across those six tabs, at whatever hour the page landed.

The configuration decays. Services change shape; thresholds set in March are wrong by August. NerdGraph makes updating conditions cheap, but noticing they need updating is still an engineer reading charts.

Correlation groups, it doesn't conclude. Five incidents in one issue is better than five pages — but "which one is the cause" remains manual work, every time.

Native tooling automates everything up to the moment understanding is required. That moment is the expensive one.

What comes next

The investigation itself — querying NRQL the way an SRE would, correlating across entities, naming the likely cause with evidence attached — is what CloudThinker agents add on top of New Relic. They connect read-only in about five minutes, pick up the same issues your workflows route, and work the incident before a human opens a single tab, with any remediation gated behind your approval. That's part three of this series.

Or see it on your own account first: Try CloudThinker free — 100 premium credits, no card required.