Automating SigNoz Alerts with Native Tools: Rules, Channels, Audits

Open the Alerts page of a SigNoz install that's two years old and count the rules. A team of eight typically finds 60–120 of them: rules named test-alert-2, rules pointing at services renamed three quarters ago, four separate rules watching the same latency metric with four different thresholds. Adding a rule takes ninety seconds; deleting one requires knowing it's safe to delete, so nobody does. The result is a SigNoz integration that pages more and means less every sprint.

In part one of this series we walked the p99 latency investigation workflow and the signals that matter on an OpenTelemetry-native stack. This part is the hands-on companion: auditing the alert rules you already have, rebuilding the ones worth keeping with SigNoz's native alerting features — query builder, PromQL, ClickHouse queries, evaluation windows, notification channels including webhooks — and knowing exactly where the DIY ceiling sits. Everything here uses only what ships with SigNoz; nothing to install.

Step 1: Audit what you have before adding anything

SigNoz exposes its alert rules over the same API the UI uses. Create a personal access token under Settings → API Keys (docs), then pull the full rule inventory:

curl -s -H "SIGNOZ-API-KEY: $SIGNOZ_TOKEN" \
  "https://signoz.your-domain.com/api/v1/rules" \
  | jq '.data.rules | length'

Dump the fields that matter for the audit:

curl -s -H "SIGNOZ-API-KEY: $SIGNOZ_TOKEN" \
  "https://signoz.your-domain.com/api/v1/rules" \
  | jq -r '.data.rules[] | [.id, .alert, .state, .disabled, .createBy, .updateAt] | @tsv' \
  | column -t -s $'\t'

Now classify every rule into one of four buckets:

Stale. Rules that reference services or attributes that no longer emit data. Their state sits at inactive permanently — not because things are healthy, but because the query returns nothing. Cross-check each rule's target against Services (the service list only shows what's actually reporting). A rule that cannot fire is worse than no rule: it looks like coverage.

Noisy. Open a suspect rule and check its History tab (Alerts → click the rule → Alert History), which shows firing frequency and duration over the past weeks. Anything firing daily that never produced a human action is noise. The usual causes: a threshold set from a bad week ("alert at 500ms because we saw 480ms once"), or an evaluation window so short that normal variance trips it.

Overlapping. Sort your dump by the metric or service each rule targets. Four rules on checkout latency at 400ms/500ms/800ms/1s means one incident produces four pages. Keep two at most: a warning-severity rule at the level someone should look, and a critical-severity rule at the level someone should act.

Keepers. Rules mapped to a symptom a user would notice (latency, error rate, saturation about to become latency), with a threshold someone can defend. These are usually 20–30% of the total.

Disable — don't delete — the first three buckets for two weeks. If nothing was missed, delete. The audit takes two to three hours and typically cuts page volume 40–60% before you've tuned anything.

Step 2: Rebuild rules with the right query type

SigNoz alert rules run against any signal — metrics, traces, logs, exceptions — and each rule is built from one of three query flavors (docs). Choosing the right one is most of the work.

Query builder: the default

Alerts → New Alert Rule → Metrics-based Alert gets you the visual builder: pick a metric, aggregation, group-by, threshold. For the standard OTel span metrics that SigNoz generates from your traces, a p99 latency rule on one service looks like: metric signoz_latency, aggregation p99, filter service_name = checkout, threshold above 800 ms.

The two settings engineers skip and shouldn't:

Evaluation window (evalWindow): how much data each evaluation considers. Five minutes is the sane default for latency; one minute makes every GC pause a page.
Match condition: at least once vs all the time vs on average vs in total within the window. For p99 latency, on average over 5 minutes pages on sustained degradation; at least once pages on a single bad interval. Most noisy latency rules from your audit are "at least once with a 1-minute window" — rebuild them as "on average over 5 minutes" and the flapping stops.

PromQL: when you already know the expression

The same rule in PromQL, for teams migrating from a Prometheus stack:

histogram_quantile(0.99,
  sum(rate(signoz_latency_bucket{service_name="checkout"}[5m])) by (le)
) > 800

The signoz_latency histogram is recorded in milliseconds, so the threshold stays in ms — the same 800 you set in the builder.

PromQL rules are the right choice when you're porting existing Prometheus alert rules verbatim, or when you need functions the builder doesn't expose (predict_linear for disk-full forecasting, absent() for missing-data alerts — which, note, is also how you make the "stale rule" failure mode from your audit page you instead of staying silent).

ClickHouse queries: alerts on raw trace and log data

Because SigNoz stores everything in ClickHouse, you can alert on queries the metric pipeline can't express — for example, p99 computed from raw spans for one endpoint only:

SELECT
  toStartOfInterval(timestamp, INTERVAL 1 MINUTE) AS interval,
  quantile(0.99)(durationNano) / 1e6 AS p99_ms
FROM signoz_traces.distributed_signoz_index_v2
WHERE serviceName = 'checkout'
  AND name = 'POST /api/pay'
  AND timestamp BETWEEN {{.start_datetime}} AND {{.end_datetime}}
GROUP BY interval;

The start_datetime / end_datetime template variables are filled in by the rule engine at evaluation time (docs). Note the table name: installs older than roughly v0.64 use signoz_index_v2 as above; newer versions moved to signoz_index_v3 with a different column layout, so run SHOW TABLES FROM signoz_traces on your cluster before copying. This power comes with a cost warning — a ClickHouse alert query runs at every evaluation interval, so an unbounded scan over a week of spans will hurt the very cluster you're monitoring. Always keep the timestamp bounds.

Log-based alerts follow the same pattern against signoz_logs.distributed_logs — the classic use case is paging when severity_text = 'ERROR' counts for one service jump above baseline.

Step 3: Notification channels — and what webhooks actually give you

Channels live under Settings → Alert Channels (docs): Slack, PagerDuty, Opsgenie, email, Microsoft Teams, and the one that matters for automation — webhook.

SigNoz's alert delivery is Alertmanager-compatible, so a webhook channel POSTs the standard payload to your endpoint:

{
  "receiver": "ops-webhook",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "checkout-p99-critical",
        "severity": "critical",
        "service_name": "checkout",
        "ruleId": "42"
      },
      "annotations": {
        "description": "P99 latency on checkout above 800ms for 5m",
        "summary": "checkout latency breach"
      },
      "startsAt": "2026-07-13T04:12:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "fingerprint": "c1f9a2b8e77d4c01"
    }
  ],
  "groupLabels": { "alertname": "checkout-p99-critical" },
  "version": "4"
}

Everything your receiving automation needs is here: fingerprint for deduplication, startsAt to anchor the incident window, labels to route on. Typical DIY builds on top of this payload: a small service that opens a Jira ticket per fingerprint, a Lambda that posts the alert plus a pre-built SigNoz trace-search URL into the incident channel, or a bridge into a runbook automation tool. Use the Test button on the channel before trusting it — a webhook receiver that 500s fails silently otherwise.

Routing convention that keeps channels sane: label rules with severity, send critical to the pager, warning to Slack only. One rule, one severity, one destination.

Step 4: Dashboards worth keeping

The same audit logic applies to dashboards — most instances accumulate dozens, and four earn their keep:

Per-service golden signals — rate, error %, p50/p99 duration from the built-in Services view; you get this free, don't rebuild it.
Alert-adjacent dashboards — one per critical alert rule, showing the alerting query plus its likely causes (latency alert → dashboard with latency, error rate, downstream call duration, host saturation). Link it in the rule's annotation so the page carries its own context.
Deploy correlation — your key latency panels with deployment markers, since "what changed" is the first triage question.
ClickHouse self-monitoring — disk, ingest rate, query latency of the SigNoz cluster itself. A self-hosted observability stack that falls over during an incident is a story you only want to live once.

Delete or archive the rest. A dashboard nobody opened in 90 days is UI debt.

Where SigNoz sits next to Datadog (the honest version)

The signoz-vs-datadog question comes up in every one of these audits, and it deserves a neutral answer. SigNoz gives you an OTel-native pipeline, your data in your own ClickHouse, and a cost model that scales with your hardware rather than per-host or per-GB list pricing — but you operate the cluster, and the alerting features above are yours to configure and maintain. Datadog gives you managed polish, a larger integration catalog, and mature correlation features out of the box — at a per-unit price that mid-market teams feel at scale, and with your telemetry living in someone else's system. Neither is wrong; the trade is control and cost predictability versus operational outsourcing. Everything in this guide is the price of the control side.

The ceiling: channels deliver alerts, they don't read traces

Run the audit, rebuild the rules, wire the webhooks, and you have a genuinely good alerting setup. Be honest about what you don't have:

The audit is point-in-time. Rules regrow. Every incident retro spawns a new rule; nobody schedules the pruning. The 90-rule pile you just cleaned is where you'll be again in 12–18 months unless auditing becomes a calendar event.

Thresholds decay. The 800ms threshold that was right at last quarter's traffic is wrong after the next architecture change. Static rules don't renegotiate themselves.

Webhooks move alerts; they don't investigate them. The payload above can open a ticket and post a link. It cannot open the trace, compare the incident window against baseline, notice that p99 broke four minutes after a deploy, or read the ClickHouse saturation panel. When the page arrives at 3 a.m., the query builder, the trace waterfall, and the correlation work are still a human's job — the exact workflow from part one, executed by whoever is on call, every time.

That last gap — between a delivered alert and an understood one — is what part three of this series addresses.

What comes next

The investigation your webhook can't do is automatable. CloudThinker agents connect to SigNoz read-only, receive the firing alert, query the traces, metrics, and logs around the incident window the way an engineer would, correlate with deploys and cloud state, and propose remediation under graduated autonomy — nothing changes without the approval level you configure. That's part three.

Or see it against your own stack now: Try CloudThinker free — 100 premium credits, no card required.