Sign inBook a Demo

How To

Prometheus Alerting: 6 Failure Modes That Bury Real Incidents

Most Prometheus alerting setups fail the same six ways: pages on causes instead of symptoms, zero-delay rules that flap, scrape targets down for weeks with nobody alerting on up == 0, label cardinality that stalls rule evaluation, a default Alertmanager config that turns one incident into forty notifications, and PromQL mistakes on counters and percentiles. Part one of our Prometheus observability series walks through each failure mode with a copy-pasteable rule or PromQL query, plus the typical noise reduction from fixing it — often 40–70% fewer pages.

ST
·
prometheusobservabilityalertingpromqlalertmanagersremonitoring
Cover Image for Prometheus Alerting: 6 Failure Modes That Bury Real Incidents

Prometheus Alerting: 6 Failure Modes That Bury Real Incidents

Last Tuesday your latency alert fired, correctly, at 2:14 a.m. The on-call engineer spent forty minutes staring at dashboards before finding the actual cause: a node exporter that had been dead for nine days. Nothing ever alerted on it — the target went down, its metrics went stale, and every threshold rule that depended on those metrics simply stopped evaluating to anything at all.

That incident contains most of what goes wrong with Prometheus alerting. The rules you wrote fire loudly on symptoms while the causes rot silently underneath, the rules you didn't write (scrape health, missing data) never fire at all, and Alertmanager forwards the resulting mess to Slack one notification at a time until everyone mutes the channel.

The failure modes are predictable. This guide covers the six that account for almost every noisy, blind, or misleading Prometheus setup — what each one looks like, why it happens, one concrete PromQL or config example, and what fixing it typically buys you.

This is part one of a three-part series. Part two is a hands-on guide to alert automation with the native Prometheus stack — rule files, Alertmanager routing, promtool tests. Part three covers putting an AI agent on the receiving end of your alerts.

1. Paging on causes instead of symptoms

What it is. Rules that page a human because CPU crossed 80%, disk crossed 70%, or a queue has "too many" items — while users are entirely unaffected. Meanwhile there's often no page at all for the thing users actually experience: error ratio and latency.

Why it happens. Cause-based rules are easy to write on day one. node_cpu is right there; your service's error ratio requires instrumentation and a real SLO conversation. So the easy alerts accumulate and the important ones never get written. The Google SRE book's monitoring chapter has been making this argument since 2016: page on symptoms, use causes for debugging context.

What good looks like. One symptom-based page per service, expressed as a ratio of counters:

- alert: HighErrorRatio
  expr: |
    sum(rate(http_requests_total{status=~"5.."}[5m]))
      /
    sum(rate(http_requests_total[5m])) > 0.05
  for: 10m
  labels:
    severity: page
  annotations:
    summary: "5xx ratio above 5% for 10 minutes on {{ $labels.job }}"

Demote the CPU and disk rules to severity: ticket — they're context, not emergencies.

Typical impact. Teams that move pages from causes to symptoms typically cut page volume by 40–70% while catching more user-facing incidents, because a symptom rule fires no matter which of a hundred causes produced it.

2. Zero-delay alerts flap

What it is. An alerting rule with no for: clause fires the instant its expression is true for a single evaluation — one garbage-collection pause, one slow scrape, one retried batch job. Thirty seconds later it resolves. Then it fires again. PagerDuty calls this pattern "flapping"; your on-call rotation calls it something less printable.

Why it happens. for: is optional, and the failure mode of omitting it isn't visible in testing — it only shows up as noise in production, weeks later, one spike at a time.

How to fix it. Give every paging rule a for: duration that the condition must hold continuously:

- alert: HighErrorRatio
  expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
  for: 10m

Rule of thumb: for: should be at least two evaluation intervals (so a single bad scrape can't page anyone), and for most pages 5–15 minutes is right. Yes, this adds detection latency — that's the deliberate trade. An incident that self-heals in four minutes did not need a human awake at 2 a.m. The alerting rules documentation covers the mechanics, including keep_firing_for for the mirror-image problem of alerts that resolve and re-fire during a slow recovery.

Typical impact. Adding sensible for: durations to a rule set that lacks them commonly eliminates half or more of all firings — the transient half nobody should have seen.

3. The silent failure: nobody alerts on up == 0

What it is. For every scrape target, Prometheus synthesizes an up metric: 1 if the scrape succeeded, 0 if it failed. When an exporter dies, up goes to 0 and — this is the dangerous part — every other series from that target goes stale within about five minutes. Your threshold alerts on those metrics don't fire. They can't. There's no data to compare against a threshold. The monitoring gap is itself invisible to the monitoring.

Why it happens. up doesn't belong to any service team, so nobody writes the rule. And the failure is quiet: dashboards show flat lines or gaps, which look a lot like "nothing happening."

How to detect it. Right now, run this in the Prometheus expression browser (or check Status → Targets in the UI):

up == 0

Then make it permanent:

- alert: TargetDown
  expr: up == 0
  for: 5m
  labels:
    severity: page
  annotations:
    summary: "{{ $labels.job }} target {{ $labels.instance }} unreachable for 5 minutes"

Typical impact. Not a noise problem — a blindness problem. In audits of real-world Prometheus setups it's common to find several percent of targets down, some for weeks, exactly like the node exporter in the opening story.

4. Label cardinality pressure

What it is. Every unique combination of label values is a separate time series. Put a user ID, raw URL path, or request UUID into a label and one metric becomes a million series. Prometheus memory climbs, queries slow down, and — the alerting-specific consequence — rule groups start missing their evaluation deadlines, which means alerts fire late or intermittently for reasons that look like ghosts.

Why it happens. A developer adds path as a label because it seems useful, and it is, right up until someone runs a URL scanner against the service and mints 400,000 series in an afternoon.

How to detect it. Check Status → TSDB Status in the Prometheus UI for the highest-cardinality metric names and label pairs, or query for the worst offenders:

topk(10, count by (__name__)({__name__=~".+"}))

(That query is expensive — run it on a quiet Prometheus, not in a dashboard refresh loop.) Watch prometheus_tsdb_head_series over time for step changes, and prometheus_rule_group_iterations_missed_total for the smoking gun that cardinality is already hurting rule evaluation.

Typical impact. A single unbounded label can multiply a metric's series count 100× or more. Cleaning up the top two or three offenders often halves head-series count and restores rule evaluation to schedule.

5. Alertmanager left on defaults

What it is. Prometheus decides what is wrong; Alertmanager decides who hears about it and how often. A near-default Alertmanager config sends every alert to one receiver with generic grouping — so when a node dies and 40 rules fire, on-call gets a wall of notifications where the one that matters (the node is down) is buried under 39 consequences of it.

Why it happens. Alertmanager configuration is a separate file, a separate mental model (routes, grouping, inhibition), and it works badly by default rather than failing loudly — so it never makes it up the backlog.

What good looks like. Group related alerts into single notifications, and inhibit the consequences when the cause is already firing:

route:
  receiver: team-default
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

inhibit_rules:
  - source_matchers:
      - alertname = TargetDown
    target_matchers:
      - severity = warning
    equal: ['instance']

With that inhibition rule, the moment TargetDown fires for an instance, every warning-level alert for the same instance is suppressed — you get one notification about the cause instead of a storm about the effects.

Typical impact. Grouping alone commonly collapses notification volume 5–10× during real incidents; inhibition removes most of what remains.

6. Most good alerts use the same three PromQL patterns

What it is. The long tail of bad rules — raw counter comparisons, averaged percentiles, thresholds on gauges that vanish — mostly comes from not knowing three PromQL patterns. These are the PromQL examples worth memorizing:

rate() on counters, always. A counter's absolute value is meaningless (it resets on restart); its rate is the signal. Never alert on http_requests_total > 1000000; alert on the rate over a window at least 4× your scrape interval:

sum by (job) (rate(http_requests_total{status=~"5.."}[5m])) > 5

absent() for data that should exist. A threshold rule on a metric that disappears evaluates to nothing and fires never. If a job's heartbeat metric matters, alert on its absence explicitly:

absent(up{job="billing"})

This catches the case section 3 doesn't: the job vanished from service discovery entirely, so there's no up series left to equal zero.

histogram_quantile() for latency. Never average percentiles across instances — it's mathematically wrong. Aggregate the histogram buckets first, then take the quantile:

histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
) > 0.5

The PromQL functions documentation covers the sharp edges of all three.

Typical impact. These patterns are the difference between rules that fire on real conditions and rules that fire on arithmetic artifacts. Most rule-set audits find 20–40% of existing rules misusing counters or percentiles.

Alert tuning never stays done

Here's the part that makes "we cleaned up our alerts last quarter" a temporary statement: every one of these failure modes regenerates. Each new service ships with cause-based rules and no for:. Each new exporter is a new target nobody watches up for. Each new label is a cardinality bet. Each new team bolts another receiver onto Alertmanager without touching grouping.

And even a perfectly tuned rule set only solves detection. When HighErrorRatio fires at 2 a.m., someone still has to run the next five PromQL queries, correlate the symptom with a cause, and decide what to do — which is exactly the forty minutes the engineer in the opening story lost.

Start by fixing the rules — part two of this series walks through the full DIY toolkit: rule files, promtool unit tests, Alertmanager routing and silences. But know that the investigation step is where the real time goes.

When the alert fires, then what?

Detection is half the job. CloudThinker agents connect to your Prometheus and Alertmanager APIs read-only, pick up firing alerts, and run the follow-up queries a human would — correlating the symptom across related series, checking scrape health, comparing against recent changes — then present a likely cause with evidence. Remediation stays gated behind your approval. Try CloudThinker free — 100 premium credits, no card required — or continue with the hands-on native-tooling guide.