RabbitMQ Monitoring: 6 Signals That Predict an Outage

A queue grew for three days before anyone noticed. Its single consumer had crashed on a Tuesday night — no exception in the app logs anyone was watching, no alert, just a channel that closed and never reopened. Publishers kept publishing. By Friday the queue held 2.1 million messages, the node hit its memory watermark, and every producer in the system blocked at once. The postmortem also surfaced a dead letter queue with 14 months of messages in it. Nobody knew it existed.

That incident is what RabbitMQ monitoring is actually for. Not dashboards for their own sake — catching the six failure patterns that broker outages are made of, days before they become outages. RabbitMQ is unusually honest about its internal state: nearly everything you need is exposed through rabbitmqctl, the management UI, and the management HTTP API. The problem is knowing which of the hundreds of numbers matter and what "bad" looks like for each.

This guide covers the six signals that matter, how each failure develops, one detection step for each, and the business impact when it's missed. This is part one of a three-part series: part two is a hands-on DIY guide to RabbitMQ monitoring with native tools only; part three covers putting an AI agent on top of the management API to watch and investigate continuously.

1. Queue depth growing faster than consumers drain it

What it is. The fundamental equation of any queue: publish rate versus delivery rate. When publishing outpaces consuming for long enough, depth grows without bound — until memory or disk pressure stops the whole broker.

Burst or leak? A burst is a spike in publish rate with a healthy consume rate behind it — depth climbs, then drains once the spike passes. A leak is a consume rate that dropped (a consumer crashed, slowed down, or is stuck) while publishing stayed constant — depth climbs in a straight line and never comes back. The distinction matters: a burst is capacity planning; a leak is an incident. You can't tell them apart from a single depth number. You need depth and both rates over time.

How to detect it. The quickest snapshot:

rabbitmqctl list_queues name messages messages_ready messages_unacknowledged consumers

Any queue with growing messages, a nonzero backlog, and consumers at 0 is your three-day incident waiting to happen. For rates, the management API exposes them per queue:

curl -s -u monitoring:password \
  'http://localhost:15672/api/queues/%2f/orders' | \
  python3 -c "import sys,json; q=json.load(sys.stdin); \
  print(q['messages'], q['message_stats']['publish_details']['rate'], \
  q['message_stats']['deliver_get_details']['rate'])"

Publish rate persistently above deliver rate is a leak, full stop. In the management UI: Queues → (queue) → Overview shows both rates on the same chart.

Typical impact. The slow-motion outage: hours of stale downstream data before anything visibly breaks, then a broker-wide failure when the backlog hits a resource limit. Consumers of that queue — order processing, notifications, billing events — silently fall behind first.

2. Unacked messages piling up

What it is. messages_unacknowledged counts messages delivered to a consumer that hasn't acked them yet. A steady low number is normal pipelining. A high, flat, non-moving number means consumers are taking deliveries and not finishing them — stuck on a dead downstream dependency, deadlocked, or holding a prefetch buffer far bigger than they can chew through.

Why it happens. Two classic causes. First, prefetch set too high (or unlimited): a consumer grabs 5,000 messages into memory, dies mid-batch, and all 5,000 get redelivered — potentially to the same broken consumer, in a loop. Second, a consumer that processes message 1 and then blocks forever on message 2, pinning everything already delivered to its channel.

How to detect it. Look at unacked counts per channel, alongside prefetch:

rabbitmqctl list_channels connection name prefetch_count messages_unacknowledged

A channel where messages_unacknowledged equals prefetch_count and stays there is a stuck consumer: it has taken its full window and acked nothing since. In the management UI, Channels shows the same columns sortable. RabbitMQ's own guidance covers sane prefetch values in the consumer prefetch docs — for most workloads a prefetch between 30 and 300 beats both extremes.

Typical impact. Throughput collapses while every surface-level metric looks alive — connections up, consumers registered, queue "being consumed." Redelivery loops can also multiply duplicate side effects (double emails, double webhooks) if consumers aren't idempotent.

3. Dead letter queues nobody drains

What it is. A dead letter exchange catches messages that were rejected, expired, or overflowed a length limit. Wiring one up is good practice. But a DLQ is a landing zone, not a destination — and in most shops, nothing consumes it and nobody looks at it. Every message in there is a real business event that failed: an order that didn't process, a payment webhook that got rejected, a provisioning job that never ran.

Why it happens. The team that added the DLX moved on. There's no consumer, no alert on depth, and no runbook for triage. The queue quietly becomes a graveyard, and — worse — a growing one signals a live bug: something upstream is producing messages that consumers reject over and over.

How to detect it. First find your dead-lettering topology — which queues route where:

rabbitmqctl list_queues name messages arguments | grep -i dead-letter

Then check depth on the DLQs themselves (they're just queues — the list_queues output above includes them). Two numbers to alert on: absolute depth above zero for more than a day, and rate of growth — a DLQ gaining messages every hour means an active failure, not historical debris. In the management UI, dead-lettered message rates show per queue under Queues → (queue) → Message rates.

Typical impact. Silent data loss with a paper trail nobody reads. Teams that open a year-old DLQ for the first time typically find hundreds to tens of thousands of lost business events — and, buried in the rejection reasons, a bug that's been live the whole time.

4. Memory and disk alarms that block publishers

What it is. When a node crosses its memory high watermark or free disk falls below disk_free_limit, RabbitMQ raises a resource alarm and blocks all connections that publish. Consumers keep running — the broker is trying to drain itself — but every producer in the system stalls. From the application side it looks like publishes hanging forever, not an error.

Why it happens. Usually as the end-stage of signal #1 or #3: a backlog someone ignored finally ate the node's memory. Disk alarms also fire on undersized VMs where logs or other services share the volume.

How to detect it. One command tells you if you're currently blocked:

rabbitmq-diagnostics alarms

Check whether connections are in the blocked state with rabbitmqctl list_connections name state, and see what's eating memory with rabbitmq-diagnostics memory_breakdown. In the management UI, an active alarm turns the node red on the Overview page — visible, but only if someone has the page open. Alert on the alarm itself, not on the UI.

Typical impact. This is the "everything stopped at once" incident. Because publishers block silently rather than erroring, application timeouts fire far from the root cause and the on-call engineer spends the first 30 minutes debugging the wrong service.

5. Cluster and quorum queue health

What it is. In a cluster, quorum queues replicate across nodes and need a majority of replicas online to accept writes. A node that's down or partitioned doesn't necessarily break anything immediately — which is exactly the danger. A three-node cluster running on two nodes is one failure away from every quorum queue rejecting publishes, and it will run in that degraded state for weeks if nothing checks.

How to detect it. Cluster membership and running nodes:

rabbitmqctl cluster_status

Then the check that matters before any maintenance or after any node event:

rabbitmq-diagnostics check_if_node_is_quorum_critical

It fails if shutting down this node would cost any quorum queue its majority. For a specific queue, rabbitmq-queues quorum_status your-queue-name shows the leader and each member's state. In the management UI, Queues lists quorum queue members and flags queues with offline replicas.

Typical impact. Degraded redundancy is invisible until the second failure — and the second failure is a hard write outage on every affected queue, plus a much messier recovery than restarting one node would have been.

6. Connection and channel churn

What it is. AMQP connections and channels are designed to be long-lived. An application that opens a connection per message — usually a framework misconfiguration or a missing connection pool — hammers the broker with constant TCP setup, TLS handshakes, and authentication. RabbitMQ calls this high connection churn and it degrades the whole node, not just the offending app.

How to detect it. The management API's overview endpoint tracks churn rates directly:

curl -s -u monitoring:password http://localhost:15672/api/overview | \
  python3 -c "import sys,json; print(json.load(sys.stdin)['churn_rates'])"

Compare connection_created_details.rate against connection_closed_details.rate — healthy apps show near-zero on both after startup. Sustained double-digit creation rates mean some client is cycling connections. rabbitmqctl list_connections name connected_at channels (or the Connections tab sorted by age) points at the culprit: hundreds of connections a few seconds old, all from one service.

Typical impact. Elevated broker CPU, exhausted file descriptors, and latency for every well-behaved application sharing the node. It also floods logs, burying the signals from problems 1–5.

Why the dashboard alone doesn't save you

The management UI shows all six of these signals. So did it during the three-day incident in the opening — the data was there the whole time. The gap is that these failures develop over hours and days, in metrics nobody stares at, and the pattern that matters (rate divergence, a flat unacked count, a slowly filling DLQ) only appears when you look at trends, not snapshots.

The fix is making these checks systematic. Part two of this series builds exactly that with native tools only: rabbitmqctl one-liners, management API queries you can cron, the built-in Prometheus plugin, and the policies that act as guardrails.

Watching these signals continuously

Everything above can also run on autopilot. CloudThinker agents connect to your RabbitMQ management API read-only and track all six signals continuously — queue depth trends, unacked buildup, DLQ growth, alarms, quorum health, churn — investigating anomalies and proposing fixes for your approval instead of paging you with a raw threshold. Try CloudThinker free — 100 premium credits, no card required — or continue with the DIY monitoring guide.