RabbitMQ Monitoring with Native Tools: rabbitmqctl, API, Prometheus

Three columns from one command settle the most common RabbitMQ argument — "is this a traffic spike or a dead consumer?":

rabbitmqctl list_queues name messages messages_unacknowledged consumers

If messages climbs while consumers is above zero and messages_unacknowledged hovers near your prefetch ceiling, your consumers are alive and simply outmatched — that's a burst. If messages climbs while consumers reads 0, nothing is draining the queue at all. And if messages_unacknowledged sits pinned at exactly prefetch times consumer count while depth grows, consumers are receiving deliveries and never acking — stuck, not slow. That one line is the core of DIY RabbitMQ monitoring, and this guide builds the rest of the toolkit around it.

In part one of this series we covered the six signals that matter: queue depth versus consumer throughput, unacked buildup, dead letter queues nobody drains, memory and disk alarms, cluster and quorum queue health, and connection churn. This article is the hands-on companion — monitoring all of them with only what ships in the box: rabbitmqctl, the management HTTP API, the built-in Prometheus plugin, policies, and the alarm subsystem. Nothing to buy, nothing to install beyond plugins RabbitMQ already bundles.

Tool 1: rabbitmqctl — the triage layer

rabbitmqctl runs on the broker host (or anywhere with the Erlang cookie) and answers questions in seconds. Full reference in the rabbitmqctl man page.

Queue listings

The extended version of the opening command adds two useful columns:

rabbitmqctl list_queues name messages messages_ready \
  messages_unacknowledged consumers consumer_utilisation

How to read it. messages_ready is work waiting; messages_unacknowledged is work in flight. consumer_utilisation is the fraction of time the queue could deliver immediately to a consumer — a value well below 1.0 on a deep queue means consumers are the bottleneck (network, prefetch too low, or slow processing), not the broker. Run it twice, sixty seconds apart, and diff the messages column: direction matters more than any absolute number.

Cluster status

rabbitmqctl cluster_status

Three things to check in the output: every node you expect appears under Running Nodes, the Network Partitions section is empty, and the Alarms section is empty. For quorum queues, one more command tells you whether it's safe to restart a node:

rabbitmq-queues check_if_node_is_quorum_critical

If that fails, taking the node down would drop some quorum queue below its quorum — reschedule the maintenance.

Connections, channels, and consumers

rabbitmqctl list_connections name user state channels recv_oct send_oct
rabbitmqctl list_channels connection number consumer_count \
  prefetch_count messages_unacknowledged
rabbitmqctl list_consumers

How to read it. A connection in state blocked means a resource alarm is stopping its publishes (more on that in Tool 5). list_channels is where stuck consumers get names: a channel whose messages_unacknowledged equals its prefetch_count and never moves is a consumer that took deliveries and stopped acking. list_consumers maps queues to consumer tags and shows prefetch_count per consumer — a prefetch of 0 (unlimited) on a fast publisher is a memory incident waiting to happen.

Tool 2: The RabbitMQ management API — the scriptable view

Everything the management UI shows is available over HTTP. Enable the plugin if you haven't (rabbitmq-plugins enable rabbitmq_management, port 15672), and create a read-only user for scripts instead of reusing an admin account (management plugin docs):

rabbitmqctl add_user monitor 'use-a-real-password'
rabbitmqctl set_user_tags monitor monitoring
rabbitmqctl set_permissions -p / monitor "^$" "^$" ".*"

Top queues by depth

The /api/queues endpoint supports column selection, sorting, and pagination server-side — use them, because on a busy vhost the unfiltered payload is huge:

curl -s -u monitor:PASS \
  'http://localhost:15672/api/queues?columns=name,vhost,messages,messages_unacknowledged,consumers&sort=messages&sort_reverse=true&page=1&page_size=20'

Threshold sweeps with jq

Flag every queue past a depth threshold, and every queue with zero consumers that has messages waiting:

# Queues deeper than 10k messages
curl -s -u monitor:PASS http://localhost:15672/api/queues |
  jq -r '.[] | select(.messages > 10000)
    | "\(.vhost)/\(.name): \(.messages) msgs, \(.consumers) consumers"'

# Backed-up queues with no consumers at all
curl -s -u monitor:PASS http://localhost:15672/api/queues |
  jq -r '.[] | select(.consumers == 0 and .messages > 0)
    | "\(.vhost)/\(.name): \(.messages) stranded"'

The queue objects also carry rates: message_stats.publish_details.rate versus message_stats.ack_details.rate is the in-versus-out comparison from part one, computed for you.

Node health and health checks

curl -s -u monitor:PASS http://localhost:15672/api/nodes |
  jq -r '.[] | "\(.name) fd=\(.fd_used)/\(.fd_total) disk_free=\(.disk_free) mem_alarm=\(.mem_alarm) disk_alarm=\(.disk_free_alarm)"'

And the purpose-built health endpoints, which return HTTP 200 when healthy and 503 when not — ideal for load balancer checks and cron:

curl -s -u monitor:PASS http://localhost:15672/api/health/checks/alarms
curl -s -u monitor:PASS http://localhost:15672/api/health/checks/node-is-quorum-critical

Tool 3: The Prometheus plugin — metrics worth alerting on

The management API is fine for scripts; for real alerting you want the built-in Prometheus plugin:

rabbitmq-plugins enable rabbitmq_prometheus
curl -s localhost:15692/metrics | head -50

By default queue metrics are aggregated per node. For per-queue alerting either scrape /metrics/per-object or set prometheus.return_per_object_metrics = true in rabbitmq.conf — with the usual caveat that per-object mode gets expensive past a few thousand queues.

The short list worth paging on:

Two PromQL examples to start from:

# Any dead letter queue accumulating messages
sum by (queue) (rabbitmq_queue_messages_ready{queue=~".*(dlq|dead).*"}) > 0

# Queue growing for 15 minutes straight
delta(rabbitmq_queue_messages_ready[15m]) > 1000

That first one matters more than it looks: a dead letter queue is only useful if its depth is an alert, not a landfill.

Tool 4: Policies — DLX, TTL, and length limits as guardrails

Monitoring tells you a queue is filling; policies put a floor under how bad it can get. Three worth setting on day one.

Dead-lettering everywhere. Rejected and expired messages should land somewhere inspectable, not vanish (DLX docs):

rabbitmqctl set_policy dlx-prod "^prod\." \
  '{"dead-letter-exchange":"dlx"}' --apply-to queues

One trap: declaring a dead letter exchange without binding a queue to it means dead-lettered messages are silently dropped. Bind a catch-all DLQ, then wire it into the Prometheus alert above.

Length limits with explicit overflow behavior (max length docs):

rabbitmqctl set_policy cap-events "^events\." \
  '{"max-length":100000,"overflow":"reject-publish"}' --apply-to queues

reject-publish pushes backpressure to producers; the default drop-head silently discards your oldest messages. Pick deliberately — for most work queues, losing the oldest jobs unnoticed is worse than a publish error.

TTL on queues holding disposable data (TTL docs):

rabbitmqctl set_policy ttl-cache "^cache\." \
  '{"message-ttl":3600000}' --apply-to queues

With a DLX policy in place, TTL expiry routes to the DLQ instead of deleting — which is exactly what you want for post-incident analysis of what didn't get processed.

Tool 5: Reading memory and disk alarms

When RabbitMQ crosses its memory watermark or free disk floor, it blocks all publishing connections cluster-wide while consumers keep working (alarms docs). It's a self-protection mechanism, and it's the single most misdiagnosed RabbitMQ failure mode: producers hang, application timeouts fire, and nothing in the broker logs looks like an error.

Check for active alarms and see where memory is actually going:

rabbitmq-diagnostics alarms
rabbitmq-diagnostics check_local_alarms
rabbitmq-diagnostics memory_breakdown --unit mb

If memory_breakdown shows queue processes dominating, the memory alarm is a queue-depth problem wearing a disguise — fix the consumers, not the watermark. Also review the thresholds themselves in rabbitmq.conf:

vm_memory_high_watermark.relative = 0.5
disk_free_limit.absolute = 5GB

The default disk free limit is a token 50MB — far too low for production, because by the time it fires you have almost no room left for the broker to page messages to disk. Set it to at least a few GB, and alert on rabbitmq_disk_space_available_bytes well before the limit.

Where DIY thresholds stop

Run everything above and you have genuinely solid coverage: depth, unacked, consumers, DLQs, alarms, cluster health. You should build it. You should also be honest about the three walls you'll hit.

Thresholds notice depth; they don't explain it. An alert tells you prod.payments has 40,000 ready messages and zero consumers. It does not tell you which consumer group stopped, whether it crashed or deadlocked, or that the deploy 20 minutes ago changed a serialization format. Someone still ssh-es in at 2 a.m. and runs the rabbitmqctl commands from Tool 1 by hand.

DLQ contents don't analyze themselves. Knowing the dead letter queue has 3,000 messages is one alert; understanding that 92% of them share a payload pattern that a schema change broke is an hour of manual message inspection.

The scripts are a system you now own. The jq sweeps, the PromQL, the cron'd health checks — they need maintenance, tuning against alert fatigue, and an on-call rotation that remembers what each one means.

What comes next

Everything in this guide can run continuously, with the investigation attached. CloudThinker agents connect to the RabbitMQ management API read-only, watch depth trends, unacked buildup, DLQ growth, and node alarms around the clock — and when something degrades, they work out which consumer stalled and why before proposing a fix for your approval. That's part three of this series.

Or see it against your own broker: try CloudThinker free — 100 premium credits, no card required.