How to Triage Kafka Consumer Lag with Kafka's Native Tools

One command answers most Kafka consumer lag questions before you open a single dashboard:

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 \
  --describe --group payments-app

Four columns in its output do the heavy lifting: CURRENT-OFFSET (what the group has committed), LOG-END-OFFSET (what the broker has accepted), LAG (the difference), and CONSUMER-ID (who owns the partition — or a dash if nobody does). Learn to read those four and you can triage the majority of lag incidents with tooling that ships in every Kafka distribution.

In part one of this series we covered the six signals that matter: what lag actually measures, the three lag shapes, rebalancing storms, partition skew, under-replicated partitions, and retention pressure. This article is the hands-on triage session — every check, using only kafka-consumer-groups.sh, kafka-topics.sh, kafka-log-dirs.sh, consumer JMX metrics, and their Confluent Cloud equivalents. Budget an hour for a first pass on a cluster you've never profiled.

Tool 1: kafka-consumer-groups.sh — the lag view

This script covers lag measurement, skew detection, and rebalance forensics in one place (Kafka docs).

Step 1: Read the per-partition lag table

Start with the describe from the intro. A healthy group looks boring — small, stable lag spread evenly across partitions:

GROUP        TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG   CONSUMER-ID
payments-app orders  0          482190          482201          11    consumer-1-...
payments-app orders  1          479002          479020          18    consumer-1-...
payments-app orders  2          481575          481590          15    consumer-2-...

Three unhealthy patterns to spot immediately:

Lag on every partition, growing on repeat runs — the group is under-provisioned or the processing path got slower. Run the command twice, 60 seconds apart, and diff the LAG column; direction matters more than the absolute number.
Lag concentrated on one or two partitions — partition skew. One hot key (a big tenant, a null key before Kafka 2.4's sticky partitioner) is funneling traffic to one partition, and no amount of consumer scaling fixes it because a partition has exactly one owner.
A dash in CONSUMER-ID — unassigned partitions. Either the group has fewer members than partitions and something crashed, or you're mid-rebalance.

To sweep every group on the cluster rather than one you already suspect:

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 \
  --describe --all-groups | awk '$6 > 10000'

Step 2: Check assignment balance with --members --verbose

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 \
  --describe --group payments-app --members --verbose

This prints each member, its #PARTITIONS count, and the exact assignment list. What you want: partition counts within one of each other. What you often find: one member holding six partitions while its peers hold two — a leftover from eager rebalancing plus uneven subscription, and a structural reason one consumer lags while the rest idle.

Step 3: Check group state and rebalance history

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 \
  --describe --group payments-app --state

Stable is the only state you want to see in steady operation. PreparingRebalance or CompletingRebalance on repeated runs means the group is churning — members are joining and leaving faster than assignments settle, which pauses consumption (fully, under the default eager protocol) every cycle. That's your cue to jump to the rebalancing triangle section below.

One honest caveat: LAG here is computed from committed offsets. A consumer that is processing fine but committing every 30 seconds will show phantom lag up to 30 seconds deep. Check auto.commit.interval.ms (or your manual commit cadence) before declaring an incident.

Tool 2: kafka-topics.sh and kafka-log-dirs.sh — the broker side

Consumer-side lag with a broker-side cause is common enough that this check belongs in every triage.

Under-replicated partitions

bin/kafka-topics.sh --bootstrap-server broker1:9092 \
  --describe --under-replicated-partitions

Empty output is the pass condition. Any rows mean followers are falling behind leaders — a broker under I/O or network pressure — and consumers reading from that broker will lag no matter how healthy the group is. Pair it with the unavailable-partitions check while you're here:

bin/kafka-topics.sh --bootstrap-server broker1:9092 \
  --describe --unavailable-partitions

Uneven partition sizes

Skew shows up on disk before it shows up in lag. Dump per-partition log sizes:

bin/kafka-log-dirs.sh --bootstrap-server broker1:9092 \
  --describe --topic-list orders

The output is JSON; pull the sizes out with jq:

bin/kafka-log-dirs.sh --bootstrap-server broker1:9092 \
  --describe --topic-list orders 2>/dev/null | tail -1 | \
  jq '[.brokers[].logDirs[].partitions[]] | map({partition, size}) | sort_by(-.size)'

If one partition of orders is 5x the size of its siblings, your producer keying is the root cause and the fix lives upstream — repartitioning or a compound key — not in the consumer group.

Tool 3: consumer JMX metrics — the trend layer

The CLI gives you snapshots; JMX gives you rates. Every Java consumer exposes these without any agent (Kafka monitoring docs). The MBeans worth scraping, whatever ships them to your metrics store:

MBean group	Metric	What it tells you
`consumer-fetch-manager-metrics`	`records-lag-max`	Worst per-partition lag this consumer sees — your primary alert metric
`consumer-fetch-manager-metrics`	`fetch-rate`, `bytes-consumed-rate`	Whether the consumer is actually pulling; a lag spike with a falling fetch rate points at the consumer, not the producer
`consumer-fetch-manager-metrics`	`records-per-request-avg`	Batch efficiency; chronically low values with high lag suggest fetch tuning, not scaling
`consumer-coordinator-metrics`	`commit-latency-avg`	Slow commits back-pressure the poll loop
`consumer-coordinator-metrics`	`rebalance-rate-per-hour`, `failed-rebalance-rate-per-hour`	Anything persistently above zero in steady state is a churn problem
`consumer-coordinator-metrics`	`last-rebalance-seconds-ago`	A value that keeps resetting to near-zero is a rebalance loop, even if each rebalance "succeeds"

The decision rule that saves the most time: lag rising while bytes-consumed-rate holds steady means the producer sped up; lag rising while bytes-consumed-rate drops means the consumer slowed down. Those are different investigations, and this pair of metrics splits them in one glance.

Tool 4: tuning the rebalancing triangle

If Tool 1 showed non-Stable states or Tool 3 showed nonzero rebalance rates, the fix is usually three consumer configs that must be tuned together (consumer configs reference):

# Liveness: how long the coordinator waits for heartbeats before evicting
session.timeout.ms=45000

# Heartbeat cadence: keep at or below one-third of session.timeout.ms
heartbeat.interval.ms=15000

# Processing deadline: max gap between poll() calls before eviction
max.poll.interval.ms=300000

# The lever that actually controls poll-loop duration
max.poll.records=500

The two distinct eviction paths people conflate: session.timeout.ms catches dead consumers (heartbeats stop — the background thread died or the network dropped), while max.poll.interval.ms catches stuck consumers (heartbeats continue but poll() never comes back because a batch is taking too long). If your rebalances correlate with slow downstream calls, raising session.timeout.ms does nothing — lower max.poll.records or raise max.poll.interval.ms so one batch reliably fits inside the deadline.

Two more configs that turn rebalances from full stops into non-events:

# Incremental cooperative rebalancing: only moved partitions pause
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

# Static membership: a restart within session.timeout.ms triggers no rebalance at all
group.instance.id=payments-app-pod-0

The cooperative-sticky assignor means a member joining or leaving revokes only the partitions that actually move, instead of pausing the whole group. Migrating a live group from the default eager assignor requires two rolling restarts (first add CooperativeStickyAssignor alongside the old strategy, then remove the old one) — don't flip it in a single deploy. Static membership via group.instance.id is the single best fix for Kubernetes environments where routine pod restarts were generating a rebalance storm per rollout.

Confluent Cloud: same triage, managed tooling

On Confluent Cloud you don't run broker-side scripts, but every check above has an equivalent.

Console. Navigate to your cluster → Clients → Consumer lag (or open a topic and use the Consumers tab). You get the same per-partition table — current offset, end offset, lag, member — with sorting, which makes skew-spotting faster than the CLI (Confluent docs).

Confluent CLI. Summarize or list lag for a group directly:

confluent kafka consumer group lag summarize payments-app --cluster lkc-abc123
confluent kafka consumer group lag list payments-app --cluster lkc-abc123

Metrics API. For alerting, scrape confluent_kafka_server_consumer_lag_offsets from the Confluent Metrics API — this is the managed replacement for the JMX layer, computed broker-side so it works even for non-Java clients.

The Apache scripts still work. kafka-consumer-groups.sh runs fine against Confluent Cloud with an API key in a properties file:

bin/kafka-consumer-groups.sh \
  --bootstrap-server pkc-xxxxx.us-east-1.aws.confluent.cloud:9092 \
  --command-config ccloud.properties \
  --describe --group payments-app

where ccloud.properties contains your sasl.jaas.config with the cluster API key and security.protocol=SASL_SSL.

Where the native-tools approach runs out

Everything above is worth doing, and none of it requires a purchase order. But after a few incidents you'll notice the same four walls:

Scripts snapshot lag; they don't explain it. LAG=180000 tells you nothing about whether a deploy slowed the consumer, a producer burst outpaced it, or a rebalance loop kept resetting progress. The correlation work — timelines, deploy history, config diffs — is still yours, at 2 a.m.

Polling isn't watching. A cron job around kafka-consumer-groups.sh catches lag on its schedule. Bursty lag that appears and drains between runs — often the early symptom of the incident that eventually sticks — never shows up.

JMX pipelines are their own project. Exporters, dashboards, alert thresholds per group and per topic: real engineering time, and thresholds go stale as traffic grows.

Diagnosis isn't remediation. Knowing that max.poll.records is too high for the new payload size still leaves a config change, a review, and a rolling restart between you and a drained backlog.

What comes next

Every check in this guide can run continuously instead of when someone remembers. CloudThinker agents connect to Kafka or Confluent Cloud read-only, watch lag, rebalance rates, skew, and replication health around the clock, and when lag grows they do the correlation work — deploy, rebalance loop, or hot partition — and propose the fix for your approval. That's part three of this series.

Or see it against your own cluster today: Try CloudThinker free — 100 premium credits, no card required.