Sign inBook a Demo

How To

SigNoz Integration: The Observability Signals That Actually Matter

A practical SigNoz integration guide for mid-market DevOps teams: how OpenTelemetry-native ingestion lands traces, metrics, and logs in ClickHouse, which per-service signals matter (p99 latency, error rate by endpoint, saturation), the three-step p99 investigation workflow from service overview to span drill-down, alert rules across all five signal types, and a neutral look at where SigNoz sits next to Datadog on cost model, self-host control, and OTel standardization. Part one of our SigNoz observability series — parts two and three cover native alert automation and an AI action layer on your alerts.

ST
·
signozobservabilityopentelemetryalertingmonitoringsre
Cover Image for SigNoz Integration: The Observability Signals That Actually Matter

SigNoz Integration: The Observability Signals That Actually Matter

At 14:07 the p99 latency on your checkout API jumps from 240ms to 2.1s. Not every endpoint — one endpoint, on one service. The trace that explains exactly why is already sitting in SigNoz, three clicks away. It stays unopened for forty minutes, because the person on call is grepping a Slack channel for "anyone else seeing slowness?"

That gap — between data that exists and a human who has read it — is where most triage time goes. A good SigNoz integration closes part of that gap: it puts traces, metrics, and logs from every service into one queryable place, on an open standard, at a cost model a mid-market team can actually sustain. This guide covers what that integration looks like, which signals deserve your attention, how a p99 investigation actually flows through the UI, how alert rules work across signal types, and — honestly — where the gap remains even after everything is wired up.

This is part one of a three-part series. Part two is a hands-on guide to alert automation with SigNoz's native features; part three covers putting an AI action layer on top of your SigNoz alerts.

How SigNoz sees your stack

SigNoz is OpenTelemetry-native end to end. There is no proprietary agent: your services emit traces, metrics, and logs through standard OTel SDKs and the OTel Collector, and SigNoz stores all three signal types in ClickHouse — a columnar database built for exactly this kind of high-cardinality, time-ordered data. One store, three signals, one query surface.

The practical consequence: instrumenting a service for SigNoz is instrumenting it for OpenTelemetry, full stop. For most languages you don't touch application code — you set environment variables and let auto-instrumentation do the work:

export OTEL_SERVICE_NAME=checkout-api
export OTEL_EXPORTER_OTLP_ENDPOINT=http://<signoz-otel-collector>:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production

Point that at your SigNoz OTel Collector (port 4318 for OTLP over HTTP, 4317 for gRPC) and the service shows up in the Services tab within minutes. The instrumentation docs cover the per-language specifics — Java and Python are near-zero-code; Go needs explicit SDK wiring.

Because it's all OTel, nothing you do here locks you in. The same instrumentation exports to any OTLP-compatible backend by changing one endpoint. That matters more than it sounds like it does — more on that in the sidebar below.

The signals that matter for a mid-market stack

You do not need hundreds of dashboards. For a stack of 10–50 services, a short list of signals catches the large majority of real incidents:

  • Per-service RED metrics — request rate, error rate, and duration (p50/p90/p99) for every service. SigNoz derives these automatically from trace data; the Services tab is this list, precomputed.
  • p99 latency, not average. Averages hide the 1% of requests that time out and page you at 2 a.m. A p99 that doubles while p50 holds steady is a classic signature of one slow dependency, one bad shard, or one pathological query.
  • Error rate by endpoint, not just by service. A service at 0.4% errors looks healthy until you see that one route is at 40% and the rest are clean.
  • External call latency — database, cache, third-party APIs. In SigNoz these show up as client spans; when your p99 spikes, the first question is almost always "us or a dependency?"
  • Saturation from infra metrics — CPU throttling, memory pressure, disk on your ClickHouse host itself. Ship these through the OTel Collector's hostmetrics receiver or the Kubernetes infra integration.
  • Error-log volume by service. A step change in ERROR-level log volume frequently precedes the user-visible symptom by minutes.

One useful check on day one: open Services and confirm every production service you expect is present. A service missing from that list is a service whose incident you'll debug blind.

Anatomy of a p99 investigation

Here is the workflow SigNoz is built around, and the one worth practicing before an incident rather than during it.

Step 1 — Service overview. Open Services → checkout-api. The overview shows latency percentiles, request rate, error rate, and Apdex over time. Confirm what you're looking at: is the spike in p99 only, or p50 too? Did the request rate change at the same moment? A p99 spike with flat p50 and flat traffic points at a subset of requests, not general overload.

Step 2 — Trace filtering. From the latency chart, drill into traces for the affected window. In the Traces explorer, filter to the guilty subset — service name, the specific route, and a duration floor so you only see the slow ones:

service.name = checkout-api
http.route = /api/v1/checkout
duration_nano >= 2000000000

Sort by duration, descending. You now have the ten worst requests of the incident in front of you, with full context each.

Step 3 — Span drill-down. Open one of those traces. The flame-graph view shows the full request tree with per-span timings. In practice the answer is visually obvious within seconds: one span holding 1.9 of the 2.1 seconds — a SELECT against the orders table, a call to a payment provider, a lock wait. The span's attributes carry the query text, the peer address, the status code. That's your lead.

Total time, for someone who knows the workflow: three to five minutes. Total time for someone who starts in Slack: measured in coffee refills. The tool is fast; the bottleneck is a human deciding to look, knowing where to look, and being awake enough to interpret what they see.

Alert rules across signal types

SigNoz supports alert rules on every signal it stores, which is genuinely useful — you're not exporting logs to a second system just to alert on them.

  • Metric-based alerts — built with the query builder or raw PromQL. The canonical p99 alert:
histogram_quantile(0.99, sum(rate(signoz_latency_bucket{service_name="checkout-api"}[5m])) by (le)) > 2000
  • Trace-based alerts — thresholds on trace-derived data, e.g. count of spans with has_error = true on a given route over the last five minutes.
  • Log-based alerts — e.g. ERROR-level log count per service exceeding a threshold, built in the logs query builder or as a ClickHouse query.
  • Exception-based alerts — fire on new or recurring exception groups from the Exceptions tab.
  • Anomaly-based alerts — deviation from a learned seasonal baseline instead of a static threshold, useful for metrics with strong daily cycles.

Two configuration habits pay for themselves. First, set evaluation windows deliberately: a p99 alert evaluated over one minute pages you for every garbage-collection pause; five-minute windows with a sustained-breach condition cut noise dramatically. Second, route by severity from day one — pages to PagerDuty or Opsgenie, warnings to Slack, everything else to a channel nobody is required to read. Part two of this series walks through the full alert-rule setup, webhook payloads, and how to audit stale rules in detail.

And audit you will need: alert rules rot. Thresholds set during an incident stay forever, services get renamed and their rules silently match nothing, and two teams independently alert on the same symptom. A quarterly pass through Alerts, sorted by last-fired time, finds rules that have either never fired (probably broken) or fire daily and get ignored (definitely broken).

Sidebar: where SigNoz sits next to Datadog

The "SigNoz vs Datadog" question comes up in almost every evaluation, and it deserves a boring, honest answer rather than a cheer for either side. They are different trade-offs, not a good product and a bad one.

  • Cost model. Datadog prices per host plus per product (APM, logs, custom metrics each metered separately), which is predictable per unit but compounds as you adopt more of the platform. SigNoz Cloud prices on usage (per GB ingested, per metric sample), and self-hosted SigNoz costs whatever your ClickHouse infrastructure costs. For log- and trace-heavy mid-market workloads the usage-based or self-hosted math often comes out substantially lower — but you should model your volumes, not trust anyone's blog.
  • Self-host control. Self-hosted SigNoz keeps telemetry in your VPC — relevant for data-residency and compliance constraints — and puts retention entirely in your hands. The cost of that control is real: you operate ClickHouse, you plan capacity, you do the upgrades.
  • OTel standardization vs. integrated polish. SigNoz bets everything on OpenTelemetry, which means portable instrumentation and no agent lock-in. Datadog supports OTel too, but its strength is the opposite bet: hundreds of turnkey integrations, mature ML-driven features, and the polish of a large managed platform. Teams that value breadth and hands-off operation reasonably choose it.

If your constraints are budget sensitivity, OTel-first architecture, or data control, SigNoz is a serious answer. If they're minimal operational load and maximal out-of-the-box coverage, a managed platform is. Neither choice closes the gap this article is actually about.

The gap that instrumentation doesn't close

Because here is the sober part. After the OTel pipelines are flowing, the Services tab is green, and the alert rules are tuned, an incident still unfolds the same way: an alert fires, a human context-switches, opens SigNoz, runs the three-step workflow above, correlates with whatever deployed recently, and decides what to do. The telemetry made that human faster. It did not replace the twenty to forty minutes of human attention per incident — at 3 a.m., with the trace that explains everything sitting three clicks away, unopened.

SigNoz's own scope ends at delivery: the alert reaches Slack or PagerDuty, and everything after that is yours. The rest of this series is about what lives on the other side of that handoff.

Where this series goes next

Part two squeezes everything possible out of SigNoz's native automation — alert rules across all five types, notification channels, webhook payloads, rule audits. Part three covers the step native tooling can't take: CloudThinker agents that connect to SigNoz read-only, receive the firing alert, run the trace investigation the way an engineer would, correlate with deploys and cloud state, and propose remediation with evidence — under autonomy levels you control. Try CloudThinker free — 100 premium credits, no card required.