Sign inBook a Demo

How To

Langfuse LLM Observability with Native Tools: Traces, Prompts, Evals

Hands-on Langfuse LLM observability with native tools only: instrument traces with sessions, users, and nested spans via the Python SDK or OpenTelemetry; move prompts out of the codebase with versioned prompt management and production/staging labels; capture user feedback and LLM-as-a-judge scores; track cost per model, feature, and user with dashboards and the Metrics API; and wire up the thin alerting layer. Part two of our LLM observability series — copy-pasteable snippets throughout, closing with where DIY hits its ceiling: Langfuse shows you the bad trace, but a human still has to read it.

ST
·
langfuseobservabilityllmopsllmtracingpromptmanagementevals
Cover Image for Langfuse LLM Observability with Native Tools: Traces, Prompts, Evals

Langfuse LLM Observability with Native Tools: Traces, Prompts, Evals

There's a difference between logging LLM calls and instrumenting them. A log line tells you a call happened. An instrumented trace tells you which user, which session, which prompt version, which retrieval step fed it, what it cost, and what came out the other end. Most teams shipping LLM features are one session_id away from a debuggable system — and don't know it until the first "the bot got worse" ticket arrives with nothing to attach to it.

This is the hands-on part of our Langfuse LLM observability series. Part one covered the failure modes: silent quality regressions after prompt changes, token cost creep, latency stacking across chained calls, prompt versions scattered through the codebase. This article is the build: setting up LLM tracing, prompt management, scores, evals, dashboards, and alerting using only Langfuse's native features. We'll close with an honest look at where the DIY ceiling sits.

Everything below uses the Python SDK (v3, OpenTelemetry-based); the TypeScript SDK mirrors it closely. Setup is three environment variables (Langfuse docs):

pip install langfuse

export LANGFUSE_PUBLIC_KEY=pk-lf-...
export LANGFUSE_SECRET_KEY=sk-lf-...
export LANGFUSE_HOST=https://cloud.langfuse.com  # or your self-hosted URL

Step 1: Instrument traces properly — sessions, users, nested spans

The fastest path to first traces is the OpenAI drop-in wrapper — change one import and every completion call is captured with model, tokens, latency, and cost:

from langfuse.openai import openai  # instead of: import openai

That gets you isolated generations. It does not get you debuggable ones. The three attributes that turn a pile of LLM calls into something you can investigate are sessions (group all calls in one conversation), users (attribute cost and quality per account), and nested spans (see the chain, not just the final call).

The @observe decorator builds the tree from your existing call structure:

from langfuse import observe, get_client

langfuse = get_client()

@observe()
def answer_ticket(ticket_id: str, question: str, user_id: str):
    langfuse.update_current_trace(
        session_id=f"ticket-{ticket_id}",
        user_id=user_id,
        tags=["support-bot", "v2-pipeline"],
    )
    docs = retrieve_context(question)      # becomes a nested span
    return generate_answer(question, docs) # becomes a nested generation

@observe()
def retrieve_context(question: str):
    ...  # vector search — timed as its own span

@observe(as_type="generation")
def generate_answer(question: str, docs: list[str]):
    ...  # the LLM call itself

Now a slow response decomposes: was it 4 seconds of retrieval or 4 seconds of generation? A bad answer decomposes too: did the model fail, or did retrieval hand it the wrong documents? Without the nesting, every incident starts with re-running the request locally and guessing.

If you already run OpenTelemetry, Langfuse accepts OTLP directly and the SDK interoperates with existing OTel instrumentation (tracing docs) — LLM spans land in the same trace as your HTTP and database spans.

In TypeScript the same shape looks like:

import { Langfuse } from "langfuse";

const langfuse = new Langfuse();
const trace = langfuse.trace({
  sessionId: `ticket-${ticketId}`,
  userId,
  tags: ["support-bot"],
});
const generation = trace.generation({
  name: "generate-answer",
  model: "gpt-4o-mini",
  input: messages,
});
// ... call the model ...
generation.end({ output: completion });

Step 2: Prompt management — get prompts out of the codebase

The prompt string hardcoded next to your API call is the reason "who changed the prompt and when" is unanswerable at most companies. Langfuse's prompt management gives you versioned prompts fetched at runtime, with labels acting like deployment pointers (prompt management docs):

# Fetch whatever version currently carries the "production" label
prompt = langfuse.get_prompt("support-answer", label="production")

messages = prompt.compile(question=question, context="\n".join(docs))

The workflow that makes this pay off:

  1. Create version 2 in the Langfuse UI (or via langfuse.create_prompt(...)) and label it staging.
  2. Point staging traffic at it — your staging deploy fetches label="staging", production still resolves label="production" to version 1. Same code, different pointer.
  3. Promote by moving the label. When v2 looks good, assign the production label to it in the UI. No deploy, instant rollback by moving the label back.

The critical detail: link the prompt to the generation so every trace records which prompt version produced it. With the OpenAI wrapper it's one argument:

completion = openai.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    langfuse_prompt=prompt,  # ties this generation to the exact prompt version
)

After a regression, you can now filter traces by prompt version and diff v1 against v2 in the UI — the single most useful move in any "answers got worse" investigation. Prompts are cached client-side by the SDK, so the fetch adds no meaningful latency on the hot path.

Step 3: Scores and evals — make "worse" measurable

Traces tell you what happened; scores tell you whether it was any good. Langfuse scores are numeric, categorical, or boolean values attached to traces or individual generations (evaluation docs). Two sources matter most in production:

User feedback. Wire your thumbs-up/down button to a score. From the browser, use the public key only (via the TS SDK's web client); from the backend:

langfuse.create_score(
    trace_id=trace_id,
    name="user_feedback",
    value=1,            # 1 = thumbs up, 0 = thumbs down
    data_type="BOOLEAN",
    comment="resolved the ticket on first answer",
)

Even 2–5% of users clicking gives you a quality trendline per prompt version — which is exactly the signal a silent regression needs to stop being silent.

LLM-as-a-judge. In the Langfuse UI under Evaluations → LLM-as-a-Judge, you configure an evaluator (built-in templates cover hallucination, relevance, toxicity, correctness) that runs against a sample of incoming production traces and writes scores back automatically. Run it on 5–10% of traffic rather than everything — judge calls cost real tokens, and a sample is enough to see a trend. The same evaluators run against datasets for offline testing: collect problem traces into a dataset, then score every new prompt version against it before it ever gets the production label.

The pattern to aim for: user feedback as the ground-truth trickle, LLM-as-a-judge as the high-volume proxy, datasets as the pre-release gate.

Step 4: Dashboards and cost tracking per model and feature

Langfuse computes token counts and USD cost per generation from model pricing definitions automatically for major providers (model usage and cost docs); custom or fine-tuned models take a pricing entry in Settings → Models.

The built-in dashboards give you cost, latency (p50/p95/p99), and score trends out of the box. Custom dashboards let you slice by what actually matters for accountability:

  • Cost by tag — tag traces per feature (support-bot, summarizer) and you get cost per feature, which turns "the OpenAI bill doubled" into "the summarizer doubled."
  • Cost by user_id — finds the one integration customer generating 40% of your spend.
  • p95 latency by trace name — latency stacking in chains shows up here first.
  • Score average by prompt version — the regression chart.

For anything programmatic, the Metrics API exposes the same aggregations (Metrics API docs):

curl -s -G "https://cloud.langfuse.com/api/public/metrics" \
  -u "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" \
  --data-urlencode 'query={"view":"traces","metrics":[{"measure":"totalCost","aggregation":"sum"}],"dimensions":[{"field":"tags"}],"fromTimestamp":"2026-07-01T00:00:00Z","toTimestamp":"2026-07-13T00:00:00Z"}'

Step 5: Alerting — the thin part

Be aware going in: alerting is the least built-out layer of the native stack. Your realistic options:

  • Spend alerts on Langfuse Cloud notify you at billing thresholds — a blunt but real backstop for cost creep.
  • Prompt webhooks fire on prompt changes (version created, label moved), so at minimum every prompt promotion lands in a Slack channel with an audit trail (webhooks docs).
  • Metrics API polling — a scheduled job (cron, GitHub Actions) that queries cost, p95 latency, or average score deltas and posts to Slack/PagerDuty when a threshold trips. This works, but you're now writing and maintaining your own alerting service.

There is no native "page me when the hallucination score drops 15% week-over-week." You build that yourself or you watch dashboards.

The honest part: what the native stack doesn't do

Run everything above and you'll have genuinely good LLM observability — better than most teams shipping AI features today. You should also know where the ceiling is:

Langfuse shows you the bad trace; someone still has to read it. A score dropping from 0.82 to 0.64 is a fact, not a diagnosis. A human still opens twenty traces, reads the outputs, and forms a hypothesis.

Correlation is manual. Was the regression the prompt change, the model version bump, the deploy that altered retrieval chunking, or the new customer with weird documents? The evidence is spread across prompt history, traces, and your deploy log — a person joins it.

Prompt diffing is a human judgment call. The UI shows v1 versus v2 side by side; deciding which changed line caused the regression, and what the fix is, is engineer-hours every time.

Watching is a job nobody has. Dashboards are pull-based. In practice they get checked after a complaint, which is precisely the failure mode part one described.

The native stack is the necessary foundation. The gap is the layer that reads the traces, diffs the versions, and proposes the fix without a human starting from zero.

What comes next

That gap — investigation and correlation on top of your Langfuse data — is what part three of this series covers: CloudThinker agents connect to Langfuse read-only, watch cost, latency, and score trends continuously, and when quality regresses they diff the prompt versions, sample the bad traces, and propose the fix under your approval. Try CloudThinker free — 100 premium credits, no card required.