What LLM observability captures that APM never did
An APM tool answers questions about the request path: how long did it take, did it error, how much throughput did the service handle. None of that tells you what the model was actually asked, what it actually said back, or whether a tool call it made mid-response even fired. LLM observability adds those fields as first-class parts of the trace, not an afterthought bolted onto a generic span.
| Classic APM signal | What LLM observability adds instead |
|---|---|
| Request latency | Latency per generation, attributable to a specific model and prompt version, not just the outer request |
| Error rate (5xx, exceptions) | A hallucination or failure signal on the completion itself - a 200 response that is still wrong |
| Throughput | Token throughput and cost per trace, since a slow trace and an expensive trace are not the same problem |
| Stack trace | The tool-call sequence inside one trace - which functions the model invoked, in what order, with what result |
Some vendors market all of this under the wider "ai observability" label, which folds in model-infrastructure and pipeline health alongside individual traces - a bigger claim than most teams need to evaluate first. This guide is scoped to the trace-level practice: get one call, or one agent run, observable end to end before worrying about the whole platform.
Tracing an agent run, not just a single call
A single LLM call is one span. An agent run - the model reasoning across several steps, calling a tool, reading the result, and deciding what to do next - is several spans that need to reconstruct as one coherent trace with a parent and children, not a pile of disconnected events with no relationship to each other. This is what "llm tracing" and "agent tracing" mean in practice: not just capturing that calls happened, but preserving the structure of how they happened.
Get this wrong and debugging an agent failure becomes archaeology - piecing together five separate log lines to guess at a sequence that should have been one readable trace. Get it right and a broken tool call is visible as exactly that: a child span, under the right parent, with the failure attributable to the right step.
Monitoring in production
Tracing tells you what happened in one run. Monitoring is the discipline of watching that signal over time in production: trace volume, cost per day, latency percentiles across model versions, and - the signal that matters most and is easiest to skip - a hallucination or quality-failure rate tracked as an ongoing metric, not a one-time spot check during development.
llm monitoring and ai agent monitoring are the same discipline applied at the platform and the agent level respectively: is the system healthy right now, and is it drifting. Hallucination detection belongs in this list as a monitored signal precisely because a model that was fine last week can degrade silently after a prompt change, a model version bump, or a shift in the kind of question users start asking it - the same class of problem section three of our companion evals guide covers in more depth.
Doing this in PostHog
PostHog's AI Observability product (referenced in its own docs interchangeably as LLM analytics, verified against posthog.com/docs/ai-observability as of this writing) captures prompt, completion, token usage, cost, latency, and tool calls per generation and stitches them into traces tied to the same person and session model as the rest of your product analytics - so a trace can sit next to session replay, feature-flag exposure, and funnel data for the same user instead of living in a separate LLM-only tool with no product context.
For the integration-and-privacy specifics - ingesting trace data from LangFuse, Helicone, or Traceloop, tracing Claude Agent SDK runs, and configuring privacy mode and retention for prompt content - see our dedicated PostHog AI-engineering integrations breakdown. This guide is the broader "what is LLM observability and why it matters" explainer; that one is the PostHog-specific how-to.
How to verify a trace is healthy
- Run one real call through the integration and confirm the trace shows prompt, completion, token counts, and cost as populated fields - not blanks that only look complete at a glance.
- For an agent or tool-calling run, confirm every tool call nests as a child under the parent trace, not as a disconnected standalone event with no relationship to the run it belongs to.
- Confirm latency is captured per generation step, not only as one wall-clock number for the outer request - a slow tool call and a slow model response need to be distinguishable.
- Trigger a monitoring alert condition (a cost spike, a latency percentile breach, or a flagged hallucination signal) and confirm it fires against real trace data, not only against a synthetic test event.
- Spot-check that a trace is attributed to the correct person or session in product analytics, not an anonymous or orphaned identity that makes the trace unusable for anything beyond itself.