The eval types, and what each one actually checks
"Evals" gets used as one word for several different checks that verify different things. Knowing which one you're running matters, because a passing offline eval and a healthy production agent are not the same claim.
| Eval type | What it actually checks |
|---|---|
| Offline evals | Run against a fixed test set before a prompt or model change ships - catches regressions before real users see them, not a substitute for production signal |
| LLM-as-a-judge | An LLM scores each generation against a rubric you define - relevance, helpfulness, hallucination, jailbreak, and toxicity are common built-in templates - good for subjective, nuanced checks a rule can't express |
| Code-based (deterministic) checks | Rule-based validation - format, keyword presence, length limits - free to run with no LLM cost, and exact wherever the check itself is exact |
| RAG evaluation | Scores retrieval quality and answer groundedness separately - did it fetch the right context, and does the answer actually rest on what was retrieved - conflating the two hides which half is actually broken |
| Agent evaluation | Scores a multi-step run as a whole - task completion, correct tool selection, whether the agent stopped when it should have - against the full trace, not a single generation |
PostHog's evaluations feature (verified against posthog.com/docs/llm-analytics/evaluations as of this writing) runs both llm-as-judge and code-based checks against built-in templates for relevance, helpfulness, jailbreak, hallucination, and toxicity, returning a pass/fail result with reasoning attached to the generation it scored - verify the current template list against PostHog's own docs before scoping a specific check against it, since this is exactly the part of the product most likely to have moved by the time this is read.
LLM evaluation metrics worth tracking over time
- Pass/fail rate per template, trended by prompt or model version - a single point-in-time score tells you nothing about whether the last change helped or hurt
- Judge-human agreement rate, spot-checked periodically - an llm-as-judge that has quietly drifted is a silent failure of exactly the kind this whole practice exists to catch
- Retrieval precision and groundedness rate, tracked separately for RAG systems, since the two failure modes need different fixes
- Task-completion rate for agent runs, plus the specific step at which failed runs actually fail - a completion rate alone doesn't tell you where to look
The bridge the market misses: an eval score is not a product outcome
A team reports "evals improved eight points" as though that settles whether the product got better. It doesn't - an eval score measures the model's output against a rubric, not what the user actually did next. Session length, task completion in the real product, retention, conversion: those are product outcomes, and none of them are guaranteed to move just because a rubric score moved.
[hypothesis] Assuming a higher eval score causes higher retention or conversion is an inference, not a measured fact, until it is tested against your own product data. It may hold. It may hold for some eval types and not others. It may hold up to a point and then plateau. The only way to know is to build the join and look, which is exactly what the next section does.
Wiring evals to activation, retention, and conversion in PostHog
Because PostHog ties each eval result to the specific generation and trace it scored, and ties traces to the same person and session model as the rest of product analytics, you can segment a retention or conversion view by eval-score band - for example, sessions where the agent's helpfulness score landed in the bottom decile - and see whether that band actually behaves differently, rather than asserting it does.
This is the same discipline that makes ai agent testing mean something beyond "the test suite passed." A passing eval suite and a better product are two different claims until they've been joined and checked against each other. For the KPI-translation side of this - how a P2 analyst defends "the agent got better" to a stakeholder who doesn't want to hear about eval rubrics - see our Amplitude practice's guides on AI consumption KPIs and measuring product agent usage.
How to verify it worked
- Confirm each eval result - pass or fail, plus reasoning - is attached to the specific generation or trace it scored, and is filterable by prompt or model version, not just visible in a separate eval-run log disconnected from the trace.
- Build one funnel or retention view segmented by eval-score band and confirm the data actually populates - an empty segment usually means the join didn't happen, not that the band has no users.
- Check whether the eval-score band correlates with the target metric on your own data. If it doesn't, the eval or the metric mapping needs rework - not the dashboard, and not the assumption that it must be a display problem.
- Re-run the same offline eval set after a prompt or model change and confirm the score moves in a direction that eventually also shows up in the linked product metric, not only in the eval log where nobody outside the team ever looks.