A standard APM dashboard will tell you your LLM endpoint returned a 200 status code in 400 milliseconds. It will not tell you the answer was confidently wrong, that retrieval pulled the wrong document, that a silent model version bump changed behavior overnight, or that cost per request quietly tripled because prompts grew longer over six months of feature additions. This is the core problem with applying traditional observability to AI systems: uptime and latency are necessary but nowhere near sufficient, because an LLM system can be 'up', fast, and still producing wrong or harmful output at scale, and none of the traditional signals will show it.
The gap traditional APM leaves wide open
Application performance monitoring was built for a world where 'working' is binary: the server responded, the response time was acceptable, the error rate was low. LLM systems break that assumption completely, because the two failure modes that matter most, a hallucinated fact and a subtly wrong retrieval, produce a perfectly healthy-looking HTTP 200 response in reasonable time. A support bot that confidently tells a customer the wrong refund policy will show up as a successful request in every traditional dashboard. The team only finds out when a customer complains, or worse, when they don't complain and just quietly stop trusting the product. This is why LLM observability has to be built as an addition to, not a replacement for, traditional APM: you still need uptime and latency, but they're now necessary and far from sufficient.
Hallucination rate: measuring what traditional monitoring can't see
Hallucination rate isn't something you can compute from a log line the way you compute error rate; it requires a ground-truth comparison, which means an evaluation set of real, representative inputs with known-correct answers, checked against production outputs on a recurring schedule. The practical version of this most teams run is a rolling sample: pull 50-200 real production requests weekly, score them against a rubric (either human-reviewed, or model-graded with periodic human spot-checks for calibration), and track the pass rate as a trend line, not a single number. What matters isn't the absolute rate, it's the trend: a hallucination rate that was acceptable at launch and has crept up 8 points over two months, because retrieval quality degraded or the underlying model was swapped by the vendor, is the kind of signal that never shows up in an uptime dashboard and directly predicts customer-visible failures.
Retrieval relevance: the upstream signal that predicts downstream failure
In any RAG system, a wrong or irrelevant retrieved document is the single most common root cause of a bad final answer, and it's detectable before generation even happens, which makes it one of the highest-leverage things to monitor. Retrieval relevance is typically tracked as a score (cosine similarity or a reranker's relevance score) between the query and the top-k retrieved chunks, aggregated as a distribution over time rather than checked per-request. A sudden drop in the median relevance score, or a growing tail of near-zero-relevance retrievals, usually means one of a few specific things: the underlying document set changed shape (new content that doesn't match old embedding assumptions), an embedding model was swapped without re-indexing everything, or query patterns shifted toward topics the knowledge base doesn't actually cover yet. Catching this at the retrieval layer, before it becomes a bad final answer, is meaningfully cheaper than catching it at the hallucination layer, because it points directly at the fix.
Prompt and output drift: catching the changes nobody announced
Two kinds of drift matter and neither shows up in a standard dashboard. Prompt drift happens on your side: a prompt template gets edited for one feature and inadvertently changes behavior on another, or accumulates so many patched-in special cases over months that nobody fully understands its current behavior. Model drift happens on the vendor's side: providers periodically update models behind a stable API name, and a model that behaved one way in March can behave differently in June with zero announcement in some cases. The defense against both is the same: version and log every prompt template with a hash or version number tied to each request, log the exact model version string returned by the API (not just the name you requested), and re-run your evaluation set whenever either changes, plus on a fixed schedule regardless, so drift gets caught by a scheduled check rather than by a customer complaint.
| Signal | Traditional APM | LLM observability |
|---|---|---|
| Is the request succeeding | Yes, HTTP status and error rate | Necessary but insufficient on its own |
| Is the answer correct | Not tracked at all | Hallucination rate against a ground-truth eval set, tracked as a trend |
| Is retrieval working | Not applicable | Relevance score distribution across retrieved chunks, tracked over time |
| Did behavior silently change | Not tracked | Prompt version hash + logged model version string, diffed against eval results |
| What did this cost | Infra cost only | Token cost per request, tracked per feature/endpoint, not just in aggregate |
| Was the response fast enough | Overall response latency | Time-to-first-token and inter-token latency, separately, for streaming UX |
Streaming latency needs its own percentiles
For a streaming chat response, overall latency is close to meaningless as a user-experience metric, because a response that takes 4 seconds to complete but starts streaming in 200ms feels fast to a user, while one that takes 2 seconds to complete but sits silent for 1.8 seconds before the first token feels slow, even though the total time is shorter. Real LLM observability tracks time-to-first-token (TTFT) and inter-token latency as separate P50/P95/P99 metrics, not folded into one number. A P95 TTFT that creeps from 300ms to 1.2s over a month is a leading indicator of either provider-side load issues or a prompt that's grown too long (more input tokens delays the first output token), and it's invisible if you're only watching total request duration.
Cost per request: the metric that silently drifts upward
Token cost per request is easy to monitor and easy to ignore until a monthly bill spikes unexpectedly. It drifts upward for boring, unglamorous reasons: a prompt template accumulates a few more few-shot examples over successive feature additions, a retrieval step starts pulling more or longer chunks, conversation history grows longer before being truncated, or a fallback to a more expensive model happens more often than intended as accuracy tuning shifts thresholds. The fix is tracking cost per request (not just aggregate monthly spend) broken down per feature or endpoint, so a creeping increase in one specific flow is visible immediately instead of being buried in a total that only tells you something's wrong after the invoice arrives.
What a real LLM observability stack looks like
In practice, a working setup has three distinct layers, and teams that build only the first one are the ones who find out about failures from customers.
- Tracing: full request-level detail, prompt version, retrieved documents, model version, token counts, latency breakdown, for every request, queryable after the fact.
- Evaluation: a recurring job that scores a sample of real production traffic against a ground-truth or rubric-based eval set, producing trend lines for hallucination rate and relevance, not just point-in-time snapshots.
- Alerting: explicit thresholds tied to the eval and drift metrics above, not just uptime and error rate, with a named owner who gets paged when hallucination rate or relevance score crosses a line.
Tooling in this space generally falls into three categories rather than being one product: LLM-specific tracing platforms (capturing the full request lifecycle including prompts and retrieved context), evaluation frameworks (running scored comparisons against a test set on a schedule), and cost/usage analytics layered on top of your existing APM. Most production setups combine at least two of these three categories rather than relying on general-purpose APM alone, because general-purpose APM genuinely cannot see inside the parts of the system where AI-specific failures happen.
