Monitor LLM Applications Beyond HTTP Status Codes

Updated 9 November 2026 · Standard formula, shown below

HTTP status codes show whether a request reached an endpoint, but they cannot tell you whether a language model produced a useful, safe, or timely answer. Monitor the full request lifecycle—including retrieval, model behavior, quality signals, and user outcomes—to spot failures that a 200 response can hide. You’ll need an instrumented application, access to its logs or telemetry, and clear rules for handling sensitive data.

What You’ll Need

An LLM application with a way to add instrumentation around requests, model calls, tools, and retrieval.
A logging, metrics, and tracing destination, such as your existing observability platform.
A defined set of service objectives, such as response-time targets and acceptable failure rates.
A representative test set or evaluation process for checking answer quality and safety.
A data-handling policy that specifies what prompts, outputs, and identifiers may be recorded.

Step-by-Step

Map the request lifecycle. Draw the path from user request to final response. Include prompt construction, model calls, retrieval, tool execution, moderation, retries, and streaming. Mark where a request can fail or return an incomplete result. This map defines what you need to observe; otherwise, a successful outer request can conceal a failed internal step.
Give each request a trace ID. Create or propagate a unique ID for each user interaction and attach it to every related event. Record parent-child relationships so you can connect a top-level request to individual model, retrieval, and tool spans. Avoid putting raw prompts or personal data into the ID. For streaming responses, record when generation starts, when the first token arrives, and when the response completes or disconnects.
Instrument each meaningful operation. Capture start time, end time, outcome, and relevant attributes for model calls, retrieval queries, and tool calls. Useful fields include model or deployment identifier, prompt-template version, token counts, retrieved document IDs, retry count, and timeout or cancellation reason. Keep fields consistent across services. Record errors as structured events, not only as text buried in a message.
Measure latency and resource use. Track total request duration as well as time spent in each stage. For generated responses, measure time to first token and completion time separately. Track input and output tokens, tool-call counts, and other usage data available to the application. Break down distributions by route, model, and operation; averages can hide slow requests, so monitor percentiles such as p50, p95, and p99.
Define outcome metrics beyond transport success. Count completed responses, empty or truncated outputs, model and tool failures, timeouts, and requests that fall back to another path. Define “success” for each workflow: a support assistant might need to answer with relevant evidence, while a structured extraction task might need valid fields. Set thresholds based on your baseline and user needs rather than treating every HTTP 200 as success.
Add quality and safety checks. Build a small, representative evaluation set and run it when prompts, models, retrieval, or tools change. Assess criteria relevant to your application, such as factual support, relevance, format validity, refusal behavior, and policy compliance. Where appropriate, collect user feedback or use human review on a sample of live results. Treat automated evaluators as signals, not ground truth, and validate them against reviewed examples.
Protect sensitive data by design. Decide whether you need prompt and output content at all. Prefer metadata, redaction, or sampling where they can answer the operational question. If content is necessary, restrict access, set retention periods, and apply approved redaction before storage. Do not place secrets, credentials, or unnecessary personal information in telemetry. Make sure trace attributes cannot accidentally expose them.
Build dashboards and alerts around user impact. Create views for latency percentiles, failure and timeout rates, token usage, quality checks, and tool or retrieval health. Segment carefully by model, feature, and deployment so a failing workflow is visible without creating high-cardinality metrics. Alert on sustained changes, not a single unusual request. Pair each alert with an owner and a runbook describing the first checks.
Test the monitoring before relying on it. In a non-production environment, simulate a model timeout, empty output, retrieval failure, malformed tool result, and slow stream. Confirm that traces connect, metrics change, sensitive content is handled as intended, and alerts reach the right person. Compare dashboards with a sample of actual requests, then update thresholds as traffic and application behavior change.

Common Mistakes

Using HTTP success as application success: A request can return 200 while the answer is empty, irrelevant, or based on failed retrieval.
Logging everything: Full prompts and outputs can expose sensitive information, increase storage needs, and make investigations harder. Collect only what is justified.
Watching averages alone: A small number of very slow requests may have little effect on the average but badly affect users. Include tail percentiles and time-to-first-token.
Alerting on noisy single events: One failed generation may be transient. Use windows, minimum traffic thresholds, and severity levels appropriate to the impact.
Treating an evaluator score as objective truth: Automated judges can be inconsistent or miss domain-specific errors. Calibrate them with human-reviewed examples.
Ignoring changes and context: Without model, prompt, and deployment identifiers, it is difficult to connect a quality drop to a release or configuration change.

Tips

Keep metric labels bounded; use trace fields for per-request details instead of labels that create a separate time series for every user or prompt.
Track retrieval quality separately from generation quality, including whether useful documents were found and whether the answer cites or uses them correctly.
For multi-step workflows, report both end-to-end success and the health of each dependency. This helps distinguish a model issue from a tool or data-source issue.
Use a consistent event schema and document what each metric means, its unit, and how it is calculated.
Review a sample of real traces regularly, with access controls and redaction in place. Dashboards show trends; traces explain individual failures.

Troubleshooting

Requests succeed but users report bad answers: Check outcome and quality signals, then inspect a permitted sample of traces. Verify retrieval results, prompt version, output validation, and whether the evaluation set covers this kind of request.

Traces are missing or disconnected: Confirm that trace context is passed through asynchronous jobs, tool adapters, and retries. Check sampling settings and ensure each service uses the same trace context rather than starting an unrelated trace.

Latency alerts fire without an obvious cause: Compare end-to-end latency with per-stage spans and percentiles. Look for slow retrieval, queueing, retries, time to first token, or a dependency whose duration recently changed.

Metrics disagree with request counts: Check whether cancelled streams, retries, background tasks, and failed requests are counted consistently. Define whether metrics count attempts or user-level interactions and use that definition everywhere.

Telemetry contains data it should not: Stop or restrict the affected collection path, follow your organization’s incident and data-handling process, and remove or expire stored data where policy allows. Add redaction tests before re-enabling collection.

Frequently Asked Questions

Which metrics matter most for an LLM application?
Start with end-to-end success, failure and timeout rates, p95 latency, time to first token, token usage, and workflow-specific quality checks. Add metrics that reflect actual user outcomes.

Should I store prompts and model responses?
Not by default. First determine whether metadata or redacted samples are sufficient. If content is needed, apply access controls, retention limits, and approved data protections.

How do I monitor answer quality in production?
Combine offline evaluations, output validation, user feedback, and human review of a suitable sample. Check evaluation results against reviewed examples and investigate meaningful changes.

How often should I run evaluations?
Run them when prompts, models, retrieval, tools, or relevant data change, and on a regular schedule for important workflows. Use production monitoring to identify new cases for the test set.

Do I need distributed tracing?
It is especially useful when a request crosses multiple services or tools. A trace helps connect the user-visible result to the model call, retrieval, and other operations that contributed to it.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.