LLM Job Records: Five Runs and the Failures That Matter

Updated 5 November 2026 · Standard formula, shown below

Running one large language model job five ways means holding the task steady while changing one execution or evaluation condition at a time; the records matter because they show whether a failure is reproducible, attributable, and important to users. A useful comparison does not merely count failed runs: it records what happened, under which conditions, and whether the difference came from the model, the prompt, the tools, or the surrounding system.

What “one job five ways” means

An LLM job might be answering a support question, extracting fields from an invoice, or producing a short summary. “Five ways” is not a standardized testing term. It can mean five model configurations, five prompts, five tool or deployment paths, or five attempts at the same task. Those interpretations answer different questions, so define the comparison before running it.

For example, to compare five prompts, keep the model, input document, output requirements, tools, and other settings fixed. To compare five models, keep the prompt and evaluation criteria fixed as far as possible. If several variables change together, a different result cannot be confidently assigned to any one of them.

Also distinguish five variants from five repeated runs. Variants test deliberate changes; repeats test whether a configuration behaves consistently. A single run for each of five variants is a useful first look, but it is not enough to estimate reliability or characterize rare failures.

What to record for each run

A record should let another person understand and, where practical, reproduce the run. Save the task input or a privacy-safe reference to it, the exact prompt and relevant configuration, the timestamp, and the outcome. Note model and tool versions when available; hosted systems may change over time, so a model name alone may not identify an immutable setup.

Run identity: unique run ID, date and time, test case ID, and the variant being tested.
Configuration: model identifier, prompt version, decoding settings, tool definitions, retrieval or context sources, and relevant software versions.
Execution: whether the request completed, latency, retries, rate limits, tool calls, and any timeout or system error.
Output and evaluation: the response or protected reference to it, expected result where one exists, evaluation method, and reviewer notes.
Impact: severity, affected user or workflow, whether a human caught the issue, and whether it caused a consequential action.

Keep operational failures separate from answer-quality failures. A timeout is not the same as a fabricated answer, and a valid answer returned after a retry is not equivalent to a job that never completed. Record both the first attempt and the final outcome, including retries, rather than silently replacing the original failure.

Failures that deserve attention

Useful failure categories describe what went wrong and why it matters. Categories should be specific enough to guide a fix, but not so elaborate that reviewers cannot apply them consistently.

Task failure: missing required fields, incorrect calculation, unsupported claims, or failure to follow a clear constraint.
Grounding failure: an answer conflicts with the supplied source, cites evidence that does not support it, or invents unavailable details.
Tool or integration failure: a malformed tool request, incorrect argument, failed retrieval, or a result that the model misreads.
Robustness failure: inconsistent behavior across equivalent inputs, sensitivity to harmless wording changes, or failure on a known edge case.
Safety or privacy failure: disclosure of protected information, unsafe assistance, or mishandling of content that should be restricted.
Operational failure: a timeout, crash, unexpected cost or latency spike, or an unhandled rate limit.

Severity depends on context. A misspelled word in an internal draft may be minor; a wrong account number in an automated payment workflow may be critical. Assess consequence, likelihood, reach, and detectability. A low-frequency failure can still matter if it is hard to notice and has serious consequences.

How to compare the five runs fairly

Start with a small, representative set of cases, including ordinary examples and known edge cases. Use the same inputs for each variant when the goal is a controlled comparison. Decide beforehand what counts as success: exact match may suit a fixed identifier, while a summary may need a rubric for factual coverage, unsupported claims, and readability.

For every variant, report the denominator as well as the failures. “Three failures” is hard to interpret without knowing whether there were five attempts or five thousand. Show completion rate, failure rate, and the number of cases in each category. For small samples, present raw counts and avoid implying that a tiny difference proves one setup is better.

For example, if one prompt produces a correct extraction on 18 of 20 cases and another on 19 of 20, that is a one-case difference in this sample, not proof of a stable improvement. Inspect the differing case, repeat tests where appropriate, and expand the dataset before making a high-impact decision. When outputs are stochastic, preserve the sampling settings and perform repeated runs if variability itself is part of the question.

Common mistakes and practical safeguards

The most common analytical mistake is changing multiple things at once. Another is treating a model’s fluent answer as evidence of correctness. Use a reference answer, source-grounded checks, or qualified human review appropriate to the task. Automated evaluators can help triage results, but they can also miss errors or favor particular wording; record the evaluator and its rubric.

Do not discard failed requests, retries, or empty outputs from the dataset.
Do not combine unlike failure types into one score without retaining the underlying categories.
Do not expose confidential prompts, personal information, or customer data in an unrestricted log; limit access and retain only what testing requires.
Do not claim broad reliability from five observations. Treat a five-run exercise as a diagnostic, not a definitive benchmark.
Do not compare results collected under materially different model versions or infrastructure without noting the change.

A concise record can be a table, structured log, or test-management entry. The format matters less than consistent fields, clear definitions, and traceability from a failure back to its input and configuration.

Frequently Asked Questions

Is five runs enough to evaluate an LLM?
Usually not for a reliable estimate of overall performance. Five runs can reveal obvious differences or reproducible defects, but broader claims need enough representative cases and, when outputs vary, repeated trials.

Should I count a retry as a success?
Record the initial failure and the eventual result separately. Whether a retry counts as successful depends on the workflow; report retry rate and final completion rate so the operational burden is visible.

What is the difference between a model error and a system failure?
A model error is a problem in the generated response, such as an unsupported claim. A system failure prevents or disrupts execution, such as a timeout. Tool failures can sit between them, so describe the observed behavior rather than forcing an unclear label.

Can I compare different models using the same prompt?
Yes, as a controlled starting point, but the prompt may suit one model better than another. Keep the first comparison fixed, then test prompt adjustments separately and report both stages.

What should I do when reviewers disagree?
Preserve the disagreement, clarify the rubric, and have reviewers assess the same examples where feasible. For consequential evaluations, use adjudication or a documented tie-breaking process instead of silently choosing one rating.

Turning records into decisions

Use the five-way comparison to find what needs investigation, not to manufacture a winner. Review high-severity failures first, identify whether they cluster around particular inputs or tools, and make one targeted change at a time. Then rerun the affected cases and a stable regression set. A dependable record makes it possible to see whether a fix solved the problem, shifted it elsewhere, or merely changed how it appears.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.