LLM Model Upgrade Tests in TypeScript

Updated 9 November 2026 · Standard formula, shown below

Before upgrading an LLM model in a TypeScript application, test whether the new model preserves task quality, response structure, safety behavior, latency, and cost across representative inputs. Run the old and new models against the same versioned evaluation set, validate outputs with your production schemas, and then verify the change in a staged rollout before switching all traffic.

Why model upgrades need testing

A model upgrade can change more than the wording of an answer. Models may differ in instruction following, tool selection, refusal behavior, context handling, and output consistency. Even when an API request still succeeds, a subtle change—such as returning an explanation around JSON—can break code that expects a particular shape.

Separate model behavior from other changes where possible. Keep the prompt, tools, sampling settings, retrieval sources, and application code fixed while comparing models. If several things change together, a failing test will not tell you which change caused it. Record the exact model identifier and configuration used for each run; a movable alias may point to a different model later.

Build a representative evaluation set

Use examples that reflect real application traffic, including routine requests and difficult edge cases. Remove secrets and personal data before saving prompts, and review examples for representativeness. A tiny set of hand-picked “easy” prompts can miss regressions that users encounter.

Include common tasks, unusual phrasing, typos, long inputs, empty or incomplete inputs, and requests outside the system’s intended scope.
For retrieval or question-answering, include cases where the answer is present, absent, ambiguous, or contradicted by the supplied context.
For tool-using systems, test when a tool should be called, when it should not be called, and how errors or empty results are handled.
Include known failure cases from support reports or production monitoring, with sensitive details removed.
Keep a held-out set for final checks so that repeated prompt tuning does not make the evaluation examples the only cases the system handles well.

Store cases as typed data rather than informal notes. For example, a test case can hold an input, expected properties, and tags:

type Case = {
  id: string;
  input: string;
  tags: string[];
  expected?: { mustMention?: string[]; mustNotContain?: string[] };
};

Use broad, testable expectations when exact wording is not required. For creative or conversational tasks, a reviewer rubric may be more useful than an exact-string comparison.

Test quality and task-specific behavior

Choose checks that match what the application actually promises. Exact matches are suitable for fixed labels or deterministic formats, but usually make poor tests for open-ended prose. Instead, check factual requirements, missing details, relevance, citation or source use where applicable, and whether the answer follows the requested format. Human review remains important for subjective quality; automated graders can help scale review, but their judgments are not ground truth.

Track results by task category as well as overall. A higher average score can conceal a serious drop in a small but important workflow. Compare the candidate with the current model on the same inputs and report the sample size, scoring method, and failures. For nondeterministic outputs, repeat selected cases and assess the range of behavior instead of treating one successful response as proof.

In TypeScript, make model output pass through runtime validation. Type annotations alone do not validate data received over a network:

const parsed: unknown = JSON.parse(responseText);
const result = OutputSchema.parse(parsed);

The example assumes a runtime schema library defines OutputSchema. Handle parse and validation failures explicitly. If the model supports constrained structured output, test both the requested format and the application’s fallback behavior when a response is incomplete or invalid.

Check safety, tools, and failure handling

Test safety behavior in the context of the product’s use, not only with generic prompts. Check that the system refuses or redirects disallowed requests when appropriate, does not expose private prompt or retrieved data, and treats untrusted user content as data rather than instructions. These tests reduce risk but cannot prove a system is safe in every situation.

For tool calls, verify the selected tool, argument types, required fields, and the application’s authorization checks. Do not let a model’s proposed arguments bypass ordinary validation or access controls. Test malformed arguments, unavailable tools, timeouts, and tool errors. Confirm that the application does not report an action as completed if the tool failed.

Also exercise operational failures: rate limits, network errors, empty responses, truncated output, and cancellation. Check retry limits and timeouts to avoid retry loops or duplicate side effects. A retry may be appropriate for a transient read-only request, but repeating a tool action can have consequences unless the action is made idempotent or otherwise protected.

Measure latency, token use, and cost

Quality is only one part of upgrade readiness. Measure end-to-end latency under realistic request sizes and concurrency, including time spent in retrieval, tools, retries, and output validation. Compare typical and slow responses, not just a single best-case call. Verify that the application’s timeout and user-interface behavior still make sense.

Track input and output token usage where the API exposes it, and estimate cost using the provider’s current published rates and the expected request volume. Rates and model limits can change, so look them up rather than hard-coding assumptions into an evaluation. Include longer prompts, larger responses, and tool cycles: average cost per request can hide expensive outliers.

Run the comparison in TypeScript

Keep the evaluation runner separate from production request handling. It should call each model with identical case data and settings, capture the model identifier, response, usage, elapsed time, and errors, then apply the same validators and scoring rules. Avoid logging sensitive prompt or response content to shared CI output.

Pin the old and candidate model identifiers and freeze the prompt, tools, and relevant settings.
Run the evaluation set against both versions, recording failures and metrics per case and category.
Review changed or ambiguous outputs manually, especially safety-sensitive and high-impact workflows.
Set release thresholds before looking at results—for example, required schema validity and acceptable latency or cost—so the decision is not made by cherry-picking.
Deploy to a limited, monitored cohort if feasible, compare real outcomes, and keep a rollback path.

For reproducibility, save a compact report with case-set version, code revision, model identifier, settings, date, aggregate metrics, and links to access-controlled failure samples. Do not assume that identical settings guarantee identical output; model services can be nondeterministic, and provider-side changes may affect behavior.

Frequently Asked Questions

Can I test an LLM upgrade with unit tests alone?
Unit tests are useful for application logic, parsing, validation, retries, and tool safeguards. They do not establish that a model still performs the intended task. Combine them with a representative model evaluation set and, where practical, staged production monitoring.

Should I compare exact responses?
Only when the output is contractually or functionally exact, such as a fixed code or label. For natural-language answers, exact-string tests are brittle; check required content, prohibited content, structure, and task-specific quality instead.

How many prompts should be in the evaluation set?
There is no universal count. Include enough examples to represent important workflows and known edge cases, and report the sample size. A small set can be a useful smoke test, but it gives limited evidence about rare failures.

Do TypeScript types guarantee valid model output?
No. TypeScript types are checked at compile time and do not validate arbitrary runtime data. Parse and validate the response at runtime, then handle invalid output without assuming the model will always follow the requested format.

What if the new model scores better but costs more?
Compare quality, latency, and cost together by workflow. Decide whether the improvement justifies the added expense for the relevant use case, and consider routing only tasks that benefit from the more capable model if the architecture supports that safely.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.