LLM JSON Validation Before Downstream Jobs
Before an LLM-generated JSON object reaches a downstream job, check that it is valid JSON, matches the expected schema, contains acceptable values, and is safe for the job to use. Treat model output as untrusted input: parse it, validate it, apply business rules, and route failures to a controlled retry or review path instead of passing them through.
Why validation matters
A response can look like JSON to a person and still fail a parser or cause incorrect work. It may include explanatory text outside the object, use a string where a number is required, omit a field, or produce a value that is syntactically valid but operationally wrong. Downstream jobs often assume inputs are consistent; an unchecked mismatch can lead to rejected records, incorrect routing, or partial processing.
Structured-output features and constrained decoding can reduce formatting errors, but they do not establish that the content is true, complete, authorized, or suitable for a particular action. Validation therefore needs more than a “does it parse?” test.
Parse the complete response
First identify exactly which part of the model response is intended to be JSON. If the integration expects a JSON object, reject extra prose, Markdown fences, or multiple objects rather than extracting a convenient substring with a loose regular expression. Substring extraction can hide malformed output or discard meaningful content.
- Use a standards-compliant JSON parser, not text matching, to parse the response.
- Require the expected top-level type, such as an object or array.
- Set reasonable limits on response size and nesting depth where the parser or application supports them.
- Handle parsing errors explicitly. Do not silently replace malformed output with an empty object or a guessed value.
JSON syntax has details that differ from common programming-language object notation: property names and strings need double quotes, trailing commas are not allowed, and values such as comments, undefined, and NaN are not JSON values. A parser should be the authority on whether the document is valid.
Validate the schema and types
Once parsing succeeds, compare the result with a documented schema. Check required properties, permitted properties, data types, array item shapes, and any fixed formats. Decide deliberately whether unknown fields should be rejected or ignored. Rejecting them can catch drift; ignoring them can support compatible additions, but only when the consumer is designed to tolerate them.
For example, a job expecting {"customer_id":"C104","priority":2} should not assume that {"customer_id":104,"priority":"high"} is equivalent. The first changes an identifier from a string to a number; the second changes a numeric priority into an unrecognized label. Type coercion may appear convenient, but implicit conversions can erase leading zeroes, accept ambiguous formats, or conceal model mistakes.
Make nullability and optionality clear. A missing field is not necessarily the same as a field set to null, an empty string, or an empty array. Define what each means to the downstream job and validate accordingly.
Check meaning and business rules
A schema verifies shape, not correctness. Add checks for the rules that make a value usable in its context. These may include allowed enum values, string length limits, date formats and ranges, numeric bounds, and relationships between fields. For instance, a start date may need to precede an end date, or a status may be required when a result is marked complete.
- Validate identifiers against the expected format, but do not assume a well-formed identifier exists or belongs to the current user.
- Check units, currency codes, time zones, and date conventions when those affect interpretation.
- Use authoritative application data to verify claims that must be true, such as whether a referenced record exists.
- Apply authorization and policy checks in the application; a model-generated field must not grant permission by itself.
Do not treat fluent or confident wording as evidence. If a downstream action depends on a factual claim, use an appropriate source of truth or a human review step rather than relying on JSON validity.
Handle unsafe content and failure paths
Valid JSON can still contain hostile or inappropriate strings. If a field will be displayed in a browser, escape it for the output context; if it will be included in a query, use parameterized queries rather than string concatenation. Validate file paths, URLs, command arguments, and template values according to the specific operation. JSON encoding alone does not prevent injection or make a value safe.
Keep model output separate from instructions and control data. A response should not be allowed to choose arbitrary tools, destinations, or privileged actions merely because it contains fields with those names. Enforce an allowlist and apply least privilege in the job that performs the action.
Define what happens when a check fails. Depending on the risk and workflow, the job can retry with a clear correction request, send the item to a dead-letter queue, request human review, or stop processing. Bound retries and avoid repeating side effects. If a job might already have acted before a later check fails, use transaction boundaries or idempotency controls so retries do not create duplicate work.
Observability, privacy, and testing
Record enough information to diagnose failures: a request or task identifier, model and prompt version where available, validation stage, and a concise error category. Avoid logging secrets or unnecessary personal data. If retaining raw responses is needed for debugging, set access controls and retention rules appropriate to the data.
Test the boundary with examples that include malformed JSON, missing keys, wrong types, unexpected fields, out-of-range values, long strings, Unicode, and valid-looking but contradictory fields. Include regression tests when a prompt, model, schema, parser, or downstream contract changes. Monitor validation failure rates and downstream outcomes; a sudden change can reveal model drift or an upstream integration change.
Frequently Asked Questions
Is valid JSON safe to send directly to a job?
No. Parsing confirms syntax, not schema compliance, factual accuracy, authorization, or safety. Validate each of those requirements before the job acts.
Should a job automatically repair malformed JSON?
Only in a controlled way. A bounded retry can ask for output matching the schema, but avoid ad hoc text edits that may change meaning. Always parse and validate the repaired response again.
Should unknown fields be rejected?
It depends on the contract. Reject them when unexpected fields could signal a bug or affect behavior; tolerate them only when the consumer intentionally supports forward-compatible additions.
Can structured output replace application validation?
No. It can help produce the requested shape, but application code still needs to check values, business rules, permissions, and downstream-specific safety conditions.
What should happen when validation fails?
Use a defined failure route, such as a limited retry, review queue, or explicit job failure. Preserve useful diagnostics, avoid silent defaults, and prevent retries from duplicating side effects.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.