Schema-Matched JSON from an LLM: How to Validate It
Schema-matched JSON gives your application structured data in a predictable shape, while validation checks that the response is valid JSON and follows the rules you set. You’ll need a schema, an LLM that can return structured output or plain text, and a validator in your application; the exact code depends on your language and provider.
What You’ll Need
- A clear definition of the fields your application expects, including their types and which are required.
- A JSON Schema or an equivalent validation model, such as a typed model supported by your programming language.
- Access to an LLM API or application that can return a response you can capture as text or structured data.
- A JSON parser and schema-validation library for your language. Check the library’s documentation for supported JSON Schema versions and configuration.
- A few representative test prompts, including one that should produce invalid or incomplete data.
Step-by-Step
- Define the output contract. Write down the exact data the next part of your application needs. For example, an event might require a title, a date, and a location. Decide whether each value is a string, number, Boolean, array, object, or nullable value. Be explicit about date formats, allowed values, and whether extra fields are acceptable. Avoid vague requirements such as “a reasonable date”; use a checkable format or rule.
- Write a schema that captures those rules. In JSON Schema, set the appropriate type, list required properties, and define each property’s type. Add constraints such as enumerated values or numeric limits only when they reflect a genuine application requirement. For example, an event object could require a string title and a date string matching a format your validator supports. Remember that declaring a format may be treated as an annotation unless format checking is enabled.
- Choose how the model will return data. If the provider offers a structured-output or schema-constrained mode, pass the schema using that feature and follow its rules for supported keywords. This can reduce formatting errors, but it does not replace validation in your application. If constrained output is unavailable, ask for one JSON object that follows the schema, and instruct the model not to add Markdown fences or commentary. Treat that instruction as guidance, not a guarantee.
- Make the prompt specific and bounded. Include the requested fields, expected types, and any important interpretations. Supply source material for the model to extract from, and say what to do when information is missing—for example, return null only if the schema allows null, or use a defined status field. Do not ask the model to invent missing values. Keep prompt instructions consistent with the schema; a prompt cannot make a disallowed value valid.
- Capture the actual response content. Read the response according to the API’s documented structure. Some structured-output APIs provide parsed data or a dedicated output field, while others return text that you must parse. Check for refusal, truncation, or other response status indicators before treating the content as JSON. Do not assume the entire response object is the JSON payload.
- Parse the response as JSON. If you received text, pass it to a standard JSON parser and handle parse errors. Do not repair malformed output with broad string replacements or extract the first braces from arbitrary prose; those shortcuts can silently change values or accept the wrong content. If parsing fails, record a safe diagnostic and decide whether to retry or return an error to the caller.
- Validate the parsed value against the schema. Run the parsed object through a validator configured for the schema version and the constraints you use. Check the validator’s result rather than assuming that successful parsing means the data is correct: syntactically valid JSON can still have missing properties, wrong types, or disallowed values. If validation fails, capture the useful error path and rule, such as
$.datehaving the wrong type. - Handle failures deliberately. Decide whether to retry, request a correction, or stop and report an error. A repair prompt should include the validation problems and ask for a complete replacement object, not a patch that your code cannot safely merge. Set a retry limit, and validate the new response from the beginning. Never bypass validation just because a retry still fails.
- Use only validated data. Pass the value onward only after parsing and validation succeed. Apply any additional business rules separately—for instance, whether a date is relevant to the task or a value is permitted for a particular user. A schema checks structure and declared constraints; it does not prove that the model’s claims are true or appropriate for your application.
- Test edge cases and monitor changes. Try missing fields, extra fields, wrong types, empty strings, boundary numbers, invalid JSON, and unusually long responses. Keep tests for expected valid and invalid cases. When changing the prompt, model, schema, or validator settings, rerun them; behavior and supported structured-output features can differ across providers and versions.
Common Mistakes
- Confusing valid JSON with valid data: Parsing checks JSON syntax. Schema validation checks the shape and constraints. You generally need both.
- Leaving the schema ambiguous: If a field may be absent, null, or an empty string, specify which cases are allowed. These are not interchangeable.
- Assuming every schema feature is enforced: Providers and validator libraries may support different JSON Schema drafts or keywords. Verify behavior and test it.
- Trusting a format annotation automatically: A date or email format may not be checked unless the validator enables the relevant format checks.
- Accepting extra prose or Markdown: A response wrapped in code fences is not itself a JSON object. Prefer structured output where available, or parse strictly and handle failure.
- Using validation as fact-checking: A response can match every field and type while containing incorrect information. Validate facts against trusted sources when accuracy matters.
Tips
- Keep the schema as small as practical, and make each constraint purposeful. Overly restrictive rules can reject useful output; overly loose rules allow unusable values.
- Use stable field names and a versioned schema when other services depend on the output. Change the schema and its consumers together.
- Return concise validation feedback to a repair attempt, but do not expose secrets, internal prompts, or sensitive source data in logs or error messages.
- Set limits for response size, retries, and processing time. These protect your application from unexpectedly large or repeated responses.
- Keep raw model output separate from validated data, and avoid logging personal or confidential content unless your data-handling rules permit it.
Troubleshooting
The response is not parseable. Check whether you captured the right response field, whether the model added prose or a code fence, and whether the output was cut off. Use the provider’s structured-output mode if available. Otherwise, retry with a clear JSON-only instruction and handle repeated failure as an error.
Parsing succeeds but validation fails. Read the error path and compare the actual value with the schema. Common causes include a number returned as a string, a missing required field, an unexpected null, or a constraint the model cannot satisfy. Correct the prompt or schema only if the intended contract has changed.
A format rule appears to be ignored. Check the validator’s documentation and configuration for format assertion support. Add explicit validation for the format if needed, and test both valid and invalid examples. Do not assume the model’s structured mode enforces every format rule.
The provider rejects the schema. Confirm the supported schema version and structured-output limitations. Reduce unsupported or unnecessary keywords, or use ordinary JSON output followed by your own validator. Keep application-side validation in either case.
Retries keep producing the same error. Include the specific failing field and expected type in a correction request, confirm the schema and prompt do not conflict, and cap retries. If the data is unavailable, make the schema and prompt define a valid missing-data response rather than encouraging a guess.
Frequently Asked Questions
Does schema-constrained output guarantee valid JSON?
It can improve compliance, but you should still check response status, parse the content when needed, and validate it in your application.
Is a JSON parser enough?
No. A parser checks syntax, not required fields, types, allowed values, or application-specific rules.
Should I use JSON Schema or a typed model?
Use the option that fits your stack and integrations. Either way, define the contract clearly and confirm that your validator enforces the rules you depend on.
What should I do if a required value is unknown?
Choose an explicit representation, such as an allowed null or a status field, and encode it in the schema. Do not let the model make up a value to fill the gap.
Can a schema verify that the answer is true?
No. It checks structure and declared constraints, not factual accuracy. Verify important claims against appropriate sources.
Should invalid output be sent to the next step?
No. Reject it, retry within a limit, or return a controlled error. Only pass data onward after it has passed parsing and validation.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.