API Test Data: LLM-Generated JSON
When testing APIs, you need JSON data that is realistic, varied, and structurally valid. Large language models (LLMs) can generate such data quickly, but they also introduce unique pitfalls like invalid types, inconsistent enums, and hallucinated fields. This article explains what to know before using LLM-generated JSON in your API tests, covering generation strategies, validation, and common failure modes.
Why Use an LLM for Test Data
Hand-writing JSON fixtures is slow, and faker libraries often produce data that is too uniform. An LLM can create a diverse set of objects with realistic names, addresses, and nested structures in seconds. It is especially useful when you need:
- Many variations of a complex schema (e.g., an order with line items, discounts, and shipping details).
- Edge cases like empty arrays, null fields, or very long strings.
- Data that mimics real user input, including typos and unusual formatting.
However, the model does not know your API contract. It guesses based on your prompt, so the output is only as good as the schema you provide and the validation you apply afterwards.
How LLMs Generate JSON
LLMs are next-token predictors. When you ask for JSON, they generate text that looks like JSON based on patterns in training data. They do not run a parser or check types. This means the output can be syntactically valid but semantically wrong—for example, a price field as a string "19.99" when your API expects a number. Some models are better at following JSON schemas than others, but none are perfectly reliable. You must treat the output as a draft, not a final fixture.
Common Problems with LLM-Generated JSON
Here are the most frequent issues testers encounter:
- Invalid JSON syntax: trailing commas, missing brackets, or unquoted keys. This happens more often with smaller models or when the prompt is vague.
- Wrong data types: a string where a number is expected, or an object instead of an array.
- Inconsistent enums: your API expects
status: "active"but the model returnsstatus: "Active"or"enabled". - Hallucinated fields: the model adds properties like
"middle_name"that your API does not recognize. Many APIs ignore unknown fields, but strict ones reject them. - Unrealistic values: an age of 300, a negative price, or a date like
2023-02-30. - Missing required fields: the model omits a field because it thinks it is optional.
How to Generate Reliable Test Data
Follow these steps to get usable JSON from an LLM:
- Provide an exact schema in the prompt. Use JSON Schema or a sample object with all required fields, types, and allowed enum values.
- Ask for a specific number of objects and a consistent structure. For example, “Return an array of 10 user objects, each with the fields: id (integer), name (string), email (string, valid format), status (one of active, inactive, pending).”
- Request variety: “Include at least one empty array, one null value, and one very long string.” This helps cover edge cases.
- Validate the output with a JSON parser and a schema validator (like Ajv for JSON Schema). Do not skip this step.
- Post-process: write a small script to coerce types, fill missing fields with defaults, or replace unrealistic values with ones from a faker library.
- Review manually for subtle issues like duplicate IDs or inconsistent references between objects.
Validation and Sanitization
Validation is non-negotiable. Even if the model returns syntactically valid JSON, it may violate your API’s contract. Use a schema validator to check types, required fields, and enum values. For example, if your API expects price as a number, the validator will catch a string. After validation, sanitize the data:
- Convert strings to numbers where needed.
- Trim whitespace from strings.
- Normalize enums to lowercase.
- Replace invalid dates with a fixed valid date.
- Regenerate any field that fails validation.
This step turns a probabilistic output into a deterministic fixture.
Comparing LLM Generation with Other Methods
LLMs are not always the best tool. Here is a quick comparison:
- Hand-written fixtures: full control, but time-consuming and often too few variations.
- Faker libraries: fast, type-safe, and consistent, but limited to predefined patterns; they cannot easily generate nested business objects.
- LLM generation: flexible and fast, but requires validation and can be non-deterministic.
- Hybrid approach: use an LLM to create a template or a few examples, then use a faker library to fill in values. This combines variety with reliability.
For most API tests, a hybrid approach is the most practical. Use the LLM to generate the structure and edge cases, then replace random values with a faker to ensure realism.
Cost and Performance Considerations
Generating large volumes of test data via an LLM API can be costly and slow. If you need thousands of objects, consider generating a few hundred and then using a script to duplicate and vary them. Also, the model’s token limit may truncate long outputs, so you may need to generate in batches. For CI/CD pipelines, caching generated data or committing validated fixtures to your repository avoids repeated API calls.
Frequently Asked Questions
Can I trust LLM-generated JSON without validation?
No. LLMs are not reliable enough to produce valid, contract-compliant JSON every time. Always validate with a JSON parser and a schema validator before using the data in tests.
What is the best prompt for generating JSON test data?
Include the exact JSON schema or a sample object, specify the number of objects, list the required fields with their types, and mention edge cases you want covered. For example: “Return 5 user objects with fields id (integer), name (string), email (string), status (one of active, inactive, pending). Include one with an empty email and one with a null status.”
Why does the model sometimes add extra fields?
LLMs generate text based on patterns, not rules. They may add fields they have seen in similar contexts. To prevent this, explicitly state that the output must contain only the listed fields.
How do I handle inconsistent enum values?
Use a schema validator to check against your allowed enum list. Alternatively, in your prompt, list the exact allowed values and ask the model to use them verbatim. Post-processing can also normalize case.
Is it better to use a faker library instead?
It depends on your needs. Faker libraries are reliable and fast for simple fields, but they cannot generate complex nested structures or domain-specific data. For those, an LLM is useful, but you must validate the output.
Can I use LLM-generated JSON for security testing?
You can use it to generate fuzzing inputs, but you should not rely on it to produce malicious payloads. For security tests, use dedicated fuzzing tools that generate random, malformed data based on your API schema.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.