Spec-Driven Testing for Independent AI Code Verification

Spec-driven testing keeps implementation and verification independent by making the specification—not the code—the shared source of expected behavior. It helps teams catch cases where an AI-generated change and its tests repeat the same mistaken assumption. You’ll need a written, testable specification, a way to run automated tests, and a review process that checks both against the requirements.

What You’ll Need

• A concise specification describing observable behavior, constraints, and important edge cases.
• The project’s test runner and a clean way to run its existing test suite.
• Access to the code and test changes, ideally reviewed separately or in separate commits.
• Representative examples, domain rules, or authoritative reference data where expected results need to be established.

Step-by-Step

1. Write the contract before asking for implementation. State what the system receives, what it must do, and what callers can observe. Include valid and invalid inputs, outputs, side effects, error behavior, and constraints that matter. For example, specify that a date-range function includes the start date but excludes the end date, rather than saying only that it “calculates a range.” Avoid prescribing internal algorithms unless a particular algorithm is required.
2. Make each requirement verifiable. Replace vague terms such as “fast,” “reasonable,” or “handles errors” with observable conditions. Define boundaries and units: whether a limit is inclusive, what happens at zero, and how missing values are treated. If a requirement cannot be checked through an observable result, clarify it before building tests around it.
3. Establish expected results independently. Derive test expectations from the specification, a trusted external reference, or a hand-checked calculation—not by running the new implementation and copying its output. For calculations, work out a few examples independently. For protocol or file behavior, use a documented format or known-valid fixture. Record why each expected value is correct so it can be reviewed.
4. Create acceptance tests from the contract. Write tests that exercise public inputs and observable outputs, rather than checking private helper functions or mirroring the implementation line by line. Cover ordinary cases, boundary values, invalid input, and important interactions. Keep the specification’s language and the test names aligned, while ensuring the expected values have their own justification.
5. Separate test authorship from implementation where practical. One person or process can turn the specification into tests, while another implements the behavior. If one AI-assisted workflow must do both, use distinct stages: freeze and review the requirements and tests first, then provide the implementation task. Do not let the implementation stage quietly rewrite expected results to make a failing solution pass.
6. Ask the AI to implement against the agreed contract. Provide the relevant specification, project conventions, and instructions to avoid changing tests or requirements without approval. Ask for a focused implementation and a summary of assumptions. Treat that summary as a review aid, not evidence that the behavior is correct.
7. Run tests against the implementation, not just against a mock. Execute the acceptance suite using the actual code path, plus the project’s existing unit and integration tests where relevant. Confirm that the tests fail when expected behavior is deliberately absent or incorrect; a test that never detects a known-bad result is not useful verification.
8. Review mismatches by returning to the contract. When a test fails, determine whether the implementation violates a clear requirement, the test misreads the requirement, or the specification is incomplete. Update the specification first when the intended behavior was genuinely undecided, then revise tests and implementation together with a visible rationale. Never treat a green test run as permission to ignore an unresolved ambiguity.
9. Check coverage of requirements and preserve the audit trail. Map every requirement to one or more tests, and note requirements that need manual review because they are difficult to automate. Keep the specification, tests, and code changes reviewable in version control. Before merging, verify that test edits are explained, all relevant checks pass, and no requirement has disappeared during implementation.

Common Mistakes

• Writing tests from the implementation. This makes tests confirm what the code already does, including its defects. Derive expected behavior from the contract or independent evidence.
• Over-specifying internals. Tests that require a particular function layout or algorithm can fail after harmless refactoring. Test externally visible behavior unless internal structure is itself a requirement.
• Using only happy-path examples. A few normal inputs can miss boundary, malformed-input, and interaction failures. Add cases based on explicit requirements and likely edge conditions.
• Letting one agent change code, tests, and specification without review. That creates a closed loop in which all three can drift together. Keep changes separated or require a reviewer to check the contract and rationale.
• Confusing coverage with correctness. High line coverage does not show that expected results are valid. Review test assertions, especially constants, units, rounding rules, and boundary conditions.

Tips

• Give each requirement a short identifier so reviewers can trace it to tests and implementation.
• Prefer small, concrete examples with expected outcomes; include at least one case near each important boundary.
• Use property-based checks for broad invariants, such as output ordering or conservation rules, but keep example-based acceptance tests for specific contract details.
• Use mutation testing or a deliberate incorrect implementation to check whether important tests actually detect defects, when your tooling supports it.
• Keep generated test fixtures and reference data versioned, with their source and assumptions documented.

Troubleshooting

Tests pass, but reviewers still distrust the result: Inspect how expected values were chosen. Recalculate representative cases independently, compare against an authoritative reference where available, and check that tests exercise the real public interface.

Tests fail after an implementation change: Reproduce the failure, identify the specific requirement involved, and compare actual behavior with the stated contract. Do not change an assertion simply to get a green build; revise it only if the contract or test was demonstrably wrong.

The specification is ambiguous: Pause implementation on the affected behavior. Ask the product or domain owner to choose a rule, document examples and boundary behavior, then update acceptance tests before proceeding.

The test suite is brittle or slow: Move most checks to focused tests of public behavior and reserve slower end-to-end tests for important integration paths. Remove assertions about incidental ordering or internal details unless the contract requires them.

AI-generated tests and code keep agreeing with each other: Have a reviewer inspect the tests without relying on the implementation, or derive a small independent set of expected cases from the specification. Keep test changes isolated long enough to review their assumptions.

Frequently Asked Questions

What does spec-driven testing mean?
It means tests are derived from a written description of required behavior, rather than inferred from the implementation being tested.

Does this guarantee AI-generated code is correct?
No. It improves the independence and traceability of verification, but incomplete requirements, weak tests, and incorrect reference data can still miss defects.

Should the specification include implementation details?
Usually not. Describe inputs, observable outcomes, constraints, and required failure behavior. Include implementation constraints only when they are part of the actual requirement.

Can the same AI write both the tests and the code?
Yes, but separate the stages and review the tests against the specification before implementation begins. Independent human or process review provides stronger protection against shared assumptions.

How many tests should each requirement have?
There is no fixed number. Use enough cases to cover normal behavior, meaningful boundaries, and relevant failure conditions, and link them back to the requirement.

What if a requirement cannot be automated?
Record it explicitly and define a manual review, inspection, or acceptance procedure. Do not imply that automated test success verifies behavior the tests cannot observe.

Related

• Digital Divide and AI Access: What It Means
• Angular AI Coding Tools: Benefits and Limits
• Best Multi-Model Coding Agents for Avoiding Rate Limits