ThisCalculator

Everyday

Google Test Study: Spec-Driven Testing vs Hidden Criteria

Updated · Standard formula, shown below

Average pace 6.00 min/km
Average speed
10.00 km/h
Result
Time to complete
30 minutes
You entered
Distance
5 km
You entered

Google’s own testing research and engineering practices support spec-driven testing as a core strategy, but they do not support the idea that hidden or undocumented criteria should be ignored. The key finding is that while writing tests from a clear specification catches most intended behavior, hidden criteria—such as edge cases, performance constraints, and security requirements—still require explicit attention and dedicated tests. This article explains what the research actually shows, how to balance both approaches, and why the debate is more nuanced than a simple either/or.

Background: What Google’s Testing Research Actually Says

Google’s engineering culture is often cited in discussions about testing strategies, largely due to the influential book Software Engineering at Google (2019) by Titus Winters, Tom Manshreck, and Hyrum Wright. The book describes how Google approaches testing at scale, emphasizing that tests should be specification-driven—that is, written from a clear, explicit description of what the code is supposed to do. This approach has been credited with helping Google maintain a massive codebase with millions of tests.

However, the book also explicitly acknowledges that specifications are rarely complete. Hidden criteria—requirements that are not in the formal spec but are still necessary for correct behavior—are a known challenge. The authors note that “the spec is not the code” and that tests must also cover behaviors that are implied by the spec, such as error handling, performance, and security. In practice, Google’s testing pyramid and its emphasis on code coverage and test size (unit, integration, end-to-end) are designed to catch hidden criteria through a combination of developer-written tests and automated analysis tools.

How Spec-Driven Testing Works

Spec-driven testing (sometimes called specification-based testing) involves deriving test cases directly from a formal or informal specification of the software’s expected behavior. The idea is that if you know the inputs and expected outputs, you can write tests that verify the implementation matches the spec.

  • Input domains: Test boundary values, typical values, and invalid values.
  • Output verification: Check that the output matches the expected result for each input.
  • State transitions: If the software has states, test that transitions occur as specified.
  • Error handling: Verify that invalid inputs produce appropriate errors.

For example, if a function is specified to return the square root of a number, the spec might say: “For any non-negative input x, return the non-negative square root.” Tests would include x=0, x=1, x=4, x=100, and x=-1 (expecting an error). This is straightforward and effective for well-defined functions.

What Are Hidden Criteria?

Hidden criteria are requirements that are not explicitly stated in the specification but are still necessary for the software to be correct, usable, or secure. They often emerge from real-world usage, integration with other systems, or non-functional requirements like performance and reliability.

  • Edge cases: Empty inputs, very large numbers, or unexpected data types.
  • Performance: The spec might not say “must respond in under 100ms,” but users expect it.
  • Security: The spec might not mention SQL injection, but the code must be safe.
  • Compatibility: The spec might not list all browsers, but the app should work in all major ones.
  • Usability: The spec might not say “button must be visible,” but it should be.

These criteria are “hidden” because they are not written down anywhere, yet they are critical to the software’s success. Spec-driven testing alone will miss them if the spec is incomplete.

Why Spec-Driven Testing Isn’t Enough

If you only write tests that match the specification, you will likely miss hidden criteria. The specification is a model of the intended behavior, but it is not the behavior itself. Real-world inputs are messier, and users interact with the software in ways the spec author didn’t anticipate.

Consider a function that calculates the average of a list of numbers. The spec might say: “Return the arithmetic mean of the list.” A spec-driven test would check [1,2,3] → 2, [10,20] → 15, and maybe [0] → 0. But what about an empty list? The spec didn’t say. Should it return 0, throw an error, or return NaN? That’s a hidden criterion. Similarly, what if the list contains very large numbers that cause overflow? Or what if the list is a generator that can only be iterated once? These are all hidden criteria that a spec-driven test would miss.

Google’s research suggests that teams should supplement spec-driven tests with property-based testing (like QuickCheck) and mutation testing to uncover hidden criteria. Property-based testing generates random inputs and checks that certain properties hold (e.g., “the average is always between the min and max”). Mutation testing introduces small changes to the code to see if tests catch them, revealing gaps in test coverage.

How to Balance Spec-Driven and Hidden Criteria

In practice, the best approach is to use spec-driven testing as the foundation, then layer on techniques to expose hidden criteria. Here’s a practical process:

  1. Write spec-driven tests first for all documented behaviors.
  2. Add boundary and edge-case tests for inputs at the edges of the specified domain.
  3. Use property-based testing to generate random inputs and verify invariants.
  4. Add integration tests to ensure the component works with other components.
  5. Include performance and security tests for non-functional requirements.
  6. Review code coverage to identify untested branches and add tests for them.
  7. Involve QA and real users to discover hidden criteria through exploratory testing.

This balanced approach is what Google actually does, despite the common oversimplification that they rely solely on spec-driven testing.

Common Misconceptions and Mistakes

One common misconception is that “spec-driven testing” means you don’t need to think about edge cases—that the spec is the only source of truth. In reality, the spec is a starting point, not a contract written in stone. Another misconception is that hidden criteria are always unknowable in advance. Many can be anticipated with experience and domain knowledge.

A common mistake is to write tests that are too tightly coupled to the implementation, which makes them brittle and less useful for catching hidden criteria. For example, testing that a function calls a specific internal method is not the same as testing that it produces the correct output. Another mistake is to rely solely on code coverage metrics, which can be gamed by writing many trivial tests that don’t actually verify behavior.

Finally, teams often forget that the specification itself can be wrong or incomplete. Tests should be reviewed by someone other than the author to catch assumptions that are hidden in the spec.

Frequently Asked Questions

Does Google’s testing research say spec-driven testing is the only way?

No. The research emphasizes that tests should be based on specifications, but it also acknowledges that specifications are incomplete. Google uses a variety of testing techniques, including property-based testing and code coverage analysis, to catch hidden criteria.

What is the difference between spec-driven testing and test-driven development (TDD)?

Spec-driven testing is about deriving tests from a specification, while TDD is a development practice where you write a failing test first, then write code to make it pass. TDD can be spec-driven if the test is based on a spec, but it can also be driven by other requirements.

How do I find hidden criteria for my software?

Start by brainstorming edge cases, then use property-based testing to explore the input space. Also, consider non-functional requirements like performance, security, and usability. Talk to users and QA to uncover behaviors they expect that aren’t in the spec.

Can hidden criteria be added to the specification later?

Yes, and that’s a good practice. When you discover a hidden criterion, you should update the specification and add a test for it. This makes the spec more complete over time and reduces the chance of future regressions.

Is property-based testing better than spec-driven testing?

They serve different purposes. Property-based testing is great for finding hidden criteria because it generates random inputs, but it’s not a replacement for spec-driven tests that verify specific documented behavior. Use both.

What if my team lacks time to write both spec-driven and hidden-criteria tests?

Prioritize based on risk. Focus on the most critical parts of the system where hidden criteria are likely to cause serious problems. Even a small set of property-based tests can catch many issues, and you can gradually expand coverage.

Conclusion

Google’s testing research supports spec-driven testing as a valuable practice, but it does not support ignoring hidden criteria. The most effective testing strategy combines spec-derived tests with techniques like property-based testing, edge-case analysis, and exploratory testing to ensure both documented and undocumented requirements are met. By understanding the limits of spec-driven testing, you can build a more robust test suite that catches real-world issues before they reach users.

Related

Formula used
in t | Time to complete | minutes | 30 | 0
in d | Distance | km | 5 | 0
out pace | Average pace | min/km | t / d | 2
out speed | Average speed | km/h | d / (t / 60) | 2

Each line is one input (in, sel, list) or one result (out: label, unit, formula, decimals). How the calculators work

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.