Pin Crash in Characterization Test Before Patching?
- Time to reproduce crash
- 5 min
- You entered
- Number of test runs
- 10
- You entered
Yes, you should pin a crash in a characterization test before patching upstream. A characterization test records the current behavior of a system, even if that behavior is buggy, so you can safely refactor or patch without accidentally changing unintended parts. This article explains why pinning the crash first is the safer sequence, how to do it, and what to watch out for.
Formula
The total time to run a characterization test multiple times is:
total = t × n
Where:
- t = time to reproduce the crash once (in minutes)
- n = number of test runs you plan to execute
This simple multiplication helps you estimate whether a full characterization suite is feasible within your time budget before you start patching.
Worked Example
Suppose reproducing the crash takes 5 minutes per run, and you want to run the test 10 times to get a stable baseline. Using the default values from the calculator:
- Multiply the time per run by the number of runs: 5 × 10 = 50
- The total time for 10 characterization runs is 50 minutes.
This matches the calculator output of 50 minutes, rounded to one decimal place.
What Is a Characterization Test?
A characterization test is a type of automated test that captures the actual behavior of a piece of software at a given moment, rather than the desired behavior. The term was popularized by Michael Feathers in his book Working Effectively with Legacy Code (2004). The key idea is to write a test that asserts what the code currently does, even if that behavior is wrong or contains a bug.
For example, imagine a function that is supposed to add two numbers but actually subtracts them. A characterization test would assert that add(2, 3) == -1, because that is what the code currently produces. Later, when you fix the bug, the characterization test will fail, alerting you that the behavior changed—and you can then update the test to reflect the corrected behavior.
Characterization tests are especially valuable when working with legacy code that has no existing tests, or when you need to refactor code that is poorly understood. They act as a safety net, catching unintended changes before they reach production.
Why Pin the Crash First?
When you encounter a crash in a system, your instinct might be to immediately patch the code. However, if you patch without first writing a characterization test that reproduces the crash, you risk:
- Fixing the crash but breaking something else that depended on the old behavior.
- Not being able to verify that the patch actually resolves the crash, because you have no automated test that reproduces it.
- Losing the ability to compare before/after behavior, making it harder to debug future regressions.
By pinning the crash with a characterization test before patching, you achieve several things:
- You create a reproducible test case that will fail until the patch is applied.
- You document the current (buggy) behavior, so you can later verify that the patch changes only the intended behavior.
- You have a baseline that can be used to ensure other parts of the system are not affected.
This approach is a standard practice in test-driven development (TDD) and legacy code refactoring. It is also recommended by Feathers in his book, where he describes the “characterization test” as a way to “pin” current behavior before making changes.
How to Write a Characterization Test for a Crash
Writing a characterization test for a crash involves a few concrete steps:
- Reproduce the crash manually. Identify the exact input or sequence of actions that causes the crash. Note the error message, stack trace, or unexpected output.
- Write a test that exercises that input. Use your unit testing framework (e.g., JUnit, pytest, Mocha) to call the relevant function or module with the crashing input.
- Assert the current behavior. If the crash produces an exception, assert that the exception is thrown. If it produces a specific incorrect output, assert that output.
- Run the test to confirm it fails (or passes) as expected. A characterization test for a crash should usually fail, because the crash is a bug. However, if the crash is intermittent, you may need to run it multiple times to get a stable baseline.
- Commit the test. Keep the test in your repository so that it becomes part of your regression suite.
Once the test is in place, you can proceed to patch the code. After patching, run the test again. If the patch fixes the crash, the test should now pass (or at least behave differently than before). If the patch causes a different failure, the characterization test will catch it.
Common Pitfalls and Misconceptions
One common misconception is that characterization tests are only for “legacy” code. In reality, they are useful for any code that lacks clear specifications or that has behavior that is not fully understood. They are also useful for third-party libraries or APIs that you depend on, where you want to document how they behave in your environment.
Another pitfall is writing a characterization test that is too broad. If you try to pin the entire system’s behavior, the test will be fragile and hard to maintain. Instead, focus on the specific crash or the specific area of code you are about to change.
A third pitfall is forgetting to update the characterization test after the patch. Once you fix the bug, the old characterization test may now fail (because the behavior changed). You should update the test to reflect the new, correct behavior. This is a normal part of the workflow—the test is a snapshot of behavior, not a permanent specification.
Finally, be aware that characterization tests can mask bugs if you never revisit them. They are a tool for safe refactoring, not a substitute for proper unit tests that assert desired behavior. Use them as a bridge, and later replace them with more meaningful tests once the code is understood.
Costs and Benefits
The main cost of pinning a crash with a characterization test is time. Writing the test, running it multiple times, and maintaining it all take effort. The calculator above helps you estimate the time cost of running the test repeatedly.
The benefits usually outweigh the costs:
- Reduced risk of introducing new bugs during patching.
- Better documentation of current behavior.
- Improved confidence in refactoring and patching.
- Faster debugging in the future, because you have a reproducible test case.
In a team setting, characterization tests also serve as communication tools. They show other developers what the code currently does, which can be especially helpful for onboarding or when dealing with poorly documented modules.
Alternatives to Characterization Tests
If you cannot write a characterization test for a crash, there are a few alternatives:
- Golden master testing: This is a form of characterization testing where you capture the output of a system for a set of inputs and store it as “golden” files. Later, you compare new output against the golden files. This is useful for systems that produce large or complex output, such as compilers or report generators.
- Property-based testing: Instead of pinning specific inputs, you define properties that should hold for a range of inputs. For example, “for any two numbers, add(a, b) == add(b, a)”. This can catch crashes that occur on unexpected inputs.
- Snapshot testing: Popular in frontend development, snapshot testing captures the rendered output of a component and compares it to a stored snapshot. It is similar to golden master testing but at a component level.
Each of these has its own strengths and weaknesses. Characterization tests are the most direct way to pin a specific crash, but the alternatives can be useful when the crash is hard to reproduce or when the system’s output is too variable.
Frequently Asked Questions
Can I skip the characterization test and just patch the crash?
Technically yes, but it is risky. Without a test that reproduces the crash, you have no way to prove that your patch actually fixes it, and you might unknowingly break other behavior. Pinning the crash first is a small investment that pays off by making your patch verifiable and safe.
What if the crash is intermittent or hard to reproduce?
If the crash is intermittent, a characterization test that runs once may not reliably fail. You can run the test multiple times (as the calculator above helps you estimate) or use a stress test approach. Alternatively, you can capture a stack trace or log from the crash and write a test that asserts the error condition based on that evidence.
Should I keep the characterization test after fixing the bug?
Yes, but update it. Once the bug is fixed, the test should be modified to assert the correct behavior. Keeping the old test would cause a failure. You can either change the assertion to match the new behavior or replace the test with a proper unit test that verifies the fix.
Is a characterization test the same as a unit test?
No. A unit test asserts expected behavior based on specifications. A characterization test asserts current behavior, which may be buggy. They serve different purposes: unit tests verify correctness, while characterization tests document reality and protect against unintended changes.
Can I use characterization tests for non-crash bugs?
Absolutely. Characterization tests are useful for any bug, not just crashes. For example, if a function returns the wrong value, you can write a characterization test that pins that wrong value, then fix the function, and the test will fail until you update it. The same principle applies.
Related
Formula used
in t | Time to reproduce crash | min | 5 | 0 in n | Number of test runs | - | 10 | 1 out total | Total time for characterization runs | min | t * n | 1
Each line is one input (in, sel, list) or one result (out: label, unit, formula, decimals). How the calculators work
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.