ThisCalculator

Everyday

Count Values Out of Tolerance: How to Calculate

Updated · Standard formula, shown below

The calculator for this page is switched off while it is checked. The formula is below, and the worked example in the article shows how to use it by hand.

This calculator works out how many values in a list fall outside a specified tolerance range. You enter your data points, the target (nominal) value, and the allowed deviation, and you get back the count of out-of-tolerance values, plus the percentage of the total they represent. It is a quick way to audit a batch of measurements, test results, or production samples against a quality limit.

Formula

For each value x in the list, compare it with the target T and the tolerance tol. A value is out of tolerance if:

|x – T| > tol

where |…| means absolute value. The count is the number of values that satisfy that condition. The percentage is:

percentage = (count / n) × 100

where n is the total number of values in the list. The units of x, T, and tol must be the same (e.g., all in millimeters, all in volts, all in grams).

Worked Example

Using the default values from the calculator:

  • Data values: 10.1, 10.3, 9.8, 10.0, 10.5, 9.9
  • Target value: 10.0
  • Tolerance: ±0.2

Step 1: For each value, find the absolute difference from 10.0:

  • 10.1 → |10.1 – 10.0| = 0.1 (within)
  • 10.3 → |10.3 – 10.0| = 0.3 (out)
  • 9.8 → |9.8 – 10.0| = 0.2 (within, since 0.2 is not greater than 0.2)
  • 10.0 → |10.0 – 10.0| = 0.0 (within)
  • 10.5 → |10.5 – 10.0| = 0.5 (out)
  • 9.9 → |9.9 – 10.0| = 0.1 (within)

Step 2: Count the out-of-tolerance values: 2 (10.3 and 10.5).

Step 3: Calculate the percentage: (2 / 6) × 100 = 33.3%.

The calculator shows count = 2 and percentage = 33.3%.

When to Use This Count

This is a basic quality-control calculation. It is useful in manufacturing when checking whether a batch of parts meets a specification, in laboratories when verifying that measurements fall within a calibration range, or in data analysis when flagging outliers beyond a fixed threshold. It is also used in education to see how many test scores deviate from a target score by more than a set amount.

The count alone does not tell you how far out of tolerance the values are. Two values that are barely outside and two that are wildly outside produce the same count. For a fuller picture, pair the count with the maximum deviation or the average deviation.

How to Read the Result

The count is the number of values that fail the tolerance check. The percentage puts that count in context relative to the total number of values. For example, 2 out of 6 is 33.3%, which is a high failure rate. In contrast, 2 out of 200 is 1%, which may be acceptable depending on the application.

Remember that the tolerance is a two-sided limit. A value equal to the target plus the tolerance exactly (e.g., 10.2 in the example) is considered within tolerance because the condition uses “greater than” not “greater than or equal to.” If your specification says “up to and including ±0.2,” then 10.2 is acceptable. If it says “must be strictly less than 0.2 away,” then 10.2 would be out. The calculator uses the strict greater-than rule, which is the common convention in quality control.

Common Mistakes

  • Unit mix-ups: If your data is in millimeters and your tolerance is in centimeters, the count will be wrong. Convert everything to the same unit first.
  • Using standard deviation instead of tolerance: Tolerance is a fixed limit you set; standard deviation is a measure of spread. They are not interchangeable.
  • Confusing “within” with “equal”: As noted, a value exactly at the boundary is within tolerance under the strict greater-than rule. Decide on your rule and apply it consistently.
  • Rounding too early: If you round each deviation before comparing, you might change the result. Compare the raw values, then round the final percentage.

Limits of This Calculation

This calculation assumes your tolerance is symmetric around the target. If your specification is one-sided (e.g., “must be at least 10.0” or “must be no more than 10.2”), this calculator will not handle it correctly. For one-sided limits, you need a different formula that checks only one direction.

The count also treats every value equally. It does not account for the magnitude of the deviation, nor does it consider the distribution of values. For a more sophisticated analysis, you might look at the mean, standard deviation, or process capability indices.

Frequently Asked Questions

What does “out of tolerance” mean? It means a value deviates from the target by more than the allowed amount. For example, if the target is 10.0 and tolerance is ±0.2, then any value below 9.8 or above 10.2 is out of tolerance.

Can I use this for one-sided tolerances? No. This calculator only handles two-sided tolerances (target ± tolerance). For a one-sided limit, you would need to check values against a single boundary, which is a different formula.

What if my data has units like inches or volts? The calculator works with any unit as long as you enter all values in the same unit. The count is unitless, and the percentage is unitless.

How is this different from standard deviation? Standard deviation measures the average spread of all values, while this count tells you how many values fall outside a fixed limit. Two datasets can have the same standard deviation but different out-of-tolerance counts, depending on the shape of the distribution.

Should I include the boundary values as out of tolerance? Usually no, if your specification is “within ±tol.” The calculator uses strict greater-than, so a value exactly at the boundary is considered within. If your specification says “strictly less than,” then you would need to adjust the condition.

What percentage is acceptable? That depends entirely on your application. In some industries, a 1% failure rate is unacceptable; in others, 5% is fine. The calculator gives you the number, but you must decide what threshold is acceptable for your situation.

Related

Formula used
in list | Data values | 10.1, 10.3, 9.8, 10.0, 10.5, 9.9 | 0
in t | Target value | 10.0 | 0
in tol | Tolerance (±) | 0.2 | 0
out count | Out-of-tolerance count | - | 0
out pct | Out-of-tolerance percentage | % | count / 6 * 100 | 1

Each line is one input (in, sel, list) or one result (out: label, unit, formula, decimals). How the calculators work

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.