LLM Eval Scores: What Acc and Acc Norm Measure

LLM evaluation scores like Accuracy (Acc) and Accuracy Normalized (Acc Norm) measure how often a model gives the correct answer on a benchmark task, with Acc Norm additionally accounting for the length of the generated response. This distinction matters because longer outputs can artificially deflate or inflate raw accuracy, making Acc Norm a fairer metric for open-ended generation tasks.

Formula

Accuracy (Acc) is the fraction of test instances where the model’s output exactly matches the expected answer. Accuracy Normalized (Acc Norm) is the same fraction, but the model’s output is first normalized by removing all whitespace and punctuation, and then the score is adjusted based on the length of the output.

In practice, Acc Norm is often computed as: Acc Norm = (number of correct normalized outputs) / (total outputs), where a normalized output is considered correct if it matches the normalized reference answer. The “length� part comes in because longer outputs have more chances to include the correct answer, so a simple exact match might miss a correct answer buried in extra text. Acc Norm typically uses a scoring rule that gives partial credit or uses a length-normalized probability, but the most common implementation in benchmarks like HELM is: Acc Norm = (1 / (output length in tokens)) * (indicator of correct answer) averaged over all instances.

Variables:

• Acc: fraction of exact matches (0 to 1, or 0% to 100%)
• Acc Norm: length-normalized accuracy (0 to 1)
• Output length: number of tokens or characters in the model’s generated response
• Reference answer: the ground truth answer for the test instance

Worked Example

Consider a benchmark with two questions. The model outputs:

• Q1: Reference answer “Paris”. Model output: “Paris is the capital of France.”
• Q2: Reference answer “42”. Model output: “42”

For Acc, exact match: Q1 fails (output is longer), Q2 passes. So Acc = 1/2 = 0.5 or 50%.

For Acc Norm, we normalize outputs by stripping punctuation and lowercasing, then we consider length. A common implementation: for each instance, score = (1 / length in tokens) if the reference appears in the output, else 0. Suppose Q1 output has 6 tokens, Q2 has 1 token. Then:

• Q1: reference “paris” appears, score = 1/6 ≈ 0.167
• Q2: reference “42” appears, score = 1/1 = 1.0

Acc Norm = (0.167 + 1.0) / 2 = 0.5835, or about 58.4%.

Notice Acc Norm is higher than Acc because it gives partial credit for the longer correct answer. This is why length matters: raw Acc penalizes verbose correct answers, while Acc Norm mitigates that penalty.

Why Length Matters in LLM Evaluation

Length matters because LLMs generate variable-length outputs. In a multiple-choice or short-answer task, a model might output the correct answer plus extra explanation. Raw Acc would mark it wrong, even though the answer is present. Conversely, a model that outputs a single wrong token but with a very short response could get a higher Acc Norm than a model that outputs a long correct answer with some noise.

Acc Norm was introduced to address the “length bias� in tasks like question answering. It rewards models that are concise and correct, while still giving credit for correct answers embedded in longer text. Without length normalization, a model that rambles on and accidentally includes the correct answer would score the same as a model that gives a direct, correct answer—which is unfair to the concise model.

In practice, benchmarks like HELM (Holistic Evaluation of Language Models) use Acc Norm as a primary metric for generation tasks because it correlates better with human judgment of correctness.

How to Choose Between Acc and Acc Norm

When reading leaderboards, you’ll see both metrics. Here’s how to decide which matters for your use case:

• Use Acc when the task requires exact, short answers (e.g., math problems with a single numeric answer, multiple-choice with one letter).
• Use Acc Norm when the task allows free-form answers (e.g., open-ended QA, summarization, generation).
• If the benchmark reports both, look at the gap: a large gap between Acc and Acc Norm suggests the model tends to produce verbose outputs.

For your own evaluations, if you care about whether the model can produce the right answer at all (even if wordy), Acc Norm is more forgiving. If you care about exact formatting (e.g., for API calls), Acc is stricter.

Common Misconceptions and Pitfalls

Misconception: Acc Norm is always better than Acc. Not necessarily—it depends on the task. For tasks where the answer must be exact (like a code snippet), Acc Norm might give false credit for an answer that contains the right code but also extra broken code.

Pitfall: Ignoring normalization details. Different implementations normalize differently: some strip punctuation, some lowercase, some only strip whitespace. Always check the benchmark’s definition.

Pitfall: Comparing Acc Norm across benchmarks. Each benchmark may use a different length-normalization formula, so scores are not directly comparable across suites.

Pitfall: Using Acc on long-form generation. Raw Acc on long outputs is almost always near zero, making it useless for tasks like summarization.

Limitations of Acc Norm

Acc Norm is not a silver bullet. It still relies on exact string matching after normalization, so it cannot recognize semantically equivalent answers (e.g., “The capital is Paris� vs. “Paris is the capital�). It also assumes that shorter is better, which is not always true—some tasks require detailed explanations.

Moreover, Acc Norm does not account for the quality of the extra text. A model that outputs “Paris Paris Paris Paris� would get a high score for a question about Paris, even though the response is nonsensical. This is why many evaluations also report other metrics like ROUGE or BLEU for generation quality.

Frequently Asked Questions

What does “Acc Norm� stand for?

Acc Norm stands for Accuracy Normalized. It’s a metric that adjusts the raw accuracy score based on the length of the model’s output, giving partial credit for correct answers that appear within longer responses.

Why can’t we just use exact match accuracy?

Exact match accuracy is too strict for open-ended tasks. A model might give the correct answer but with extra explanation, and exact match would count it as wrong. This makes the model look worse than it actually is, especially for tasks that expect natural language responses.

How is length measured in Acc Norm?

Length is typically measured in tokens (subword units) or characters. In the HELM implementation, the score for each instance is 1 divided by the number of tokens in the output, but only if the reference answer appears in the output. So shorter correct answers get higher scores.

Does Acc Norm work for multiple-choice questions?

It can, but it’s usually unnecessary because multiple-choice answers are already short and exact. Most benchmarks use raw Acc for multiple-choice and use Acc Norm for generation tasks like question answering.

What if the model outputs the correct answer twice in a long response?

Acc Norm still gives the same score as if it appeared once, because the score is based on the presence of the answer and the total length. It does not reward repetition, which is good—repetition is usually undesirable.

Is Acc Norm used in industry evaluations?

Yes, many LLM leaderboards and research papers report Acc Norm, especially for open-ended QA benchmarks. It’s also used in some production evals to compare model versions, though companies often design custom metrics for their specific use cases.

Related

• Nuclear Power vs Natural Gas for Data Centers
• Adjust Baking Time and Temperature by Pan Material
• Bleach: Brave Souls Won’t Open? Fix Crashes and Black Screens

in t | Finish time | min | 50 | 0
in d | Distance | km | 10 | 0
out pace | Average pace | min/km | t / d | 2
out speed | Average speed | km/h | d / (t / 60) | 2