Claude Opus 4.5 Benchmarks: Terminal Harness Differences
Yes, Claude Opus 4.5 (the model you refer to as “Opus 4 8”) scored differently across two terminal benchmark harnesses, primarily because the harnesses measure different skills: one tests raw coding ability on isolated tasks, while the other tests end-to-end software engineering with real repositories and hidden tests. The gap is not a bug or a sign of inconsistency; it reflects what each harness actually evaluates.
What the Two Harnesses Actually Measure
The two most commonly cited terminal-based benchmark harnesses for large language models are SWE-bench and Terminal-Bench. They are often conflated because both involve a command-line interface and a coding task, but their designs diverge sharply.
- SWE-bench (and its variants like SWE-bench Verified) presents a model with a real GitHub issue and the corresponding codebase. The model must produce a patch that fixes the issue, and the patch is evaluated against a set of hidden unit tests that were added by the repository maintainers. It measures whether the model can understand an existing codebase, locate the bug, and produce a minimal, correct change.
- Terminal-Bench gives the model a shell environment and a task description (for example, “install this package and run its test suite”). The model must execute commands, read outputs, and iterate until the task is complete. The evaluation checks whether the final state of the environment matches the expected outcome. It measures command-line fluency, tool use, and the ability to recover from errors in a live session.
Because the tasks, evaluation methods, and scoring rubrics differ, a model that excels at writing a patch may struggle with interactive terminal navigation, and vice versa. Claude Opus 4.5 is not unique in this regard—every frontier model shows a spread across harnesses.
Why the Scores Differ: Task Design and Scoring
The most important reason for the discrepancy is that the two harnesses reward different behaviors. SWE-bench rewards a single, correct patch; there is no partial credit for a patch that fixes some tests but not others. Terminal-Bench rewards the final state of the environment, so a model that takes a long, meandering path but eventually succeeds scores the same as one that finishes quickly.
Additionally, the difficulty distributions are not aligned. SWE-bench Verified contains a curated subset of issues that are known to be solvable by human developers, but many are quite hard. Terminal-Bench tasks tend to be more procedural—installing dependencies, configuring files, running scripts—which can be easier for a model that has memorized common command sequences. A model with strong code generation but weaker shell knowledge will score higher on SWE-bench, while a model with broad command recall but weaker debugging skills will score higher on Terminal-Bench.
Another factor is the evaluation environment. SWE-bench runs offline: the model sees the code, produces a patch, and the patch is applied to a fresh checkout. Terminal-Bench runs live: the model can execute commands, see errors, and correct itself. This means Terminal-Bench rewards iterative trial-and-error, while SWE-bench rewards getting it right the first time. A model that is cautious and double-checks its work may do better on SWE-bench but slower on Terminal-Bench, where speed and willingness to try commands matter.
How to Read the Numbers Correctly
When comparing scores across harnesses, the absolute number is less meaningful than the relative ranking of models. A 40% on SWE-bench and a 60% on Terminal-Bench are not directly comparable—they are not on the same scale. Instead, look at how each model ranks within each harness.
For example, if Claude Opus 4.5 scores 50% on SWE-bench Verified and 70% on Terminal-Bench, while a competing model scores 55% and 65% respectively, the conclusion is that the competitor is slightly better at patch generation but worse at interactive terminal work. That is a nuanced, useful comparison. If you only looked at one harness, you would draw a misleading conclusion about which model is “better.”
Also note that benchmark scores are reported with specific versions of the harness. SWE-bench has multiple versions (Full, Verified, Multimodal), and Terminal-Bench has its own updates. A model’s score can change by several points just by switching from one version to another, because the task set changes. Always check which version was used before comparing scores.
Common Misconceptions About the Gap
A frequent misunderstanding is that a large gap between harness scores indicates the model is “overfitted” to one benchmark. While overfitting is a real concern in AI evaluation, a gap alone is not evidence of it. The two harnesses measure different capabilities, and a model can legitimately be stronger in one area.
Another misconception is that the harness with the higher score is the “easier” one. This is not necessarily true. A model might score higher on Terminal-Bench because it has memorized common command patterns, but that does not make the tasks trivial. Conversely, a model might score lower on SWE-bench because the hidden tests are unforgiving, not because the tasks are harder in a general sense.
Finally, some readers assume the scores are deterministic—that running the same model on the same harness twice gives the same result. In practice, many harnesses use sampling (the model generates multiple responses and the best one is scored), and the temperature setting can change outcomes. A score is a point estimate with variance, not a fixed property of the model.
Which Harness Should You Trust?
The answer depends on your use case. If you are hiring a model to work on a large existing codebase and fix bugs, SWE-bench is the more relevant signal. If you are using a model as a terminal assistant—someone who can install packages, run scripts, and manage files—Terminal-Bench is more representative.
For most practical purposes, you should look at both, but weight them according to your workflow. A developer who spends most of the day in an IDE will care more about SWE-bench. A DevOps engineer or system administrator who lives in the shell will care more about Terminal-Bench. There is no single “best” score; there is only the score that matches your task.
It is also worth noting that both harnesses are static snapshots. They were built from a fixed set of tasks, and models may have seen similar tasks during training. A model that scores well on a benchmark does not guarantee good performance on novel, unseen problems. Benchmarks are a useful but incomplete proxy for real-world capability.
How to Compare Models Fairly
If you want to compare Claude Opus 4.5 to another model, follow these steps:
- Find the official scores for both models on the same version of the same harness (e.g., SWE-bench Verified, not SWE-bench Full).
- Check the evaluation settings: temperature, number of samples, and whether the best-of-n was used.
- Look at the task breakdown, if available. Some harnesses report scores by task category (e.g., “bug fix” vs. “feature addition”), which can reveal strengths and weaknesses.
- Run your own small evaluation with tasks similar to your actual work. A 20-task custom test is often more informative than a 2,000-task benchmark.
This approach gives you a fuller picture than relying on a single headline number.
Frequently Asked Questions
Why did Claude Opus 4.5 score lower on Terminal-Bench than on SWE-bench?
Likely because Terminal-Bench requires interactive command execution and error recovery, which is a different skill set from writing a patch. The model may be stronger at code comprehension than at shell navigation, or the specific tasks in that version of Terminal-Bench were harder for it.
Is a higher SWE-bench score always better?
No. SWE-bench measures patch generation, not general coding ability. A model that is excellent at writing patches but poor at using a terminal would score high on SWE-bench but low on Terminal-Bench. The better model depends on the task.
Can I reproduce these scores myself?
Yes, both harnesses are open-source and can be run locally, but they require significant compute and setup. SWE-bench needs Docker containers and a codebase checkout; Terminal-Bench needs a sandboxed shell. Most people rely on published scores rather than running them.
Do the scores change over time?
Yes. Harness maintainers periodically add tasks or fix evaluation bugs, which can shift scores. Also, model providers may update their models silently, so the score you see on a blog post may not match the current model. Always check the date of the evaluation.
What is the best single benchmark for coding models?
There is no best single benchmark. SWE-bench Verified is the most cited for code repair, and Terminal-Bench is the most cited for interactive shell use, but they measure different things. Use the one that matches your use case, or both for a fuller picture.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.