Measure AI Coding Agent Efficiency by Cost and Success

Updated 8 November 2026 · Standard formula, shown below

Measuring an AI coding agent by cost, token use, and task success shows whether it completes useful work efficiently—not merely whether it produces code quickly. You will compare repeated, clearly defined tasks using recorded usage and the same success checks, then calculate cost per successful task and tokens per success.

What You’ll Need

Access to the coding agent’s usage records or API logs, including input and output tokens and any billed tool or model charges.
A set of representative coding tasks, a fixed repository state, and an independent way to check results, such as tests and human review.
A spreadsheet or other simple record for runs, outcomes, token counts, costs, and notes.
A consistent model, agent configuration, tool permissions, and pricing source for the comparison. If comparing configurations, record each one separately.

Step-by-Step

Define what “success” means. Write down objective acceptance criteria before running the agent. For a bug fix, that might mean the targeted test passes, existing tests still pass, and the change does not introduce an out-of-scope behavior. For a feature, list required behavior and constraints. Decide whether partial completion counts; a simple approach is to record pass or fail for every required criterion and count a task as successful only when all critical criteria pass.
Build a representative task set. Choose tasks that resemble real work in scope and difficulty. Include different task types, such as debugging, implementation, and refactoring, rather than selecting only easy examples. Keep the task descriptions and starting code identical across agents or configurations. Do not reveal test answers or evaluation details that the agent would not normally have.
Set a fair, repeatable run protocol. Fix the repository commit, environment, model, agent settings, context limits, and available tools. Give each run the same prompt and time or interaction budget. Decide in advance whether retries are allowed and how they count. For stochastic agents, run each task more than once where practical; treat each run as a result, and do not quietly discard failures.
Record token use consistently. For every run, capture input and output tokens separately if available. Include tokens from retries, agent planning, tool-call follow-ups, and any additional model calls that are part of completing the task. Record cache usage separately if the provider reports it, and follow the provider’s billing definitions. Tool output may increase context tokens even when it is not billed as model output; preserve the raw counts so this distinction is visible.
Capture the full cost. Use the actual billed amount for the run when available. Otherwise calculate an estimate from the applicable input, output, and cached-token rates, plus separately billed tools or services. Check the pricing unit and date: rates may be quoted per million tokens, and different token categories can have different rates. Include all attempts in the task’s total cost. Label estimates as estimates, and do not mix them with actual charges without saying so.
Evaluate the result independently. Run the predefined automated checks in a clean, consistent environment, then review requirements that tests cannot cover. Record pass or fail for each criterion, defects, and whether a human had to make material corrections. Do not judge success from the agent’s explanation or its claim that work is complete. Keep evaluator decisions and evidence, such as test logs, with the run record.
Calculate the key measures. For a group of runs, calculate success rate as successful runs divided by total runs, multiplied by 100%. Calculate cost per successful task as total run cost divided by successful runs, and tokens per successful task as total tokens divided by successful runs. Also report average cost and tokens per attempt; these show resource use even when runs fail. If there are zero successes, report cost per success as unavailable rather than dividing by zero.
Compare like with like and report uncertainty. Compare agents on the same tasks and protocol. Show results by task type as well as overall; an overall average can hide a weak area. Report the number of runs, success rate, average cost, and cost per success together. For repeated runs, include a range or median alongside the mean so one unusually expensive run does not obscure typical performance. State pricing date, token-count source, and any exclusions.
Choose based on the trade-off that matters. A lower cost per successful task is useful only if success quality is acceptable. Set a minimum success threshold before reviewing results, then compare cost and tokens among configurations that meet it. Consider review and repair effort if it differs substantially: record that effort separately rather than treating model tokens as the entire cost of completing software work.

Common Mistakes

Counting only successful runs in total usage. Failed attempts still consume resources. Include their costs and tokens in the numerator; only successful runs belong in the denominator for cost per success.
Using incomparable tasks or environments. Different repositories, prompts, test suites, or permissions can dominate the comparison. Hold them constant or clearly describe the difference.
Calling code generation success. A patch that does not meet acceptance criteria is not a successful task, even if it compiles or looks plausible.
Mixing token counts and bills. Token totals are not money. Rates, cached inputs, model choice, and separately billed tools affect charges. Keep usage and cost as distinct measurements.
Ignoring human effort and quality. An inexpensive result that needs extensive repair may not be efficient overall. Record intervention and defects, and avoid presenting token efficiency as total engineering productivity.

Tips

Use a run ID and save the prompt, configuration, commit, logs, test result, and cost source together.
Report both per-attempt and per-success figures. The first describes consumption; the second incorporates reliability.
Keep raw measurements and state whether counts cover one model call or the entire agent session.
Repeat runs when output varies, and compare medians and ranges rather than relying on a single demonstration.
Recheck prices when rates change, and avoid extrapolating a small task sample to every kind of engineering work.

Troubleshooting

Usage logs do not match the bill: Confirm the billing period, model, token categories, cache treatment, and any non-model charges. Use the billed total for actual cost, and retain token counts as a separate measure.
Token data is missing: Check whether the agent or provider exposes per-session usage. If not, mark the figure unavailable; do not infer exact tokens from text length. You can still compare recorded charges and task outcomes.
Results vary widely: Increase repeat runs, verify that the starting commit and settings match, and report the distribution. Do not remove an outlier unless a documented protocol error invalidated that run.
Tests pass but review finds defects: Add the missed requirement to the acceptance criteria and evaluation process for future runs. Re-evaluate all configurations consistently if the criteria materially change.
No run succeeds: Report the observed failure rate and total spend, then inspect task clarity, environment setup, and agent constraints. Do not calculate a finite cost per success until a successful run exists.

Frequently Asked Questions

Is tokens per task enough to measure efficiency?
No. It shows usage, not whether the task was completed correctly. Pair it with success rate, cost, and quality checks.

Should failed attempts count toward cost per successful task?
Yes. Include costs and tokens from every attempt in the total, then divide by successful runs.

How many times should I run each task?
There is no universal number. Use enough repeats to expose variability, apply the same count to each configuration, and report the sample size.

How should I handle cached tokens?
Record them separately when available and follow the provider’s billing treatment. Do not assume cached and uncached tokens have the same cost.

Can I compare agents with different models?
Yes, if the goal is to compare complete configurations. Keep tasks and evaluation rules fixed, and report model and settings so readers can interpret the result.

What if one agent needs more human correction?
Record correction effort and defects separately. Cost per successful run does not capture all the labor needed to deliver acceptable code.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.