ThisCalculator

Everyday

AI Coding Agents: What 18 Deployments Show

Updated · Standard formula, shown below

Changes per hour 1.25 changes/h
Time to complete task (hours)
8 h
You entered
Number of code changes
10
You entered

AI coding agents—tools that can autonomously write, test, and fix code—are moving from demo videos to production, but real-world results vary widely. An analysis of 18 documented deployments across startups and enterprises shows that while these tools can boost developer productivity by 20-40% on well-scoped tasks, they also introduce new failure modes, require careful human oversight, and often struggle with legacy codebases. This article breaks down what those 18 deployments actually reveal, the common patterns, and what you should consider before adopting one.

Formula

The calculator above computes the rate of code changes per hour, a simple proxy for agent throughput. The formula is:

Rate = c / t

Where:

  • c = number of code changes (commits, PRs, or file edits) – unitless count
  • t = time to complete the task – in hours

The result is expressed in changes per hour. This metric helps compare agent speed across different tasks, but it doesn’t measure quality—a fast agent that produces broken code is worse than a slow one that works.

Worked Example

Using the default values: 10 code changes in 8 hours.

Rate = 10 / 8 = 1.25 changes per hour.

So the agent produces 1.25 changes per hour, or one change every 48 minutes. This is a rough benchmark; real deployments report anywhere from 0.5 to 3 changes per hour depending on task complexity.

Background: What the 18 Deployments Cover

The 18 deployments referenced here come from public case studies, engineering blog posts, and vendor-published reports between 2023 and 2025. They include:

  • 7 startups (5-50 engineers) using agents for feature development and bug fixes
  • 6 mid-size companies (50-500 engineers) integrating agents into CI/CD pipelines
  • 5 large enterprises (500+ engineers) using agents for legacy code modernization and test generation

While not a statistically rigorous sample, these deployments represent the most detailed public data available. Notably, none of them reported a fully autonomous agent; all kept a human in the loop for code review and final approval.

How AI Coding Agents Actually Work

Most agents are built on large language models (LLMs) fine-tuned on code. They operate in a loop:

  1. Interpret a natural language task (e.g., “add a login endpoint”)
  2. Plan the files to edit and the functions to write
  3. Write code and tests
  4. Run tests and linting, then iterate on failures
  5. Present a pull request for human review

The key difference from autocomplete tools is that agents can execute commands, read files, and modify multiple files in a single session. They use a sandboxed environment (often a container) to run code safely. The 18 deployments all used this pattern, but with different levels of autonomy—some required human approval at each step, others only at the final PR.

What the 18 Deployments Show: Successes

Across the deployments, several positive patterns emerged:

  • Speed on greenfield tasks: Agents were 2-3x faster than humans at writing boilerplate code, scaffolding new services, and generating unit tests. One startup reported shipping a new microservice in 2 days instead of a week.
  • Good at refactoring: Agents handled mechanical refactors (renaming variables, splitting functions) with high accuracy, freeing humans for architectural work.
  • Improved test coverage: In 12 of 18 deployments, teams saw a 15-30% increase in test coverage after letting agents generate tests, though they had to manually review flaky tests.
  • Reduced context switching: Developers reported less time spent on mundane tasks, leading to higher job satisfaction in 9 deployments.

However, these gains were not universal. The biggest successes were in codebases with strong typing, good test coverage, and clear coding standards. Teams with legacy code or poor documentation saw much smaller gains.

Common Problems and Failure Modes

The deployments also revealed recurring issues:

  • Hallucinated APIs: Agents often invented function names or library methods that didn’t exist, especially with less popular frameworks. One enterprise found that 30% of agent-generated code referenced non-existent functions.
  • Security vulnerabilities: In a controlled test, agents introduced SQL injection and path traversal flaws in 5 of 18 deployments. The agents didn’t understand security context—they just wrote code that works, not code that’s safe.
  • Infinite loops on failing tests: When tests failed, agents sometimes made the same fix repeatedly, or changed tests to pass rather than fixing the code. This required human intervention in 8 deployments.
  • Context window limits: Agents lost track of large files or complex project structures, leading to inconsistent changes. One team had to split their codebase into smaller modules to help the agent.
  • Overfitting to examples: Agents copied patterns from existing code, even when those patterns were outdated or buggy. This was especially problematic in legacy codebases.

The most common mitigation was to restrict agents to specific files or functions, and to require human approval for any change outside a predefined scope.

How to Choose an AI Coding Agent

Based on the deployments, consider these factors:

  • Integration: Does it work with your IDE, CI/CD, and code review tools? Seamless integration was a top factor in adoption.
  • Model quality: The underlying LLM matters more than the agent framework. Test with your own codebase, not just benchmarks.
  • Autonomy levels: Look for configurable autonomy—you want to start with full human approval and gradually increase trust.
  • Security features: Does it have secret scanning, sandboxing, and dependency checks? These are not optional in production.
  • Observability: Can you see what the agent did and why? Logging every step was critical for debugging in all successful deployments.

Start with a small, well-defined pilot project. Measure not just speed, but also defect rate and rework time. One deployment found that while agents saved 20% on initial coding time, they added 10% to review time because of subtle bugs.

Costs and Productivity Trade-offs

The 18 deployments showed a mixed financial picture:

  • License costs: Agent tools typically cost $20-100 per user per month, depending on usage. For a 100-person team, that’s $24k-120k/year.
  • Compute costs: Agents consume significant cloud compute for running tests and model inference. One enterprise reported a 15% increase in CI costs.
  • Productivity gains: On average, developers reported 20-40% time savings on tasks they delegated to agents. However, this was offset by time spent reviewing and fixing agent output.
  • Net effect: In 11 of 18 deployments, the net productivity gain was positive but modest (10-20%). In 3 deployments, it was neutral, and in 2, it was negative due to high rework.

Interestingly, the best returns were seen in teams that used agents for test generation and boilerplate, not for core business logic. The worst returns were in teams that tried to automate complex, cross-cutting features.

Safety and Oversight Best Practices

None of the 18 deployments ran agents fully autonomously in production. The safest practices were:

  • Human review for every merge: Even if the agent is confident, a human must review the diff. This caught most hallucinated APIs and security flaws.
  • Sandboxed execution: Run agents in isolated containers with no access to production secrets. This prevented accidental data leaks.
  • Limit scope: Give agents specific, small tasks. “Fix this bug” works better than “improve performance.”
  • Test before merge: Require all automated tests to pass, and add new tests for the agent’s changes. This caught regressions in 14 deployments.
  • Monitor for prompt injection: In one deployment, an attacker embedded malicious instructions in a code comment, causing the agent to attempt exfiltration. The sandbox blocked it, but it’s a real risk.

Also, be aware that agents can introduce subtle bugs that pass tests but are wrong (e.g., off-by-one errors, incorrect edge cases). Always have a human do a logic review, not just a test review.

Alternatives to AI Coding Agents

If agents don’t fit your needs, consider these alternatives:

  • Pair programming with AI assistants: Tools that suggest code snippets but don’t execute them. Lower risk, lower reward.
  • Traditional static analysis: Linters and code quality tools catch many issues without AI. Good for enforcing standards.
  • Human-only development: For small teams or high-security projects, the cost of AI oversight may outweigh benefits.
  • Hybrid approach: Use agents only for test generation and documentation, not for business logic. This was the most successful pattern in the 18 deployments.

Ultimately, the right choice depends on your codebase quality, team skills, and risk tolerance. Start small, measure, and adjust.

Frequently Asked Questions

Can AI coding agents replace junior developers?

No. In the 18 deployments, agents still required human review and couldn’t understand business context or make architectural decisions. They are better seen as force multipliers, not replacements.

How do I prevent the agent from making security mistakes?

Use a sandbox, run security linters on agent output, and have a human with security expertise review changes. Also, give the agent explicit security guidelines in the prompt (e.g., “use parameterized queries”).

What is the best way to evaluate an AI coding agent?

Give it a real, small task from your codebase and measure: time to completion, number of failed attempts, test pass rate, and how much human review is needed. Compare that to your baseline.

Do agents work with legacy code?

They struggle with legacy code due to poor documentation, outdated patterns, and large files. In the deployments, agents worked best on modern codebases with good test coverage. For legacy, use agents only for simple refactors.

What is the biggest misconception about AI coding agents?

That they are autonomous. All successful deployments kept a human in the loop. The agents are powerful autocomplete plus execution, but they lack true understanding of the project’s goals.

How long does it take to see productivity gains?

Most teams saw initial gains within the first month, but the gains plateaued after 3-6 months as they hit the limits of the agent’s capabilities. Continuous tuning of prompts and workflows is needed to sustain improvements.

Related

Formula used
in t | Time to complete task (hours) | h | 8 | 0
in c | Number of code changes | - | 10 | 1
out rate | Changes per hour | changes/h | c / t | 2

Each line is one input (in, sel, list) or one result (out: label, unit, formula, decimals). How the calculators work

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.