Test AI Coding Agents Safely in a Sandbox
This guide shows you how to evaluate an AI coding agent’s skills without risking your real codebase. You’ll set up an isolated environment, design meaningful tasks, and measure results—all while keeping your systems and data safe.
What You’ll Need
- A computer with Docker installed (or a cloud VM you can easily reset)
- A Git account (GitHub, GitLab, or Bitbucket) for creating test repositories
- Access to the AI coding agent you want to test (API key or CLI tool)
- Basic familiarity with command line and version control
- A timer or stopwatch (optional, for timing tasks)
Step-by-Step
- Create a dedicated sandbox environment. Use Docker to spin up a container with the programming language and tools you want to test. For example, run
docker run -it --rm -v $(pwd)/test-repo:/workspace python:3.11 bashto get a Python environment. This isolates the agent from your main system. - Set up a test repository. Create a new, empty repository on your Git host. Clone it into the sandbox. Add a simple README and a basic project skeleton (e.g., a single file with a function). This gives the agent context to work with.
- Define clear, isolated tasks. Write 3–5 tasks that cover different skills: bug fixing, feature implementation, refactoring, and writing tests. Each task should be self-contained. For example: “Fix the off-by-one error in the
calculate_totalfunction” or “Add a new functionvalidate_emailthat returns True for valid emails.” - Create a test harness. Write a script that runs automated tests against the agent’s output. For Python, use
pytest; for JavaScript, usejest. The harness should check that the code passes existing tests and any new tests you write for the task. - Run the agent in the sandbox. Give the agent the task description and the repository path. Ensure the agent’s working directory is inside the Docker container or VM. Do not give it access to your host filesystem or network credentials.
- Monitor the agent’s actions. Watch the agent’s output. Does it attempt to run commands? Does it try to access the internet? Use
docker logsor a screen recorder. Note any suspicious behavior, like trying to read files outside the repo. - Evaluate the results. After the agent finishes, run your test harness. Check if the code passes tests, if it’s readable, and if it follows best practices (e.g., no hardcoded secrets, proper error handling). Record the time taken and the number of attempts.
- Repeat with different tasks. To get a reliable assessment, run the agent on multiple tasks. Vary the difficulty and the language. Keep a log of results to compare performance over time.
- Clean up. After testing, destroy the container or VM. Delete the temporary repository. This ensures no leftover code or credentials remain.
Common Mistakes
- Giving the agent real credentials. Never put API keys, passwords, or SSH keys in the sandbox. The agent might use them, or worse, leak them.
- Testing on a copy of your production code. Even if you think it’s safe, a copy may contain sensitive data or complex dependencies that could cause unexpected behavior.
- Not defining clear acceptance criteria. If you don’t know what “done” looks like, you can’t judge the agent’s performance. Always write tests first.
- Ignoring the agent’s resource usage. An agent might run an infinite loop or consume all memory. Set resource limits in Docker (e.g.,
--memory=1g --cpus=1). - Forgetting to reset the environment between tasks. If the agent modifies the system (installs packages, changes configs), it could affect subsequent tasks. Start fresh each time.
Tips
- Use a version control branch. Have the agent work on a branch, then review the diff. This makes it easy to see exactly what changed.
- Include a task that requires reading documentation. For example, “Use the
requestslibrary to fetch a URL. Read the docs if needed.” This tests the agent’s ability to learn. - Time the tasks. Use a stopwatch or
timecommand. Speed is a factor, but don’t prioritize it over correctness. - Test with a known bug. Introduce a deliberate bug and see if the agent finds it. This is a good test of debugging skills.
- Keep a scoring rubric. Score each task on a scale of 1–5 for correctness, code quality, and efficiency. This helps you compare different agents or versions.
Troubleshooting
- The agent doesn’t produce any code. Check the task description for ambiguity. Make sure the agent has the right permissions to write files in the sandbox. Try a simpler task first.
- The agent’s code fails tests. Look at the error messages. It might be a missing dependency or a misunderstanding of the task. Provide more context in the task description, or check if the agent installed required packages.
- The agent tries to access the internet. Some agents do this to fetch libraries or documentation. If you want to prevent it, run the sandbox with
--network nonein Docker. But note that some tasks might require internet access (e.g., fetching a URL). - The sandbox runs out of memory. Increase the memory limit in Docker, or reduce the size of the test repository. Also, check for infinite loops in the agent’s code.
- The agent modifies files outside the repo. This is a red flag. Ensure the agent’s working directory is strictly the repo. In Docker, you can mount only the repo directory, and run the container with a non-root user.
Frequently Asked Questions
Can I test an AI coding agent on my actual project?
It’s risky. Even with a copy, your project may have secrets or complex dependencies. It’s safer to use a small, synthetic project that mimics the structure of your real code.
How many tasks should I use?
At least 5–10 tasks for a reliable assessment. More tasks give you a better picture, but balance with time. Focus on variety over quantity.
What if the agent writes code that passes tests but is insecure?
Include security checks in your evaluation. Look for SQL injection, hardcoded passwords, or unsafe file operations. You can also run a linter or security scanner like Bandit for Python.
Do I need to watch the agent in real-time?
Not necessarily. You can review the logs and the final code. But real-time monitoring helps you spot risky behavior early, like attempts to access sensitive files.
Can I test multiple agents at once?
Yes, but run them in separate sandboxes to avoid interference. Compare their results side by side using the same tasks.
How do I know if the agent is “good”?
Define your own criteria. Typically, a good agent produces correct, readable, and maintainable code, completes tasks in reasonable time, and asks clarifying questions when needed.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.