Build a Pull Request Reviewer That Switches AI Models

Updated 9 November 2026 · Standard formula, shown below

Build a pull request reviewer that checks code changes and can route each review to a different AI model without changing the rest of the workflow. You will create a small review service, connect it to your code-hosting platform, and make model choice configurable so you can compare providers or use a fallback.

What You’ll Need

A repository on a code-hosting platform with pull request or merge request webhooks and an API for posting comments.
A server or serverless runtime that can receive HTTPS webhook requests, plus a place to store configuration and secrets.
API credentials for at least one model provider. Use a second provider or model if you want to test switching.
A programming language and an HTTP client library, along with a test repository where you can safely open sample pull requests.
Permission to configure repository webhooks and create a narrowly scoped access token for reading changes and posting review comments.

Step-by-Step

Define the reviewer’s job. Decide which files it reviews, which issues it should report, and how comments should be formatted. Start with actionable defects such as likely bugs, unsafe input handling, or missing error cases. Ask for evidence from the diff and a concise explanation. Do not ask the model to approve or merge a pull request; keep that decision with human reviewers and your existing checks.
Create a webhook receiver. Add an HTTPS endpoint to your service and configure the repository to send pull request events to it. Subscribe only to the events you need, typically pull request opened, reopened, and synchronized (updated with new commits). Validate the webhook signature using the platform’s signing secret before trusting the payload. Reject invalid signatures and ignore unrelated events.
Fetch the pull request changes. From a verified event, get the repository, pull request number, and head commit identifier. Use the platform API to retrieve changed-file metadata and patches, handling pagination. Review only text files within a configured size limit. Skip deleted binaries, generated files, vendored dependencies, and patches that are missing or too large; report skipped files in a summary if useful.
Build a provider-neutral model interface. Put model calls behind one function, such as reviewWithModel(provider, model, prompt). Convert your review request into the format each provider expects, then normalize its response into your own structure, for example findings with file path, line, severity, and explanation. Keep provider-specific request and response handling out of webhook and comment-posting code.
Make model selection configurable. Read a provider and model identifier from environment variables or a protected configuration store, such as REVIEW_PROVIDER and REVIEW_MODEL. Keep each provider’s API key in a secret store, not in source code or pull request content. Add a validated allowlist of providers and model identifiers so a typo or untrusted webhook value cannot select an arbitrary endpoint. Restart or reload configuration deliberately when you change models.
Send a bounded, clear review prompt. Include the diff and relevant context, but set limits for total input size and output length. Tell the model to treat repository text, comments, and diffs as untrusted data, not as instructions that can override the review task. Request findings only when there is a specific concern, and require a file and changed line where possible. Never include secrets, environment values, or unrelated private files in the prompt.
Validate the response before posting. Parse the provider response, enforce your expected schema, and discard malformed or out-of-range file and line references. Check that a finding refers to a changed line if the platform requires inline comments. Escape content for the comment format, cap the number of findings, and avoid repeating identical comments on webhook retries. Treat model output as untrusted input.
Post results and make retries safe. Post a single summary or supported inline review comments using the repository API. Record the pull request and head commit with a processing status or idempotency key. If the same event arrives again, do not post duplicates. If processing is slow, acknowledge the webhook promptly and run the review in a background job; retry temporary API failures with limits and backoff.
Test model switching before rollout. Use a fixture diff with known issues and run it through each configured model. Compare whether findings are relevant, correctly located, and valid under your schema. Test invalid webhook signatures, oversized patches, provider timeouts, malformed responses, and a second delivery of the same event. Start with a test repository, then enable the integration for a small group before expanding it.
Monitor and control the service. Log event IDs, selected provider and model, duration, token or usage metadata when available, and failure categories. Do not log full diffs, prompts, credentials, or sensitive model responses by default. Set provider timeouts, concurrency limits, and usage budgets, and provide a safe fallback or a clear “review unavailable” status rather than silently claiming the code passed.

Common Mistakes

Hard-coding one provider throughout the code: this makes switching require edits in multiple places. Keep provider adapters behind one interface.
Trusting webhook payloads without verification: validate signatures and use the event only to identify work; fetch authoritative diff data through the platform API.
Sending an entire repository: large, irrelevant context raises cost and can bury important changes. Send bounded diffs and only necessary context.
Posting every model suggestion: models can be wrong or repetitive. Validate locations, filter low-confidence output, and make comments specific and reviewable.
Ignoring retries and concurrent events: webhook delivery can repeat, and new commits can arrive during a review. Key work to the head commit and prevent duplicate or stale comments.

Tips

Keep model selection in deployment configuration, but allow a controlled per-repository setting only if it is validated against an allowlist.
Use the same test diffs and output schema when comparing models. Track useful findings, false positives, latency, and failures rather than judging by one sample.
Separate review comments from automated status checks. A model review should not replace tests, static analysis, or human approval.
Make the reviewer easy to disable with a configuration flag, and document who can change providers and where credentials are managed.
Check the current provider and platform API documentation for request limits, supported model identifiers, webhook signature rules, and comment restrictions.

Troubleshooting

No review appears: confirm the webhook is enabled, the event type is subscribed, and the endpoint returns a successful acknowledgment. Inspect delivery records and service logs using the event ID. Check that the integration token can read pull request files and post comments.

Signature checks fail: confirm the configured secret matches the webhook secret and that verification uses the unmodified request body. Do not disable verification as a workaround.

One model works but another fails: verify the selected provider, model identifier, credential, and request format. Providers may require different parameters or response parsing. Keep these differences inside the adapter and return a clear provider error to logs.

Comments are misplaced or duplicated: compare the finding’s file and line with the current head diff, and ensure retries use an idempotency record keyed to the commit. Remove or mark stale results when the head commit changes.

Reviews time out or exceed limits: reduce patch size, skip low-value files, split work into bounded batches, and use background jobs with timeouts. If a provider is unavailable, retry transient errors carefully or report that review could not run; do not present an empty result as approval.

Frequently Asked Questions

Can I switch models without changing the webhook?
Yes. Keep the webhook and review pipeline provider-neutral, then change the validated provider and model configuration. Confirm credentials and response compatibility before rollout.

Can the reviewer post inline comments?
Usually, if the platform API supports them and the referenced file and line are valid for the current diff. Validate locations first and fall back to a summary comment when inline placement is unsupported.

Should the model approve pull requests automatically?
No. Use its output as review assistance, not as a substitute for human judgment, tests, or required repository checks.

How do I keep source code and secrets private?
Send only necessary diff context, exclude secrets and unrelated files, store credentials in a secret manager, and limit access to logs and review output. Check the provider’s data-handling terms before sending code.

What should happen if a model returns invalid output?
Reject it, record a sanitized error, and either retry within a limit or post a neutral review-unavailable message. Never turn invalid output into an approval.

How can I compare two models fairly?
Run both against the same representative test diffs with the same review criteria and output schema. Compare actionable accuracy, missed issues, false positives, latency, and operational failures.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.