Build a Code Reviewer That Remembers Past PRs

A code reviewer that remembers past pull requests can compare new changes with established decisions, recurring bugs, and project conventions. Build it by storing useful review context, retrieving only relevant examples, and giving a reviewer clear rules for how to use that context.

What You’ll Need

• A Git repository and access to pull request diffs, comments, and review outcomes through your hosting platform or an exported dataset.
• A small service or script that can fetch PR data, create searchable records, and run when a PR is opened or updated.
• A database for records and, optionally, a vector index for semantic search. A relational database with text search can be enough for a small project.
• A code-review model or analysis service, plus credentials stored in a secrets manager or protected environment variables.
• Permission to process the repository’s PR data. Decide what data may be retained and who can access it before indexing.

Step-by-Step

1. Define what “remember” means. Choose a narrow goal, such as spotting a previously fixed bug pattern, applying a documented API convention, or avoiding repeated comments that maintainers rejected. Avoid asking the system to imitate every historical reviewer: old comments can be incorrect, outdated, or specific to a change that no longer applies. Write down the expected output, such as a short list of potential findings with links to supporting PRs.
2. Collect useful PR history. Fetch merged and closed PRs, their changed files, review comments, final review state, and relevant commit information. Include only data your team has approved for this use. Exclude bot noise, sensitive files, and comments that contain secrets or personal information. Use the platform’s documented API or an export, and respect pagination and rate limits. Keep PR identifiers and URLs so each stored lesson can be traced back to its source.
3. Turn history into small, reviewable lessons. Do not store a whole PR as one giant memory. For each useful finding, create a record with fields such as repository, file or component, language, issue pattern, the lesson, resolution, source PR, and date. For example: “When adding a retry loop to the payment client, cap attempts and avoid retrying non-transient errors.” Keep the original comment and surrounding code available for verification, but distinguish them from a concise, normalized lesson.
4. Filter and validate lessons. Favor comments that led to a code change, were affirmed by a maintainer, or match a documented convention. Mark disputed or superseded guidance as inactive rather than silently treating it as truth. Have a maintainer review generated summaries before they become searchable. Record when a lesson was created and last confirmed; codebase rules change.
5. Build retrieval for a new PR. When a PR opens or changes, extract its title, description, changed paths, and diff. Search memory using both lexical terms and semantic similarity if available. Narrow results by repository, language, directory, and recency where useful. Retrieve a small number of high-confidence lessons rather than dumping the entire archive into a prompt. Include the source PR links with every result.
6. Give the reviewer explicit instructions. Ask it to inspect the current diff, compare it with retrieved lessons, and report only actionable issues that apply to the new code. Require each finding to include the affected file and line, why it matters, and the supporting memory link. Tell it to say when no relevant lesson was found, and not to invent project rules or treat old discussion as authoritative without checking current code.
7. Keep memory separate from instructions. Treat PR descriptions, comments, and code as untrusted input. They may contain misleading text or instructions unrelated to review. The review service should use retrieved records as evidence, not execute them or let them override its fixed review policy. Do not place secrets in prompts or indexed records, and restrict access to the index and logs.
8. Test offline before posting comments. Run the system against a sample of past PRs that were not used to create its lessons. Check whether it finds relevant examples, cites them correctly, and avoids repeating rejected or obsolete comments. Ask maintainers to label findings as useful, irrelevant, or wrong. Tune retrieval and filters before enabling any automatic comment publishing.
9. Integrate gradually and monitor it. Start with a report visible only to maintainers or a draft review. Once quality is acceptable, allow carefully scoped comments, while keeping a human in control of consequential decisions. Log retrieved lesson IDs, model output, and reviewer feedback without logging secrets. Add a way to correct or retire bad memories, and periodically review stale records.

Common Mistakes

• Indexing every comment as a rule: discussions include speculation, rejected suggestions, and one-off decisions. Track outcomes and require validation.
• Retrieving too much: a long collection of loosely related lessons distracts the reviewer. Prefer a few precise matches with clear sources.
• Using similarity as proof: similar wording does not mean the same bug or design constraint. The reviewer must verify each lesson against the current diff.
• Ignoring change over time: an old workaround can become harmful after a refactor. Store dates and provide a retirement process.
• Publishing comments immediately: false positives erode trust. Begin in advisory or draft mode and measure feedback before expanding.
• Keeping more data than necessary: PR history may contain confidential details. Minimize retained content, control access, and follow repository retention requirements.

Tips

• Use structured fields and stable IDs so a lesson can be updated without losing its source.
• Make retrieval explainable: show maintainers which past PRs influenced each suggestion.
• Separate general project conventions from examples tied to a specific component.
• Track precision and usefulness, not just the number of comments produced. A quiet reviewer that finds strong, relevant precedents is often more useful.
• Let maintainers mark a memory as outdated, incorrect, or useful, and feed those corrections into the curation workflow.

Troubleshooting

It retrieves unrelated PRs: add filters for repository, file path, language, and lesson type. Try lexical search alongside semantic search, and increase the minimum relevance threshold. Inspect the actual query and returned records to find whether the problem is query construction or poor metadata.

It repeats old or rejected advice: check whether comments were indexed without their review outcome. Add status fields, exclude rejected guidance, and mark outdated records inactive. Rebuild the index after correcting source records.

It finds good examples but gives vague feedback: require a specific file and line, a concrete risk, and a source link. Confirm that the diff and retrieved lesson are both included in the reviewer’s input, and test with a small example.

It misses relevant history: verify that the PR collector handles pagination and includes the right repositories and comments. Check whether changed-file paths are normalized consistently, then test search with exact terms and paraphrases.

It is slow or fails on large PRs: limit diff size, split files into manageable sections, and retrieve memories per file or component. Handle API errors and rate limits with retries and clear logs; do not silently review an incomplete diff.

It exposes sensitive context: disable publishing, restrict index access, and inspect stored records and logs. Remove exposed data according to your retention process, rotate any credentials that were included, and resume only after the data flow is corrected.

Frequently Asked Questions

Does the reviewer need a vector database?
No. A small project can start with tags, database filters, and full-text search. Add semantic search if maintainers need to find lessons expressed in different words.

Should it learn from every new PR automatically?
Not without checks. Queue candidate lessons for review, and add them to shared memory only after confirming they are accurate, reusable, and not superseded.

Can it review code without sending the full repository?
Often, yes. Provide the diff and only the relevant surrounding code and retrieved lessons. The reviewer still needs enough context to understand symbols and behavior.

How can I tell whether memory improves reviews?
Compare results on held-out PRs with memory enabled and disabled. Have maintainers judge relevance, correctness, and usefulness, and track false positives as well as useful findings.

Should it automatically approve or block a PR?
Start with advisory findings. Historical precedent is context, not a reliable substitute for tests, current code, or human judgment, so consequential decisions should remain reviewable.

Related

• Precision Physics: When Measurements Change a Theory
• Low Cost Tech Gifts That Are Useful and Easy to Choose
• Save the World Codes: Current Rewards and How to Redeem