AI code review uses a language model to read a pull request the way a reviewer would: it looks at what changed, works out what the change is meant to do, and comments where the code is likely to be wrong. The goal is the same as human review. Catch the defects that matter before they merge, and say nothing about the rest.
This guide explains what an AI reviewer does, where it sits next to the tools you already run, and how to tell a useful one from a noisy one before you roll it out.
What an AI reviewer does
A typical AI review runs when a pull request opens or gets a new push. The reviewer receives the diff and some surrounding code, reasons about it, and posts inline comments on the code host. A good comment names the file and line, explains the problem in terms of what will happen at runtime, and often suggests a change.
What separates AI review from earlier automation is judgment. A model can notice that a guard was removed, that a new branch never releases a lock, or that a function now returns an empty list where callers expect an error. None of those break a syntax rule. They break an assumption, and spotting a broken assumption requires reading the code with intent in mind.
That same flexibility is the risk. A model that can comment on anything will comment on everything unless it is held to a bar. The rest of this guide is mostly about that bar.
How it differs from linters and static analysis
Linters, type checkers and static analysis tools are deterministic. They apply fixed rules and give the same answer every time. AI review is probabilistic and reasons about intent. They solve different problems, and a team usually wants both.
| Tool | What it checks | Strengths | Limits |
|---|---|---|---|
| Linter and formatter | Style, simple patterns, unused code | Fast, exact, cheap | No understanding of what the code is for |
| Type checker and compiler | Type and interface agreement | Proves whole classes of errors are absent | Only what the type system can express |
| Static analysis | Data flow, known vulnerability patterns | Repeatable, explainable rules | Rules must be written in advance; can be noisy on unusual code |
| AI code review | Logic, intent, contracts between parts of the code, missing checks | Finds bugs no one wrote a rule for | Can be wrong; needs context and verification to be trusted |
| Human review | Design, product intent, ownership, trade-offs | Knows the business and the history of decisions | Time-limited; attention drops on large changes |
Some checks need no judgment at all. A committed secret, a SQL query built by concatenating values, or a disabled TLS check is wrong regardless of context. Rules catch these more reliably and more cheaply than a model. MergeHarbor runs built-in deterministic checks for these before any model reads the change, and they use no credits.
The useful division of labour is: rules for what can be decided mechanically, the AI reviewer for logic and contracts, and people for design and intent.
Diff-only versus codebase-aware review
The diff shows what changed. It does not show what depends on it. A one-line change to a function’s return value looks harmless until you see the twelve callers that dereference the result.
A diff-only reviewer sees the changed lines and perhaps a few lines around them. It can catch local mistakes, such as an off-by-one error or a typo in a condition, but it has to guess about everything else.
A codebase-aware reviewer can read beyond the diff. It can look up the callers of a changed function, find the configuration that reads a renamed setting, search for similar code that already solved the same problem, and check the file’s history. That context turns guesses into findings with evidence.
MergeHarbor indexes each repository into a code graph, a text index and a semantic index, and gives the reviewer read-only tools to search them. The guide on diff-only versus codebase-aware review walks through concrete examples of bugs a diff hides.
Verification and noise
The fastest way to lose a team’s trust is to post comments that turn out to be wrong. Once engineers learn that half the comments are noise, they stop reading all of them, including the correct ones.
There are three common sources of noise:
- False positives. The reviewer reports a bug that is not there, often because it did not see the code that handles the case.
- Low-value notes. Comments about naming, style or documentation on a change where nothing is actually wrong.
- Duplicates. The same issue reported in several places, or repeated after someone already raised it.
A reviewer that takes noise seriously does more than tune its prompt. It checks its own work. MergeHarbor runs a second pass that tries to disprove each finding, citing the code it checked, and never posts a finding the pass disproves. It merges duplicates, holds back style notes unless you ask for them, and holds every review tier to the same posting bar, so a deeper review investigates more without commenting more.
You can see how much this matters in your own organization: the share of draft findings the verifier stopped appears under Insights → Quality as Stopped by verifier.
How to evaluate an AI code review tool
Demos tend to show the best case. To compare tools, or to decide whether one is worth adopting, measure them on changes you understand.
What to measure
- Catch rate. Of the known bugs in your test set, how many did the reviewer report at the right place?
- Precision. Of the comments it posted, how many pointed at a real problem?
- Comments on clean changes. On pull requests with nothing wrong, how often did it comment anyway? The right answer is no comment.
- Cost per review. What does a typical review cost, and what does an expensive one cost?
- Time to first comment. How long does a developer wait for a result?
How to build a fair test
The benchmark method we publish is one example of how to set this up:
- Use real bugs. Start from merged bug fixes and revert them, so each pull request reintroduces a bug that really shipped.
- Give no hints. Use neutral titles and keep the fix’s commit message out of the history the reviewer can see.
- Include clean changes. Small merged changes with no bug measure how much the reviewer says when it should stay quiet.
- Match strictly. Count a bug as caught only when a comment lands on the same file near the defect. A vague comment elsewhere is a miss.
- Publish every case. Read the per-case table, not just the headline. A small set moves a lot when one case changes.
Run each tool with default settings first, then with the configuration you would actually use. Keep in mind that reverting a fix often removes a guard, which can be easier to spot than a bug written from scratch, and that well-known open-source projects may appear in a model’s training data. Your own repositories are the most honest test.
Rolling it out
A careful rollout gives the team time to calibrate their trust before the tool affects merges.
- Start with a few repositories. Pick active ones with owners who will give feedback. Reviews start right away, and codebase context is added as indexing completes.
- Write down your conventions. Rules in plain English, such as “amounts are integers in cents”, stop the reviewer from flagging things your team does on purpose. MergeHarbor reads rules from the console and from
.mergeharbor/folders in the repository. - Watch how people respond. Which comments get fixed, which get dismissed, and why. MergeHarbor learns from this feedback and proposes rules for an administrator to approve; nothing it learns applies on its own.
- Report before you enforce. A merge gate in advisory mode, the default, shows a verdict without blocking anything. Switch to enforce once the team agrees the verdict is reliable.
- Set a budget. Decide which changes deserve a deeper review and cap spend per month and per pull request. MergeHarbor’s review tiers use 1, 3 or 10 credits, and Auto picks one per pull request and says why.
A conservative starting point for the merge gate in the first month:
# .mergeharbor.yml
merge_gate:
enabled: true
mode: advisory # report only; nothing is blockedWhat AI review does not replace
AI review does not decide whether a feature should exist, whether an abstraction will age well, or whether a change fits where the product is going. It does not own the code. Keep human approval on every change that matters, and use the AI reviewer to make sure those humans spend their attention on design rather than on hunting for a missing null check.
Used that way, AI review is a second reader that gives every line of every pull request the same attention, and shows its evidence when it speaks up.
Further reading
- How MergeHarbor reviews a pull request, stage by stage
- Benchmarks: every case, per review tier, and how to reproduce the run
- A code review checklist for human and AI-assisted review