A first prototype of an LLM code reviewer takes an afternoon. Fetch a pull request’s diff, send it to a model with a prompt that says “review this code”, and post the answer as a comment. It works well enough to be convincing in a demo.
The gap between that prototype and a reviewer a whole engineering organization trusts is large, and most of it has nothing to do with the prompt. This guide lists the components a production reviewer needs, the work that continues after launch, and the situations where building your own is still the right decision.
The prototype and where it stops
The afternoon version usually looks like this:
diff = code_host.get_diff(pr)
reply = llm.complete(f"Review this diff and list bugs:\n{diff}")
code_host.post_comment(pr, reply)Within a week of real use, the problems show up:
- Comments repeat on every push, because nothing remembers what was already said.
- The model flags things your team does on purpose, because it does not know your conventions.
- It misses bugs whose cause sits outside the diff, because it never saw the callers.
- Some comments are confidently wrong, and people start ignoring all of them.
- Large pull requests exceed the context window or cost more than anyone expected.
- Someone asks whether code is being sent to a model provider with secrets in it, and nobody is sure.
Each of these has a fix. Together, the fixes are a product.
The components you would need
Code host integration
Each code host has its own webhooks, diff format, comment API, status checks, thread resolution and permission model, and self-managed versions often differ from the cloud ones. You need to verify webhook signatures, handle retries and duplicate deliveries, deal with force-pushes and rebases, anchor comments to the right line after the code moves, and respect rate limits. If your organization uses more than one code host, multiply this work.
MergeHarbor supports several code hosts, including self-managed editions, and every feature below works across them.
Repository indexing
To review with more than the diff, the reviewer needs to search the codebase quickly. That means keeping a mirror of each repository, parsing it into a code graph of definitions, calls and types, building a text index for exact search and a semantic index for search by meaning, and updating all three incrementally after each push. Indexing must not block reviews while it runs.
Retrieval and context budgeting
An index is only useful if the right parts of it reach the model. You need to decide which callers, implementations and similar code to include, merge results from several kinds of search, remove duplicates, and trim to a token budget so the important parts survive when space is tight. Generated, vendored and lock files should stay out of the prompt.
An agent loop with tools
A single prompt cannot know in advance what it will need to read. A reviewer that investigates gets tools (read a file, search code, find callers, read history) and decides its own next step, within limits on steps and tokens. The tools must be read-only and scoped to one repository and commit.
Verification and a posting bar
This is the part prototypes skip and teams notice most. Each candidate finding should be checked by a second pass that tries to prove it wrong. Duplicates should be merged, issues already raised should not be repeated, and low-value notes should be held back. MergeHarbor’s review pipeline shows one way to arrange these stages.
Deterministic checks
Some defects need no model: committed secrets, injection patterns, unsafe settings, migrations that lock tables, breaking API contract changes. Rules for these are cheaper and more reliable than a model, but each rule needs tests on clean code so it stays quiet when nothing is wrong.
Cost control
Model spend grows with pull request size and volume. Keeping it predictable takes several mechanisms working together:
- routing routine changes to small models and escalating only risky ones;
- prompt caching, so the stable parts of a prompt (instructions, rules) are reused across reviews;
- batch processing for work that can wait;
- reviewing only what changed since the last push;
- budgets and per-pull-request caps, with a defined behaviour when they are reached;
- per-review cost reporting so you can see where the money goes.
MergeHarbor documents how it does each of these in Review costs.
Security and compliance
Before code reaches a model, secrets should be removed from diffs, file contents and tool results. You will need to answer which providers see which data, how long they keep it, and whether they train on it. An enterprise rollout adds single sign-on, roles, an audit log, data export, possibly self-hosting and data residency, and evidence for auditors that each change was reviewed. See How MergeHarbor uses AI and your code for the level of detail a security review will ask for.
Evaluation
Without a test set, every prompt change is a guess. You need cases with known bugs and clean cases that should get no comment, a way to score catch rate and precision, and a habit of running the set before every change ships. Model providers release new versions regularly, and each one needs to be measured before you switch. Our benchmark method describes one way to build such a set.
Feedback and learning
Teams react to comments: they fix some, dismiss others, and reply with explanations. Capturing that, turning patterns into rules, and letting an administrator approve or undo them keeps the reviewer aligned with the team over time.
Operations
Queues, retries, provider outages and fallbacks, monitoring, upgrades and on-call. A reviewer that blocks merges when a model provider is down becomes an incident of its own.
The ongoing cost
Building is a one-time cost. Running is not. Expect continuing work on:
- Model changes. New models change behaviour. Each switch needs re-evaluation and often prompt changes.
- Price changes. Provider prices move, and routing that was cheap last quarter may not be now.
- Code host changes. APIs, webhook payloads and permission models change.
- Quality drift. As your codebase and conventions change, rules and context need upkeep.
- Support. Someone has to answer “why did it say that?” and “why didn’t it catch this?”.
Model spend is the most visible cost, but not the only one. For reference, on our published benchmark of small real-world changes, MergeHarbor’s model cost per review was about $0.04 at the Standard tier, $0.16 at Deep and $0.21 at Max. Your own changes will differ, and larger pull requests use more tokens. The engineering time to build and maintain the components above is the other part of the cost, and only you can estimate it for your team.
What buying costs
For comparison, MergeHarbor’s plans are Essentials at $25, Team at $55 and Advanced at $85 per active developer per month ($20, $45 and $70 per seat billed annually upfront). Each active developer adds 100, 200 or 300 credits to an allowance pooled across the organization. A Standard review uses 1 credit, Deep 3 and Max 10 (Deep from Team, Max from Advanced), and extra credits are $0.25 each, with a hard monthly limit on extra spend that admins set. An active developer is someone whose pull request received a review that used credits that month. New organizations get a 14-day free trial of Team with no card required; single sign-on and SCIM are part of Enterprise. There is no free plan; public open-source repositories are reviewed free through the open source program. Enterprise is priced by agreement.
When building makes sense
Building your own reviewer is a reasonable choice when:
- The check is narrow and specific to you. A focused check on one internal framework or one class of mistake can be a small script on a model API. A general reviewer is a different scale of project.
- Review is your product or research area. If you want to learn how to build this, building is the point.
- Your platform is unusual. An in-house version control system or a workflow no vendor supports may leave no option.
- You already run the surrounding platform. A team that already operates code search, an evaluation harness and model routing has many of the pieces.
Building is usually the wrong choice when the main reasons are control over data or cost, because those can often be met without writing the reviewer yourself.
The middle path: buy and extend
Many teams buy the reviewer and customize the parts that make it theirs:
- Your rules. Plain-English rules in the console or in
.mergeharbor/folders next to the code they govern. - Your review depth. Choose a review tier per path or size of change and set credit caps. On the hosted service, MergeHarbor runs and pays for the models, so there are no model keys to manage.
- Your infrastructure. On Enterprise, self-host on Kubernetes or AWS, so code stays in your cloud.
- Your tools. The CLI writes SARIF and JSON for CI, and an MCP server lets coding agents read and fix findings.
- Your checks. Turn off any built-in rule that does not fit, per organization, repository or folder.
Whichever way you go, decide how you will measure the result before you start. A reviewer you cannot measure is one you cannot improve.