How we measure review quality
We put 34 real bugs from 24 open-source projects back into their code, in 33 pull requests, and asked MergeHarbor to review each change, along with 7 clean changes it should leave alone.
- pull requests
- 40
- pull requests
- languages
- 7
- languages
- projects
- 24
- projects
- run date
- October 21, 2026
- run date
Results by review tier
Every tier runs the shipped pipeline: investigation, verification and filtering. A bug counts as caught when a comment lands on the same file within five lines of it.
Standard
40 pull requests91.2%
of bugs caught
- Comments that were real bugs
- 91.2%
- Clean changes commented on
- 0%
- Cost per review
- $0.04
- Median time
- 18 s
By severity
- major23 of 24
- minor8 of 10
Deep
40 pull requests85.3%
of bugs caught
- Comments that were real bugs
- 93.6%
- Clean changes commented on
- 0%
- Cost per review
- $0.16
- Median time
- 45 s
By severity
- major22 of 24
- minor7 of 10
Max
34 of 40 pull requests85.3%
of bugs caught
- Comments that were real bugs
- 90.6%
- Clean changes commented on
- 0%
- Cost per review
- $0.21
- Median time
- 69 s
By severity
- major21 of 24
- minor8 of 10
Max reviewed all 33 pull requests with bugs (34 bugs) but only 1 of the 7 clean changes before the run reached its spending cap of $15, so its clean-change figure rests on one change. Standard is the default for most pull requests; see review tiers.
Where it catches bugs, and what it costs
- Go
- 5 of 6
- Java
- 6 of 6
- Python
- 6 of 6
- Ruby
- 4 of 4
- Rust
- 4 of 4
- TypeScript
- 3 of 4
- JavaScript
- 3 of 3
Model cost per review
US dollars, average
- Standard
- $0.0375
- Deep
- $0.1557
- Max
- $0.2136
Review time
Median to 95th percentile, seconds
- Standard
- 18 s to 26 s
- Deep
- 45 s to 86 s
- Max
- 69 s to 97 s
Every case
Each row is a real bug fix. Open the fix to see the original change and the bug it repaired. One pull request carries a second bug; its row shows the first one.
| Project | Bug | Severity | Standard | Deep | Max | Fix |
|---|---|---|---|---|---|---|
| prometheus/prometheus | Nil pointer dereference: inside the `e.RHS == nil` branch the code calls e.RHS.PositionRange(); it should use e.LHS | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| prometheus/prometheus | Off-by-one overflow guard: float64(math.MaxInt64) rounds to 2^63, so ts == 2^63 (and NaN) pass `>` and time.Duration(ts) overflows; use >= and reject NaN | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| etcd-io/etcd | time.After inside a long-running for/select loop creates a new timer every iteration that is not released until it fires; reuse a single time.Timer | minor | ✗Missed | ✗Missed | ✗Missed | Fix (opens in a new tab) |
| gin-gonic/gin | Data race: c.Keys is read before c.mu.RLock(), while Set writes the map under the lock; the copy can miss keys or race | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| go-chi/chi | Missing return after next.ServeHTTP in the empty-router branch: execution falls through and the next handler is invoked a second time | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| caddyserver/caddy | Error from ca.Root.Load() is not checked before rootCertChain[0]: a failed load panics with index out of range instead of returning the error | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| gin-gonic/gin | Clean change, no bug | — | ✓No comment | ✓No comment | ✓No comment | Fix (opens in a new tab) |
| Project | Bug | Severity | Standard | Deep | Max | Fix |
|---|---|---|---|---|---|---|
| google/guava | Overflow check happens before the multiply-and-add, so values just above Integer.MAX_VALUE (e.g. 2147483648) wrap to a negative int instead of returning -1 | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| apache/commons-lang | Only a single factor of two is removed and only when the denominator is MIN_VALUE; MIN_VALUE numerators/denominators reach sign negation and overflow (ArithmeticException or wrong value) | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| apache/commons-io | skip() adds the requested n (not the number actually skipped) and does not clamp to the remaining bound, so it can skip past the limit and corrupt charsRead (int overflow for large n) | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| apache/commons-lang | Caller's mutable TimeZone is stored in the static cache key and formatter without a defensive clone; later mutation makes lookups return formatters with stale zone rules | minor | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| keycloak/keycloak | Missing resource returns null (404) before any permission check, while existing ones return 403, letting callers without view permission probe which ids exist | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| spring-projects/spring-framework | Length pre-computation calls element.toString() and throws NullPointerException for null elements; use String.valueOf(element) | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| google/guava | Clean change, no bug | — | ✓No comment | ✓No comment | — | Fix (opens in a new tab) |
| Project | Bug | Severity | Standard | Deep | Max | Fix |
|---|---|---|---|---|---|---|
| axios/axios | No 'goaway' listener: a session that received GOAWAY stays in the pool and is reused, so new requests fail; also the idle timer can re-run removal for an already removed session | major | ✓Caught | ✓Caught | ✗Missed | Fix (opens in a new tab) |
| axios/axios | filter is invoked with `this` of a plain module function (undefined) instead of the passed-in headers context | minor | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| expressjs/express | Content-Length is populated even when a Transfer-Encoding header is set; sending both is invalid HTTP (RFC 9112) and enables request/response smuggling ambiguities | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| expressjs/express | Clean change, no bug | — | ✓No comment | ✓No comment | — | Fix (opens in a new tab) |
| Project | Bug | Severity | Standard | Deep | Max | Fix |
|---|---|---|---|---|---|---|
| psf/requests | Empty entries from a trailing or doubled ';' in ProxyOverride become an empty regex that matches every host, so the proxy is bypassed for all hosts; filter out empty strings | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| psf/requests | A netrc entry with all-empty fields is a truthy tuple, so empty login/password are returned as auth; check that some field is non-empty | minor | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| pallets/flask | Key order reversed: itsdangerous signs with the last key in the list, so placing secret_key first and fallbacks after makes new sessions signed with a retired fallback key | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| pallets/werkzeug | Wrong boolean logic: requiring subdomain_matching or host_matching to be enabled means a map with neither enabled never returns a plain path and always builds an external URL | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| encode/httpx | Unescaped '.' in the IPv4 regex matches any character, so hosts like '1x2x3x4' are treated as IPv4 literals; escape the dots | minor | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| fastapi/fastapi | Recursive jsonable_encoder calls for dict keys and values pass exclude_unset and exclude_none but drop exclude_defaults, so nested models inside dicts ignore it | minor | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| psf/requests | Clean change, no bug | — | ✓No comment | ✓No comment | — | Fix (opens in a new tab) |
| Project | Bug | Severity | Standard | Deep | Max | Fix |
|---|---|---|---|---|---|---|
| sinatra/sinatra | Splitting the client-controlled If-None-Match/If-Match header with /\s*,\s*/ backtracks polynomially on long whitespace runs (ReDoS); split on ',' and strip | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| rack/rack | path.index(url) == 0 is a plain string-prefix test, so urls: ['/assets'] also routes '/assets-private/...' to the static file server; require an exact match or a trailing '/' | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| rails/rails | 10**options[:precision] raises TypeError when precision is nil (allowed for positive numbers); guard the nil case | minor | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| rails/rails | Stray .compact (copied from the filter: true branch) removes nil elements although filter: false is documented to keep every element | minor | ✓Caught | ✗Missed | ✓Caught | Fix (opens in a new tab) |
| rack/rack | Clean change, no bug | — | ✓No comment | ✓No comment | — | Fix (opens in a new tab) |
| Project | Bug | Severity | Standard | Deep | Max | Fix |
|---|---|---|---|---|---|---|
| tokio-rs/tokio | Adding the unread-buffer adjustment to a SeekFrom::Current offset overflows i64 for extreme offsets (e.g. i64::MIN): panic in debug, wrong seek in release; use checked_add | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| tokio-rs/tokio | Rejected task is shut down (dropping the user's closure) while the blocking-pool mutex is held; a Drop impl that calls spawn_blocking re-locks it and deadlocks. Drop the guard first | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| tokio-rs/tokio | chan.send wakes the receiver before mem::forget(self); if the waker panics, unwinding drops the Permit and releases its slot a second time (semaphore over-release) | major | ✓Caught | ✗Missed | ✗Missed | Fix (opens in a new tab) |
| hyperium/hyper | HeaderMap::insert replaces earlier values, so repeated trailer fields keep only the last one; use append | minor | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| hyperium/hyper | Clean change, no bug | — | ✓No comment | ✓No comment | — | Fix (opens in a new tab) |
| Project | Bug | Severity | Standard | Deep | Max | Fix |
|---|---|---|---|---|---|---|
| honojs/hono | serveStatic no longer checks the request method: POST/PUT/DELETE requests are answered with static files instead of falling through to next(); only GET and HEAD should be served | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| honojs/hono | Empty-segment check only covers the end of the path: '/u//posts' matches '/u/:id/posts' with id = '' (other routers return 404) | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| honojs/hono | Header keys are lowercased but the user-supplied retainedHeaders list is not, so mixed-case names like 'X-Custom' are never retained on 304 responses | minor | ✗Missed | ✗Missed | ✗Missed | Fix (opens in a new tab) |
| vitejs/vite | hmrProxy aliases the user's hmr config object, so Object.defineProperty installs accessors on the shared/default config; copy it first ({ ...hmrConfig }) | major | ✓Caught | ✓Caught | ✓Caught | Fix (opens in a new tab) |
| honojs/hono | Clean change, no bug | — | ✓No comment | ✓No comment | — | Fix (opens in a new tab) |
✓ caught the bug, or left a clean change alone. ✗ missed the bug, or commented on a clean change. — not run.
How we ran it
- Real bugs, put back. Each case starts from a merged bug fix in a permissively licensed project (MIT, Apache-2.0 or BSD). The pull request under review undoes the fix, so it reintroduces a bug that really shipped.
- No hints. The reviewer sees only the code change and a neutral title such as “Update src/parser.go”. Test files added by the fix stay in place, and the commit history shows nothing about the fix.
- Clean changes too. 7 small merged changes with no bug measure noise: the right result is no comment at all.
- Default settings. No custom rules, no learned memory, no repository index: the reviewer reads the changed files and whatever else its tools return.
- Strict matching. A bug is caught only by a comment on the same file within five lines. One correct comment ten lines away counted as a miss and a false positive.
What to keep in mind
- Undoing a fix often removes a guard or a check, which can be easier to spot than a bug written from scratch. Some fixes also added comments that the reverted change deletes.
- These are well-known projects, and models may have seen their history during training.
- 34 bugs is a small sample: one bug moves a tier's catch rate by about three points. Read the case table, not only the headline.
- We publish our own results only. Other tools use different cases and rules, so their numbers are not comparable to these.
Controlled set (in-sample)
We also run 61 small pull requests written for testing (48 with one bug each, 13 clean) on every change to the reviewer, so regressions show up before release. The prompts and posting bar were tuned on these cases, so these figures are regression checks, not an estimate of quality on your pull requests.
- Standard
- 100% caught
- 10% clean commented
- Deep
- 100% caught
- 10% clean commented
- Max
- 100% caught
- 10% clean commented
See the comments it wrote
Every example is MergeHarbor's own review comment from this run, next to the real fix.
Try it on your own pull request
Paste a public GitHub pull request and see MergeHarbor's review in a few minutes. No sign-up, nothing posted.
Review every pull request with the context of your whole codebase
Connect a code host in minutes. Reviews start right away, and codebase context is added as indexing completes.