# How we measure review quality

We put 34 real bugs from 24 open-source projects back into their code, in 33 pull requests, and asked MergeHarbor to review each change, along with 7 clean changes it should leave alone.

40

pull requests

7

languages

24

projects

October 21, 2026

run date

Read this first

-   This is a pilot: one live review per case and tier on October 21, 2026, 40 cases in all. The numbers are point estimates with no confidence intervals.
-   The cases were not split into a tuning set and a held-out set, so this is not an out-of-sample result. A larger run on a fixed held-out split, with confidence intervals, replaces it when it is recorded.
-   The controlled set further down is in-sample: the reviewer's prompts and posting bar were tuned on it.

## Results by review tier

Every tier runs the shipped pipeline: investigation, verification and filtering. A bug counts as caught when a comment lands on the same file within five lines of it.

### Standard

40 pull requests

91.2%

of bugs caught

Comments that were real bugs

91.2%

Clean changes commented on

0%

Cost per review

$0.04

Median time

18 s

By severity

-   major 23 of 24

-   minor 8 of 10

### Deep

40 pull requests

85.3%

of bugs caught

Comments that were real bugs

93.6%

Clean changes commented on

0%

Cost per review

$0.16

Median time

45 s

By severity

-   major 22 of 24

-   minor 7 of 10

### Max

34 of 40 pull requests

85.3%

of bugs caught

Comments that were real bugs

90.6%

Clean changes commented on

0%

Cost per review

$0.21

Median time

69 s

By severity

-   major 21 of 24

-   minor 8 of 10

Max reviewed all 33 pull requests with bugs (34 bugs) but only 1 of the 7 clean changes before the run reached its spending cap of $15, so its clean-change figure rests on one change. Standard is the default for most pull requests; see [review tiers](https://mergeharbor.com/docs/review-tiers).

## Where it catches bugs, and what it costs

Bugs caught by language Pull requests with a bug that the Standard tier caught, out of all bug pull requests per language

Go

5 of 6

Java

6 of 6

Python

6 of 6

Ruby

4 of 4

Rust

4 of 4

TypeScript

3 of 4

JavaScript

3 of 3

Model cost and review time by review tier 34 real bugs in 33 pull requests, plus 7 clean changes. Benchmark run October 21, 2026.

### Model cost per review

US dollars, average

Standard

$0.0375

Deep

$0.1557

Max

$0.2136

### Review time

Median to 95th percentile, seconds

Standard

18 s to 26 s

Deep

45 s to 86 s

Max

69 s to 97 s

## Every case

Each row is a real bug fix. Open the fix to see the original change and the bug it repaired. One pull request carries a second bug; its row shows the first one.

GoJavaJavaScriptPythonRubyRustTypeScript

| Project | Bug | Severity | Standard | Deep | Max | Fix |
| --- | --- | --- | --- | --- | --- | --- |
| prometheus/prometheus | Nil pointer dereference: inside the \`e.RHS == nil\` branch the code calls e.RHS.PositionRange(); it should use e.LHS | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/prometheus/prometheus/commit/e4e65d1247bb6f92fb1fd1ded5d0ed3486136ed3) |
| prometheus/prometheus | Off-by-one overflow guard: float64(math.MaxInt64) rounds to 2^63, so ts == 2^63 (and NaN) pass \`>\` and time.Duration(ts) overflows; use >= and reject NaN | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/prometheus/prometheus/commit/ba860bfd0ea767a5da2f301caada12b29b1d2526) |
| etcd-io/etcd | time.After inside a long-running for/select loop creates a new timer every iteration that is not released until it fires; reuse a single time.Timer | minor | ✗Missed | ✗Missed | ✗Missed | [Fix](https://github.com/etcd-io/etcd/commit/ed01ee1e5ecd6dc46054c37b34c3b1b4e015a287) |
| gin-gonic/gin | Data race: c.Keys is read before c.mu.RLock(), while Set writes the map under the lock; the copy can miss keys or race | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/gin-gonic/gin/commit/1bd0ecf2d3e08da37efa270bb7be20c1eddb0d68) |
| go-chi/chi | Missing return after next.ServeHTTP in the empty-router branch: execution falls through and the next handler is invoked a second time | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/go-chi/chi/commit/05f1ef7bb50b8a8cb33a9dd3ba1c5b94bff0f723) |
| caddyserver/caddy | Error from ca.Root.Load() is not checked before rootCertChain\[0\]: a failed load panics with index out of range instead of returning the error | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/caddyserver/caddy/commit/9738f49b952a337cdaf42175a0e92b01974c0e3c) |
| gin-gonic/gin | *Clean change, no bug* | — | ✓No comment | ✓No comment | ✓No comment | [Fix](https://github.com/gin-gonic/gin/commit/488f8c3ffa579a8d19beb2bae95ff8ef36b3d53f) |

| Project | Bug | Severity | Standard | Deep | Max | Fix |
| --- | --- | --- | --- | --- | --- | --- |
| google/guava | Overflow check happens before the multiply-and-add, so values just above Integer.MAX\_VALUE (e.g. 2147483648) wrap to a negative int instead of returning -1 | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/google/guava/commit/6b3a8d7498a7fde122967ba37a023ba3dcb47792) |
| apache/commons-lang | Only a single factor of two is removed and only when the denominator is MIN\_VALUE; MIN\_VALUE numerators/denominators reach sign negation and overflow (ArithmeticException or wrong value) | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/apache/commons-lang/commit/e992a498d711fc4302d5ad89e80511db41db2f11) |
| apache/commons-io | skip() adds the requested n (not the number actually skipped) and does not clamp to the remaining bound, so it can skip past the limit and corrupt charsRead (int overflow for large n) | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/apache/commons-io/commit/e806e6b5926993ec6348a78542b1e1377a765e88) |
| apache/commons-lang | Caller's mutable TimeZone is stored in the static cache key and formatter without a defensive clone; later mutation makes lookups return formatters with stale zone rules | minor | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/apache/commons-lang/commit/55dcbc4bb1449aa9f7c6ee422cc22b465a35fa2f) |
| keycloak/keycloak | Missing resource returns null (404) before any permission check, while existing ones return 403, letting callers without view permission probe which ids exist | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/keycloak/keycloak/commit/0aa156ebd74ad90e7faccb77b7ca1844bfbec812) |
| spring-projects/spring-framework | Length pre-computation calls element.toString() and throws NullPointerException for null elements; use String.valueOf(element) | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/spring-projects/spring-framework/commit/0d6cc122744df002e35f0788eab28bbd492c46ed) |
| google/guava | *Clean change, no bug* | — | ✓No comment | ✓No comment | — | [Fix](https://github.com/google/guava/commit/faff9ff208f0850f93baf75b069f88366ce7fc03) |

| Project | Bug | Severity | Standard | Deep | Max | Fix |
| --- | --- | --- | --- | --- | --- | --- |
| axios/axios | No 'goaway' listener: a session that received GOAWAY stays in the pool and is reused, so new requests fail; also the idle timer can re-run removal for an already removed session | major | ✓Caught | ✓Caught | ✗Missed | [Fix](https://github.com/axios/axios/commit/43968f294eaf79dabf0fcc2c05fd9add6a4faf12) |
| axios/axios | filter is invoked with \`this\` of a plain module function (undefined) instead of the passed-in headers context | minor | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/axios/axios/commit/8d415e6af9cd50a8ae27331a53b4aaf9f7e24612) |
| expressjs/express | Content-Length is populated even when a Transfer-Encoding header is set; sending both is invalid HTTP (RFC 9112) and enables request/response smuggling ambiguities | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/expressjs/express/commit/18e5985b8a9d5e8423db0a9121f22bdaecd5b120) |
| expressjs/express | *Clean change, no bug* | — | ✓No comment | ✓No comment | — | [Fix](https://github.com/expressjs/express/commit/9d8223d92ee81137a50a28eb6ad55a096791091d) |

| Project | Bug | Severity | Standard | Deep | Max | Fix |
| --- | --- | --- | --- | --- | --- | --- |
| psf/requests | Empty entries from a trailing or doubled ';' in ProxyOverride become an empty regex that matches every host, so the proxy is bypassed for all hosts; filter out empty strings | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/psf/requests/commit/1604e20fc87ce212cf3a02474a6d7509b640cbbb) |
| psf/requests | A netrc entry with all-empty fields is a truthy tuple, so empty login/password are returned as auth; check that some field is non-empty | minor | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/psf/requests/commit/47914226c2968802f2cdee3f6f0f6e06e4472f64) |
| pallets/flask | Key order reversed: itsdangerous signs with the last key in the list, so placing secret\_key first and fallbacks after makes new sessions signed with a retired fallback key | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/pallets/flask/commit/fb54159861708558b5f5658ebdc14709d984361c) |
| pallets/werkzeug | Wrong boolean logic: requiring subdomain\_matching or host\_matching to be enabled means a map with neither enabled never returns a plain path and always builds an external URL | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/pallets/werkzeug/commit/884437aef93e543ebaff3b9cdbe857ee47e7e00c) |
| encode/httpx | Unescaped '.' in the IPv4 regex matches any character, so hosts like '1x2x3x4' are treated as IPv4 literals; escape the dots | minor | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/encode/httpx/commit/e63b6594f2863b7c8274eb0991ebc6cad63661f7) |
| fastapi/fastapi | Recursive jsonable\_encoder calls for dict keys and values pass exclude\_unset and exclude\_none but drop exclude\_defaults, so nested models inside dicts ignore it | minor | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/fastapi/fastapi/commit/aadfcce76380ab169fe172d5cda21722e53c4924) |
| psf/requests | *Clean change, no bug* | — | ✓No comment | ✓No comment | — | [Fix](https://github.com/psf/requests/commit/b684dcb9bbf3aa557d1238e72062c4a29737dd1c) |

| Project | Bug | Severity | Standard | Deep | Max | Fix |
| --- | --- | --- | --- | --- | --- | --- |
| sinatra/sinatra | Splitting the client-controlled If-None-Match/If-Match header with /\\s\*,\\s\*/ backtracks polynomially on long whitespace runs (ReDoS); split on ',' and strip | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/sinatra/sinatra/commit/3fe8c38dc405586f7ad8f2ac748aa53e9c3615bd) |
| rack/rack | path.index(url) == 0 is a plain string-prefix test, so urls: \['/assets'\] also routes '/assets-private/...' to the static file server; require an exact match or a trailing '/' | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/rack/rack/commit/6d434bbccdc2ffc3086dbce3930c3816945a35ec) |
| rails/rails | 10\*\*options\[:precision\] raises TypeError when precision is nil (allowed for positive numbers); guard the nil case | minor | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/rails/rails/commit/21c60f81a3c72635ba9b0fdb863608952d4766a8) |
| rails/rails | Stray .compact (copied from the filter: true branch) removes nil elements although filter: false is documented to keep every element | minor | ✓Caught | ✗Missed | ✓Caught | [Fix](https://github.com/rails/rails/commit/30d0cdf6767d050cb7cfd2b5df6038d3eea29059) |
| rack/rack | *Clean change, no bug* | — | ✓No comment | ✓No comment | — | [Fix](https://github.com/rack/rack/commit/bd9db1ac67492b4886cc7793b40bd590bb482d45) |

| Project | Bug | Severity | Standard | Deep | Max | Fix |
| --- | --- | --- | --- | --- | --- | --- |
| tokio-rs/tokio | Adding the unread-buffer adjustment to a SeekFrom::Current offset overflows i64 for extreme offsets (e.g. i64::MIN): panic in debug, wrong seek in release; use checked\_add | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/tokio-rs/tokio/commit/2fc5972b8bd6f51d1ad6df3bcc1a1927d1702663) |
| tokio-rs/tokio | Rejected task is shut down (dropping the user's closure) while the blocking-pool mutex is held; a Drop impl that calls spawn\_blocking re-locks it and deadlocks. Drop the guard first | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/tokio-rs/tokio/commit/436faa2da3970817653fef98c9abc6c8b4367f51) |
| tokio-rs/tokio | chan.send wakes the receiver before mem::forget(self); if the waker panics, unwinding drops the Permit and releases its slot a second time (semaphore over-release) | major | ✓Caught | ✗Missed | ✗Missed | [Fix](https://github.com/tokio-rs/tokio/commit/5b758f5b5bc5b05ee3a47004aadaeadc9e2f9cb2) |
| hyperium/hyper | HeaderMap::insert replaces earlier values, so repeated trailer fields keep only the last one; use append | minor | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/hyperium/hyper/commit/876effe10fd8f8ad4535ade1f84e88f199f2cf6b) |
| hyperium/hyper | *Clean change, no bug* | — | ✓No comment | ✓No comment | — | [Fix](https://github.com/hyperium/hyper/commit/da3b1b197c954ac5454a92562fe3d729a87cba56) |

| Project | Bug | Severity | Standard | Deep | Max | Fix |
| --- | --- | --- | --- | --- | --- | --- |
| honojs/hono | serveStatic no longer checks the request method: POST/PUT/DELETE requests are answered with static files instead of falling through to next(); only GET and HEAD should be served | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/honojs/hono/commit/588ec13fa68ea44fed1b5e1b503341db2bf09d70) |
| honojs/hono | Empty-segment check only covers the end of the path: '/u//posts' matches '/u/:id/posts' with id = '' (other routers return 404) | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/honojs/hono/commit/00ee875728e2369b98a9ab7dda0690b77afedcff) |
| honojs/hono | Header keys are lowercased but the user-supplied retainedHeaders list is not, so mixed-case names like 'X-Custom' are never retained on 304 responses | minor | ✗Missed | ✗Missed | ✗Missed | [Fix](https://github.com/honojs/hono/commit/c3053ccf14f1c4c70a0c2888eed6121b9f6c8f95) |
| vitejs/vite | hmrProxy aliases the user's hmr config object, so Object.defineProperty installs accessors on the shared/default config; copy it first ({ ...hmrConfig }) | major | ✓Caught | ✓Caught | ✓Caught | [Fix](https://github.com/vitejs/vite/commit/574d5e351905fe99891364d03c2440531378f7fd) |
| honojs/hono | *Clean change, no bug* | — | ✓No comment | ✓No comment | — | [Fix](https://github.com/honojs/hono/commit/21b44b6db6118fc77fabb49832fc4cadd0ef8ec8) |

✓ caught the bug, or left a clean change alone. ✗ missed the bug, or commented on a clean change. — not run.

## How we ran it

-   **Real bugs, put back.** Each case starts from a merged bug fix in a permissively licensed project (MIT, Apache-2.0 or BSD). The pull request under review undoes the fix, so it reintroduces a bug that really shipped.
-   **No hints.** The reviewer sees only the code change and a neutral title such as “Update src/parser.go”. Test files added by the fix stay in place, and the commit history shows nothing about the fix.
-   **Clean changes too.** 7 small merged changes with no bug measure noise: the right result is no comment at all.
-   **Default settings.** No custom rules, no learned memory, no repository index: the reviewer reads the changed files and whatever else its tools return.
-   **Strict matching.** A bug is caught only by a comment on the same file within five lines. One correct comment ten lines away counted as a miss and a false positive.

## What to keep in mind

-   Undoing a fix often removes a guard or a check, which can be easier to spot than a bug written from scratch. Some fixes also added comments that the reverted change deletes.
-   These are well-known projects, and models may have seen their history during training.
-   34 bugs is a small sample: one bug moves a tier's catch rate by about three points. Read the case table, not only the headline.
-   We publish our own results only. Other tools use different cases and rules, so their numbers are not comparable to these.

### Controlled set (in-sample)

We also run 61 small pull requests written for testing (48 with one bug each, 13 clean) on every change to the reviewer, so regressions show up before release. The prompts and posting bar were tuned on these cases, so these figures are regression checks, not an estimate of quality on your pull requests.

Standard

100% caught

10% clean commented

Deep

100% caught

10% clean commented

Max

100% caught

10% clean commented

## See the comments it wrote

Every example is MergeHarbor's own review comment from this run, next to the real fix.

[View examples](https://mergeharbor.com/examples)

## Try it on your own pull request

Paste a public GitHub pull request and see MergeHarbor's review in a few minutes. No sign-up, nothing posted.

[Review a pull request](https://mergeharbor.com/try)

## Review every pull request with the context of your whole codebase

Connect a code host in minutes. Reviews start right away, and codebase context is added as indexing completes.

[Start free trial](https://app.mergeharbor.com/signup) [Book a demo](https://mergeharbor.com/contact?topic=sales&source=demo)