Puan
28,728
Çözümler
0
- Konum
- Freanchey
- Mesajlar
- 3,049
- Katılım
- 2 Eki 2026
- Tepkime puanı
- 3,565
- Puan
- 28,728
- Web sitesi
- glomaxgpt.com
Michelle Zhou & Alejandro Carderera de Diego
Agentic code review is becoming an essential piece of how development happens. It helps you inspect pull requests, catch issues, and decide what deserves attention before code ships.
But the quality of existing AI reviewers can be hard to measure, and you need to know the strengths of a reviewer before you know if it will help you. Some reviewers surface more issues, some produce less noise, and some are stronger at catching critical problems while others surface smaller improvements, too. You may need code review to do different things within your workflow.
That makes it important to understand how reviewers actually compare: what different systems catch, what they miss, and the tradeoffs they make. A good code review benchmark should reflect the diversity of real pull requests, capture a broad set of review findings, and support meaningful breakdowns by severity, category, and precision-recall preferences. For teams building code review agents, the benchmark should also provide an offline signal that reliably tracks whether changes are likely to improve the experience in production. Existing benchmarks often make tradeoffs between label quality, coverage, and how well they represent real-world code review, leaving a gap for a rigorous and reproducible evaluation methodology that brings these pieces together.
We built ReviewBench, a new code review offline benchmark, to address that gap, and it is available for you to use today. It follows the language, repo size, and size distribution of pull requests, modeled after over 100 million real pull requests on GitHub. It uses a multi-source golden set and a consistent evaluation rubric and has been independently validated by senior engineers. Just as important, with the help of ReviewBench, our offline evaluation of Copilot code review (CCR) has become more effective at anticipating the direction of production experiments, giving us greater confidence that measured improvements reflect meaningful gains for users.
In this post, we’ll walk through how ReviewBench is constructed, how it establishes reliable ground truth and scoring, and how to onboard your own code review system and submit results.
1What we builtA realistic, comprehensive benchmark for AI code review agents103.9MGitHub pull requestsAnalyze distributions by language, repository size, and change shape.Representative benchmark corpus219 public pull requests across 19 languages, aligned to GitHub-wide distributions while preserving substantive review cases.Multi-source golden setHuman reviewersFrontier LLMsStatic analysisStructured findingsEvery finding is labeled for severity and category, enabling user-tailored slices.SeverityCriticalMediumLowCategoryCorrectnessSecurityReliabilityMaintainabilityTesting......Evaluation metricsFour metrics measure both known and newly discovered issues.Grounded precisionGrounded recallAugmented precisionAugmented recallObjective evaluationMeasure improvement and compare across agents objectively. Help users choose the reviewer that fits their needs the best.2How we keep it trustworthyAn auditable chain from rubric to expert validation and production checksPublished rubricOne explicit standard for all findings.Human-labeled dev setSenior engineers establish ground truth.Calibrated graderAligned with human judgment.Uniform labelingSame standard across all sources.Published agreementExpert audit of benchmark quality.Auditable end to end96.6% agreementSenior engineers independently labeled golden true-positives before release.Offline signals that anticipate productionBenchmark movement is checked against online experiments.Improvements tend to show up onlineRegressions tend to show up online too
Our benchmark is built around five principles:
1. Representative pull requests, not a demo set
We analyzed 103.9 million GitHub pull requests to characterize the real-world distribution of code review workloads. ReviewBench contains 219 pull requests from 187 public open source licensed repositories spanning 19 languages, with its language and repository-size distributions closely matching GitHub overall. The complete benchmark dataset is publicly available.
We make one deliberate adjustment to this distribution: while language and repository size mirror GitHub directly, pull request size is weighted toward the reviewable middle and tail. This reduces the overrepresentation of tiny, single-file changes while preserving more substantive, multi-file pull requests where review quality matters most.
2. Broad ground truth discovery, independently judged
Agentic code review is becoming an essential piece of how development happens. It helps you inspect pull requests, catch issues, and decide what deserves attention before code ships.
But the quality of existing AI reviewers can be hard to measure, and you need to know the strengths of a reviewer before you know if it will help you. Some reviewers surface more issues, some produce less noise, and some are stronger at catching critical problems while others surface smaller improvements, too. You may need code review to do different things within your workflow.
That makes it important to understand how reviewers actually compare: what different systems catch, what they miss, and the tradeoffs they make. A good code review benchmark should reflect the diversity of real pull requests, capture a broad set of review findings, and support meaningful breakdowns by severity, category, and precision-recall preferences. For teams building code review agents, the benchmark should also provide an offline signal that reliably tracks whether changes are likely to improve the experience in production. Existing benchmarks often make tradeoffs between label quality, coverage, and how well they represent real-world code review, leaving a gap for a rigorous and reproducible evaluation methodology that brings these pieces together.
We built ReviewBench, a new code review offline benchmark, to address that gap, and it is available for you to use today. It follows the language, repo size, and size distribution of pull requests, modeled after over 100 million real pull requests on GitHub. It uses a multi-source golden set and a consistent evaluation rubric and has been independently validated by senior engineers. Just as important, with the help of ReviewBench, our offline evaluation of Copilot code review (CCR) has become more effective at anticipating the direction of production experiments, giving us greater confidence that measured improvements reflect meaningful gains for users.
In this post, we’ll walk through how ReviewBench is constructed, how it establishes reliable ground truth and scoring, and how to onboard your own code review system and submit results.
1What we builtA realistic, comprehensive benchmark for AI code review agents103.9MGitHub pull requestsAnalyze distributions by language, repository size, and change shape.Representative benchmark corpus219 public pull requests across 19 languages, aligned to GitHub-wide distributions while preserving substantive review cases.Multi-source golden setHuman reviewersFrontier LLMsStatic analysisStructured findingsEvery finding is labeled for severity and category, enabling user-tailored slices.SeverityCriticalMediumLowCategoryCorrectnessSecurityReliabilityMaintainabilityTesting......Evaluation metricsFour metrics measure both known and newly discovered issues.Grounded precisionGrounded recallAugmented precisionAugmented recallObjective evaluationMeasure improvement and compare across agents objectively. Help users choose the reviewer that fits their needs the best.2How we keep it trustworthyAn auditable chain from rubric to expert validation and production checksPublished rubricOne explicit standard for all findings.Human-labeled dev setSenior engineers establish ground truth.Calibrated graderAligned with human judgment.Uniform labelingSame standard across all sources.Published agreementExpert audit of benchmark quality.Auditable end to end96.6% agreementSenior engineers independently labeled golden true-positives before release.Offline signals that anticipate productionBenchmark movement is checked against online experiments.Improvements tend to show up onlineRegressions tend to show up online too
Our benchmark is built around five principles:
1. Representative pull requests, not a demo set
We analyzed 103.9 million GitHub pull requests to characterize the real-world distribution of code review workloads. ReviewBench contains 219 pull requests from 187 public open source licensed repositories spanning 19 languages, with its language and repository-size distributions closely matching GitHub overall. The complete benchmark dataset is publicly available.
We make one deliberate adjustment to this distribution: while language and repository size mirror GitHub directly, pull request size is weighted toward the reviewable middle and tail. This reduces the overrepresentation of tiny, single-file changes while preserving more substantive, multi-file pull requests where review quality matters most.
2. Broad ground truth discovery, independently judged