FrontierCode Leaderboard

Read the methodologyExplore a task
Current revision. Runs flagged for unfair internet use are zeroed.
Loading leaderboard…

Methodology

FrontierCode is the first benchmark to measure mergeability: would the maintainer actually merge this PR?

Our criteria assess end-to-end code quality (correctness, test quality, scope discipline, style, and adherence to codebase standards) using an ensemble of grading techniques including unit tests, rubrics, and new types of verifiers.

Every task is crafted by the open-source maintainers of the repos it comes from: 20+ world-class developers built realistic, diverse, and challenging tasks, spending more than 40 hours per task, and they define what “mergeable” means in their repo. Rubric grading is subjective, so we built an extensive QC pipeline with adversarial testing, calibration, and multi-stage review, where every task is manually reviewed by a Cognition researcher.

FrontierCode also restricts internet usage. Models may use the internet the way an engineer would, reading documentation and searching for error messages, but runs that consult solution-bearing sources such as the original pull request are detected and scored zero.

Methodology revisions

Read how the benchmark is constructed, graded, and refined across revisions.

FrontierCode 1.1Current07.07.26Refines the methodology to distinguish legitimate internet use from unfair use: runs flagged for consulting solution-bearing sources are zeroed. Also audits blocker criteria and deprecates the Diamond subset.Read the methodology Introducing FrontierCodeOriginal06.08.26Introduces the benchmark: mergeability as the grading standard, tasks crafted by open-source maintainers, and the ensemble of unit tests, rubrics, and verifiers behind the score.Read the methodology

What a task looks like

Each FrontierCode task pairs a maintainer-written brief with reviewer-defined grading criteria. Explore a real task below: switch between models, run the eval, and click a failed criterion to jump to the offending part of the diff.

Task description

Encapsulate all warning logs in a new auto LOG_WARNING() -> std::ostream & method in src/logger.h such that:

  • Warnings are always printed to standard error
  • Warnings are always printed, independently of --verbose
  • The helper automatically prints the warning: prefix

Use this new function in every instance of warning: <message> messages throughout the codebase.

Test guidelines

Run make and ensure no code changes remain. If there are more code changes, then it means that the code was not formatted properly.

Unless you are sure that the code change is already covered by an existing test case, always edit or create relevant tests (in the ./test directory) to confirm the changes work and prevent regressions.

The tests are written using GoogleTest and POSIX shell scripts (not bash) and must be registered in the test/CMakeLists.txt build definition to run.

Lint guidelines

Run make configure compile to compile and format the code in-place. The compile step comes with a large amount of linter-like checks.

Style guidelines

You are already on the correct base commit. Create your branch from this commit. Do not rebase or start from master, main, or any other branch.

Press "Run eval" to generate Opus 4.8's patch for this task.

The graded rubric appears here after the run.

Interactive: run the FrontierCode grading pipeline against each model’s output and inspect how the patch maps to rubric pass/fail.