Definition
p is 1-run success (the source Pass@1 point estimate), m is mean duration in minutes per attempt, and c is mean cost in dollars per attempt. The two nonnegative priorities are normalized to sum to one. The defaults are 60% amortized agent time per pass and 40% amortized cost per pass.
The anchors describe an index value of 100: 40 minutes amortized agent time per pass and $10 amortized cost per pass. Anchors affect the displayed scale, not ordering.
The leaderboard considers only configurations meeting the selected point-estimate 1-run success (Pass@1) floor, which defaults to 60%. This is an eligibility preference, not a statistical confidence threshold.
Relative value
The interface presents Formula v1 as relative value: each score divided by the highest finite score meeting the current success-rate floor. The leader is 1.00×. Model visibility and the table-only Pareto filter do not change this reference; changing priorities or the success-rate floor can change it. Relative value preserves Formula v1 ordering and ties. It is a presentation normalization, not an efficiency probability or a new formula version. A later dataset with a better leader can lower existing configurations’ relative values without changing their measurements, so relative values should only be compared within the same dataset and settings.
What the priorities mean
Amortized agent time per pass is m / p; amortized cost per pass is c / p. These divide aggregate per-attempt averages by the aggregate 1-run success rate. Expanding the formula gives:
Reliability therefore has an effective elasticity of one: a lower pass rate reduces both aggregate time and cost efficiency. Formula v1 gives amortized agent time moderately more priority than cost for developer-directed, semi-asynchronous work. Agent time represents feedback-loop latency and throughput, not continuous developer waiting time. This is an editorial starting point, not an empirically calibrated optimum; the control allows either outcome to receive more priority.
4-run success (Pass@4)
The leaderboard calls the source Pass@4 field “4-run success”: the observed share of attempted tasks solved in at least one of four published runs. This helps distinguish a configuration that repeatedly misses the same tasks from one that succeeds on more tasks across multiple runs.
4-run success is a supplemental persistence diagnostic. It does not affect Formula v1 or Pareto status, because the aggregate feed does not describe a sequential policy that stops after success or the time and cost consumed before each task first passes. Rows published with a run count other than four remain visible but are not compared in the 4-run success sort.
Interpretation limits
Formula v1 is a descriptive point-estimate index, not a calibrated utility function. It does not express uncertainty, the value of developer time, or the severity of a failed result.
Amortized agent time and cost describe aggregate benchmark throughput. They do not estimate the time, cost, or wall-clock latency of retrying a task until success. Correlated or deterministic failures are visible only indirectly through the repeated-run diagnostic. Upstream exclusions can also produce slightly different task coverage between configurations; the leaderboard keeps those configurations rankable and displays their coverage.
Exceptional values
- Pass@1 of zero receives a score of zero and infinite amortized time and cost per pass. It remains in an unfiltered table but is omitted from the finite plot.
- Nonpositive or nonfinite mean duration or cost is unpriced and excluded from scoring and Pareto calculations.
- Equal scores remain tied. Configuration ID is only the deterministic ordering fallback.
Pareto frontier
A configuration is point-estimate Pareto-efficient when no other rankable configuration has at most as much amortized agent time per pass and at most as much amortized cost per pass—with at least one strict improvement. This is not statistical dominance.
The chart displays every finite selected configuration above the Pass@1 floor. The shared model filter limits both chart and table, and the Pareto frontier is recalculated over that selected, floor-eligible set. Pareto status therefore means efficient among selected models. The Pareto control filters only the table and is off by default.
Version and data
Formula v1 is immutable. A future methodology must use a new formula module
and version. The stable ?formula=v1 selector chooses this
calculation; it does not pin or replay a historical dataset.
The site applies Formula v1 to the latest fully validated daily copy of the published benchmark data from the pinned v1.1 aggregate feed. This independent project is not affiliated with or endorsed by the benchmark publisher. A failed refresh cannot replace the last valid copy.