Measured capability
Software Engineering
Repository repair on exact public issues, graded against isolated hidden tests.
3 measured exams4 recorded lanesLatest measurement 2026-09-02Published campaign scopeNo category composite
Certified exam
xmltodict issue 257
4 measured model lanes
Model lane · condition 1 of 2
cli/claude-sonnet
9/1090% certified
Lanecli/claude-sonnetDeliveryDelivery not establishedConditionsUnproven
Model failures1Lane errors0Withheld / invalid0 / 0
Elapsed median155.0 seconds · 10/10 observedSteps median5 · 10/10 observedTokens medianNot recordedCost median (USD)Not recorded
delivery equivalence is unproven; delivery is not proved; lane configuration is not recorded; recorded engine or harness conditions differ
Inspect measurement evidence →
Model lane · condition 2 of 2
cli/claude-sonnet
1/1Certified passes · rate not published
Lanecli/claude-sonnetDeliveryDelivery not establishedConditionsUnproven
Model failures0Lane errors0Withheld / invalid0 / 0
Elapsed median110.2 seconds · 1/1 observedSteps median5 · 1/1 observedTokens medianNot recordedCost median (USD)Not recorded
delivery equivalence is unproven; delivery is not proved; lane configuration is not recorded; recorded engine or harness conditions differ
Inspect measurement evidence →
Model lane · condition 1 of 2
cli/codex
10/10100% certified
Lanecli/codexDeliveryDelivery not establishedConditionsUnproven
Model failures0Lane errors0Withheld / invalid0 / 0
Elapsed median40.7 seconds · 10/10 observedSteps median6 · 10/10 observedTokens medianNot recordedCost median (USD)Not recorded
delivery equivalence is unproven; delivery is not proved; lane configuration is not recorded; recorded engine or harness conditions differ
Inspect measurement evidence →
Model lane · condition 2 of 2
cli/codex
1/1Certified passes · rate not published
Lanecli/codexDeliveryDelivery not establishedConditionsUnproven
Model failures0Lane errors0Withheld / invalid0 / 0
Elapsed median40.1 seconds · 1/1 observedSteps median6 · 1/1 observedTokens medianNot recordedCost median (USD)Not recorded
delivery equivalence is unproven; delivery is not proved; lane configuration is not recorded; recorded engine or harness conditions differ
Inspect measurement evidence →
Model lane · condition 1 of 2
codestral-latest
0/100% certified
Lanecodestral-latestDeliveryDelivery not establishedConditionsUnproven
Model failures10Lane errors0Withheld / invalid0 / 0
Elapsed median10.9 seconds · 10/10 observedSteps median5 · 10/10 observedTokens median16868.5 · 10/10 observedCost median (USD)Not recorded
delivery equivalence is unproven; delivery is not proved; lane configuration is not recorded; recorded engine or harness conditions differ
Inspect measurement evidence →
Model lane · condition 2 of 2
codestral-latest
0/1Certified passes · rate not published
Lanecodestral-latestDeliveryDelivery not establishedConditionsUnproven
Model failures1Lane errors0Withheld / invalid0 / 0
Elapsed median12.5 seconds · 1/1 observedSteps median5 · 1/1 observedTokens median19214 · 1/1 observedCost median (USD)Not recorded
delivery equivalence is unproven; delivery is not proved; lane configuration is not recorded; recorded engine or harness conditions differ
Inspect measurement evidence →
Model lane
openai/gpt-oss-120b
Not evaluatedNo graded attempts
Laneopenai/gpt-oss-120bDeliveryDelivery not establishedConditionsUnproven
Model failures0Lane errors1Withheld / invalid0 / 0
Elapsed medianNot recordedSteps medianNot recordedTokens medianNot recordedCost median (USD)Not recorded
delivery equivalence is unproven; delivery is not proved; lane configuration is not recorded; recorded engine or harness conditions differ
Inspect measurement evidence →Certified exam
toolz issue 602
4 measured model lanes
Model lane
cli/claude-sonnet
1/1Certified passes · rate not published
Lanecli/claude-sonnetDeliveryDelivery not establishedConditionsUnproven
Model failures0Lane errors0Withheld / invalid0 / 0
Elapsed median93.2 seconds · 1/1 observedSteps median9 · 1/1 observedTokens medianNot recordedCost median (USD)Not recorded
delivery equivalence is unproven; delivery is not proved; lane configuration is not recorded
Inspect measurement evidence →
0/1Certified passes · rate not published
Lanecli/codexDeliveryDelivery not establishedConditionsUnproven
Model failures1Lane errors0Withheld / invalid0 / 0
Elapsed median54.8 seconds · 1/1 observedSteps median7 · 1/1 observedTokens medianNot recordedCost median (USD)Not recorded
delivery equivalence is unproven; delivery is not proved; lane configuration is not recorded
Inspect measurement evidence →
Model lane
codestral-latest
0/1Certified passes · rate not published
Lanecodestral-latestDeliveryDelivery not establishedConditionsUnproven
Model failures1Lane errors0Withheld / invalid0 / 0
Elapsed median16.7 seconds · 1/1 observedSteps median6 · 1/1 observedTokens median40054 · 1/1 observedCost median (USD)Not recorded
delivery equivalence is unproven; delivery is not proved; lane configuration is not recorded
Inspect measurement evidence →
Model lane
openai/gpt-oss-120b
Not evaluatedNo graded attempts
Laneopenai/gpt-oss-120bDeliveryDelivery not establishedConditionsUnproven
Model failures0Lane errors1Withheld / invalid0 / 0
Elapsed medianNot recordedSteps medianNot recordedTokens medianNot recordedCost median (USD)Not recorded
delivery equivalence is unproven; delivery is not proved; lane configuration is not recorded
Inspect measurement evidence →Certified exam
parse issue 102
4 measured model lanes
Model lane
cli/claude-sonnet
1/1Certified passes · rate not published
Lanecli/claude-sonnetDeliveryDelivery not establishedConditionsUnproven
Model failures0Lane errors0Withheld / invalid0 / 0
Elapsed median255.5 seconds · 1/1 observedSteps median11 · 1/1 observedTokens medianNot recordedCost median (USD)Not recorded
delivery equivalence is unproven; delivery is not proved; lane configuration is not recorded
Inspect measurement evidence →
0/1Certified passes · rate not published
Lanecli/codexDeliveryDelivery not establishedConditionsUnproven
Model failures1Lane errors0Withheld / invalid0 / 0
Elapsed median72.9 seconds · 1/1 observedSteps median11 · 1/1 observedTokens medianNot recordedCost median (USD)Not recorded
delivery equivalence is unproven; delivery is not proved; lane configuration is not recorded
Inspect measurement evidence →
Model lane
codestral-latest
0/1Certified passes · rate not published
Lanecodestral-latestDeliveryDelivery not establishedConditionsUnproven
Model failures1Lane errors0Withheld / invalid0 / 0
Elapsed median22.3 seconds · 1/1 observedSteps median9 · 1/1 observedTokens median73260 · 1/1 observedCost median (USD)Not recorded
delivery equivalence is unproven; delivery is not proved; lane configuration is not recorded
Inspect measurement evidence →
Model lane
openai/gpt-oss-120b
Not evaluatedNo graded attempts
Laneopenai/gpt-oss-120bDeliveryDelivery not establishedConditionsUnproven
Model failures0Lane errors1Withheld / invalid0 / 0
Elapsed medianNot recordedSteps medianNot recordedTokens medianNot recordedCost median (USD)Not recorded
delivery equivalence is unproven; delivery is not proved; lane configuration is not recorded
Inspect measurement evidence →