gameability v0.2
Python package. Runs on a laptop, no model or API key.
Finds problems in a code benchmark that accept broken solutions. It breaks each problem's reference solution in one small place, confirms the result is really wrong, then checks whether the problem's own tests notice.
How it works
The tool evaluates benchmark test sensitivity through five sequential steps:
- Load problem & reference solution: Retrieve the task prompt, canonical reference implementation, and official test suite.
- Apply single small edit: Break the reference solution in one small place (e.g. operator mutation, off-by-one boundary, branch perturbation).
- Confirm wrongness: Confirm that the perturbed result is genuinely incorrect against semantic specification before proceeding.
- Execute problem's own tests: Run the problem's official test harness against the broken code.
- Check for detection: Record whether the test suite notices. If the broken solution passes every visible test, flag the problem as gameable.
Full Scoreboard
Problems that accept at least one broken solution, reported with 95% confidence intervals:
| Benchmark | Flagged Rate | Fraction | 95% Interval | 95% Interval Visual |
|---|---|---|---|---|
| MBPP | 17.9% | 169 of 942 | [15.6, 20.5] |
|
| MBPP sanitized | 13.8% | 57 of 413 | [10.8, 17.5] |
|
| HumanEval | 10.3% | 16 of 155 | [6.5, 16.1] |
|
| KodCode RL training set (1,000-row random sample) | 9.7% | 45 of 466 | [7.3, 12.7] |
|
| HumanEval+ | 3.4% | 5 of 147 | [1.5, 7.7] |
|
| MBPP+ | 2.0% | 7 of 353 | [1.0, 4.0] |
|
Confirmation: re-run against EvalPlus's independent tests, 95.2% of flagged MBPP problems and 84.6% of flagged HumanEval problems are confirmed.
Limits
- The attacker is a single small edit (a floor, not a ceiling).
- It audits tests, not models.
- It does not predict where a specific model will cheat.
Example output
Real tool output rendered as diffs:
415 -- 6 of 11 broken solutions pass every visible test
- if arr_len < 2:
+ if arr_len <= 2:
a test that catches it: assert max_Product([4, 9]) == (4, 9)Reproduce commands
Commands to reproduce the scoreboard numbers locally: