Research method
A system can only improve as fast as it can tell a real improvement from a fooled check. Our verification standards enforce that every reported finding reflects genuine measurement.
Pre-registration before running
Evaluation protocols, test sets, perturbation operators, and target metrics are committed prior to running experiments. Committing the evaluation harness before viewing results prevents post-hoc selection, metric shopping, and stopping rules based on favorable noise.
Two independent routes for every reported number
No headline figure is published from a single script or execution run. Every result must be arrived at via two independent implementation routes—independent code paths, separate data loaders, or alternative verification harnesses—and verified to agree before publication.
Measurement vs inference labelled
We strictly separate what was directly observed from what is inferred:
- Measurement: Direct, reproducible empirical output (e.g. 169 of 942 reference solution perturbations passed visible tests).
- Inference: Interpretations, extrapolations, or theoretical limits. All inferences are explicitly labelled as such and never conflated with raw experimental data.
Authors notified first
Claims about others' work go to the authors privately first. Before releasing an audit of any public artefact or benchmark, findings are sent directly to the original creators to allow review, clarification, or resolution of misunderstandings.
Negative results are published too
Verification tools that find no gameability or interventions that fail to produce a measurable difference are published on the same cadence as positive findings. Suppressing negative results distorts the verification baseline.
Intervals and limits with every number
Point estimates without intervals are unscientific. Every measurement is accompanied by its empirical interval (such as 95% intervals) and an explicit statement of its limits (what the tool measures, what it cannot measure, and where the floor lies).