Empyrean
Tools // Release 2 // MIT License

rsi-audit v0.1

Python package. No dependencies.

Checks a self-improvement or agent-leaderboard claim from its public artefacts.

Quick Install
$ pip install rsi-audit
Audit Capabilities

The five checks

rsi-audit evaluates verification rigor across five distinct axes:

1. selection

Was the reported system picked on the graded tasks? 

$ rsi-audit selection devlog_example.csv --graded graded_test --submitted harness_v2
Graded set: graded_test -- 4 development evaluation(s), 4 distinct candidate(s) scored on it.
  harness_v2                     0.7100   <- submitted
  harness_v2_finetuned           0.6900
  harness_v1                     0.6200
  baseline                       0.3100
VERDICT: SELECTION ON TEST

2. luck

How many record claims could pass by chance? 

$ rsi-audit luck --total 400 --top-max 50 --rate 0.001
Expected luck-only passes: 0.400 across the study; 0.050 in the most-claimed headline run.
VERDICT: INCONCLUSIVE

3. ranking

Does the order survive resampling tasks? 

$ rsi-audit ranking devlog_example.csv
Resampling tasks to test ranking stability:
Order does not survive bootstrap task resampling.
VERDICT: UNSTABLE RANKING

4. visibility

Can the agent read the grader's tests or answers? 

$ rsi-audit visibility artifact_manifest.json
Checking isolation between agent runtime and evaluation grader:
Agent execution environment has read visibility into grader test files.
VERDICT: VISIBILITY LEAK DETECTED

5. card

12 requirements a self-improvement number needs. 

$ rsi-audit card claims_card.json
Auditing against the 12 card requirements:
Status: 2 requirements failed, 1 requirement unknown.
VERDICT: CARD INCOMPLETE
Standardization

The 12 card requirements

A self-improvement number requires all twelve conditions to establish verified progress:

YES matched control
YES matched compute
YES repeated training runs
YES intervals
YES held-out final evaluation
YES attempts reported
NO correctness gate covers what is scored
NO cheats scored as failures
UNKNOWN hidden grader
YES resampled ranking
YES pre-committed task set
YES disclosed harness
Boundary Conditions

Limits

Audit Protocol Limits

rsi-audit evaluates published artefacts, devlogs, and evaluation harnesses. It identifies unverified reporting paths, selection on test, and missing gates; it audits artefacts, not unreleased weights.

Audits

Audit write-ups

Each audit goes privately to the authors first; the write-up is published here once they have had the chance to reply.

← Back to overview