Superintelligence may be the most important event since the universe began, and the last invention humanity ever needs to make.
It should be built by the best minds alive, and built to benefit all of humanity. Empyrean works on the five problems that stand between today's models and that future: experiential learning, continual learning, sample efficiency, recursive self-improvement, and verification.
Every result we report is pre-registered, run against matched controls, and published with its interval and its limits, including the results that come out negative.
What brings what
None of the five works alone. Each arrow is a dependency: the problem at its tail supplies something the problem at its head cannot get any other way.
- Verification brings a signal that cannot be gamed. It feeds experiential learning and gates every self-update.
- Sample efficiency brings more learned per checked attempt. It makes experiential learning affordable.
- Experiential learning brings new data past where human data ends. It feeds continual learning.
- Continual learning brings gains that add up instead of overwriting each other. It feeds recursive self-improvement.
- Recursive self-improvement brings a learner that improves the loop itself. It is the road to superintelligence.
Five problems, one engine
Verification is the engine of the other four: a model can only teach itself, keep what it learned, or improve its own learning if the signal it learns from cannot be fooled.
Machines that check what humans cannot. We measure how often today's checks are fooled, in benchmarks, RL environments and self-improvement claims, with an independent oracle, and we build the checks that are not.
A small model trains only on its own attempts, each checked exactly, on program-induction tasks whose difficulty has no ceiling. It chooses its own level, and it is compared against the same recipe with coin-flip reward, a fixed curriculum, and a frozen copy. The claim we test is the slope, levels climbed per unit of experience, not a gap to any one model.
Weights that keep changing with every verified experience, without forgetting what came before. The verifier decides which experiences are safe to keep. Three studies so far, none positive: a forgetting effect we predicted came out in the opposite direction; a per-item learning cap was uninformative under its own pre-set rule; and an agent reflecting on feedback with frozen weights did not clear the bar we set for learning.
How much does one checked attempt teach? In our reinforcement-learning run, 60 training steps moved depth-4 accuracy from 5 in 100 to 15 in 100. Every run we register now reports its gain per verified attempt, so efficiency becomes a measured curve rather than a claim.
A model that improves the process that improves it. In small controlled loops, the learner improved when its problems and its checking came from outside it; at 1.5B, fine-tuning on its own verified successes produced no measurable gain. The closed reinforcement-learning loop is registered next, against a matched control.
What is measured
Correct signal teaches; random signal does not. In our own reinforcement-learning run on a 1.5B model, reward from a code-running checker raised depth-4 composition accuracy by +0.10 (95% interval [+0.04, +0.17]) over a frozen copy, while an otherwise identical run with coin-flip reward did not improve. It did not transfer to an outside math benchmark, and we report that too. Read the full write-up →
Benchmarks accept broken solutions. Our open tool gameability finds that
17.9% of MBPP problems (169 of 942) and 10.3% of HumanEval problems
(16 of 155) accept a deliberately broken solution; 95.2% and 84.6% of
those flags are confirmed by EvalPlus's independent tests.
The harness moves the score. The same checkpoint, on the same 250 problems, scored 0.608 under one evaluation scaffold and 0.460 under another, both choices a competent team could defend: a 0.148 gap, against a 0.088 noise floor, with identical weights. A benchmark number reported without its harness cannot be read to better than that.
Vendors' verifiers, audited. We read the public reward and scoring code of the 42 companies that sell RL environments to frontier labs, against a checklist fixed before reading, and confirmed each flag by running the vendor's own code. Results are published here after every affected vendor has been told privately and had the chance to reply.
Tools anyone can run
gameability finds the problems in a code benchmark that accept broken solutions. It breaks each
problem's reference solution in one small place, confirms the result is really wrong, then checks whether the
problem's own tests notice. Runs on a laptop, no model or API key.
Scoreboard & documentation →
rsi-audit checks a self-improvement or agent-leaderboard claim from its public artefacts:
whether the ranking survives resampling, whether the agent could see the grader, whether the reported system
was picked on the graded tasks, how many record claims could pass by chance, and the 12 requirements a
self-improvement number needs. The five checks & documentation →
How we work
Every experiment's decision rule is written before the code that runs it.
Every claim of improvement is made against a matched control.
Every number ships with its interval and its limits.
Claims about others' work go to the authors privately first.
Negative results are published too.
Contact
Source: gameability · rsi-audit
Packages: PyPI: gameability · PyPI: rsi-audit
Reach us: github.com/Empyrean-Intelligence · X · LinkedIn