Recursive self-improvement
Study whether changes to a system's own methods produce measurable gains under matched compute and counterfactual controls.
What recursive self-improvement is, and how to test it →Independent AI research
Intelligence Research Project
Exploring how intelligence evolves.
Building the evidence to understand it.
Recursive self-improvement, how to measure it, and program synthesis — investigated in the open.
THE SEARCH LANDSCAPEFIG. 01
GENERATE · EVALUATE · REFINE
01 / The approach
Can AI improve its own methods? The answer begins with evidence. This project builds systems that test their own results — and publishes the failures alongside the progress.
Independent & unaffiliated · open-source experiments · design directed, agent‑built
02 / Research directions
Three paths.
One open question.
Three connected research directions, with code and evaluation evidence available for inspection.
Measuring self-improvement
Score a self-improving system on several axes at once, so one number cannot hide where a gain came from.
Explore the benchmark →Validation-gated program synthesis
Generate candidate programs and make acceptance depend on explicit verification gates.
Explore program synthesis →03 / Selected work
Ideas, put to the test.
“Validated” means the result survives a stated counterfactual or held-out test I can point to — not a badge I award myself. Each entry links to that evidence.
27issues in 23 open-source projects
5reproduced or implemented by others
7recurring failure patterns
Projects include NousResearch hermes-agent-self-evolution, the Darwin Gödel Machine, Microsoft SkillOpt and STOP, GEPA, Google Research reasoning-bank and Meta HyperAgents · most issues are still open · an issue is a report, not a confirmed defect
Read the findings, the responses and the limits →
When a self-improving system also writes or selects the test that admits its changes, a gain can pass without being real. This record looks for exactly that in public code: an optimizer that never touched its target, a selection mode that kept the worst candidates, a mean-score gate that accepted evaluator gaming while a task regressed. Two other users independently reproduced the first; a project collaborator implemented negative-control probes from another.
+1.52tasks · memory carried vs wiped · p = 0.0011
+1.78tasks · vs a frozen improver · p = 0.0006
+0.05growth over later problems · not significant
Pre‑registered · run once on 300 untouched seeds after the design was frozen · unseen task families · about +2.4% of the final metric · the advantage does not keep growing
Read the results, the controls, and the limits →
LLM-free, deterministic, counterfactually controlled recursive self-improvement engine. Its latest experiment asks whether experience at improving carries over. An improver that keeps its improvement memory across ten independent problems beats the identical learner whose memory is wiped between problems, on task families neither has seen. An earlier pre-registered test found that five recursive rounds beat one round with five times the compute (+0.40 of 24 tasks). The project’s first headline was retracted by its own audit and stays in the failure log below.
A multi-axis benchmark for recursive self-improvement — scoring self-modification depth, improvement trajectories, operator discovery, meta-adaptation, safety, and autonomy in one suite.
Validation-gated program synthesis runtime: meta-meta learning loops, analogical transfer, grammar-mediated expansion, and anti-cheat verification built into the search loop itself — not bolted on after. 286 tests: a quick CI suite runs on every push, and the full evidence battery whenever the runtime changes.
04 / The complete record
Progress includes
what didn’t work.
Three headline results didn't survive their own verification gates, and several pre-registered hypotheses came back null. Publishing them here is the point — a validation system that never rejects its own results isn't one.
rsi-metaforge-core · phase ae
A 15/15 adoption result. Traced back to the task generator and the search engine having been co-designed — the result was structurally guaranteed, not earned. Caught by the same claim-boundary discipline linked above before it ever shipped, so there's no separate postmortem file — only the standing rule that produced the catch.
gated-self-improvement
A compounding-improvement claim resting on a 5× compute mismatch between control and treatment runs (the original chain measured −2.08 tasks of self‑harm, p<1e‑4). Found the mismatch, repaired the mechanism, and retracted the original claim. The repaired result that replaced it was retracted too; see the next entry.
gated-self-improvement · repaired chain
The repaired chain’s +1.55 tasks per seed over a single round at 5× compute, once the featured result on this page. An audit found that held-out evaluation seeded each arm’s search with the arm’s own name, and that the single-round control saw 5 distinct source tasks where the chain saw 25. Rerun with shared random streams, the repaired chain fell below the untrained baseline (−0.70, p = 0.0005), and a single round over the same 25 tasks beat it (−0.59, p = 0.002).
gated-self-improvement · pre-registered nulls
Three pre-registered hypotheses failed and stay on record. Adapting the improvement policy within one run did not beat a frozen policy (v2, −0.08). A meta-predictor that only ranked candidates carried nothing detectable across problems, so its go/no-go said NO_GO and the confirmatory seeds were left untouched (v3, first design). And the carried-memory advantage does not keep growing over later problems (v3, +0.05).
23
External projects audited
27
Audit issues filed
2
Pre-registered experiments
3
Retractions published
05 / Support independent research
Make the next
experiment possible.
Support goes to research funding: the compute and time the next experiments need. There is no fixed target, and any amount moves the next run forward.
Today each run is bounded by the compute it can afford. In the latest experiment the improvement process gets about 50 decisions per run, where measuring its options would take roughly 400. Funding removes that ceiling.
Read the research plan
First run it funds: the question the latest result leaves open. Carried improvement memory helps, but its advantage stops growing after about one problem. Longer sequences of problems, and runs long enough for the process options to be measured, test whether that advantage can keep growing or has reached a ceiling. Then stronger fixed controls: the report notes that untested fixed policies might beat the one it used.
This work isn’t one long run — it’s hundreds of short, seeded ones. What limits it is wall‑clock time and the number of seeds and decisions a run can afford, not peak throughput. I’ll publish the scale limits alongside the numbers rather than around them.
Like the last two, the run will be pre-registered and built to be able to kill the claim. If it does, the retraction or the null goes up here with the others — with the code, the seeds, and the writeup.
Email is the fastest route. Any amount is welcome; how to send it is settled in the reply.
Fund the research
Contribute to the compute and time the next experiments need. There is no minimum and no fixed target. Any amount helps.
Email to contributeSponsor one run
Sponsor the growth test: longer sequences of problems and longer runs, to see whether carried memory keeps improving the improver or levels off. You get the full results, including a null if it levels off.
Email about this runReproduce, critique, cite
Researchers can help without money. Rerun a result, send a critique, or cite a note. Every note has a pinned revision and a BibTeX file.
Get in touch- Every result is published with code and a pinned revision.
- Failures and retractions go up next to the wins.