* feat(learning): lessons say where the fix goes; reworded tasks learn together (ADR-511)
The §7 evaluation (#4581) showed three limits on what the verified-outcome
loop can learn:
- a lesson named an agent, nothing the model can act on;
- tasks naming different files merged into one lesson;
- the same task worded differently never learned (15 reworded verified
successes left 15 patterns at one success each, zero lessons).
Changes:
- mods: a verified turn records its screened edited source files; a lesson
names the three its passing fixes edited most and leads with them; pick()
weighs rare keywords (IDF) and drops matches far below the best.
ruflo-mods 0.13.0.
- cli: files are counted per pattern (successes only) into learned-lessons.json;
SONA takes them as pattern anchors (never keywords), so reworded tasks that
fixed the same file reinforce one pattern; tasks naming disjoint paths never
merge. Lesson building moves to verified-lessons.ts.
- eval: the harness gains --ablate, --test-index, --concurrency, --lessons,
--save-lessons, --eval-sample, --calibrate and --max-turns;
scripts/eval-learning-loop-corpus.mjs writes a validated 15+30-task corpus.
Proved offline (tests): 15 reworded prompts form 5 lessons, up from 0; the
first run's prompts form 5 lessons, up from 1; the right lesson ranks first
for 30 of 30 eval prompts, with 1.07 lessons shown per prompt.
Paid pilots (haiku, about $6.80) are directional and did not reach the gate:
- 8-turn cap: off 2/10, on 4/10, lessons without files 1/10;
- uncapped: 6/6 vs 6/6, cost within 1%.
ADR-511 §5 has the details.
Co-Authored-By: RuFlo <ruv@ruv.net>
* docs(adr): ADR-511 cites PR #4603
Co-Authored-By: RuFlo <ruv@ruv.net>
---------
Co-authored-by: r <r@x>
scripts/eval-learning-loop.mjs now uses ruflo's MetaHarness flywheel
sequential-evidence gate (anytime-valid McNemar e-process) as its decision
rule for #4581 §7:
- the plan prints the minimum number of tasks the loop must pass where the
baseline fails (9 for a first test at family-wise alpha 0.05), and --run
refuses an eval split that can never reach it, before any spend;
- the report carries task-level pairedOutcomes and the gate verdict, with a
cost verdict as a secondary sign check only;
- each run's claude -p JSON result is saved next to its work directory, and
--keep leaves the directories in place for later inspection;
- --help prints the whole header comment instead of a fixed line range.
Applied to the 2026-10-30 run: success e-value 1.0 (0/30 informative pairs,
threshold 32.9), cost e-value 9.7, both not significant.
Co-authored-by: r <r@x>
* feat(learning-loop): gaming defences for verified outcomes (ADR-503)
The verified-outcome loop (ADR-502) trusts a verifier's exit status, but the
model chooses what that verifier checks. Four downgrade-only layers:
- mod: a verified-success record carries taints when the turn edited a test
file (edited-tests), a verifier config (edited-verifier-config), or decided
on a weakened/narrowed command (narrowed-verifier); a per-process session
token is written on every record. The outcome word never changes.
- CLI: a success with a taint not listed in .claude-flow/config.json
learningLoop.allowTaint is counted under `tainted` and never reaches SONA.
- CLI: opt-in re-verify (learningLoop.reverify, off by default) re-runs a
rotating allowlisted manifest check in a temp detached git worktree at HEAD
with a timeout; failure, timeout or error downgrades to unknown. Only an
explicit verified-ingest re-verifies; hot callers defer.
- CLI: anomaly gate holds would-be lessons that are perfect-and-tainted,
only-narrowed, or from one session, listed as `held: <id> (<reason>)`.
Adds scripts/eval-learning-loop.mjs (manual, plan-only unless --run).
Co-Authored-By: RuFlo <ruv@ruv.net>
* docs(adr): ADR-503 status names PR #4590
Co-Authored-By: RuFlo <ruv@ruv.net>
* fix(learning-loop): shell edits of tests taint; test the real re-verify runner (ADR-503)
- A Bash command that writes, moves or deletes a test file or verifier
config (rm, mv, cp, tee, truncate, sed -i/perl -i, output redirection,
git rm/checkout/restore/...) now sets edited-tests /
edited-verifier-config like an Edit of that file. Before this change,
Edit src/a.ts + `sed -i 's/it(/it.skip(/' src/a.test.ts` + `npm test`
produced an untainted verified-success. Downgrade-only: reads and test
runs earn nothing.
- gitWorktreeRunner is now tested against a throwaway git repo: pass,
fail, missing script, timeout (hung script's process verified dead),
non-repo error. After each run only the main worktree remains and no
ruflo-reverify-* dir is left. A negative control (kill only the direct
child) makes the timeout test fail.
- ADR-503 implementation notes record both, plus the interpreter-write gap.
Co-Authored-By: RuFlo <ruv@ruv.net>
* test(learning-loop): give the hung-check re-verify test an 8s budget for cold CI runners
Co-Authored-By: RuFlo <ruv@ruv.net>
* chore(ruflo-mods): 0.9.0 for ADR-503
Co-Authored-By: RuFlo <ruv@ruv.net>
---------
Co-authored-by: r <r@x>
