From c2069ed6f606271a91bd66e1b82c3ae5597a81c2 Mon Sep 17 00:00:00 2001 From: longxinjun-ai Date: Tue, 20 Oct 2026 18:50:00 +0800 Subject: [PATCH] fix(metaharness): revive the cost dimension and single-source the verdict bands (ADR-152) (#3520) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reconciling the shipped similarity code against docs/metaharness-user-guide.md turned up three divergences between what ADR-152 specifies, what the code does, and what the docs promise. 1. Cost normalization (vector index 8) was inert below $0.999. ADR-152 Table 1 specified `log10(usd + 0.001) / log10(10)` clamped to [0,1]. `Math.log10(10)` is 1, so the division was a no-op and the expression reduced to `clamp(log10(usd + 0.001), 0, 1)`. Because log10 is negative for every usd < 0.999, the clamp flattened that entire range to a constant 0 — including the $0.048 baseline the plugin README records. Two harnesses differing only in cost scored `cosine === 1`; the documented 9-dim vector was effectively 8-dim. Replaced with an explicit log band: $0.001 -> 0, $10 -> 1, log-interpolated between, monotonic and non-finite-safe. Exported as `normalizeCost()` with the band in `COST_BAND_USD`. 2. The verdict bands had two encodings, and they disagreed. `audit-trend.mjs` inlined the bands with a 0.80 `minor-drift` floor while the user guide documented 0.85; `drift-from-history` prints that verdict, so the two diverged in the 0.80-0.85 band. The bands now live once as `VERDICT_THRESHOLDS` + `verdictFor()` in `_similarity.mjs`, `audit-trend.mjs` imports them, and `similarity()` returns `verdict` beside `overall`. The documented 0.85 floor is canonical. Deliberate user-visible change: structural similarity in [0.80, 0.85) now reports `moderate-drift` instead of `minor-drift`. `--threshold` comparison is unaffected (it tests `overall`), but consumers branching on the verdict string should be re-checked. 3. The user guide's weight table said 0.4/0.3/0.3; ADR-152 and the code both say 0.6/0.25/0.15. Corrected the guide, which was the outlier. The ADR gains a dated Revision section recording both fixes and the behaviour change. The §Spike-result figures are unchanged (the LEGAL/SUPPORT fixtures are near-identical on the cost axis), so test-similarity.mjs Phase 8 keeps pinning the original anchors; new Phase 7b/7c/7d coverage pins the corrected semantics. Note this does NOT address the coarse-cosine limitation ADR-152 itself flags ("numerical cosine alone is too coarse"): uniform quality degradation still reads as high similarity. Drift gates that must catch absolute loss should pair structural similarity with the absolute gates rather than rely on --threshold. Verification: node plugins/ruflo-metaharness/scripts/test-similarity.mjs -> 128 passed, 0 failed (was 102) node plugins/ruflo-metaharness/scripts/_spike-similarity.mjs -> both invariants hold Co-authored-by: longxinjun-ai Co-authored-by: rUv --- docs/metaharness-user-guide.md | 12 ++-- .../ruflo-metaharness/scripts/_similarity.mjs | 64 ++++++++++++++++++- .../ruflo-metaharness/scripts/audit-trend.mjs | 10 +-- .../ruflo-metaharness/scripts/similarity.mjs | 1 + .../scripts/test-similarity.mjs | 63 ++++++++++++++++++ .../adr/ADR-152-genome-similarity-search.md | 38 ++++++++++- 6 files changed, 175 insertions(+), 13 deletions(-) diff --git a/docs/metaharness-user-guide.md b/docs/metaharness-user-guide.md index 4c99d948d1..f6be6a05de 100644 --- a/docs/metaharness-user-guide.md +++ b/docs/metaharness-user-guide.md @@ -328,11 +328,13 @@ A **pure-TS, zero-`@metaharness/*`-dep** similarity engine. Weighted blend: | Component | Weight | What it compares | |---|---|---| -| **cosine** | 0.4 | 9 numerics: `harnessFit`, `compileConfidence`, `taskCoverage`, `toolSafety`, `memoryUsefulness`, `risk_score`, `test_confidence`, `publish_readiness`, `estCostPerRunUsd` | -| **categorical** | 0.3 | 4 enums: `repo_type`, `recommendedMode`, `archetype`, `template` | -| **jaccard** | 0.3 | `agent_topology` (set of declared roles) | +| **cosine** | 0.6 | 9 numerics: `harnessFit`, `compileConfidence`, `taskCoverage`, `toolSafety`, `memoryUsefulness`, `risk_score`, `test_confidence`, `publish_readiness`, `estCostPerRunUsd` | +| **categorical** | 0.25 | 4 enums: `repo_type`, `recommendedMode`, `archetype`, `template` | +| **jaccard** | 0.15 | `agent_topology` (set of declared roles) | -`overall = w_c · cosine + w_k · categorical + w_j · jaccard`, all in `[0, 1]`. +`overall = 0.6 · cosine + 0.25 · categorical + 0.15 · jaccard`, all in `[0, 1]`. These are the ADR-152 §Decision defaults; `similarity.mjs --format json` echoes them under `weights`, so a consumer never has to hard-code them. + +The cost dimension is normalized on a log band (`$0.001–$10` → `0–1`), so costs inside the realistic range stay distinguishable rather than collapsing to a single value. **Verdict thresholds:** @@ -343,7 +345,7 @@ A **pure-TS, zero-`@metaharness/*`-dep** similarity engine. Weighted blend: | ≥ 0.5 | `moderate-drift` | | < 0.5 | `major-drift` | -These are the structural-distance verdicts surfaced by `audit-trend` and `drift-from-history`. +These are the structural-distance verdicts surfaced by `audit-trend` and `drift-from-history`. The bands live in `plugins/ruflo-metaharness/scripts/_similarity.mjs` as `VERDICT_THRESHOLDS`; `audit-trend` imports them rather than re-encoding the cutoffs, so this table and the implementation cannot drift apart. --- diff --git a/plugins/ruflo-metaharness/scripts/_similarity.mjs b/plugins/ruflo-metaharness/scripts/_similarity.mjs index f92469b63d..5c93e7987c 100644 --- a/plugins/ruflo-metaharness/scripts/_similarity.mjs +++ b/plugins/ruflo-metaharness/scripts/_similarity.mjs @@ -13,10 +13,13 @@ // // CONTRACT (PUBLIC) // projectToVec(input) → 9-dim numerical feature vector +// normalizeCost(usd) → [0,1] log-band normalization (vector index 8) // cosine(a, b) → [0,1] // categoricalAgreement(a, b) → [0,1] over 4 enum fields // jaccard(a, b) → [0,1] over agent_topology[] -// similarity(a, b, opts?) → { overall, components, perDimension? } +// verdictFor(overall) → 'near-identical' | 'minor-drift' | 'moderate-drift' | 'major-drift' +// VERDICT_THRESHOLDS → the bands verdictFor() walks, highest first +// similarity(a, b, opts?) → { overall, verdict, components, perDimension? } // // WEIGHT DEFAULTS (from ADR-152 §Decision) // overall = 0.60·cosine + 0.25·categorical + 0.15·jaccard @@ -29,6 +32,59 @@ const DEFAULT_WEIGHTS = Object.freeze({ cosine: 0.6, categorical: 0.25, jaccard: const CATEGORICAL_FIELDS = Object.freeze(['repo_type', 'archetype', 'template', 'recommendedMode']); +// ───────────────────────────────────────────────────────────────────── +// Cost normalization — vector index 8. +// +// ADR-152 Table 1 specified `log10(usd + 0.001) / log10(10)` clamped to +// [0,1]. `Math.log10(10)` is 1, so the division was a no-op, and the raw +// log is negative for every cost below $0.999 — the outer clamp then +// flattened that whole range to 0. Every realistic per-run cost collapsed +// to the same value, so the documented 9-dim vector was effectively +// 8-dim and `estCostPerRunUsd` could not influence any score. +// +// The replacement is a real log band: costs at or below `min` map to 0, +// at or above `max` map to 1, log-interpolated in between. It is +// monotonic and live across the whole realistic range ($0.001–$10), and +// non-finite or missing input degrades to the band floor rather than NaN. +// ───────────────────────────────────────────────────────────────────── + +export const COST_BAND_USD = Object.freeze({ min: 0.001, max: 10 }); + +export function normalizeCost(usd) { + const lo = Math.log10(COST_BAND_USD.min); + const hi = Math.log10(COST_BAND_USD.max); + const finite = typeof usd === 'number' && Number.isFinite(usd) ? usd : COST_BAND_USD.min; + const clamped = Math.min(COST_BAND_USD.max, Math.max(COST_BAND_USD.min, finite)); + return (Math.log10(clamped) - lo) / (hi - lo); +} + +// ───────────────────────────────────────────────────────────────────── +// Verdict bands — single source of truth. +// +// These bands previously lived inline in `audit-trend.mjs` with a +// `minor-drift` floor of 0.80, while `docs/metaharness-user-guide.md` +// documented 0.85 for the same band. Two encodings of one policy, and +// they disagreed. Consumers now import this table instead of re-encoding +// the bands, so the primitive and its documentation cannot drift apart. +// +// Ordered highest-first; `verdictFor` returns the first band whose `min` +// the score reaches. +// ───────────────────────────────────────────────────────────────────── + +export const VERDICT_THRESHOLDS = Object.freeze([ + Object.freeze({ min: 0.95, verdict: 'near-identical' }), + Object.freeze({ min: 0.85, verdict: 'minor-drift' }), + Object.freeze({ min: 0.5, verdict: 'moderate-drift' }), + Object.freeze({ min: Number.NEGATIVE_INFINITY, verdict: 'major-drift' }), +]); + +export function verdictFor(overall) { + for (const band of VERDICT_THRESHOLDS) { + if (overall >= band.min) return band.verdict; + } + return 'major-drift'; +} + // ───────────────────────────────────────────────────────────────────── // 9-dim feature vector. The mapping mirrors ADR-152 §Decision Table 1. // Missing fields default to 0 — that's the graceful-degradation path. @@ -46,7 +102,7 @@ export function projectToVec(input) { g.risk_score ?? 0, g.test_confidence ?? 0, g.publish_readiness ?? 0, - Math.max(0, Math.min(1, Math.log10((s.estCostPerRunUsd ?? 0) + 0.001) / Math.log10(10))), + normalizeCost(s.estCostPerRunUsd), ]; } @@ -99,6 +155,10 @@ export function similarity(a, b, opts = {}) { const result = { overall: round4(overall), + // The verdict is derived here, next to the score it classifies, so + // every consumer (similarity.mjs, audit-trend.mjs) reports the same + // band for the same number. + verdict: verdictFor(round4(overall)), components: { cosine: round4(cos), categorical: round4(cat), diff --git a/plugins/ruflo-metaharness/scripts/audit-trend.mjs b/plugins/ruflo-metaharness/scripts/audit-trend.mjs index 7e50f04686..e0b50fa325 100755 --- a/plugins/ruflo-metaharness/scripts/audit-trend.mjs +++ b/plugins/ruflo-metaharness/scripts/audit-trend.mjs @@ -31,7 +31,10 @@ import { runRufloCli } from './_invoke.mjs'; // iter 38 — structural-distance drift via ADR-152 §3.1 production module. // Falls back to null if either record predates iter-38 oia-audit (no // fingerprint field) — graceful degradation, never throws. -import { similarity } from './_similarity.mjs'; +// The verdict bands come from the module, not a local literal: this file +// previously re-encoded them with a 0.80 minor-drift floor while +// docs/metaharness-user-guide.md documented 0.85. +import { similarity, verdictFor } from './_similarity.mjs'; // iter 63 — shared SEVERITY_RANK from _harness.mjs (was a local literal // missing info/warn/error/critical, which caused NaN-compare hazards). import { SEVERITY_RANK, rankSeverity } from './_harness.mjs'; @@ -168,10 +171,7 @@ function main() { // Distance is the complement of similarity in [0,1] distance: Number((1 - sim.overall).toFixed(4)), components: sim.components, - verdict: sim.overall >= 0.95 ? 'near-identical' - : sim.overall >= 0.80 ? 'minor-drift' - : sim.overall >= 0.50 ? 'moderate-drift' - : 'major-drift', + verdict: verdictFor(sim.overall), }; } else { structuralDistance = { diff --git a/plugins/ruflo-metaharness/scripts/similarity.mjs b/plugins/ruflo-metaharness/scripts/similarity.mjs index c7dedfcd65..88d8eaa202 100755 --- a/plugins/ruflo-metaharness/scripts/similarity.mjs +++ b/plugins/ruflo-metaharness/scripts/similarity.mjs @@ -147,6 +147,7 @@ function main() { console.log(`| jaccard | ${components.jaccard.toFixed(4)} | ${weights.jaccard} | ${(components.jaccard * weights.jaccard).toFixed(4)} |`); console.log(''); console.log(`**Overall:** ${payload.overall.toFixed(4)}`); + console.log(`**Verdict:** ${payload.verdict}`); if (payload.alert) { console.log(''); console.log(payload.alert.triggered ? `⚠ ALERT: ${payload.alert.reason}` : `✓ ${payload.alert.reason}`); diff --git a/plugins/ruflo-metaharness/scripts/test-similarity.mjs b/plugins/ruflo-metaharness/scripts/test-similarity.mjs index 4b0ae17029..11447d28ae 100755 --- a/plugins/ruflo-metaharness/scripts/test-similarity.mjs +++ b/plugins/ruflo-metaharness/scripts/test-similarity.mjs @@ -23,6 +23,7 @@ import { projectToVec, cosine, categoricalAgreement, jaccard, similarity, + normalizeCost, COST_BAND_USD, verdictFor, VERDICT_THRESHOLDS, } from './_similarity.mjs'; // iter 64 — also test the iter-63 shared severity primitives import { SEVERITY_RANK, rankSeverity, parseMcpScanText } from './_harness.mjs'; @@ -174,6 +175,68 @@ assert(typeof hf.contribution === 'number', 'perDimension has numeric contributi const sNoPD = similarity(A, B); assert(!('perDimension' in sNoPD), 'perDimension omitted by default'); +console.log('\nPhase 7b — normalizeCost (vector index 8, ADR-152 Table 1)'); +// The pre-fix transform was `log10(usd + 0.001) / log10(10)` clamped to +// [0,1]. `Math.log10(10)` is 1, so the division was a no-op and the raw +// log — negative for every cost below $0.999 — clamped to a flat 0. That +// dead zone covered the whole realistic per-run range, including ruflo's +// own $0.048 Phase-0 baseline. +assert(normalizeCost(COST_BAND_USD.min) === 0, 'band floor → 0'); +assert(normalizeCost(COST_BAND_USD.max) === 1, 'band ceiling → 1'); +assert(normalizeCost(50) === 1, 'above band ceiling clamps to 1'); +assert(normalizeCost(0) === 0, 'zero cost clamps to 0'); +assert(normalizeCost(-5) === 0, 'negative cost clamps to 0'); +// The regression that motivated the fix: realistic costs must not collide. +assert(normalizeCost(0.048) !== normalizeCost(0.11), + 'ruflo baseline $0.048 and $0.11 must not collapse to the same value (dead-zone regression)'); +assert(normalizeCost(0.048) > 0, + '$0.048 (ruflo Phase-0 baseline) must be strictly inside the band'); +// Monotonic non-decreasing across the band, sampled log-evenly. +const costSamples = [0.001, 0.005, 0.02, 0.048, 0.11, 0.5, 1, 5, 10]; +const costVec = costSamples.map(normalizeCost); +assert(costVec.every((v, i) => i === 0 || v >= costVec[i - 1]), + 'normalizeCost is monotonic non-decreasing across the band'); +assert(costVec.every((v) => v >= 0 && v <= 1), 'normalizeCost bounded to [0, 1]'); +// End-to-end: the dimension must now reach the composite. Two harnesses +// identical except for cost previously scored cosine === 1 because both +// collapsed to 0; they must now differ. +const costTwin = (usd) => ({ + score: { ...A.score, estCostPerRunUsd: usd }, genome: A.genome, +}); +const cheapTwin = similarity(costTwin(0.048), A); +const priceyTwin = similarity(costTwin(1), A); +assert(cheapTwin.components.cosine !== priceyTwin.components.cosine, + 'cost now influences cosine (was a dead dimension below $0.999)'); +assert(priceyTwin.components.cosine < cheapTwin.components.cosine, + 'a same-shaped harness with a higher cost is less similar to a cheap one'); + +console.log('\nPhase 7c — verdictFor + VERDICT_THRESHOLDS (single source of truth)'); +assert(VERDICT_THRESHOLDS.length === 4, 'four verdict bands'); +assert(VERDICT_THRESHOLDS.every((b, i) => i === 0 || b.min < VERDICT_THRESHOLDS[i - 1].min), + 'bands are ordered highest-first'); +const LAST_BAND = VERDICT_THRESHOLDS[VERDICT_THRESHOLDS.length - 1]; +assert(LAST_BAND.min === Number.NEGATIVE_INFINITY, + 'final band is unbounded so verdictFor always returns a verdict'); +assert(verdictFor(1) === 'near-identical', 'overall 1 → near-identical'); +assert(verdictFor(0.95) === 'near-identical', 'boundary 0.95 → near-identical (inclusive)'); +assert(verdictFor(0.9499) === 'minor-drift', 'just below 0.95 → minor-drift'); +assert(verdictFor(0.85) === 'minor-drift', 'boundary 0.85 → minor-drift (inclusive, the documented floor)'); +assert(verdictFor(0.8499) === 'moderate-drift', 'just below 0.85 → moderate-drift'); +assert(verdictFor(0.5) === 'moderate-drift', 'boundary 0.5 → moderate-drift (inclusive)'); +assert(verdictFor(0.4999) === 'major-drift', 'just below 0.5 → major-drift'); +assert(verdictFor(0) === 'major-drift', 'overall 0 → major-drift'); +// Regression: audit-trend.mjs used to inline a 0.80 minor-drift floor while +// docs/metaharness-user-guide.md documented 0.85. 0.80 must resolve to +// moderate-drift now that the bands live in one place. +assert(verdictFor(0.80) === 'moderate-drift', + '0.80 resolves via the shared bands (was a divergent inline 0.80 floor in audit-trend.mjs)'); + +console.log('\nPhase 7d — verdict on the similarity() return shape'); +assert(typeof sAA.verdict === 'string', 'similarity() includes a verdict string'); +assert(sAA.verdict === 'near-identical', 'self-match verdict is near-identical'); +assert(sPD.verdict === verdictFor(sPD.overall), + 'similarity().verdict agrees with verdictFor(overall)'); + console.log('\nPhase 8 — round-trip with the iter-35 spike fixtures (regression anchor)'); // Hard-coded spike numbers from `_spike-similarity.mjs` — ANY change // here means the production module drifted from the spike's frozen diff --git a/v3/docs/adr/ADR-152-genome-similarity-search.md b/v3/docs/adr/ADR-152-genome-similarity-search.md index de472f3ed6..6622e55624 100644 --- a/v3/docs/adr/ADR-152-genome-similarity-search.md +++ b/v3/docs/adr/ADR-152-genome-similarity-search.md @@ -52,7 +52,7 @@ Project the structured genome + scorecard into a fixed-length numerical vector: | 5 | genome | `risk_score` | already 0..1 | | 6 | genome | `test_confidence` | already 0..1 | | 7 | genome | `publish_readiness` | already 0..1 | -| 8 | score | `estCostPerRunUsd` | log-transformed: `log10(usd + 0.001) / log10(10)` clamped to 0..1 | +| 8 | score | `estCostPerRunUsd` | log band: `$0.001` → `0`, `$10` → `1`, log-interpolated between (revised 2026-09-28 — see [Revision](#revision--2026-09-28)) | Cosine similarity over these 9 dims gives a `[0, 1]` score where 1 = identical and 0 = orthogonal. Cheap, deterministic, byte-identical for identical inputs. @@ -87,6 +87,7 @@ Return shape: ```typescript interface SimilarityResult { overall: number; // [0,1] + verdict: 'near-identical' | 'minor-drift' | 'moderate-drift' | 'major-drift'; // added 2026-09-28 components: { cosine: number; // [0,1] categorical: number; // [0,1] @@ -169,6 +170,41 @@ The spike script: `plugins/ruflo-metaharness/scripts/_spike-similarity.mjs`. Liv - Should the function ALSO emit a Mermaid diagram visualizing the breakdown? Probably yes via an optional `--format mermaid`. Cheap to add post-spike. - How does similarity interact with the iter-15 `audit-trend` script? `audit-trend` already diffs two timestamps of the SAME harness; this ADR adds a separate primitive for diffing two DIFFERENT harnesses. Both should live as siblings, not subtypes. +## Revision — 2026-09-28 + +Two defects in this ADR's §Decision were found while reconciling the shipped code against [the MetaHarness user guide](../../docs/metaharness-user-guide.md). Both are corrected in `plugins/ruflo-metaharness/scripts/_similarity.mjs`; nothing else consumes the affected semantics. + +### 1. The cost normalization (index 8) was inert below $0.999 + +The original transform was `log10(usd + 0.001) / log10(10)` clamped to `[0, 1]`. `Math.log10(10)` is `1`, so the division was a no-op and the expression reduced to `clamp(log10(usd + 0.001), 0, 1)`. Since `log10` is negative for every `usd < 0.999`, the outer clamp flattened that entire range to a constant `0`: + +| `estCostPerRunUsd` | original index 8 | revised index 8 | +|---:|---:|---:| +| 0.005 | 0 | 0.1747 | +| 0.048 | 0 | 0.4203 | +| 0.11 | 0 | 0.5103 | +| 0.5 | 0 | 0.6747 | +| 1 | 0.0004 | 0.75 | +| 5 | 0.6991 | 0.9247 | + +Every realistic per-run cost — including the $0.048 Phase-0 baseline recorded in the plugin README — landed in the dead zone, so the "9-dim" vector was effectively 8-dim and a 2× cost regression was invisible to `cosine`. Two harnesses differing *only* in cost scored `cosine === 1`. + +The replacement is an explicit log band: `$0.001 → 0`, `$10 → 1`, log-interpolated in between, monotonic and non-finite-safe (missing/`NaN`/negative input degrades to the band floor, never `NaN`). Exported as `normalizeCost()` with the band in `COST_BAND_USD`. + +### 2. The verdict bands had two encodings, and they disagreed + +This ADR's return shape never specified a verdict, but `audit-trend.mjs` inlined one using a **0.80** `minor-drift` floor while the user guide documented **0.85**. `drift-from-history` prints that verdict to users, so the two diverged in the 0.80–0.85 band. + +The bands now live once, in `_similarity.mjs` as `VERDICT_THRESHOLDS`, with `verdictFor()` as the only classifier; `audit-trend.mjs` imports it and `similarity()` returns `verdict` alongside `overall`. The documented `0.85` floor is canonical — it is the published contract, and the guide's own `audit-trend` example already gates on `--alert-on-distance-below 0.85`. + +**This is a deliberate user-visible behaviour change:** a structural similarity in `[0.80, 0.85)` now reports `moderate-drift` instead of `minor-drift`. `drift-from-history --threshold` semantics are unchanged (the threshold is compared against `overall`, not the verdict), but downstream consumers that branch on the verdict string should be re-checked. + +### Effect on the frozen spike numbers + +The §Spike-result figures are unchanged (`0.8296` for LEGAL×SUPPORT, `0.9987` cosine): the LEGAL/`$0.04` and SUPPORT/`$0.05` fixtures are near-identical on the cost axis, so the revived dimension perturbs their cosine by less than the 4-decimal reporting precision. The regression anchor in `test-similarity.mjs` Phase 8 therefore still pins the original values, and new Phase 7b/7c/7d coverage pins the corrected cost and verdict semantics directly. + +The defect this ADR's §Consequences already anticipated — "numerical cosine alone is too coarse" (§Spike result) — is **not** addressed here. Cosine over mostly-high normalized dimensions stays insensitive to uniform degradation; drift gates that must catch absolute quality loss should pair structural similarity with the absolute gates (`score --alert-on-fit-below`, `mcp-scan --fail-on`) rather than relying on `--threshold` alone. + ## References - [ADR-151](ADR-151-harness-intelligence-layer.md) — Phase 3 scope shell (parent)