fix(metaharness): revive the cost dimension and single-source the verdict bands (ADR-152) (#3520)

Reconciling the shipped similarity code against docs/metaharness-user-guide.md
turned up three divergences between what ADR-152 specifies, what the code does,
and what the docs promise.

1. Cost normalization (vector index 8) was inert below $0.999.

   ADR-152 Table 1 specified `log10(usd + 0.001) / log10(10)` clamped to [0,1].
   `Math.log10(10)` is 1, so the division was a no-op and the expression reduced
   to `clamp(log10(usd + 0.001), 0, 1)`. Because log10 is negative for every
   usd < 0.999, the clamp flattened that entire range to a constant 0 — including
   the $0.048 baseline the plugin README records. Two harnesses differing only in
   cost scored `cosine === 1`; the documented 9-dim vector was effectively 8-dim.

   Replaced with an explicit log band: $0.001 -> 0, $10 -> 1, log-interpolated
   between, monotonic and non-finite-safe. Exported as `normalizeCost()` with the
   band in `COST_BAND_USD`.

2. The verdict bands had two encodings, and they disagreed.

   `audit-trend.mjs` inlined the bands with a 0.80 `minor-drift` floor while the
   user guide documented 0.85; `drift-from-history` prints that verdict, so the
   two diverged in the 0.80-0.85 band. The bands now live once as
   `VERDICT_THRESHOLDS` + `verdictFor()` in `_similarity.mjs`, `audit-trend.mjs`
   imports them, and `similarity()` returns `verdict` beside `overall`. The
   documented 0.85 floor is canonical.

   Deliberate user-visible change: structural similarity in [0.80, 0.85) now
   reports `moderate-drift` instead of `minor-drift`. `--threshold` comparison is
   unaffected (it tests `overall`), but consumers branching on the verdict string
   should be re-checked.

3. The user guide's weight table said 0.4/0.3/0.3; ADR-152 and the code both say
   0.6/0.25/0.15. Corrected the guide, which was the outlier.

The ADR gains a dated Revision section recording both fixes and the behaviour
change. The §Spike-result figures are unchanged (the LEGAL/SUPPORT fixtures are
near-identical on the cost axis), so test-similarity.mjs Phase 8 keeps pinning
the original anchors; new Phase 7b/7c/7d coverage pins the corrected semantics.

Note this does NOT address the coarse-cosine limitation ADR-152 itself flags
("numerical cosine alone is too coarse"): uniform quality degradation still
reads as high similarity. Drift gates that must catch absolute loss should pair
structural similarity with the absolute gates rather than rely on --threshold.

Verification:
  node plugins/ruflo-metaharness/scripts/test-similarity.mjs   -> 128 passed, 0 failed (was 102)
  node plugins/ruflo-metaharness/scripts/_spike-similarity.mjs -> both invariants hold

Co-authored-by: longxinjun-ai <longxinjun-ai@users.noreply.github.com>
Co-authored-by: rUv <ruv@ruv.net>
This commit is contained in:
authored and GitHub committed 2026-10-20 06:50:00 -04:00
1 parent d7f2286835
commit c2069ed6f6
6 files changed
+175 -13

No files matched your search

+7 -5
View File
@@ -328,11 +328,13 @@ A **pure-TS, zero-`@metaharness/*`-dep** similarity engine. Weighted blend:
| Component | Weight | What it compares |
|---|---|---|
| **cosine** | 0.4 | 9 numerics: `harnessFit`, `compileConfidence`, `taskCoverage`, `toolSafety`, `memoryUsefulness`, `risk_score`, `test_confidence`, `publish_readiness`, `estCostPerRunUsd` |
| **categorical** | 0.3 | 4 enums: `repo_type`, `recommendedMode`, `archetype`, `template` |
| **jaccard** | 0.3 | `agent_topology` (set of declared roles) |
| **cosine** | 0.6 | 9 numerics: `harnessFit`, `compileConfidence`, `taskCoverage`, `toolSafety`, `memoryUsefulness`, `risk_score`, `test_confidence`, `publish_readiness`, `estCostPerRunUsd` |
| **categorical** | 0.25 | 4 enums: `repo_type`, `recommendedMode`, `archetype`, `template` |
| **jaccard** | 0.15 | `agent_topology` (set of declared roles) |
`overall = w_c · cosine + w_k · categorical + w_j · jaccard`, all in `[0, 1]`.
`overall = 0.6 · cosine + 0.25 · categorical + 0.15 · jaccard`, all in `[0, 1]`. These are the ADR-152 §Decision defaults; `similarity.mjs --format json` echoes them under `weights`, so a consumer never has to hard-code them.
The cost dimension is normalized on a log band (`$0.001–$10` → `0–1`), so costs inside the realistic range stay distinguishable rather than collapsing to a single value.
**Verdict thresholds:**
@@ -343,7 +345,7 @@ A **pure-TS, zero-`@metaharness/*`-dep** similarity engine. Weighted blend:
| ≥ 0.5 | `moderate-drift` |
| < 0.5 | `major-drift` |
These are the structural-distance verdicts surfaced by `audit-trend` and `drift-from-history`.
These are the structural-distance verdicts surfaced by `audit-trend` and `drift-from-history`. The bands live in `plugins/ruflo-metaharness/scripts/_similarity.mjs` as `VERDICT_THRESHOLDS`; `audit-trend` imports them rather than re-encoding the cutoffs, so this table and the implementation cannot drift apart.
---
@@ -13,10 +13,13 @@
//
// CONTRACT (PUBLIC)
// projectToVec(input) → 9-dim numerical feature vector
// normalizeCost(usd) → [0,1] log-band normalization (vector index 8)
// cosine(a, b) → [0,1]
// categoricalAgreement(a, b) → [0,1] over 4 enum fields
// jaccard(a, b) → [0,1] over agent_topology[]
// similarity(a, b, opts?) → { overall, components, perDimension? }
// verdictFor(overall) → 'near-identical' | 'minor-drift' | 'moderate-drift' | 'major-drift'
// VERDICT_THRESHOLDS → the bands verdictFor() walks, highest first
// similarity(a, b, opts?) → { overall, verdict, components, perDimension? }
//
// WEIGHT DEFAULTS (from ADR-152 §Decision)
// overall = 0.60·cosine + 0.25·categorical + 0.15·jaccard
@@ -29,6 +32,59 @@ const DEFAULT_WEIGHTS = Object.freeze({ cosine: 0.6, categorical: 0.25, jaccard:
const CATEGORICAL_FIELDS = Object.freeze(['repo_type', 'archetype', 'template', 'recommendedMode']);
// ─────────────────────────────────────────────────────────────────────
// Cost normalization — vector index 8.
//
// ADR-152 Table 1 specified `log10(usd + 0.001) / log10(10)` clamped to
// [0,1]. `Math.log10(10)` is 1, so the division was a no-op, and the raw
// log is negative for every cost below $0.999 — the outer clamp then
// flattened that whole range to 0. Every realistic per-run cost collapsed
// to the same value, so the documented 9-dim vector was effectively
// 8-dim and `estCostPerRunUsd` could not influence any score.
//
// The replacement is a real log band: costs at or below `min` map to 0,
// at or above `max` map to 1, log-interpolated in between. It is
// monotonic and live across the whole realistic range ($0.001–$10), and
// non-finite or missing input degrades to the band floor rather than NaN.
// ─────────────────────────────────────────────────────────────────────
export const COST_BAND_USD = Object.freeze({ min: 0.001, max: 10 });
export function normalizeCost(usd) {
const lo = Math.log10(COST_BAND_USD.min);
const hi = Math.log10(COST_BAND_USD.max);
const finite = typeof usd === 'number' && Number.isFinite(usd) ? usd : COST_BAND_USD.min;
const clamped = Math.min(COST_BAND_USD.max, Math.max(COST_BAND_USD.min, finite));
return (Math.log10(clamped) - lo) / (hi - lo);
}
// ─────────────────────────────────────────────────────────────────────
// Verdict bands — single source of truth.
//
// These bands previously lived inline in `audit-trend.mjs` with a
// `minor-drift` floor of 0.80, while `docs/metaharness-user-guide.md`
// documented 0.85 for the same band. Two encodings of one policy, and
// they disagreed. Consumers now import this table instead of re-encoding
// the bands, so the primitive and its documentation cannot drift apart.
//
// Ordered highest-first; `verdictFor` returns the first band whose `min`
// the score reaches.
// ─────────────────────────────────────────────────────────────────────
export const VERDICT_THRESHOLDS = Object.freeze([
Object.freeze({ min: 0.95, verdict: 'near-identical' }),
Object.freeze({ min: 0.85, verdict: 'minor-drift' }),
Object.freeze({ min: 0.5, verdict: 'moderate-drift' }),
Object.freeze({ min: Number.NEGATIVE_INFINITY, verdict: 'major-drift' }),
]);
export function verdictFor(overall) {
for (const band of VERDICT_THRESHOLDS) {
if (overall >= band.min) return band.verdict;
}
return 'major-drift';
}
// ─────────────────────────────────────────────────────────────────────
// 9-dim feature vector. The mapping mirrors ADR-152 §Decision Table 1.
// Missing fields default to 0 — that's the graceful-degradation path.
@@ -46,7 +102,7 @@ export function projectToVec(input) {
g.risk_score ?? 0,
g.test_confidence ?? 0,
g.publish_readiness ?? 0,
Math.max(0, Math.min(1, Math.log10((s.estCostPerRunUsd ?? 0) + 0.001) / Math.log10(10))),
normalizeCost(s.estCostPerRunUsd),
];
}
@@ -99,6 +155,10 @@ export function similarity(a, b, opts = {}) {
const result = {
overall: round4(overall),
// The verdict is derived here, next to the score it classifies, so
// every consumer (similarity.mjs, audit-trend.mjs) reports the same
// band for the same number.
verdict: verdictFor(round4(overall)),
components: {
cosine: round4(cos),
categorical: round4(cat),
@@ -31,7 +31,10 @@ import { runRufloCli } from './_invoke.mjs';
// iter 38 — structural-distance drift via ADR-152 §3.1 production module.
// Falls back to null if either record predates iter-38 oia-audit (no
// fingerprint field) — graceful degradation, never throws.
import { similarity } from './_similarity.mjs';
// The verdict bands come from the module, not a local literal: this file
// previously re-encoded them with a 0.80 minor-drift floor while
// docs/metaharness-user-guide.md documented 0.85.
import { similarity, verdictFor } from './_similarity.mjs';
// iter 63 — shared SEVERITY_RANK from _harness.mjs (was a local literal
// missing info/warn/error/critical, which caused NaN-compare hazards).
import { SEVERITY_RANK, rankSeverity } from './_harness.mjs';
@@ -168,10 +171,7 @@ function main() {
// Distance is the complement of similarity in [0,1]
distance: Number((1 - sim.overall).toFixed(4)),
components: sim.components,
verdict: sim.overall >= 0.95 ? 'near-identical'
: sim.overall >= 0.80 ? 'minor-drift'
: sim.overall >= 0.50 ? 'moderate-drift'
: 'major-drift',
verdict: verdictFor(sim.overall),
};
} else {
structuralDistance = {
@@ -147,6 +147,7 @@ function main() {
console.log(`| jaccard | ${components.jaccard.toFixed(4)} | ${weights.jaccard} | ${(components.jaccard * weights.jaccard).toFixed(4)} |`);
console.log('');
console.log(`**Overall:** ${payload.overall.toFixed(4)}`);
console.log(`**Verdict:** ${payload.verdict}`);
if (payload.alert) {
console.log('');
console.log(payload.alert.triggered ? `⚠ ALERT: ${payload.alert.reason}` : `✓ ${payload.alert.reason}`);
@@ -23,6 +23,7 @@
import {
projectToVec, cosine, categoricalAgreement, jaccard, similarity,
normalizeCost, COST_BAND_USD, verdictFor, VERDICT_THRESHOLDS,
} from './_similarity.mjs';
// iter 64 — also test the iter-63 shared severity primitives
import { SEVERITY_RANK, rankSeverity, parseMcpScanText } from './_harness.mjs';
@@ -174,6 +175,68 @@ assert(typeof hf.contribution === 'number', 'perDimension has numeric contributi
const sNoPD = similarity(A, B);
assert(!('perDimension' in sNoPD), 'perDimension omitted by default');
console.log('\nPhase 7b — normalizeCost (vector index 8, ADR-152 Table 1)');
// The pre-fix transform was `log10(usd + 0.001) / log10(10)` clamped to
// [0,1]. `Math.log10(10)` is 1, so the division was a no-op and the raw
// log — negative for every cost below $0.999 — clamped to a flat 0. That
// dead zone covered the whole realistic per-run range, including ruflo's
// own $0.048 Phase-0 baseline.
assert(normalizeCost(COST_BAND_USD.min) === 0, 'band floor → 0');
assert(normalizeCost(COST_BAND_USD.max) === 1, 'band ceiling → 1');
assert(normalizeCost(50) === 1, 'above band ceiling clamps to 1');
assert(normalizeCost(0) === 0, 'zero cost clamps to 0');
assert(normalizeCost(-5) === 0, 'negative cost clamps to 0');
// The regression that motivated the fix: realistic costs must not collide.
assert(normalizeCost(0.048) !== normalizeCost(0.11),
'ruflo baseline $0.048 and $0.11 must not collapse to the same value (dead-zone regression)');
assert(normalizeCost(0.048) > 0,
'$0.048 (ruflo Phase-0 baseline) must be strictly inside the band');
// Monotonic non-decreasing across the band, sampled log-evenly.
const costSamples = [0.001, 0.005, 0.02, 0.048, 0.11, 0.5, 1, 5, 10];
const costVec = costSamples.map(normalizeCost);
assert(costVec.every((v, i) => i === 0 || v >= costVec[i - 1]),
'normalizeCost is monotonic non-decreasing across the band');
assert(costVec.every((v) => v >= 0 && v <= 1), 'normalizeCost bounded to [0, 1]');
// End-to-end: the dimension must now reach the composite. Two harnesses
// identical except for cost previously scored cosine === 1 because both
// collapsed to 0; they must now differ.
const costTwin = (usd) => ({
score: { ...A.score, estCostPerRunUsd: usd }, genome: A.genome,
});
const cheapTwin = similarity(costTwin(0.048), A);
const priceyTwin = similarity(costTwin(1), A);
assert(cheapTwin.components.cosine !== priceyTwin.components.cosine,
'cost now influences cosine (was a dead dimension below $0.999)');
assert(priceyTwin.components.cosine < cheapTwin.components.cosine,
'a same-shaped harness with a higher cost is less similar to a cheap one');
console.log('\nPhase 7c — verdictFor + VERDICT_THRESHOLDS (single source of truth)');
assert(VERDICT_THRESHOLDS.length === 4, 'four verdict bands');
assert(VERDICT_THRESHOLDS.every((b, i) => i === 0 || b.min < VERDICT_THRESHOLDS[i - 1].min),
'bands are ordered highest-first');
const LAST_BAND = VERDICT_THRESHOLDS[VERDICT_THRESHOLDS.length - 1];
assert(LAST_BAND.min === Number.NEGATIVE_INFINITY,
'final band is unbounded so verdictFor always returns a verdict');
assert(verdictFor(1) === 'near-identical', 'overall 1 → near-identical');
assert(verdictFor(0.95) === 'near-identical', 'boundary 0.95 → near-identical (inclusive)');
assert(verdictFor(0.9499) === 'minor-drift', 'just below 0.95 → minor-drift');
assert(verdictFor(0.85) === 'minor-drift', 'boundary 0.85 → minor-drift (inclusive, the documented floor)');
assert(verdictFor(0.8499) === 'moderate-drift', 'just below 0.85 → moderate-drift');
assert(verdictFor(0.5) === 'moderate-drift', 'boundary 0.5 → moderate-drift (inclusive)');
assert(verdictFor(0.4999) === 'major-drift', 'just below 0.5 → major-drift');
assert(verdictFor(0) === 'major-drift', 'overall 0 → major-drift');
// Regression: audit-trend.mjs used to inline a 0.80 minor-drift floor while
// docs/metaharness-user-guide.md documented 0.85. 0.80 must resolve to
// moderate-drift now that the bands live in one place.
assert(verdictFor(0.80) === 'moderate-drift',
'0.80 resolves via the shared bands (was a divergent inline 0.80 floor in audit-trend.mjs)');
console.log('\nPhase 7d — verdict on the similarity() return shape');
assert(typeof sAA.verdict === 'string', 'similarity() includes a verdict string');
assert(sAA.verdict === 'near-identical', 'self-match verdict is near-identical');
assert(sPD.verdict === verdictFor(sPD.overall),
'similarity().verdict agrees with verdictFor(overall)');
console.log('\nPhase 8 — round-trip with the iter-35 spike fixtures (regression anchor)');
// Hard-coded spike numbers from `_spike-similarity.mjs` — ANY change
// here means the production module drifted from the spike's frozen
@@ -52,7 +52,7 @@ Project the structured genome + scorecard into a fixed-length numerical vector:
| 5 | genome | `risk_score` | already 0..1 |
| 6 | genome | `test_confidence` | already 0..1 |
| 7 | genome | `publish_readiness` | already 0..1 |
| 8 | score | `estCostPerRunUsd` | log-transformed: `log10(usd + 0.001) / log10(10)` clamped to 0..1 |
| 8 | score | `estCostPerRunUsd` | log band: `$0.001` → `0`, `$10` → `1`, log-interpolated between (revised 2026-09-28 — see [Revision](#revision--2026-09-28)) |
Cosine similarity over these 9 dims gives a `[0, 1]` score where 1 = identical and 0 = orthogonal. Cheap, deterministic, byte-identical for identical inputs.
@@ -87,6 +87,7 @@ Return shape:
```typescript
interface SimilarityResult {
overall: number; // [0,1]
verdict: 'near-identical' | 'minor-drift' | 'moderate-drift' | 'major-drift'; // added 2026-09-28
components: {
cosine: number; // [0,1]
categorical: number; // [0,1]
@@ -169,6 +170,41 @@ The spike script: `plugins/ruflo-metaharness/scripts/_spike-similarity.mjs`. Liv
- Should the function ALSO emit a Mermaid diagram visualizing the breakdown? Probably yes via an optional `--format mermaid`. Cheap to add post-spike.
- How does similarity interact with the iter-15 `audit-trend` script? `audit-trend` already diffs two timestamps of the SAME harness; this ADR adds a separate primitive for diffing two DIFFERENT harnesses. Both should live as siblings, not subtypes.
## Revision — 2026-09-28
Two defects in this ADR's §Decision were found while reconciling the shipped code against [the MetaHarness user guide](../../docs/metaharness-user-guide.md). Both are corrected in `plugins/ruflo-metaharness/scripts/_similarity.mjs`; nothing else consumes the affected semantics.
### 1. The cost normalization (index 8) was inert below $0.999
The original transform was `log10(usd + 0.001) / log10(10)` clamped to `[0, 1]`. `Math.log10(10)` is `1`, so the division was a no-op and the expression reduced to `clamp(log10(usd + 0.001), 0, 1)`. Since `log10` is negative for every `usd < 0.999`, the outer clamp flattened that entire range to a constant `0`:
| `estCostPerRunUsd` | original index 8 | revised index 8 |
|---:|---:|---:|
| 0.005 | 0 | 0.1747 |
| 0.048 | 0 | 0.4203 |
| 0.11 | 0 | 0.5103 |
| 0.5 | 0 | 0.6747 |
| 1 | 0.0004 | 0.75 |
| 5 | 0.6991 | 0.9247 |
Every realistic per-run cost — including the $0.048 Phase-0 baseline recorded in the plugin README — landed in the dead zone, so the "9-dim" vector was effectively 8-dim and a 2× cost regression was invisible to `cosine`. Two harnesses differing *only* in cost scored `cosine === 1`.
The replacement is an explicit log band: `$0.001 → 0`, `$10 → 1`, log-interpolated in between, monotonic and non-finite-safe (missing/`NaN`/negative input degrades to the band floor, never `NaN`). Exported as `normalizeCost()` with the band in `COST_BAND_USD`.
### 2. The verdict bands had two encodings, and they disagreed
This ADR's return shape never specified a verdict, but `audit-trend.mjs` inlined one using a **0.80** `minor-drift` floor while the user guide documented **0.85**. `drift-from-history` prints that verdict to users, so the two diverged in the 0.80–0.85 band.
The bands now live once, in `_similarity.mjs` as `VERDICT_THRESHOLDS`, with `verdictFor()` as the only classifier; `audit-trend.mjs` imports it and `similarity()` returns `verdict` alongside `overall`. The documented `0.85` floor is canonical — it is the published contract, and the guide's own `audit-trend` example already gates on `--alert-on-distance-below 0.85`.
**This is a deliberate user-visible behaviour change:** a structural similarity in `[0.80, 0.85)` now reports `moderate-drift` instead of `minor-drift`. `drift-from-history --threshold` semantics are unchanged (the threshold is compared against `overall`, not the verdict), but downstream consumers that branch on the verdict string should be re-checked.
### Effect on the frozen spike numbers
The §Spike-result figures are unchanged (`0.8296` for LEGAL×SUPPORT, `0.9987` cosine): the LEGAL/`$0.04` and SUPPORT/`$0.05` fixtures are near-identical on the cost axis, so the revived dimension perturbs their cosine by less than the 4-decimal reporting precision. The regression anchor in `test-similarity.mjs` Phase 8 therefore still pins the original values, and new Phase 7b/7c/7d coverage pins the corrected cost and verdict semantics directly.
The defect this ADR's §Consequences already anticipated — "numerical cosine alone is too coarse" (§Spike result) — is **not** addressed here. Cosine over mostly-high normalized dimensions stays insensitive to uniform degradation; drift gates that must catch absolute quality loss should pair structural similarity with the absolute gates (`score --alert-on-fit-below`, `mcp-scan --fail-on`) rather than relying on `--threshold` alone.
## References
- [ADR-151](ADR-151-harness-intelligence-layer.md) — Phase 3 scope shell (parent)