ruflo

9,763 Commits 919 Branches 1,708 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
longxinjun-ai longxinjun-ai rUv	c2069ed6f6	fix(metaharness): revive the cost dimension and single-source the verdict bands (ADR-152) (#3520 ) Reconciling the shipped similarity code against docs/metaharness-user-guide.md turned up three divergences between what ADR-152 specifies, what the code does, and what the docs promise. 1. Cost normalization (vector index 8) was inert below $0.999. ADR-152 Table 1 specified `log10(usd + 0.001) / log10(10)` clamped to [0,1]. `Math.log10(10)` is 1, so the division was a no-op and the expression reduced to `clamp(log10(usd + 0.001), 0, 1)`. Because log10 is negative for every usd < 0.999, the clamp flattened that entire range to a constant 0 — including the $0.048 baseline the plugin README records. Two harnesses differing only in cost scored `cosine === 1`; the documented 9-dim vector was effectively 8-dim. Replaced with an explicit log band: $0.001 -> 0, $10 -> 1, log-interpolated between, monotonic and non-finite-safe. Exported as `normalizeCost()` with the band in `COST_BAND_USD`. 2. The verdict bands had two encodings, and they disagreed. `audit-trend.mjs` inlined the bands with a 0.80 `minor-drift` floor while the user guide documented 0.85; `drift-from-history` prints that verdict, so the two diverged in the 0.80-0.85 band. The bands now live once as `VERDICT_THRESHOLDS` + `verdictFor()` in `_similarity.mjs`, `audit-trend.mjs` imports them, and `similarity()` returns `verdict` beside `overall`. The documented 0.85 floor is canonical. Deliberate user-visible change: structural similarity in [0.80, 0.85) now reports `moderate-drift` instead of `minor-drift`. `--threshold` comparison is unaffected (it tests `overall`), but consumers branching on the verdict string should be re-checked. 3. The user guide's weight table said 0.4/0.3/0.3; ADR-152 and the code both say 0.6/0.25/0.15. Corrected the guide, which was the outlier. The ADR gains a dated Revision section recording both fixes and the behaviour change. The §Spike-result figures are unchanged (the LEGAL/SUPPORT fixtures are near-identical on the cost axis), so test-similarity.mjs Phase 8 keeps pinning the original anchors; new Phase 7b/7c/7d coverage pins the corrected semantics. Note this does NOT address the coarse-cosine limitation ADR-152 itself flags ("numerical cosine alone is too coarse"): uniform quality degradation still reads as high similarity. Drift gates that must catch absolute loss should pair structural similarity with the absolute gates rather than rely on --threshold. Verification: node plugins/ruflo-metaharness/scripts/test-similarity.mjs -> 128 passed, 0 failed (was 102) node plugins/ruflo-metaharness/scripts/_spike-similarity.mjs -> both invariants hold Co-authored-by: longxinjun-ai <longxinjun-ai@users.noreply.github.com> Co-authored-by: rUv <ruv@ruv.net>	2026-10-20 06:50:00 -04:00
ruv	9de004b3f6	feat(metaharness): ADR-152 spike landed — similarity invariants pass, ADR Accepted (iter 35) ADR-152 §"Smallest demonstrable spike" gate REACHED. Both invariants verified on synthetic fixtures; ADR-152 graduates Proposed → Accepted. MEASURED RESULTS similarity(LEGAL, LEGAL).overall = 1.0000 ✓ Invariant 1 (self-match exact) similarity(LEGAL, SUPPORT) = 0.8296 similarity(LEGAL, DEVOPS) = 0.5840 ✓ Invariant 2 (vertical affinity) Per-component (LEGAL vs SUPPORT): cosine=0.9987 categorical=0.75 jaccard=0.2857 Per-component (LEGAL vs DEVOPS): cosine=0.9734 categorical=0 jaccard=0 NOTABLE SPIKE FINDINGS - Numerical cosine alone is too coarse: LEGAL vs DEVOPS scored cosine 0.9734 despite being unrelated verticals. The numerics cluster around similar score ranges. Categorical + jaccard pulled the composite correctly to 0.58 vs 0.83. - The 0.6/0.25/0.15 weighting from ADR-152 §Decision reproduces the intended ordering on the synthetic LEGAL/SUPPORT/DEVOPS fixtures. - `categorical: 0` for LEGAL/DEVOPS correctly fires because they share no enum field (different archetypes, templates, modes). This is a strong feature of the design — categorical disagreement is a clean kill-switch for similarity. THE SPIKE (50 LOC) IMPLEMENTS - projectToVec() — 9-dim numerical feature vector - cosine() — over the 9 numerics - categoricalAgreement() — 4-field enum agreement / 4 - jaccard() — set similarity over agent_topology - similarity() — weighted composite per ADR-152 ARCHITECTURAL CONSTRAINTS (all four preserved) - Removable ✓ pure-TS, reads JSON the CLI emits - Optional ✓ no new @metaharness/* dep - Graceful ✓ guards on missing fields, never throws - CI gate ✓ standalone unit-runnable, no npx required LIFECYCLE NOTE The spike STAYS in the codebase as the regression-suite anchor for the two invariants. When the full feature lands (ADR-152 implementation PR), it goes into `_similarity.mjs`; this `_spike-similarity.mjs` remains as the gate that any future weight tweak must still pass. DOCS UPDATED - ADR-152 status: Proposed → Accepted, with measured spike results and per-component breakdown inlined - ADR-151 §3.1 marker upgraded: "DRAFTED iter 34" → "ACCEPTED iter 35 — spike landed, both invariants pass" SMOKE STEP 17x - Spike script exists, parses, declares the 3-component formula literal - Both invariants explicitly named in source - LEGAL/SUPPORT/DEVOPS fixtures present (anti-regression on the spike) - Fail-closed exit - ADR-152 status = Accepted in source - ADR-151 §3.1 marker upgraded Smoke: 39 → 40. Fleet: 33/33. The Phase-3 critical path is now unblocked. ADR-153 (Recommendation Engine), ADR-154 (Fleet Drift), and ADR-156 (Plugin Compat) can begin once the user/maintainer decides to start them — they all consume similarity() as the iter-35 spike implements it. Co-Authored-By: RuFlo <ruv@ruv.net>	2026-06-16 16:17:05 -04:00
ruv	06852f991b	docs(adr): ADR-152 — Genome Similarity Search (Phase 3 §3.1 critical path, iter 34) ADR-151 named §3.1 (Genome Similarity Search) as the critical-path dependency for §3.2/§3.3/§3.5 — three of four Phase-3 consumers. This ADR drafts the specific decision: how to compute similarity between two MetaHarness genome+score JSON blobs, with what weights, returning what shape. THE DECISION (one pure-TS function, 3 components, weighted composite) overall = 0.6 · cosine + 0.25 · categorical + 0.15 · jaccard cosine 9-dim numerical feature vector projected from: score: harnessFit, compileConfidence, taskCoverage, toolSafety, memoryUsefulness (each ÷ 100) genome: risk_score, test_confidence, publish_readiness score: estCostPerRunUsd (log-transformed, clamped) categorical enum agreement over repo_type / archetype / template / recommendedMode (4 fields, sum/4) jaccard set similarity over agent_topology[] arrays RETURN SHAPE (codified, so consumers can branch deterministically) { overall, components: {cosine, categorical, jaccard}, perDimension: { ... contribution per field ... } } ARCHITECTURAL INVARIANTS (all inherited from ADR-150 via ADR-151) - Pure-TS function, no @metaharness/* dep - Reads JSON the existing CLI emits — zero new instrumentation - Failure modes limited to garbage-in → low-confidence-out, never throw - Standalone unit-testable, CI gate satisfied trivially SMALLEST DEMONSTRABLE SPIKE (the implementation gate) Before ADR-152 → Accepted, ship a 30-LOC proof that: 1. similarity(X, X) === 1 2. similarity(legal, devops) < similarity(legal, support) using the 13 metaharness verticals as fixtures Spike script lives in plugin, deleted after graduation. WEIGHTS RATIONALE - 0.6 cosine: most signal density (9 dims of structured numerics) - 0.25 categorical: high-signal but low-cardinality (4 enum fields) - 0.15 jaccard: tie-breaker (interpretable but coarse) - Defaults documented + CLI-overridable, fixtures lock the weighting as a regression suite so future weight changes are visible THREE ALTERNATIVES CONSIDERED + REJECTED - A: sentence embeddings over README/prompts — deferred to §3.4 (graph) where it's actually needed; would break ADR-150 removability - B: skip cosine, categorical+jaccard only — loses 9-dim signal - C: train an ML similarity model — circular (recommender produces the labels), premature OPEN QUESTIONS (for spike phase) - Per-org weight tuning via mcp-policy.json? Lean no — global default - Optional mermaid diagram output? Probably yes post-spike, cheap CROSS-LINKS UPDATED - ADR-151 §3.1 marker upgraded from "to be drafted" → "DRAFTED iter 34" - Smoke step 17w locks the 9 numerical-feature names + the weight formula literal + the spike contract section + the parent back-ref Smoke: 38 → 39. Fleet: 33/33. No code yet — this is scope-only per ADR-151's gate. Spike is the next move when work starts. ADR-CHAIN STATUS ADR-150 [Implemented] — Phase 1+2 shipped ADR-151 [Proposed] — Phase 3 scope (5 sub-capabilities) ADR-152 [Proposed] — §3.1 Genome Similarity Search (this) ADR-153 [-] — §3.2 Recommendation Engine (consumes 152) ADR-154 [-] — §3.3 Fleet Drift Detection (consumes 152) ADR-155 [-] — §3.4 Capability Graph ADR-156 [-] — §3.5 Plugin Compatibility (consumes 152) Co-Authored-By: RuFlo <ruv@ruv.net>	2026-06-16 16:11:46 -04:00

Author

SHA1

Message

Date

c2069ed6f6

fix(metaharness): revive the cost dimension and single-source the verdict bands (ADR-152) (#3520 )

Reconciling the shipped similarity code against docs/metaharness-user-guide.md
turned up three divergences between what ADR-152 specifies, what the code does,
and what the docs promise.

1. Cost normalization (vector index 8) was inert below $0.999.

   ADR-152 Table 1 specified `log10(usd + 0.001) / log10(10)` clamped to [0,1].
   `Math.log10(10)` is 1, so the division was a no-op and the expression reduced
   to `clamp(log10(usd + 0.001), 0, 1)`. Because log10 is negative for every
   usd < 0.999, the clamp flattened that entire range to a constant 0 — including
   the $0.048 baseline the plugin README records. Two harnesses differing only in
   cost scored `cosine === 1`; the documented 9-dim vector was effectively 8-dim.

   Replaced with an explicit log band: $0.001 -> 0, $10 -> 1, log-interpolated
   between, monotonic and non-finite-safe. Exported as `normalizeCost()` with the
   band in `COST_BAND_USD`.

2. The verdict bands had two encodings, and they disagreed.

   `audit-trend.mjs` inlined the bands with a 0.80 `minor-drift` floor while the
   user guide documented 0.85; `drift-from-history` prints that verdict, so the
   two diverged in the 0.80-0.85 band. The bands now live once as
   `VERDICT_THRESHOLDS` + `verdictFor()` in `_similarity.mjs`, `audit-trend.mjs`
   imports them, and `similarity()` returns `verdict` beside `overall`. The
   documented 0.85 floor is canonical.

   Deliberate user-visible change: structural similarity in [0.80, 0.85) now
   reports `moderate-drift` instead of `minor-drift`. `--threshold` comparison is
   unaffected (it tests `overall`), but consumers branching on the verdict string
   should be re-checked.

3. The user guide's weight table said 0.4/0.3/0.3; ADR-152 and the code both say
   0.6/0.25/0.15. Corrected the guide, which was the outlier.

The ADR gains a dated Revision section recording both fixes and the behaviour
change. The §Spike-result figures are unchanged (the LEGAL/SUPPORT fixtures are
near-identical on the cost axis), so test-similarity.mjs Phase 8 keeps pinning
the original anchors; new Phase 7b/7c/7d coverage pins the corrected semantics.

Note this does NOT address the coarse-cosine limitation ADR-152 itself flags
("numerical cosine alone is too coarse"): uniform quality degradation still
reads as high similarity. Drift gates that must catch absolute loss should pair
structural similarity with the absolute gates rather than rely on --threshold.

Verification:
  node plugins/ruflo-metaharness/scripts/test-similarity.mjs   -> 128 passed, 0 failed (was 102)
  node plugins/ruflo-metaharness/scripts/_spike-similarity.mjs -> both invariants hold

Co-authored-by: longxinjun-ai <longxinjun-ai@users.noreply.github.com>
Co-authored-by: rUv <ruv@ruv.net>

2026-10-20 06:50:00 -04:00

ruv

9de004b3f6

feat(metaharness): ADR-152 spike landed — similarity invariants pass, ADR Accepted (iter 35)

ADR-152 §"Smallest demonstrable spike" gate REACHED. Both invariants
verified on synthetic fixtures; ADR-152 graduates Proposed → Accepted.

MEASURED RESULTS
  similarity(LEGAL, LEGAL).overall = 1.0000  ✓ Invariant 1 (self-match exact)
  similarity(LEGAL, SUPPORT)       = 0.8296
  similarity(LEGAL, DEVOPS)        = 0.5840  ✓ Invariant 2 (vertical affinity)

  Per-component (LEGAL vs SUPPORT):  cosine=0.9987  categorical=0.75  jaccard=0.2857
  Per-component (LEGAL vs DEVOPS):   cosine=0.9734  categorical=0     jaccard=0

NOTABLE SPIKE FINDINGS
- Numerical cosine alone is too coarse: LEGAL vs DEVOPS scored cosine
  0.9734 despite being unrelated verticals. The numerics cluster around
  similar score ranges. Categorical + jaccard pulled the composite
  correctly to 0.58 vs 0.83.
- The 0.6/0.25/0.15 weighting from ADR-152 §Decision reproduces the
  intended ordering on the synthetic LEGAL/SUPPORT/DEVOPS fixtures.
- `categorical: 0` for LEGAL/DEVOPS correctly fires because they share
  no enum field (different archetypes, templates, modes). This is a
  strong feature of the design — categorical disagreement is a clean
  kill-switch for similarity.

THE SPIKE (50 LOC) IMPLEMENTS
- projectToVec()         — 9-dim numerical feature vector
- cosine()               — over the 9 numerics
- categoricalAgreement() — 4-field enum agreement / 4
- jaccard()              — set similarity over agent_topology
- similarity()           — weighted composite per ADR-152

ARCHITECTURAL CONSTRAINTS (all four preserved)
- Removable      ✓ pure-TS, reads JSON the CLI emits
- Optional       ✓ no new @metaharness/* dep
- Graceful       ✓ guards on missing fields, never throws
- CI gate        ✓ standalone unit-runnable, no npx required

LIFECYCLE NOTE
The spike STAYS in the codebase as the regression-suite anchor for
the two invariants. When the full feature lands (ADR-152 implementation
PR), it goes into `_similarity.mjs`; this `_spike-similarity.mjs`
remains as the gate that any future weight tweak must still pass.

DOCS UPDATED
- ADR-152 status: Proposed → Accepted, with measured spike results
  and per-component breakdown inlined
- ADR-151 §3.1 marker upgraded: "DRAFTED iter 34" → "ACCEPTED iter 35
  — spike landed, both invariants pass"

SMOKE STEP 17x
- Spike script exists, parses, declares the 3-component formula literal
- Both invariants explicitly named in source
- LEGAL/SUPPORT/DEVOPS fixtures present (anti-regression on the spike)
- Fail-closed exit
- ADR-152 status = Accepted in source
- ADR-151 §3.1 marker upgraded

Smoke: 39 → 40. Fleet: 33/33.

The Phase-3 critical path is now unblocked. ADR-153 (Recommendation
Engine), ADR-154 (Fleet Drift), and ADR-156 (Plugin Compat) can begin
once the user/maintainer decides to start them — they all consume
similarity() as the iter-35 spike implements it.

Co-Authored-By: RuFlo <ruv@ruv.net>

2026-06-16 16:17:05 -04:00

ruv

06852f991b

docs(adr): ADR-152 — Genome Similarity Search (Phase 3 §3.1 critical path, iter 34)

ADR-151 named §3.1 (Genome Similarity Search) as the critical-path
dependency for §3.2/§3.3/§3.5 — three of four Phase-3 consumers. This
ADR drafts the specific decision: how to compute similarity between
two MetaHarness genome+score JSON blobs, with what weights, returning
what shape.

THE DECISION (one pure-TS function, 3 components, weighted composite)

  overall = 0.6 · cosine + 0.25 · categorical + 0.15 · jaccard

  cosine       9-dim numerical feature vector projected from:
                  score:  harnessFit, compileConfidence, taskCoverage,
                          toolSafety, memoryUsefulness (each ÷ 100)
                  genome: risk_score, test_confidence, publish_readiness
                  score:  estCostPerRunUsd (log-transformed, clamped)
  categorical  enum agreement over repo_type / archetype / template /
               recommendedMode (4 fields, sum/4)
  jaccard      set similarity over agent_topology[] arrays

RETURN SHAPE (codified, so consumers can branch deterministically)
  { overall, components: {cosine, categorical, jaccard},
    perDimension: { ... contribution per field ... } }

ARCHITECTURAL INVARIANTS (all inherited from ADR-150 via ADR-151)
- Pure-TS function, no @metaharness/* dep
- Reads JSON the existing CLI emits — zero new instrumentation
- Failure modes limited to garbage-in → low-confidence-out, never throw
- Standalone unit-testable, CI gate satisfied trivially

SMALLEST DEMONSTRABLE SPIKE (the implementation gate)
Before ADR-152 → Accepted, ship a 30-LOC proof that:
  1. similarity(X, X) === 1
  2. similarity(legal, devops) < similarity(legal, support)
     using the 13 metaharness verticals as fixtures
Spike script lives in plugin, deleted after graduation.

WEIGHTS RATIONALE
- 0.6 cosine: most signal density (9 dims of structured numerics)
- 0.25 categorical: high-signal but low-cardinality (4 enum fields)
- 0.15 jaccard: tie-breaker (interpretable but coarse)
- Defaults documented + CLI-overridable, fixtures lock the weighting
  as a regression suite so future weight changes are visible

THREE ALTERNATIVES CONSIDERED + REJECTED
- A: sentence embeddings over README/prompts — deferred to §3.4 (graph)
     where it's actually needed; would break ADR-150 removability
- B: skip cosine, categorical+jaccard only — loses 9-dim signal
- C: train an ML similarity model — circular (recommender produces
     the labels), premature

OPEN QUESTIONS (for spike phase)
- Per-org weight tuning via mcp-policy.json? Lean no — global default
- Optional mermaid diagram output? Probably yes post-spike, cheap

CROSS-LINKS UPDATED
- ADR-151 §3.1 marker upgraded from "to be drafted" → "DRAFTED iter 34"
- Smoke step 17w locks the 9 numerical-feature names + the weight
  formula literal + the spike contract section + the parent back-ref

Smoke: 38 → 39. Fleet: 33/33. No code yet — this is scope-only per
ADR-151's gate. Spike is the next move when work starts.

ADR-CHAIN STATUS
  ADR-150 [Implemented]   — Phase 1+2 shipped
  ADR-151 [Proposed]      — Phase 3 scope (5 sub-capabilities)
  ADR-152 [Proposed]      — §3.1 Genome Similarity Search (this)
  ADR-153 [-]             — §3.2 Recommendation Engine (consumes 152)
  ADR-154 [-]             — §3.3 Fleet Drift Detection (consumes 152)
  ADR-155 [-]             — §3.4 Capability Graph
  ADR-156 [-]             — §3.5 Plugin Compatibility (consumes 152)

Co-Authored-By: RuFlo <ruv@ruv.net>

2026-06-16 16:11:46 -04:00