diff --git a/docs/dream-cycle/LEDGER.md b/docs/dream-cycle/LEDGER.md index 3cd198a309..34499b14dc 100644 --- a/docs/dream-cycle/LEDGER.md +++ b/docs/dream-cycle/LEDGER.md @@ -188,3 +188,4 @@ as a first-class, high-priority candidate, not a background note. | 2026-09-09 | swarm | AgentPool.performHealthChecks() stamped lastHeartbeat=now on every "still healthy" tick, and updateAgentHeartbeat() (the only other legitimate way to advance it) has zero callers anywhere in v3/ — so the check's own tick was the sole thing keeping every pooled agent's heartbeat fresh, meaning timeSinceLastActivity could never exceed one interval and replaceUnhealthyAgent() could never fire for any agent, in any pool, ever; removed the self-stamp | #3242 | #3243 | yes | ACCEPT-scoped | 223/223 tests (+3 new), baseline-fails-2-of-3/candidate-passes-3-of-3 via stash isolation; adversarial critic confirmed independently but disclosed @claude-flow/swarm has no reachable path through the shipped @claude-flow/cli today (blast radius is the swarm package's own UnifiedSwarmCoordinator, not an active CLI hot path) | 008c96a0... | 08-24..09-02 backlog unchanged (see note above); #3169/#3184/#3221 merged since 09-05 | | 2026-09-01 | security | `maxToolCallsPerTurn` was session-lifetime cumulative, never reset (disclosed bug in #3139); wall-clock sliding window fix, new `turnWindowMs` policy field (follow-up to #3139, stacked PR) | #3151 | #3152 | yes | ACCEPT | 30/30 new-suite tests, 52/56 broader MCP sweep (4 pre-existing unbuilt-sibling-package env failures, unrelated); independent critic CONFIRMED-WITH-CAVEATS, 2 minor findings fixed pre-commit | 7d93a482... | rebased onto main (post #3086/#3094/#3103/#3110/#3119/#3123/#3130/#3139 merges), awaiting fresh CI as of 2026-09-14 | | 2026-09-02 | intelligence | `LearningBridge.consolidate()` reward-blind fix (hardcoded `1.0` → real insight confidence via `getByKey`; round-1 fix had a wrong id/key lookup — `entryId` is the caller-assigned `key`, not the backend's internal `entry.id` — caught by adversarial critic, corrected round 2) | #3159 | #3160 | yes | ACCEPT | 56/56→62/62 tests (candidate); 464/465 full package (1 pre-existing environmental failure); independent critic round 2 CONFIRMED | bead6402... | rebased onto main (post #3086/#3094/#3103/#3110/#3119/#3123/#3130/#3139/#3152 merges), awaiting fresh CI as of 2026-09-14 | +| 2026-09-08 | memory | `MemoryConsolidator.dedup()` deduplicated via byte-exact SHA-256 hash only, though every entry already carries a computed `.embedding` and `dedup()` already holds a handle to the adapter's incrementally-synced `HNSWIndex` (used only for `removePoint` bookkeeping, never queried for similarity) — added a second embedding-cosine near-duplicate pass (threshold 0.95, zero extra backend round-trips); independent adversarial critique found and this session fixed forward a real convergence bug (a duplicate cluster larger than the 8-entry search window split into multiple survivors in one call, self-healing across repeated `runAll()` but under-converging on a single call) | #3231 | #3232 | yes | ACCEPT | 472/473 full-suite green (1 pre-existing unrelated failure, unchanged); baseline-fails/candidate-passes via stash isolation for both the original fix and the convergence fix-forward; `tsc` clean | f5eaa461... | rebased onto main (post #3086/#3094/#3103/#3110/#3119/#3123/#3130/#3139/#3152/#3160 merges), awaiting fresh CI as of 2026-09-14 | diff --git a/docs/dream-cycle/dream-gist-2026-09-08.md b/docs/dream-cycle/dream-gist-2026-09-08.md new file mode 100644 index 0000000000..5c53eae7af --- /dev/null +++ b/docs/dream-cycle/dream-gist-2026-09-08.md @@ -0,0 +1,124 @@ +# Memory SOTA Report — 2026-09-08 + +TL;DR: Tonight's memory deep-dive confirmed and closed a gap flagged (but not selected) by last night's memory-scan finding: `MemoryConsolidator.dedup()` (`v3/@claude-flow/memory/src/consolidator.ts`) only ever collapsed duplicates via byte-exact SHA-256 content hashing, even though every stored entry already carries a computed `.embedding` and the method already holds a handle to the adapter's incrementally-synced `HNSWIndex` — used only for point-removal bookkeeping, never queried for similarity. This is the same "signal already computed, live path discards it for a worse proxy" bug class as three recently-fixed/merged findings (2026-09-03 MMR/#3169, 2026-08-25 PQ-dispatch/#3094, 2026-08-28 HybridBackend-weights/#3119). Five parallel research roles converged independently on the same candidate and patch shape. An orphaned second implementation, `MemoryDomainService.consolidate()` (~700 LOC, zero tests, unreachable from the package's public exports), already does real embedding-cosine grouping but was confirmed NOT directly reusable (it's bound to an injected `IMemoryRepository` requiring a real per-entry `searchByVector()` round-trip; the in-scope `HNSWIndex` handle is strictly cheaper). Fixed by adding an embedding near-duplicate pass directly in `consolidator.ts`, reusing the already-loaded embedding and index with zero extra backend round-trips. An independent adversarial critic then found and this session fixed a real, bounded convergence bug in the first cut of the patch (duplicate clusters wider than the 8-entry search window split into multiple survivors within one call) — documented in full below, including the reproduction and the fix. + +## What's New in 2026 + +| Finding | Source | Confidence | +|---|---|---| +| SemDedup: embed → cluster → intra-cluster cosine-threshold removal, up to ~50% dedup with minimal quality loss | [NVIDIA NeMo Curator SemDedup docs](https://docs.nvidia.com/nemo/curator/curate-text/process-data/deduplication/semdedup) (implements arXiv:2303.09540) | A — reproducible open-source pipeline | +| CrewAI's native `Memory` class: explicit 0.85 cosine-similarity gate → LLM adjudicates keep/update/delete/insert | [CrewAI docs v1.15.2](https://docs.crewai.com/v1.15.2/en/concepts/memory) | B — official vendor docs | +| Mem0: embed-retrieve-similar → single LLM call decides ADD/UPDATE/DELETE/NOOP; LOCOMO 26% relative LLM-judge improvement over OpenAI Memory | arXiv:2504.19413 (ECAI 2025) | B — published, vendor-run benchmark | +| Zep/Graphiti: bi-temporal knowledge graph, entity resolution at write-time, contradictions close a validity window rather than delete; DMR 94.8% vs MemGPT 93.4% | arXiv:2501.13956 | B — published, vendor-run benchmark | +| TiMem Temporal Memory Tree: hierarchical "semantic-guided consolidation... without fine-tuning"; LoCoMo 75.30% with 52.2% reduction in recalled memory length | arXiv:2601.02845 (Jan 2026) | B — single arXiv preprint, concrete numbers, not independently reproduced | +| Qdrant maintainer, on record: no out-of-the-box memory-dedup feature is planned — "each dataset require[s] individual calibration" | [Qdrant GH discussion #3268](https://github.com/orgs/qdrant/discussions/3268) | B — primary-source maintainer statement | +| Mandol: agglomerative clustering + LLM cluster-synthesis for conversational agent memory, benchmarked against Mem0/Zep | arXiv:2606.29778 (Jun 2026) | C — single paper, not yet reproduced | +| Hindsight practitioner blog: "Most agent memory systems are explicit about retrieval and silent about consolidation policy. That silence is where production failures live." Four-lever framework (Importance/Merge/Decay/Eviction) | [hindsight.vectorize.io, 2026-05-21](https://hindsight.vectorize.io/blog/2026/05/21/agent-memory-consolidation) | C — vendor blog, self-reported numbers | +| Mem0's Apr-2026 rework reportedly drops UPDATE/DELETE for single-pass ADD-only accumulation — a direction *opposite* embedding-based consolidation, an open debate not settled practice | [mem0.ai blog, "State of AI Agent Memory 2026"](https://mem0.ai/blog/state-of-ai-agent-memory-2026) | C — vendor blog, unverified | + +Dominant 2025-2026 direction (SemDedup, TiMem, Zep, CrewAI, Mem0) is cluster/embed-then-threshold dedup — exactly what the orphaned `MemoryDomainService` already does and what tonight's fix adds to the wired path. This independently corroborates last night's diagnosis rather than pointing Ruflo toward a different approach. + +## Ruflo Current Capability + +`MemoryConsolidator.dedup()` (`v3/@claude-flow/memory/src/consolidator.ts:140-210` pre-fix) buckets every entry by `sha256(content)` and merges byte-exact matches only — structurally blind to paraphrases/reformattings. It's invoked via `runAll()` (sweep → dedup → compact), reached by `UnifiedMemoryService`'s background timer (opt-in, `config.consolidator.autoRun`) and by the AgentDB `nightlyLearner` controller. + +At the call site, `entries: Map` is already fully resident with each entry's `.embedding?: Float32Array` (`types.ts:66`) inline, and `index: HNSWIndex` is already an in-scope, incrementally-synced handle exposing `.search(query, k, ef?)` — confirmed via direct read of `agentdb-adapter.ts` (index kept in sync on every write/delete) and `hnsw-index.ts` (`search()` is pure in-process graph traversal, no I/O). Fetching a similarity signal here costs zero extra backend round-trips. + +Ruflo also ships a second, orphaned consolidation implementation: `MemoryDomainService.consolidate()` (`domain/services/memory-domain-service.ts:135-331`) does real embedding-cosine grouping (thresholds 0.95 dedupe / 0.8 merge) via `repository.searchByVector()` — but `repository: IMemoryRepository` is an injected interface requiring a genuine async round-trip *per entry* for a real backend, and the concrete backing repository (`HybridMemoryRepository`) does its own brute-force O(n) linear scan, not even HNSW-backed. The whole domain+application DDD chain dead-ends at its own barrel exports — confirmed via repo-wide import grep — and is explicitly excluded from the package's public surface (`index.test.ts:43` asserts `HybridMemoryRepository` must not be exported), consistent with a deliberate ADR-125-style quarantine rather than pure oversight. Zero test files exist for any file in this chain. **Conclusion, independently reached by two research roles**: wiring in the full domain-service architecture is not the smallest fix and was not selected; the in-scope `HNSWIndex` + already-resident `.embedding` is a strictly cheaper, more direct fix. + +One more live gap surfaced while verifying the above (not previously documented): `dedup()`'s per-entry removal loop deletes IDs from `namespaceIndex`/`tagIndex` but — unlike `sweepExpired()`, which explicitly prunes now-empty `Set`s — never removes an emptied `Set` from its parent `Map`, so these maps grow unboundedly across many dedup cycles in a long-lived process. Not fixed tonight (separate, smaller, lower-value finding — see Additional Candidates below). + +## Competitor Comparison + +| System | Mechanism | Evidence Grade | +|---|---|---| +| LangGraph / LangMem SDK | LLM-based background consolidation (`BackgroundMemoryManager`), parallel LLM tool calls decide create/update/delete; no fixed numeric threshold | B | +| AutoGen | No native consolidation; delegates entirely to plugged-in Mem0/Zep | B | +| CrewAI | Embedding-cosine gate (0.85) → LLM adjudicates keep/update/delete/insert | B | +| Mem0 | Embed-retrieve-similar → LLM decides ADD/UPDATE/DELETE/NOOP | B | +| Zep / Graphiti | Bi-temporal knowledge-graph entity resolution + validity-window supersession | B | +| Qdrant / Weaviate / Milvus / LanceDB | **None shipped, deliberately** — maintainer on record: no out-of-the-box solution, "each dataset requires individual calibration" | B (deliberate-avoidance, not oversight) | +| NVIDIA NeMo Curator SemDedup | Embed + cluster dedup for *training* corpora — never adapted to live agent memory in anything found tonight (a genuine, undisputed gap) | B for SemDedup itself; the "agent-memory gap" is a negative result across the search | + +Ruflo's hash-exact-only baseline sat below every framework-layer competitor here and at parity with the vector-DB layer's default of doing nothing. Tonight's fix — an explicit, documented, conservative cosine threshold (0.95) feeding the existing keeper-selection logic — moves Ruflo onto the same mechanism class as CrewAI's 0.85 gate, the most reproducible and simplest-to-benchmark option in this table (versus Mem0/Zep/LangMem's non-deterministic LLM-mediated judging, or Graphiti's much heavier graph-structural machinery). It does not reach Mem0/Zep's *contradiction resolution* (rewriting superseded content, not just dropping a near-duplicate) — a legitimate next rung, out of scope tonight. + +## Hypothesis + +> Given `MemoryConsolidator.dedup()` operating on entries that already carry a resident `.embedding: Float32Array` and an already-populated, incrementally-synced `HNSWIndex` handle it holds but never queries for similarity, when a near-duplicate detection pass is added that queries `index.search()` for each hash-pass-surviving entry with an embedding and merges any group whose cosine similarity is at or above a configurable `similarityThreshold` (default 0.95, matching the orphaned domain-service's own constant) using the existing keeper-selection strategies, then `dedup()` should catch semantic near-duplicates (paraphrases/reformattings) that byte-exact SHA-256 hashing structurally cannot, relative to baseline (hash-exact-only, near-duplicates never merged), subject to: (1) all pre-existing hash-exact-dedup tests and behavior remain unchanged; (2) the near-dup pass is strictly additive — no survivor is dropped without an above-threshold match, and hash-exact duplicates are still merged even without embeddings; (3) entries without `.embedding`, or an index configured with a non-cosine metric, fall back to exactly today's hash-only behavior; (4) a single `dedup()` call fully converges a duplicate cluster of any size to one survivor, not just clusters within the search-neighborhood width; (5) $0 evaluation cost, fully deterministic test-only evaluation, zero LLM calls. + +Frozen before evaluation began. Condition (4) was added mid-session after the adversarial critic (STEP 10) reproduced a violation of it in the first cut of the implementation — see Evaluation below for why this is a legitimate hypothesis refinement (a genuinely new, previously-unstated condition the critique surfaced) rather than a post-hoc loosening: the fix was applied to satisfy the *stronger* condition, not to relax the hypothesis to match a weaker result. + +## Benchmarks / Evaluation + +**evaluated: accepted (ACCEPT).** Real evaluator: Vitest 4.1.8, deterministic, zero LLM calls, $0 cost. + +**Candidate diff** (2 commits on `dream/2026-09-08-memory`): `v3/@claude-flow/memory/src/consolidator.ts` — `selectKeeper`/`mergeGroup` helpers factored out of the original hash-pass logic and reused by a new embedding near-duplicate pass, looped to a fixed point (see Adversarial Critique below for why). New `ConsolidatorOptions.similarityThreshold` (default 0.95; `>= 1` disables the pass entirely). `v3/@claude-flow/memory/src/consolidator.test.ts` — 7 new tests. + +**Baseline vs. candidate, isolated via `git stash` of just the source file (test file kept):** +- First cut (single-round near-dup pass): baseline fails exactly the 2 tests requiring real cosine-merge behavior (`expected 0 to be 1`); candidate passes all 15. 13/15 pass on baseline (all pre-existing + 4 new guard/fallback tests that hold under both old and new code). +- After the convergence fix: re-ran the same isolation against the *first-cut* candidate specifically for the new 15-entry-cluster regression test — it fails (`expected 13 to be 14`, i.e. 2 leftover survivors instead of 1), reproducing the critic's exact finding; the fixed-point-loop version passes (16/16). + +**Full `@claude-flow/memory` package suite**: **472/473 passing** (466 pre-existing + 7 new; 1 pre-existing, unrelated, environmental failure — `auto-memory-bridge.test.ts`, a chmod-based read-only-file test that can't enforce permissions in this root-owned sandbox, confirmed identical with and without this diff and matching the architecture-reviewer's independently-measured pre-change baseline of 465/466). `tsc --noEmit`: zero errors. + +**Adversarial critique (STEP 10, independent subagent, no authoring context)** — verdict **CONFIRMED-WITH-CAVEATS**, full 14-point checklist run and answered with evidence (re-derived from the diff, not trusted from any prior claim): +- No test weakened (diff purely additive on the test file — confirmed via direct diff of the pre-existing 9 tests, byte-identical). +- No gold/expected data touched; no cherry-picking (ran the full suite itself: 471/472 on the first cut, matching what's claimed above pre-convergence-fix). +- No cost/latency/I/O regression — read `hnsw-index.ts`'s `search()`/`removePoint()` directly and confirmed pure in-memory graph traversal. +- Only one production call site (`controller-registry.ts:1021`, `nightlyLearner`) plus `MemoryService.getConsolidator()` — confirmed via repo-wide grep. **Disclosed, not hidden**: `getConsolidator()` does not override `similarityThreshold`, so the near-dup pass is **on by default at 0.95 for every existing `MemoryService` consumer**, not opt-in — a real behavior change for current users of `runAll()`/the background auto-run timer, not just new callers. Judged acceptable given the conservative default and the precedent set by 2026-09-03's MMR fix (also enabled by default), but explicitly flagged here rather than left implicit. +- No undocumented caching, no threshold gaming, no leaked expected answers, no security-boundary crossing (pure `Map`/`HNSWIndex` operations). +- Dimension-agnostic by inspection (no literal dimension anywhere in the new code path). +- **Found a real bug** (the convergence gap described above) — reported with a precise reproduction (15 pairwise-identical-embedding entries, `merged: 13` instead of 14, i.e. 2 survivors instead of 1) and fixed in this session per the report, with a new discriminating regression test added and confirmed to fail pre-fix / pass post-fix. + +## Darwin Results + +Real interface confirmed tonight (`ruvector harness darwin --help`): `darwin --execute`, JSON-genome-driven, requires an explicit `--execute` flag to authorize candidate execution. **Skipped, deliberately, with a reason distinct from most recent nights' skips**: unlike the last several accepted candidates (binary invariant/wiring fixes with no continuous parameter), tonight's `similarityThreshold` *is* a genuine continuous tunable with a real quality/false-positive-rate tradeoff — in principle Darwin-eligible. It was not run tonight because doing so honestly would require a labeled near-duplicate benchmark corpus (true-duplicate vs. true-distinct pairs with ground truth) that does not exist in this repo, and building one is scoped out tonight per STEP 0.6's budget protection (research depth over new evaluation infrastructure). The shipped default (0.95) is not arbitrary — it matches the orphaned domain-service's own already-chosen constant and CrewAI's closely-adjacent 0.85 gate — but a future `memory` DEEP night building a small labeled near-dup corpus and running bounded Darwin evolution over this exact parameter is a concrete, well-scoped follow-up (see Recommended Next Steps). + +## Flywheel Evidence + +Real interface confirmed tonight (`ruvector harness flywheel --help`): `flywheel verify ` / `flywheel gate `. No signed `@metaharness/flywheel` bundle exists for this candidate — the schema targets retrieval-policy candidates evaluated against an LLM-task corpus via paired bootstrap statistics; this is a deterministic code-correctness fix with no such corpus, consistent with every accepted dream-cycle night since 2026-08-18. Evidence retained as: 7 new deterministic tests (including the critic-driven convergence regression) + this gist + the linked issue + the two stash-isolated baseline-fails/candidate-passes comparisons (first-cut and convergence-fix) + the independent adversarial critique. + +Classified: **OBSERVATION** (byte-exact-hash-only dedup, `HNSWIndex` handle held but never queried for similarity — confirmed via direct read + repo-wide grep by two independent research roles) / **MEASUREMENT** (baseline fails exactly the discriminating tests both before and after the convergence fix; full-suite pass count 472/473, unchanged failure identity from the pre-existing baseline) / **INFERENCE** (the orphaned `MemoryDomainService` is not directly reusable — its `IMemoryRepository` abstraction requires real per-entry round-trips; the in-scope `HNSWIndex` is strictly cheaper) / **DECISION** (ship the minimal in-file wiring fix, not the domain-service swap; fix the convergence bug forward rather than merely disclosing it, since a working fix was cheap and available) / **REJECTION** (Darwin evolution of `similarityThreshold`, deferred — no labeled corpus exists; not attempted with a fabricated one). + +## Reward Hack Check + +No standalone generic reward-hack CLI reachable this session (`weight-eft` resolves to a bin name but is a LoRA-distillation tool per its own doctor description, not a diff/benchmark scanner — consistent with every recent night's note). Manual checklist, independently re-verified by the adversarial critic with fresh eyes: no test weakened (purely additive, confirmed via diff); no gold-label leakage (no LLM/gold-answer path in this code); no cherry-picking (full package suite reported both ways, honestly, including the one pre-existing unrelated failure); no seed manipulation (fully deterministic PRNG-seeded embeddings throughout); zero cost (no model calls anywhere); no undocumented caching. The one caveat disclosed rather than hidden: the near-dup pass is on-by-default for existing `MemoryService` consumers (see Evaluation above) — a real behavior change, not a hidden side effect, but worth a reviewer's explicit attention. + +## Security Review + +Not the primary focus, but directly relevant to data-retention correctness: this pass *deletes* entries judged near-duplicate, so an adversary who could influence embedding generation to force a coincidental high-cosine collision with a legitimate memory could, in principle, cause that memory to be silently dropped during consolidation. Mitigations already in place: the default threshold (0.95) is conservative and requires near-exact vector alignment, not merely "related" content; the pass only runs during consolidation (background timer or `nightlyLearner`), not on the synchronous write/store path, so it isn't reachable from a single malicious write in real time; and it operates purely on already-trusted, already-embedded internal state — no new network, filesystem, or credential surface is introduced (confirmed line-by-line by the adversarial critic). No fix required tonight; flagging the threshold-as-attack-surface framing for future `security`-DEEP consideration if Ruflo's embedding-generation path itself is ever exposed to less-trusted input. + +## Scan Findings: plugins + +Re-verified last night's finding (`PluginIntegrityVerifier` unwired, registry `checksum` fields placeholder, `requireVerification` enforced at the registry-listing level only) — **confirmed still accurate**, with new concrete artifacts found tonight: (1) `commands/plugins.ts`'s `--verify` flag (`default: true`, advertised to users) is completely dead — the captured `verify` variable is never read again, so `--verify` and `--no-verify` are byte-identical no-ops; (2) six more `PluginStoreConfig` fields (`requireSecurityAudit`, `minTrustLevel`, `trustedAuthors`, `blockedPlugins`, `requirePermissionPrompt`, `allowedPermissions`) are declared, defaulted, and never read anywhere; (3) `trust-anchors.json` ships a placeholder all-zero public key, so even wiring the verifier in today would validate nothing real. Competitor check: npm's own provenance/Sigstore attestation and VS Code Marketplace's install-time repository-signature verification both show install-time artifact verification is the industry norm for a plugin marketplace shaped like Ruflo's (Grade A sources, both). **Explicit novelty verdict: this is a refinement of a direction already surfaced 3+ times in the ledger** (#2254 2026-06-01, #2630 2026-07-11, last night 2026-09-07) — not filed as a new top-level issue; recorded here as supporting evidence for a future `security`-DEEP night, per STEP 1.1. + +## Scan Findings: automation + +`dream-cycle-backlog-guard.yml` (referenced in the ledger's own notes as addressing the dream-cycle-PR-backlog visibility gap) **does not exist on `main`** — it exists only inside still-open, still-draft PR #3205 (created 2026-09-05, zero comments, untouched since creation — 3 days stale as of tonight). Read directly from the PR diff: it is a **dashboard, not a fix** — a daily cron job that lists aging `dream/*` PRs into a GitHub Step Summary with a `::warning::` annotation past a configurable age threshold; its own header comment says so explicitly ("does not fix the append mechanism... adds the missing visibility"). No other open PR or issue addresses the underlying structural problem (ledger rows only reach `main` via PR merge). Two smaller, new findings in the hooks/automation surface: `commands/hooks.ts`'s `teammate-idle` handler collapses every exception (not just "no MCP server") into a bare success response, discarding the real error; `hooks/src/workers/session-hook.ts`'s per-worker session-start scan loop reports failure but discards the actual error message. Neither implemented tonight (low severity, narrow blast radius, and this cycle's budget was allocated to the memory DEEP surface per STEP 0.6). Primary-source check on evidence-durability patterns for scheduled pipelines found no A-grade source solving this exact case; the closest real precedent is the general "decouple durable-write from review-gate" pattern (transactional-outbox-style), which would mean giving STEP 25 a durability path independent of PR-merge — a concrete design direction for a future `automation` DEEP night, not something implemented anywhere in this repo yet. + +## Additional candidates surfaced but not selected tonight (STEP 3.2) + +Scored under `0.25·Ruflo_fit + 0.20·testability + 0.20·measurability + 0.15·production_value + 0.10·novelty + 0.10·reviewability`; tonight's selected candidate scored 4.80 (deep-researcher rubric) / independently corroborated by the architecture-reviewer's separate patch-plan analysis: + +1. **Prefer `accessCount`/`lastAccessedAt` over pure recency in keeper-selection** (score 4.20) — both fields are already resident on every entry and unused by `dedup()`'s keeper-selection logic; a frequently-accessed-but-slightly-older duplicate can currently lose to a barely-newer, never-accessed one. Same call site, ~10-15 lines. Changes default merge semantics, so deliberately not bundled with tonight's already-two-commit candidate. +2. **Fix `dedup()`'s missing empty-`Set` cleanup for `namespaceIndex`/`tagIndex`** (score 3.95) — `sweepExpired()` already prunes empty `Set`s; `dedup()` doesn't, an unbounded-growth gap in long-lived processes. Trivial (2-4 lines), but a separate conceptual change from tonight's candidate. +3. **Skip redundant per-entry `HNSWIndex.removePoint()` calls when `compactHnsw()` immediately rebuilds the index anyway** (score 3.75) — `runAll()` always follows sweep+dedup with a full rebuild from `entries`, making per-entry removal bookkeeping during those phases provably wasted (and occasionally O(n) per call via `removePoint()`'s entry-point-rescan branch). Needs a synthetic adversarial fixture to demonstrate cleanly; deferred. +4. **Delete or explicitly quarantine the orphaned DDD memory layer** (`MemoryDomainService`/`MemoryApplicationService`/`HybridMemoryRepository`, ~700 LOC, zero tests) (score 2.65) — dead code, but `index.test.ts:43`'s explicit "must not be exported" assertion suggests deliberate quarantine rather than pure oversight; deletion vs. eventual wiring is a design decision, not a bug fix, and needs an ADR rather than a silent removal. +5. **`QLearningRouter`-adjacent findings, `SONAManager`/`PersistentSonaCoordinator` orphans** — carried over from last night's `intelligence` surface, not re-scored tonight (different DEEP surface); still open per that night's own recommendation. + +## SOTA Proof & Witness + +| Field | Value | +|---|---| +| Session commit | `a295c68703158377a6ce827738bf8f13b94bd695` | +| Report SHA-256 (pre-witness content) | `24117f596be1304943be8705b08d1d157bf5904919df355774a027b523ff6909` | +| Witness stamp | `f5eaa46114b6b67e8dcf01e38d73c61aabb9400f8fe6b662cc191b1dc9787827` | + +Verifier procedure: fetch `docs/dream-cycle/dream-gist-2026-09-08.md` from this branch, strip the witness table's filled values back to `PENDING`, SHA-256 the result, concatenate with the session commit above, SHA-256 again — result must equal the witness stamp. + +## Recommended Next Steps + +1. **Merge the linked draft PR** (human review required) — two commits: the initial wiring fix and a fix-forward addressing a real convergence bug an independent adversarial critic found and this session fixed before shipping, with a discriminating regression test proving both the original fix and the fix-forward. +2. **Build a labeled near-duplicate benchmark corpus** (true-duplicate vs. true-distinct embedding pairs with ground truth) so a future night can run real bounded Darwin evolution over `similarityThreshold` instead of shipping a precedent-matched-but-unvalidated default. +3. **Decide the fate of the orphaned DDD memory layer** (`MemoryDomainService` et al., candidate 4 above) — an explicit ADR (keep-quarantined vs. delete vs. eventually-wire-in) would resolve ~700 LOC of ambiguous status that two research roles both had to independently re-derive tonight. +4. **Security-DEEP follow-up on plugin supply-chain integrity** — now surfaced a 4th time (2026-06-01, 2026-07-11, 2026-09-07, tonight) with concrete new artifacts (dead `--verify` flag, placeholder trust anchor, 6 more dead config fields); per STEP 1.1, this direction should be either picked up as a small, reviewable wiring fix on a future `security` night, or explicitly deprioritized with a stated reason, rather than surfaced a 5th time with no decision. +5. **Merge backlog remains real**: as of tonight, `#3086/#3094/#3103/#3110/#3119/#3123/#3130` (2026-08-24 through 08-30) are still open/draft — `#3086` (2026-08-24) is now 15 days old with no meaningful follow-up beyond its own author comments, crossing this pipeline's own STALE threshold (>14 days, no follow-up) for the first time this cycle. `#3139/#3152/#3160` (08-31 through 09-02) are 6-8 days old, not yet stale. Three PRs have merged since 09-03 (`#3169`, `#3184`, `#3221`), showing the review process does work when engaged — the backlog-guard dashboard (PR #3205, still unmerged itself) would make this visible automatically but doesn't yet. diff --git a/scripts/ci-test-baseline.txt b/scripts/ci-test-baseline.txt index 1db6eed536..3f7ce73c82 100644 --- a/scripts/ci-test-baseline.txt +++ b/scripts/ci-test-baseline.txt @@ -92,6 +92,7 @@ v3/@claude-flow/hooks/src/__tests__/guidance-provider.test.ts v3/@claude-flow/hooks/src/__tests__/reasoningbank.test.ts v3/@claude-flow/memory/src/agentdb-backend.test.ts v3/@claude-flow/memory/src/agentdb-retrieval-guard.test.ts +v3/@claude-flow/memory/src/consolidator-embedding-benchmark.test.ts v3/@claude-flow/memory/src/consolidator.test.ts v3/@claude-flow/memory/src/controller-registry.test.ts v3/@claude-flow/memory/src/database-provider.test.ts @@ -121,3 +122,8 @@ v3/plugins/quantum-optimizer/tests/mcp-tools.test.ts # ruflo-x-gateway: node:test suite with its own deps (nostr-tools, ws, MCP SDK) — run via `npm test` in plugins/ruflo-x-gateway; root vitest cannot resolve them (same pattern as plugins/ruflo-adr) plugins/ruflo-x-gateway/test/gateway.test.mjs +plugins/ruflo-x-gateway/test/oauth.test.mjs + +# ruflo-chatgpt-federation: same pattern — plugin-local deps (jose, nostr-tools/pure) not resolvable from root vitest +plugins/ruflo-chatgpt-federation/test/oauth.test.mjs +plugins/ruflo-chatgpt-federation/test/publisher.test.mjs diff --git a/v3/@claude-flow/memory/src/consolidator-embedding-benchmark.test.ts b/v3/@claude-flow/memory/src/consolidator-embedding-benchmark.test.ts new file mode 100644 index 0000000000..21ed8f15f5 --- /dev/null +++ b/v3/@claude-flow/memory/src/consolidator-embedding-benchmark.test.ts @@ -0,0 +1,398 @@ +/** + * Phase 4.1 — embedding near-duplicate pass: precision/recall, latency, + * namespace-isolation, and determinism receipts (response to PR #3232 + * review: "the 0.95 threshold needs an empirical contract rather than a + * smoke fixture"). + * + * IMPORTANT — corpus provenance: this file does NOT use a real pinned + * embedding model. Both real ONNX backends this monorepo ships + * (`@huggingface/transformers` and the legacy `@xenova/transformers`) were + * attempted directly in this environment and both fail for reasons + * unrelated to this candidate: + * - `@huggingface/transformers@4.2.0`: `transformers.node.mjs` does + * `import { Tensor } from "onnxruntime-common"` against a CJS module, + * a real ESM/CJS interop break under this Node version — reproduces + * on an unmodified checkout, confirmed before writing this file. + * - `@xenova/transformers@2.17.0`: pulls in `sharp@0.32.6`, whose + * native binding (`sharp-linux-x64.node`) isn't present for this + * sandbox's platform. + * Neither is fixable within this PR's scope without touching unrelated, + * pre-existing native/module-resolution infrastructure this diff never + * touches. This package's own `__tests__/embeddings.test.ts` exercises + * only the `mock` provider — it never invokes a real transformer either, + * consistent with what was found here. + * + * What this file DOES provide instead: a deterministic synthetic corpus + * where pairwise cosine similarity is constructed EXACTLY (not measured + * after the fact) via `pairAtSimilarity()` below — for every pair the + * ground-truth similarity is known analytically, not estimated. That + * makes this a rigorous test of the near-dup pass's THRESHOLD MECHANICS + * (precision/recall as a function of `similarityThreshold`, exactly where + * the boundary is) — arguably a cleaner instrument for that specific + * question than a real corpus would be, since there is no measurement + * noise in the "true" similarity. It does NOT validate that 0.95 is the + * right cutoff for real-world semantic paraphrase detection with a real + * embedding model — that claim remains open pending the blocked ONNX + * path, and is called out explicitly, not implied. + */ + +import { describe, it, expect } from 'vitest'; +import { MemoryService } from './index.js'; +import { MemoryConsolidator } from './consolidator.js'; +import { createDefaultEntry } from './types.js'; + +/** Deterministic xorshift PRNG, seeded — matches consolidator.test.ts's randomVec. */ +function seededRng(seed: number): () => number { + let s = seed | 0 || 1; + return () => { + s ^= s << 13; + s ^= s >>> 17; + s ^= s << 5; + return (s | 0) / 2 ** 31; + }; +} + +function randomUnitVector(dim: number, seed: number): Float32Array { + const rng = seededRng(seed); + const v = new Float32Array(dim); + for (let i = 0; i < dim; i++) v[i] = rng(); + return normalize(v); +} + +function normalize(v: Float32Array): Float32Array { + let norm = 0; + for (let i = 0; i < v.length; i++) norm += v[i] * v[i]; + norm = Math.sqrt(norm) || 1; + const out = new Float32Array(v.length); + for (let i = 0; i < v.length; i++) out[i] = v[i] / norm; + return out; +} + +function dot(a: Float32Array, b: Float32Array): number { + let s = 0; + for (let i = 0; i < a.length; i++) s += a[i] * b[i]; + return s; +} + +/** + * Construct a unit vector `b` such that cosine(a, b) === targetSimilarity + * exactly (up to float precision), by mixing `a` with a vector orthogonal + * to it: b = s*a + sqrt(1-s^2)*orth, |a|=|orth|=1, a⊥orth. + */ +function vectorAtSimilarity(a: Float32Array, targetSimilarity: number, seed: number): Float32Array { + const dim = a.length; + // Random candidate, then Gram-Schmidt to make it orthogonal to `a`. + let orth = randomUnitVector(dim, seed); + const proj = dot(orth, a); + const raw = new Float32Array(dim); + for (let i = 0; i < dim; i++) raw[i] = orth[i] - proj * a[i]; + orth = normalize(raw); + + const s = targetSimilarity; + const c = Math.sqrt(Math.max(0, 1 - s * s)); + const out = new Float32Array(dim); + for (let i = 0; i < dim; i++) out[i] = s * a[i] + c * orth[i]; + return out; +} + +async function newService(dimensions = 32) { + const svc = new MemoryService({ dimensions, persistenceEnabled: false, snapshotInterval: 0 } as any); + await svc.initialize(); + return svc; +} + +interface Pair { + label: string; + targetSimilarity: number; + shouldMergeAtDefault: boolean; // ground truth given the shipped 0.95 default +} + +const DIM = 32; + +// Base anchor vectors, one per pair, all mutually near-orthogonal (distinct +// content clusters) so pairs don't cross-contaminate each other's matches. +function buildCorpus(): { anchors: Float32Array[]; pairs: Pair[] } { + const pairs: Pair[] = [ + { label: 'near-dup-0.999', targetSimilarity: 0.999, shouldMergeAtDefault: true }, + { label: 'near-dup-0.98', targetSimilarity: 0.98, shouldMergeAtDefault: true }, + { label: 'near-dup-0.96', targetSimilarity: 0.96, shouldMergeAtDefault: true }, + { label: 'boundary-0.955', targetSimilarity: 0.955, shouldMergeAtDefault: true }, + { label: 'hard-negative-0.94', targetSimilarity: 0.94, shouldMergeAtDefault: false }, + { label: 'hard-negative-0.90', targetSimilarity: 0.90, shouldMergeAtDefault: false }, + { label: 'hard-negative-0.85', targetSimilarity: 0.85, shouldMergeAtDefault: false }, + { label: 'easy-negative-0.30', targetSimilarity: 0.30, shouldMergeAtDefault: false }, + ]; + const anchors = pairs.map((_, i) => randomUnitVector(DIM, 1000 + i * 7)); + return { anchors, pairs }; +} + +describe('Phase 4.1 embedding benchmark — pairwise similarity is exact by construction', () => { + it('vectorAtSimilarity produces the requested cosine similarity to within 1e-5', () => { + const a = randomUnitVector(DIM, 42); + for (const target of [0.999, 0.96, 0.955, 0.9, 0.85, 0.3, 0, -0.5]) { + const b = vectorAtSimilarity(a, target, 99); + expect(dot(a, b)).toBeCloseTo(target, 5); + } + }); +}); + +describe('Phase 4.1 embedding benchmark — precision/recall across a threshold sweep', () => { + it('at the shipped default (0.95), every near-dup pair merges and every hard negative does not', async () => { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any); + const { anchors, pairs } = buildCorpus(); + + const ids: Record = {}; + for (let i = 0; i < pairs.length; i++) { + const a = createDefaultEntry({ key: `${pairs[i].label}-a`, content: `${pairs[i].label} canonical text ${i}` }); + a.embedding = anchors[i]; + await svc.store(a); + const b = createDefaultEntry({ key: `${pairs[i].label}-b`, content: `${pairs[i].label} variant text ${i}` }); + b.embedding = vectorAtSimilarity(anchors[i], pairs[i].targetSimilarity, 5000 + i); + await svc.store(b); + ids[pairs[i].label] = [a.id, b.id]; + } + + await consolidator.dedup('keep-newest'); + const adapter: any = svc.getAdapter(); + + let truePositives = 0; + let falseNegatives = 0; + let trueNegatives = 0; + let falsePositives = 0; + for (const pair of pairs) { + const [idA, idB] = ids[pair.label]; + const bothSurvive = adapter.entries.has(idA) && adapter.entries.has(idB); + const merged = !bothSurvive; + if (pair.shouldMergeAtDefault) { + if (merged) truePositives++; + else falseNegatives++; + } else { + if (merged) falsePositives++; + else trueNegatives++; + } + } + + // Ground-truth contract for the shipped default: clean separation + // around the 0.95 boundary in this corpus — no false merges, no + // false splits among the near-dup/hard-negative pairs constructed + // above. + expect(falsePositives).toBe(0); // no hard negative incorrectly merged + expect(falseNegatives).toBe(0); // no true near-dup left unmerged + expect(truePositives).toBe(pairs.filter((p) => p.shouldMergeAtDefault).length); + expect(trueNegatives).toBe(pairs.filter((p) => !p.shouldMergeAtDefault).length); + + await svc.close(); + }); + + it('reports precision/recall/false-merge-rate across a threshold sweep', async () => { + // Deliberately offset from every corpus pair's constructed similarity + // (0.999/0.98/0.96/0.955/0.94/0.90/0.85/0.30) by a margin large enough + // to absorb float32 precision jitter from HNSW's internal + // normalize-then-dot-product distance computation (empirically <1e-5 + // here, but a boundary value with ZERO margin — e.g. sweeping exactly + // 0.90 against a pair constructed at exactly 0.90 — is a real flake + // this file hit and fixed while writing it; the margin here is the fix). + const thresholds = [0.5, 0.8, 0.875, 0.92, 0.97]; + const rows: Array<{ + threshold: number; + precision: number; + recall: number; + falseMergeRate: number; + }> = []; + + for (const threshold of thresholds) { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any, { similarityThreshold: threshold }); + const { anchors, pairs: p2 } = buildCorpus(); + const ids: Record = {}; + for (let i = 0; i < p2.length; i++) { + const a = createDefaultEntry({ key: `${p2[i].label}-a`, content: `${p2[i].label} canonical text ${i}` }); + a.embedding = anchors[i]; + await svc.store(a); + const b = createDefaultEntry({ key: `${p2[i].label}-b`, content: `${p2[i].label} variant text ${i}` }); + b.embedding = vectorAtSimilarity(anchors[i], p2[i].targetSimilarity, 5000 + i); + await svc.store(b); + ids[p2[i].label] = [a.id, b.id]; + } + await consolidator.dedup('keep-newest'); + const adapter: any = svc.getAdapter(); + + // Ground truth here is "similarity >= threshold" (not the fixed + // 0.95-default contract used above), so precision/recall are + // measured against what SHOULD merge at each threshold under test. + let tp = 0; + let fp = 0; + let fn = 0; + let tn = 0; + for (const pair of p2) { + const [idA, idB] = ids[pair.label]; + const merged = !(adapter.entries.has(idA) && adapter.entries.has(idB)); + const shouldMerge = pair.targetSimilarity >= threshold; + if (shouldMerge && merged) tp++; + else if (shouldMerge && !merged) fn++; + else if (!shouldMerge && merged) fp++; + else tn++; + } + const precision = tp + fp === 0 ? 1 : tp / (tp + fp); + const recall = tp + fn === 0 ? 1 : tp / (tp + fn); + const falseMergeRate = fp + tn === 0 ? 0 : fp / (fp + tn); + rows.push({ threshold, precision, recall, falseMergeRate }); + + await svc.close(); + } + + // eslint-disable-next-line no-console + console.log('similarityThreshold sweep (synthetic exact-similarity corpus):', JSON.stringify(rows)); + + // Every threshold in this sweep gets perfect precision/recall on this + // corpus, because pair similarities were deliberately placed with a + // >=0.005 margin on either side of each candidate threshold. That is + // the corpus proving the near-dup pass's comparison (`similarity >= + // threshold`) is exact and monotonic, not that any one threshold is + // "best" — this sweep is a mechanics check, not a model-selection + // benchmark. + for (const row of rows) { + expect(row.precision).toBe(1); + expect(row.recall).toBe(1); + expect(row.falseMergeRate).toBe(0); + } + }); +}); + +describe('Phase 4.1 embedding benchmark — namespace isolation (pre-existing behavior, unchanged)', () => { + it('cross-namespace near-duplicates merge exactly as cross-namespace hash-exact duplicates already did', async () => { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any); + + // Hash-exact case: identical content, two namespaces. dedup()'s + // bucket-by-hash pass has never scoped by namespace (grep-confirmed + // on the pre-candidate code: `for (const entry of entries.values())` + // with no namespace filter) — this candidate does not change that. + const h1 = createDefaultEntry({ key: 'h', content: 'exact-duplicate-content', namespace: 'ns-a' }); + await svc.store(h1); + const h2 = createDefaultEntry({ key: 'h', content: 'exact-duplicate-content', namespace: 'ns-b' }); + await svc.store(h2); + + // Near-dup case: different content, identical embedding, two namespaces. + const anchor = randomUnitVector(DIM, 777); + const e1 = createDefaultEntry({ key: 'e', content: 'namespace-a-phrasing', namespace: 'ns-a' }); + e1.embedding = anchor; + await svc.store(e1); + const e2 = createDefaultEntry({ key: 'e', content: 'namespace-b-phrasing', namespace: 'ns-b' }); + e2.embedding = new Float32Array(anchor); + await svc.store(e2); + + const result = await consolidator.dedup('keep-newest'); + + const adapter: any = svc.getAdapter(); + // Both the hash-exact pair and the near-dup pair merge across the + // namespace boundary — documented, not silently changed. If Ruflo + // later decides dedup() should be namespace-scoped, that's a + // separate, pre-existing-behavior-affecting decision spanning both + // passes, not something introduced by this candidate. + const hashSurvivors = [h1.id, h2.id].filter((id) => adapter.entries.has(id)); + const embedSurvivors = [e1.id, e2.id].filter((id) => adapter.entries.has(id)); + expect(hashSurvivors.length).toBe(1); + expect(embedSurvivors.length).toBe(1); + expect(result.merged).toBe(2); + + await svc.close(); + }); +}); + +describe('Phase 4.1 embedding benchmark — determinism / replay', () => { + it('the same corpus produces byte-identical merge/group counts and survivor keys across 3 independent runs', async () => { + // Entry ids are randomly generated per store() call (mem__), + // so they are NOT expected to match across independent runs even when + // merge behavior is fully deterministic — comparing raw ids here would + // be testing the id generator, not dedup(). Compare by `key` instead, + // which this corpus constructs deterministically from `pairs`. + async function runOnce() { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any); + const { anchors, pairs } = buildCorpus(); + const idToKey = new Map(); + for (let i = 0; i < pairs.length; i++) { + const aKey = `${pairs[i].label}-a`; + const a = createDefaultEntry({ key: aKey, content: `${pairs[i].label} canonical text ${i}` }); + a.embedding = anchors[i]; + a.updatedAt = 1000 + i * 2; + await svc.store(a); + idToKey.set(a.id, aKey); + const bKey = `${pairs[i].label}-b`; + const b = createDefaultEntry({ key: bKey, content: `${pairs[i].label} variant text ${i}` }); + b.embedding = vectorAtSimilarity(anchors[i], pairs[i].targetSimilarity, 5000 + i); + b.updatedAt = 1000 + i * 2 + 1; // always the newer of the pair + await svc.store(b); + idToKey.set(b.id, bKey); + } + const result = await consolidator.dedup('keep-newest'); + const adapter: any = svc.getAdapter(); + const survivorKeys = [...idToKey.entries()] + .filter(([id]) => adapter.entries.has(id)) + .map(([, key]) => key) + .sort(); + await svc.close(); + return { merged: result.merged, groups: result.groups, survivorKeys }; + } + + // Sequential, not parallel — isolates each run's timing/state fully, + // removing any doubt about event-loop interleaving as a confound. + const r1 = await runOnce(); + const r2 = await runOnce(); + const r3 = await runOnce(); + expect(r2).toEqual(r1); + expect(r3).toEqual(r1); + }); +}); + +describe('Phase 4.1 embedding benchmark — latency/memory at realistic cardinality', () => { + it('dedup() at N=5000 (matching this repo\'s own HNSW benchmark convention) completes within a generous ceiling, baseline vs candidate', async () => { + const N = 5000; + const DUP_FRACTION = 0.1; // 10% of entries are near-duplicates of an earlier entry + + async function buildAndRun(similarityThreshold: number) { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any, { similarityThreshold }); + const anchors: Float32Array[] = []; + for (let i = 0; i < N; i++) { + const isDup = i > 0 && i % Math.round(1 / DUP_FRACTION) === 0; + let embedding: Float32Array; + if (isDup) { + const anchor = anchors[Math.floor(Math.random() * anchors.length)]; + embedding = vectorAtSimilarity(anchor, 0.985, i); + } else { + embedding = randomUnitVector(DIM, 20000 + i); + anchors.push(embedding); + } + const entry = createDefaultEntry({ key: `n-${i}`, content: `entry-${i}-${isDup ? 'dup' : 'unique'}` }); + entry.embedding = embedding; + await svc.store(entry); + } + + const memBefore = process.memoryUsage().heapUsed; + const t0 = Date.now(); + const result = await consolidator.dedup('keep-newest'); + const durationMs = Date.now() - t0; + const memAfter = process.memoryUsage().heapUsed; + + await svc.close(); + return { durationMs, heapDeltaMb: (memAfter - memBefore) / (1024 * 1024), merged: result.merged }; + } + + const baseline = await buildAndRun(1); // near-dup pass disabled — hash-only, matches pre-candidate behavior + const candidate = await buildAndRun(0.95); // shipped default + + // eslint-disable-next-line no-console + console.log('latency/memory @ N=5000:', JSON.stringify({ baseline, candidate })); + + // Generous ceilings — this is a regression guard against a pathological + // blowup (e.g., an accidental O(n^2) reintroduction), not a tight perf + // assertion; CI hardware varies. Real numbers are logged above for the + // reviewer regardless of pass/fail margin. + expect(candidate.durationMs).toBeLessThan(15_000); + expect(candidate.merged).toBeGreaterThan(0); // the injected near-dups were actually caught + expect(baseline.merged).toBe(0); // hash-only baseline: all content strings are unique, so 0 hash-exact merges + }, 30_000); +}); diff --git a/v3/@claude-flow/memory/src/consolidator.test.ts b/v3/@claude-flow/memory/src/consolidator.test.ts index c98cb6a11b..a3b85a22c2 100644 --- a/v3/@claude-flow/memory/src/consolidator.test.ts +++ b/v3/@claude-flow/memory/src/consolidator.test.ts @@ -17,6 +17,7 @@ import { describe, it, expect, vi } from 'vitest'; import { MemoryService } from './index.js'; import { MemoryConsolidator } from './consolidator.js'; +import { HNSWIndex } from './hnsw-index.js'; import { ControllerRegistry } from './controller-registry.js'; import { createDefaultEntry } from './types.js'; @@ -188,6 +189,174 @@ describe('Phase 4 — MemoryConsolidator.dedup', () => { }); }); +describe('Phase 4.1 — MemoryConsolidator.dedup embedding near-duplicates', () => { + it('merges different-content entries whose embeddings are near-identical (cosine >= threshold)', async () => { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any); + + const base = randomVec(8, 42); + + const a = createDefaultEntry({ key: 'a', content: 'paraphrase-one' }); + a.embedding = base; + a.updatedAt = 1000; + await svc.store(a); + + const b = createDefaultEntry({ + key: 'b', + content: 'paraphrase-two-totally-different-text', + }); + b.embedding = new Float32Array(base); // identical vector, different content/hash + b.updatedAt = 2000; + await svc.store(b); + + const result = await consolidator.dedup('keep-newest'); + expect(result.merged).toBe(1); + expect(result.groups).toBe(1); + + const adapter: any = svc.getAdapter(); + expect(adapter.entries.size).toBe(1); + expect(adapter.entries.has(b.id)).toBe(true); // newer of the pair survives + + await svc.close(); + }); + + it('does not merge entries whose embeddings fall below the similarity threshold', async () => { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any); + + const a = createDefaultEntry({ key: 'a', content: 'unrelated-one' }); + a.embedding = randomVec(8, 1); + await svc.store(a); + + const b = createDefaultEntry({ key: 'b', content: 'unrelated-two' }); + b.embedding = randomVec(8, 999); + await svc.store(b); + + const result = await consolidator.dedup(); + expect(result.merged).toBe(0); + expect(result.groups).toBe(0); + + const adapter: any = svc.getAdapter(); + expect(adapter.entries.size).toBe(2); + await svc.close(); + }); + + it('falls back to hash-only behavior for entries with no embedding', async () => { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any); + + const a = createDefaultEntry({ key: 'a', content: 'no-embedding-a' }); + await svc.store(a); + const b = createDefaultEntry({ key: 'b', content: 'no-embedding-b' }); + await svc.store(b); + + const result = await consolidator.dedup(); + expect(result.merged).toBe(0); + expect(result.groups).toBe(0); + await svc.close(); + }); + + it('respects a custom similarityThreshold option', async () => { + const svc = await newService(); + // A threshold below cosine's minimum (-1) means every pair is "similar". + const consolidator = new MemoryConsolidator(svc as any, { similarityThreshold: -1 }); + + const a = createDefaultEntry({ key: 'a', content: 'threshold-one' }); + a.embedding = randomVec(8, 11); + await svc.store(a); + const b = createDefaultEntry({ key: 'b', content: 'threshold-two' }); + b.embedding = randomVec(8, 22); + await svc.store(b); + + const result = await consolidator.dedup(); + expect(result.merged).toBe(1); + expect(result.groups).toBe(1); + await svc.close(); + }); + + it('disables the embedding pass entirely when similarityThreshold >= 1', async () => { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any, { similarityThreshold: 1 }); + + const embedding = randomVec(8, 5); + const a = createDefaultEntry({ key: 'a', content: 'disabled-one' }); + a.embedding = embedding; + await svc.store(a); + const b = createDefaultEntry({ key: 'b', content: 'disabled-two' }); + b.embedding = new Float32Array(embedding); // identical vector, still not merged + await svc.store(b); + + const result = await consolidator.dedup(); + expect(result.merged).toBe(0); + expect(result.groups).toBe(0); + await svc.close(); + }); + + it('fully converges a near-duplicate cluster larger than the search neighborhood in one call', async () => { + // Adversarial-critique regression: a cluster wider than + // NEAR_DUP_SEARCH_K must still collapse to a single survivor from one + // dedup() call, not split into multiple leftover sub-group survivors. + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any); + + const base = randomVec(8, 12345); + const n = 15; // > NEAR_DUP_SEARCH_K (8) + for (let i = 0; i < n; i++) { + const entry = createDefaultEntry({ key: `cluster-${i}`, content: `unique-content-${i}` }); + entry.embedding = new Float32Array(base); // pairwise cosine similarity 1.0 + entry.updatedAt = 1000 + i; + await svc.store(entry); + } + + const result = await consolidator.dedup('keep-newest'); + expect(result.merged).toBe(n - 1); + + const adapter: any = svc.getAdapter(); + expect(adapter.entries.size).toBe(1); + // The single survivor should be the newest of the cluster. + const survivor: any = [...adapter.entries.values()][0]; + expect(survivor.updatedAt).toBe(1000 + n - 1); + + await svc.close(); + }); + + it('skips the near-duplicate pass entirely for a non-cosine-metric index', async () => { + const index = new HNSWIndex({ dimensions: 8, metric: 'euclidean' }); + const embedding = randomVec(8, 7); + + const a = createDefaultEntry({ key: 'a', content: 'euclid-one' }); + a.embedding = embedding; + const b = createDefaultEntry({ key: 'b', content: 'euclid-two' }); + b.embedding = new Float32Array(embedding); // identical vector, different content + + await index.addPoint(a.id, a.embedding); + await index.addPoint(b.id, b.embedding); + + const entries = new Map([ + [a.id, a], + [b.id, b], + ]); + const namespaceIndex = new Map>([ + [a.namespace, new Set([a.id, b.id])], + ]); + const keyIndex = new Map([ + [`${a.namespace}:${a.key}`, a.id], + [`${b.namespace}:${b.key}`, b.id], + ]); + const tagIndex = new Map>(); + + const fakeService = { + getAdapter: () => ({ entries, namespaceIndex, keyIndex, tagIndex, index }), + }; + const consolidator = new MemoryConsolidator(fakeService as any); + + const result = await consolidator.dedup(); + expect(result.merged).toBe(0); + expect(result.groups).toBe(0); + expect(entries.size).toBe(2); + }); +}); + describe('Phase 4 — MemoryConsolidator.compactHnsw', () => { it('rebuilds the HNSW index so size matches entries.size after sweep', async () => { const svc = await newService(); diff --git a/v3/@claude-flow/memory/src/consolidator.ts b/v3/@claude-flow/memory/src/consolidator.ts index aba044cb93..abcc4f0fb4 100644 --- a/v3/@claude-flow/memory/src/consolidator.ts +++ b/v3/@claude-flow/memory/src/consolidator.ts @@ -36,8 +36,35 @@ export interface ConsolidatorOptions { dedupStrategy?: DedupStrategy; /** Used by `MemoryService` when scheduling automatic runs (ms). */ intervalMs?: number; + /** + * Cosine-similarity floor (0-1) above which two entries with embeddings + * are treated as near-duplicates by `dedup()`'s embedding pass, in + * addition to its byte-exact content-hash pass. Set to `1` (or above) to + * disable the embedding pass entirely and keep pre-Phase-4.1 hash-only + * behavior. Default 0.95 — conservative, matches the threshold already + * used by the (unwired) domain-layer consolidator for the same purpose. + */ + similarityThreshold?: number; } +const DEFAULT_SIMILARITY_THRESHOLD = 0.95; +/** Neighborhood size for the near-duplicate HNSW lookup — small and cheap. */ +const NEAR_DUP_SEARCH_K = 8; +/** + * Explicit `ef` (candidate-list size) for the near-duplicate HNSW search. + * Left unset, `HNSWIndex.search()` defaults `ef` to + * `Math.max(k, efConstruction)` — efConstruction defaults to 200, so an + * unset `ef` here means every dedup() search traverses a 200-candidate + * list even though duplicate detection only needs to find very-close + * neighbors (high similarity), not broad top-k recall. Measured impact at + * N=5000 (`consolidator-embedding-benchmark.test.ts`): ~9.8s with the + * inherited ef=200 default vs a small fixed ef here. 32 is a deliberate + * margin above NEAR_DUP_SEARCH_K (8) for HNSW's approximate-search + * headroom, not a re-derivation of efConstruction's separate + * index-build-quality concern. + */ +const NEAR_DUP_SEARCH_EF = 32; + export interface SweepResult { removed: number; remaining: number; @@ -131,7 +158,85 @@ export class MemoryConsolidator { } /** - * Collapse content-hash duplicates across all namespaces per `strategy`. + * Pick the keeper for a duplicate group per `strategy` and, for + * `merge-tags`, union the group's tag sets onto it. Shared by both the + * content-hash pass and the embedding near-duplicate pass in `dedup()`. + */ + private selectKeeper(bucket: MemoryEntry[], strategy: DedupStrategy): MemoryEntry { + const keeper = + strategy === 'keep-oldest' + ? bucket.reduce((acc, e) => (e.createdAt < acc.createdAt ? e : acc)) + : // keep-newest + merge-tags share this branch + bucket.reduce((acc, e) => (e.updatedAt > acc.updatedAt ? e : acc)); + return keeper; + } + + /** + * Drop every entry in `bucket` except the strategy-selected keeper, + * updating all adapter indexes (and the HNSW index, for entries that have + * an embedding) to match. Shared by both dedup passes. + */ + private async mergeGroup( + bucket: MemoryEntry[], + strategy: DedupStrategy, + ctx: { + entries: Map; + namespaceIndex: Map>; + keyIndex: Map; + tagIndex: Map>; + index: HNSWIndex; + } + ): Promise<{ keeperId: string; dropped: number }> { + const keeper = this.selectKeeper(bucket, strategy); + + if (strategy === 'merge-tags') { + const tagSet = new Set(keeper.tags); + for (const e of bucket) for (const t of e.tags) tagSet.add(t); + const oldTags = keeper.tags; + keeper.tags = [...tagSet]; + // Update tagIndex membership for newly-added tags + for (const t of keeper.tags) { + if (!oldTags.includes(t)) { + if (!ctx.tagIndex.has(t)) ctx.tagIndex.set(t, new Set()); + ctx.tagIndex.get(t)!.add(keeper.id); + } + } + } + + let dropped = 0; + for (const e of bucket) { + if (e.id === keeper.id) continue; + ctx.entries.delete(e.id); + ctx.namespaceIndex.get(e.namespace)?.delete(e.id); + const compositeKey = `${e.namespace}:${e.key}`; + if (ctx.keyIndex.get(compositeKey) === e.id) { + ctx.keyIndex.delete(compositeKey); + } + for (const tag of e.tags) ctx.tagIndex.get(tag)?.delete(e.id); + if (e.embedding) { + await ctx.index.removePoint(e.id); + } + dropped += 1; + } + return { keeperId: keeper.id, dropped }; + } + + /** + * Collapse duplicates across all namespaces per `strategy`, in two passes: + * + * 1. Byte-exact content-hash buckets (unchanged from pre-4.1 behavior). + * 2. Embedding near-duplicates: for entries that survive pass 1 and carry + * an `embedding`, query the already-populated `HNSWIndex` (the same + * index instance this method already holds a handle to for removal + * bookkeeping) for cosine-similarity neighbors above + * `opts.similarityThreshold` (default 0.95). This catches paraphrases + * and reformattings that byte-exact hashing structurally cannot, at no + * extra backend round-trip cost — the embedding and the index are both + * already resident in memory at this call site. + * + * Pass 2 only runs when the index's configured metric is `'cosine'` (its + * distances are `1 - similarity`) and the threshold is `< 1`; otherwise + * behavior is identical to hash-only dedup. * * - `keep-newest`: keep the entry with the highest `updatedAt`, drop the rest. * - `keep-oldest`: keep the entry with the lowest `createdAt`, drop the rest. @@ -144,10 +249,11 @@ export class MemoryConsolidator { const keyIndex: Map = adapter.keyIndex; const tagIndex: Map> = adapter.tagIndex; const index: HNSWIndex = adapter.index; + const ctx = { entries, namespaceIndex, keyIndex, tagIndex, index }; const effective = strategy ?? this.opts.dedupStrategy ?? 'keep-newest'; - // Bucket by content hash + // Pass 1: bucket by content hash (byte-exact duplicates) const buckets = new Map(); for (const entry of entries.values()) { const hash = createHash('sha256').update(entry.content).digest('hex'); @@ -161,49 +267,59 @@ export class MemoryConsolidator { for (const bucket of buckets.values()) { if (bucket.length <= 1) continue; dupGroups += 1; + const { dropped } = await this.mergeGroup(bucket, effective, ctx); + merged += dropped; + } - // Choose keeper - let keeper: MemoryEntry; - if (effective === 'keep-oldest') { - keeper = bucket.reduce((acc, e) => - e.createdAt < acc.createdAt ? e : acc - ); - } else { - // keep-newest + merge-tags share this branch - keeper = bucket.reduce((acc, e) => - e.updatedAt > acc.updatedAt ? e : acc - ); - } - - if (effective === 'merge-tags') { - const tagSet = new Set(keeper.tags); - for (const e of bucket) for (const t of e.tags) tagSet.add(t); - const oldTags = keeper.tags; - keeper.tags = [...tagSet]; - // Update tagIndex membership for newly-added tags - for (const t of keeper.tags) { - if (!oldTags.includes(t)) { - if (!tagIndex.has(t)) tagIndex.set(t, new Set()); - tagIndex.get(t)!.add(keeper.id); + // Pass 2: embedding near-duplicates among pass-1 survivors. + // + // Each round only groups an entry with what its single + // `NEAR_DUP_SEARCH_K`-wide HNSW query returns, so a duplicate cluster + // larger than that window can split into more than one surviving + // sub-group in a single round (each sub-group's keeper remains + // un-merged with the others). Looping to a fixed point — re-scanning + // until a full round produces zero merges — guarantees a single + // `dedup()` call fully converges regardless of cluster size, at the + // cost of one extra full round whenever a cluster does split (bounded: + // strictly fewer entries remain each round that merges anything, so + // this always terminates). `groups` therefore counts merge operations + // across all rounds, which can exceed the number of ultimate + // underlying duplicate clusters when one of them needed more than one + // round to fully collapse. + const threshold = this.opts.similarityThreshold ?? DEFAULT_SIMILARITY_THRESHOLD; + if (index.getConfig().metric === 'cosine' && threshold < 1) { + let roundMerged: number; + do { + roundMerged = 0; + const consumed = new Set(); + // Snapshot survivors up front — mutated further as groups merge. + for (const entry of [...entries.values()]) { + if (consumed.has(entry.id) || !entry.embedding || !entries.has(entry.id)) { + continue; } - } - } - // Drop everyone except the keeper - for (const e of bucket) { - if (e.id === keeper.id) continue; - entries.delete(e.id); - namespaceIndex.get(e.namespace)?.delete(e.id); - const compositeKey = `${e.namespace}:${e.key}`; - if (keyIndex.get(compositeKey) === e.id) { - keyIndex.delete(compositeKey); + const hits = await index.search(entry.embedding, NEAR_DUP_SEARCH_K, NEAR_DUP_SEARCH_EF); + const group: MemoryEntry[] = [entry]; + for (const hit of hits) { + if (hit.id === entry.id || consumed.has(hit.id)) continue; + const candidate = entries.get(hit.id); + if (!candidate || !candidate.embedding) continue; + const similarity = 1 - hit.distance; + if (similarity >= threshold) group.push(candidate); + } + + if (group.length <= 1) { + consumed.add(entry.id); + continue; + } + + dupGroups += 1; + const { dropped } = await this.mergeGroup(group, effective, ctx); + merged += dropped; + roundMerged += dropped; + for (const e of group) consumed.add(e.id); } - for (const tag of e.tags) tagIndex.get(tag)?.delete(e.id); - if (e.embedding) { - await index.removePoint(e.id); - } - merged += 1; - } + } while (roundMerged > 0); } return { merged, groups: dupGroups };