From e036e35dd74c19e61c240c75b80bbdfdee0d781e Mon Sep 17 00:00:00 2001 From: rUv Date: Mon, 14 Sep 2026 23:13:14 +0000 Subject: [PATCH] dream(memory): #3231 wire embedding near-dup detection into MemoryConsolidator.dedup() (evaluated, ACCEPT) (#3232) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * dream(memory): wire embedding near-duplicate detection into MemoryConsolidator.dedup() MemoryConsolidator.dedup() only ever deduplicated entries via byte-exact SHA-256 content hashing, even though every entry already carries a computed .embedding and the adapter's HNSWIndex (already held in scope for removal bookkeeping) is incrementally kept in sync. Paraphrases and reformattings of the same memory were never caught. Adds a second pass: for hash-pass survivors with an embedding, query the already-populated HNSWIndex for cosine-similarity neighbors above a configurable similarityThreshold (default 0.95, matching the unwired domain-layer consolidator's own constant), and merge using the same keeper-selection strategies (now factored into shared selectKeeper/ mergeGroup helpers). Guarded to cosine-metric indexes only; disabled via similarityThreshold >= 1. Evaluation: 471/472 passing in @claude-flow/memory (1 pre-existing, unrelated, environmental failure); baseline (git-stash-isolated source) fails exactly the 2 new discriminating tests and passes the rest, confirming the fix is additive and non-regressive. tsc --noEmit clean. Co-Authored-By: RuFlo Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG * dream(memory): fix-forward — converge near-dup pass to a fixed point in one dedup() call Independent adversarial critique (STEP 10) reproduced a real, bounded gap in the prior commit: a near-duplicate cluster larger than NEAR_DUP_SEARCH_K (8) split into multiple leftover sub-group survivors within a single dedup() call, because each round's `consumed` bookkeeping permanently excluded a group's keeper from further matching even though it was still fully present in the index. 15 pairwise-identical-embedding entries collapsed to 2 survivors (merged: 13) instead of 1 (merged: 14). It self-healed across repeated runAll() calls (background timer / nightlyLearner), so this was never a permanent-data-loss bug, but a single manual dedup() call under-converged relative to its documented "collapse duplicates" contract. Fix: loop pass 2 to a fixed point (re-scan until a full round produces zero merges) instead of a single scan. Always terminates — entries strictly decrease each round that merges anything. `groups` now counts merge operations across all rounds, which can exceed the number of underlying duplicate clusters when one needed more than one round; documented in the method's doc comment. Added a discriminating regression test reproducing the critic's exact 15-entry scenario; confirmed it fails against the pre-fix single-round code (merged: 13) and passes against this fix (merged: 14, single survivor). Full package suite: 472/473 passing (1 pre-existing, unrelated, environmental failure, unchanged from before this commit). tsc --noEmit clean. Co-Authored-By: RuFlo Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG * dream(memory): 2026-09-08 research gist — embedding near-dup consolidation Full SOTA report: 5-role parallel research fan-out, ledger check with GitHub-verified fates for the last 14 rows (3 merged since 09-03, one PR now crossing the 14-day stale threshold for the first time this cycle), competitor comparison (CrewAI/Mem0/Zep/LangMem/Qdrant/SemDedup), plugins and automation scan findings, adversarial critique with a real bug found and fixed, and witness stamp. No gist-creation MCP tool available in this session (GitHub MCP tools cover issues/PRs/repos, not gists) — committing to docs/dream-cycle/ instead, matching every dream-cycle night since 2026-08-14. Co-Authored-By: RuFlo Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG * dream(memory): #3231 update LEDGER.md for 2026-09-08 Co-Authored-By: RuFlo Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG * dream(memory): #3231 empirical receipts for dedup() near-dup pass (response to review) Response to PR #3232's INCONCLUSIVE review, which asked for a threshold contract rather than a smoke fixture: a contamination-checked embedding corpus with pinned model, precision/recall including hard negatives and cross-namespace cases, baseline vs. candidate latency/memory at realistic cardinality, and deterministic replay + unchanged exact-dedup behavior. Real pinned-model corpus: attempted directly against both ONNX backends this monorepo ships. Both fail for reasons unrelated to this candidate, reproduced on an unmodified checkout before writing anything: - @huggingface/transformers@4.2.0: transformers.node.mjs does `import { Tensor } from "onnxruntime-common"` against a CJS module — a real ESM/CJS interop break under this sandbox's Node version. - @xenova/transformers@2.17.0: pulls in sharp@0.32.6, whose native binding (sharp-linux-x64.node) isn't present for this platform — the same failure independently reproduced moments later by this package's own nightlyLearner test, which falls back to mock embeddings. Neither is fixable within this PR's scope without touching unrelated, pre-existing native/module-resolution infrastructure. Documented in detail at the top of the new test file rather than silently worked around or omitted. What's delivered instead, in v3/@claude-flow/memory/src/consolidator-embedding-benchmark.test.ts: - A deterministic synthetic corpus where pairwise cosine similarity is constructed EXACTLY via vectorAtSimilarity() (Gram-Schmidt against a random orthogonal vector), not measured after the fact — a rigorous instrument for threshold mechanics specifically, disclosed as not validating real-world semantic accuracy. - Precision/recall/false-merge-rate across a 5-point threshold sweep, every point at a deliberate margin from every corpus similarity value after an exact-boundary collision (sweep value == corpus value) proved float32-jitter-sensitive while writing this — documented as a finding, not silently patched around. - Namespace isolation: proved cross-namespace near-dup merging matches cross-namespace hash-exact merging exactly (both pre-existing, unscoped-by-namespace behavior — this candidate doesn't change it). - Determinism: 3 independent runs, compared by deterministic entry KEY (not raw id, which is randomly generated per store() and was a second bug this file's own first draft had to fix). - Latency/memory at N=5000 (matching this repo's own HNSW-benchmark convention): initially measured 9.8s for the candidate pass, which traced to an unset `ef` search parameter defaulting to efConstruction (200) — every dedup() search traversed a 200-candidate list for a duplicate-detection task that only needs very-close neighbors. Fixed by passing an explicit NEAR_DUP_SEARCH_EF=32 in consolidator.ts, re-measured at 3.7s (~2.7x), precision/recall unchanged (still exact). Real numbers logged in the test output either way, not asserted away. Full package suite: 478/479 passing (472 pre-existing + 7 new; same 1 pre-existing unrelated environmental failure as every prior commit on this branch). tsc --noEmit clean. Co-Authored-By: RuFlo Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG * dream(memory): #3231 add new benchmark file to CI test-ratchet baseline CI on a2e99cf failed with "1 unexpected failing file(s): + v3/@claude-flow/memory/src/consolidator-embedding-benchmark.test.ts". Root-caused via the vitest JSON report artifact (test-results-ubuntu-latest, downloaded and parsed directly, not guessed): a suite-level collection failure, "Cannot find package '@claude-flow/security' imported from '.../v3/@claude-flow/memory/src/agentdb-retrieval-guard.ts'" — the same unbuilt-sibling-package gap already accepted in scripts/ci-test-baseline.txt for 9 other @claude-flow/memory test files, including consolidator.test.ts itself (line 95). Every file in this package fails identically in this CI checkout state; my new file just wasn't in the baseline yet because it's new. Not a regression this PR introduced — confirmed by the failure being a pre-import-time module-resolution error, not a test assertion in the new file's own logic (which passes locally in a checkout with @claude-flow/ memory's dependencies actually built: see the 6/6 local run in the prior commit's message). Fix: add the new file to the baseline, alongside its 9 memory-package siblings already there, matching this repo's own established convention for this exact gap rather than working around it some other way. Co-Authored-By: RuFlo Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG * dream(memory): #3231 baseline 2 more federation/gateway plugin test files CI on 20afdc8 failed (run 34905889339) after main was rebased in — 3 unexpected failing files, all pre-existing, unrelated to this PR's diff: plugins/ruflo-chatgpt-federation/test/oauth.test.mjs Cannot find package 'jose' plugins/ruflo-chatgpt-federation/test/publisher.test.mjs Cannot find package 'nostr-tools/pure' plugins/ruflo-x-gateway/test/oauth.test.mjs Cannot find package 'jose' Verified via the vitest.json artifact (downloaded and parsed, not guessed): all three are plugin-local-dependency collection failures, identical in kind to the already-baselined sibling plugins/ruflo-x-gateway/test/gateway.test.mjs ("node:test suite with its own deps... root vitest cannot resolve them"). Confirmed these are new on main itself (main's own ci-test-baseline.txt at the rebase point does not list them either) and unrelated to v3/@claude-flow/memory — nothing this PR touches. Re-verified locally: re-ran the full @claude-flow/memory suite against the rebased main (496/497 passing, same 1 pre-existing unrelated failure as always) and recomputed the ratchet's unexpected-set directly against the CI run's vitest.json artifact — 0 unexpected after this change. Co-Authored-By: RuFlo Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG --------- Co-authored-by: Claude --- docs/dream-cycle/LEDGER.md | 1 + docs/dream-cycle/dream-gist-2026-09-08.md | 124 ++++++ scripts/ci-test-baseline.txt | 6 + .../consolidator-embedding-benchmark.test.ts | 398 ++++++++++++++++++ .../memory/src/consolidator.test.ts | 169 ++++++++ v3/@claude-flow/memory/src/consolidator.ts | 198 +++++++-- 6 files changed, 855 insertions(+), 41 deletions(-) create mode 100644 docs/dream-cycle/dream-gist-2026-09-08.md create mode 100644 v3/@claude-flow/memory/src/consolidator-embedding-benchmark.test.ts diff --git a/docs/dream-cycle/LEDGER.md b/docs/dream-cycle/LEDGER.md index 3cd198a309..34499b14dc 100644 --- a/docs/dream-cycle/LEDGER.md +++ b/docs/dream-cycle/LEDGER.md @@ -188,3 +188,4 @@ as a first-class, high-priority candidate, not a background note. | 2026-09-09 | swarm | AgentPool.performHealthChecks() stamped lastHeartbeat=now on every "still healthy" tick, and updateAgentHeartbeat() (the only other legitimate way to advance it) has zero callers anywhere in v3/ — so the check's own tick was the sole thing keeping every pooled agent's heartbeat fresh, meaning timeSinceLastActivity could never exceed one interval and replaceUnhealthyAgent() could never fire for any agent, in any pool, ever; removed the self-stamp | #3242 | #3243 | yes | ACCEPT-scoped | 223/223 tests (+3 new), baseline-fails-2-of-3/candidate-passes-3-of-3 via stash isolation; adversarial critic confirmed independently but disclosed @claude-flow/swarm has no reachable path through the shipped @claude-flow/cli today (blast radius is the swarm package's own UnifiedSwarmCoordinator, not an active CLI hot path) | 008c96a0... | 08-24..09-02 backlog unchanged (see note above); #3169/#3184/#3221 merged since 09-05 | | 2026-09-01 | security | `maxToolCallsPerTurn` was session-lifetime cumulative, never reset (disclosed bug in #3139); wall-clock sliding window fix, new `turnWindowMs` policy field (follow-up to #3139, stacked PR) | #3151 | #3152 | yes | ACCEPT | 30/30 new-suite tests, 52/56 broader MCP sweep (4 pre-existing unbuilt-sibling-package env failures, unrelated); independent critic CONFIRMED-WITH-CAVEATS, 2 minor findings fixed pre-commit | 7d93a482... | rebased onto main (post #3086/#3094/#3103/#3110/#3119/#3123/#3130/#3139 merges), awaiting fresh CI as of 2026-09-14 | | 2026-09-02 | intelligence | `LearningBridge.consolidate()` reward-blind fix (hardcoded `1.0` → real insight confidence via `getByKey`; round-1 fix had a wrong id/key lookup — `entryId` is the caller-assigned `key`, not the backend's internal `entry.id` — caught by adversarial critic, corrected round 2) | #3159 | #3160 | yes | ACCEPT | 56/56→62/62 tests (candidate); 464/465 full package (1 pre-existing environmental failure); independent critic round 2 CONFIRMED | bead6402... | rebased onto main (post #3086/#3094/#3103/#3110/#3119/#3123/#3130/#3139/#3152 merges), awaiting fresh CI as of 2026-09-14 | +| 2026-09-08 | memory | `MemoryConsolidator.dedup()` deduplicated via byte-exact SHA-256 hash only, though every entry already carries a computed `.embedding` and `dedup()` already holds a handle to the adapter's incrementally-synced `HNSWIndex` (used only for `removePoint` bookkeeping, never queried for similarity) — added a second embedding-cosine near-duplicate pass (threshold 0.95, zero extra backend round-trips); independent adversarial critique found and this session fixed forward a real convergence bug (a duplicate cluster larger than the 8-entry search window split into multiple survivors in one call, self-healing across repeated `runAll()` but under-converging on a single call) | #3231 | #3232 | yes | ACCEPT | 472/473 full-suite green (1 pre-existing unrelated failure, unchanged); baseline-fails/candidate-passes via stash isolation for both the original fix and the convergence fix-forward; `tsc` clean | f5eaa461... | rebased onto main (post #3086/#3094/#3103/#3110/#3119/#3123/#3130/#3139/#3152/#3160 merges), awaiting fresh CI as of 2026-09-14 | diff --git a/docs/dream-cycle/dream-gist-2026-09-08.md b/docs/dream-cycle/dream-gist-2026-09-08.md new file mode 100644 index 0000000000..5c53eae7af --- /dev/null +++ b/docs/dream-cycle/dream-gist-2026-09-08.md @@ -0,0 +1,124 @@ +# Memory SOTA Report — 2026-09-08 + +TL;DR: Tonight's memory deep-dive confirmed and closed a gap flagged (but not selected) by last night's memory-scan finding: `MemoryConsolidator.dedup()` (`v3/@claude-flow/memory/src/consolidator.ts`) only ever collapsed duplicates via byte-exact SHA-256 content hashing, even though every stored entry already carries a computed `.embedding` and the method already holds a handle to the adapter's incrementally-synced `HNSWIndex` — used only for point-removal bookkeeping, never queried for similarity. This is the same "signal already computed, live path discards it for a worse proxy" bug class as three recently-fixed/merged findings (2026-09-03 MMR/#3169, 2026-08-25 PQ-dispatch/#3094, 2026-08-28 HybridBackend-weights/#3119). Five parallel research roles converged independently on the same candidate and patch shape. An orphaned second implementation, `MemoryDomainService.consolidate()` (~700 LOC, zero tests, unreachable from the package's public exports), already does real embedding-cosine grouping but was confirmed NOT directly reusable (it's bound to an injected `IMemoryRepository` requiring a real per-entry `searchByVector()` round-trip; the in-scope `HNSWIndex` handle is strictly cheaper). Fixed by adding an embedding near-duplicate pass directly in `consolidator.ts`, reusing the already-loaded embedding and index with zero extra backend round-trips. An independent adversarial critic then found and this session fixed a real, bounded convergence bug in the first cut of the patch (duplicate clusters wider than the 8-entry search window split into multiple survivors within one call) — documented in full below, including the reproduction and the fix. + +## What's New in 2026 + +| Finding | Source | Confidence | +|---|---|---| +| SemDedup: embed → cluster → intra-cluster cosine-threshold removal, up to ~50% dedup with minimal quality loss | [NVIDIA NeMo Curator SemDedup docs](https://docs.nvidia.com/nemo/curator/curate-text/process-data/deduplication/semdedup) (implements arXiv:2303.09540) | A — reproducible open-source pipeline | +| CrewAI's native `Memory` class: explicit 0.85 cosine-similarity gate → LLM adjudicates keep/update/delete/insert | [CrewAI docs v1.15.2](https://docs.crewai.com/v1.15.2/en/concepts/memory) | B — official vendor docs | +| Mem0: embed-retrieve-similar → single LLM call decides ADD/UPDATE/DELETE/NOOP; LOCOMO 26% relative LLM-judge improvement over OpenAI Memory | arXiv:2504.19413 (ECAI 2025) | B — published, vendor-run benchmark | +| Zep/Graphiti: bi-temporal knowledge graph, entity resolution at write-time, contradictions close a validity window rather than delete; DMR 94.8% vs MemGPT 93.4% | arXiv:2501.13956 | B — published, vendor-run benchmark | +| TiMem Temporal Memory Tree: hierarchical "semantic-guided consolidation... without fine-tuning"; LoCoMo 75.30% with 52.2% reduction in recalled memory length | arXiv:2601.02845 (Jan 2026) | B — single arXiv preprint, concrete numbers, not independently reproduced | +| Qdrant maintainer, on record: no out-of-the-box memory-dedup feature is planned — "each dataset require[s] individual calibration" | [Qdrant GH discussion #3268](https://github.com/orgs/qdrant/discussions/3268) | B — primary-source maintainer statement | +| Mandol: agglomerative clustering + LLM cluster-synthesis for conversational agent memory, benchmarked against Mem0/Zep | arXiv:2606.29778 (Jun 2026) | C — single paper, not yet reproduced | +| Hindsight practitioner blog: "Most agent memory systems are explicit about retrieval and silent about consolidation policy. That silence is where production failures live." Four-lever framework (Importance/Merge/Decay/Eviction) | [hindsight.vectorize.io, 2026-05-21](https://hindsight.vectorize.io/blog/2026/05/21/agent-memory-consolidation) | C — vendor blog, self-reported numbers | +| Mem0's Apr-2026 rework reportedly drops UPDATE/DELETE for single-pass ADD-only accumulation — a direction *opposite* embedding-based consolidation, an open debate not settled practice | [mem0.ai blog, "State of AI Agent Memory 2026"](https://mem0.ai/blog/state-of-ai-agent-memory-2026) | C — vendor blog, unverified | + +Dominant 2025-2026 direction (SemDedup, TiMem, Zep, CrewAI, Mem0) is cluster/embed-then-threshold dedup — exactly what the orphaned `MemoryDomainService` already does and what tonight's fix adds to the wired path. This independently corroborates last night's diagnosis rather than pointing Ruflo toward a different approach. + +## Ruflo Current Capability + +`MemoryConsolidator.dedup()` (`v3/@claude-flow/memory/src/consolidator.ts:140-210` pre-fix) buckets every entry by `sha256(content)` and merges byte-exact matches only — structurally blind to paraphrases/reformattings. It's invoked via `runAll()` (sweep → dedup → compact), reached by `UnifiedMemoryService`'s background timer (opt-in, `config.consolidator.autoRun`) and by the AgentDB `nightlyLearner` controller. + +At the call site, `entries: Map` is already fully resident with each entry's `.embedding?: Float32Array` (`types.ts:66`) inline, and `index: HNSWIndex` is already an in-scope, incrementally-synced handle exposing `.search(query, k, ef?)` — confirmed via direct read of `agentdb-adapter.ts` (index kept in sync on every write/delete) and `hnsw-index.ts` (`search()` is pure in-process graph traversal, no I/O). Fetching a similarity signal here costs zero extra backend round-trips. + +Ruflo also ships a second, orphaned consolidation implementation: `MemoryDomainService.consolidate()` (`domain/services/memory-domain-service.ts:135-331`) does real embedding-cosine grouping (thresholds 0.95 dedupe / 0.8 merge) via `repository.searchByVector()` — but `repository: IMemoryRepository` is an injected interface requiring a genuine async round-trip *per entry* for a real backend, and the concrete backing repository (`HybridMemoryRepository`) does its own brute-force O(n) linear scan, not even HNSW-backed. The whole domain+application DDD chain dead-ends at its own barrel exports — confirmed via repo-wide import grep — and is explicitly excluded from the package's public surface (`index.test.ts:43` asserts `HybridMemoryRepository` must not be exported), consistent with a deliberate ADR-125-style quarantine rather than pure oversight. Zero test files exist for any file in this chain. **Conclusion, independently reached by two research roles**: wiring in the full domain-service architecture is not the smallest fix and was not selected; the in-scope `HNSWIndex` + already-resident `.embedding` is a strictly cheaper, more direct fix. + +One more live gap surfaced while verifying the above (not previously documented): `dedup()`'s per-entry removal loop deletes IDs from `namespaceIndex`/`tagIndex` but — unlike `sweepExpired()`, which explicitly prunes now-empty `Set`s — never removes an emptied `Set` from its parent `Map`, so these maps grow unboundedly across many dedup cycles in a long-lived process. Not fixed tonight (separate, smaller, lower-value finding — see Additional Candidates below). + +## Competitor Comparison + +| System | Mechanism | Evidence Grade | +|---|---|---| +| LangGraph / LangMem SDK | LLM-based background consolidation (`BackgroundMemoryManager`), parallel LLM tool calls decide create/update/delete; no fixed numeric threshold | B | +| AutoGen | No native consolidation; delegates entirely to plugged-in Mem0/Zep | B | +| CrewAI | Embedding-cosine gate (0.85) → LLM adjudicates keep/update/delete/insert | B | +| Mem0 | Embed-retrieve-similar → LLM decides ADD/UPDATE/DELETE/NOOP | B | +| Zep / Graphiti | Bi-temporal knowledge-graph entity resolution + validity-window supersession | B | +| Qdrant / Weaviate / Milvus / LanceDB | **None shipped, deliberately** — maintainer on record: no out-of-the-box solution, "each dataset requires individual calibration" | B (deliberate-avoidance, not oversight) | +| NVIDIA NeMo Curator SemDedup | Embed + cluster dedup for *training* corpora — never adapted to live agent memory in anything found tonight (a genuine, undisputed gap) | B for SemDedup itself; the "agent-memory gap" is a negative result across the search | + +Ruflo's hash-exact-only baseline sat below every framework-layer competitor here and at parity with the vector-DB layer's default of doing nothing. Tonight's fix — an explicit, documented, conservative cosine threshold (0.95) feeding the existing keeper-selection logic — moves Ruflo onto the same mechanism class as CrewAI's 0.85 gate, the most reproducible and simplest-to-benchmark option in this table (versus Mem0/Zep/LangMem's non-deterministic LLM-mediated judging, or Graphiti's much heavier graph-structural machinery). It does not reach Mem0/Zep's *contradiction resolution* (rewriting superseded content, not just dropping a near-duplicate) — a legitimate next rung, out of scope tonight. + +## Hypothesis + +> Given `MemoryConsolidator.dedup()` operating on entries that already carry a resident `.embedding: Float32Array` and an already-populated, incrementally-synced `HNSWIndex` handle it holds but never queries for similarity, when a near-duplicate detection pass is added that queries `index.search()` for each hash-pass-surviving entry with an embedding and merges any group whose cosine similarity is at or above a configurable `similarityThreshold` (default 0.95, matching the orphaned domain-service's own constant) using the existing keeper-selection strategies, then `dedup()` should catch semantic near-duplicates (paraphrases/reformattings) that byte-exact SHA-256 hashing structurally cannot, relative to baseline (hash-exact-only, near-duplicates never merged), subject to: (1) all pre-existing hash-exact-dedup tests and behavior remain unchanged; (2) the near-dup pass is strictly additive — no survivor is dropped without an above-threshold match, and hash-exact duplicates are still merged even without embeddings; (3) entries without `.embedding`, or an index configured with a non-cosine metric, fall back to exactly today's hash-only behavior; (4) a single `dedup()` call fully converges a duplicate cluster of any size to one survivor, not just clusters within the search-neighborhood width; (5) $0 evaluation cost, fully deterministic test-only evaluation, zero LLM calls. + +Frozen before evaluation began. Condition (4) was added mid-session after the adversarial critic (STEP 10) reproduced a violation of it in the first cut of the implementation — see Evaluation below for why this is a legitimate hypothesis refinement (a genuinely new, previously-unstated condition the critique surfaced) rather than a post-hoc loosening: the fix was applied to satisfy the *stronger* condition, not to relax the hypothesis to match a weaker result. + +## Benchmarks / Evaluation + +**evaluated: accepted (ACCEPT).** Real evaluator: Vitest 4.1.8, deterministic, zero LLM calls, $0 cost. + +**Candidate diff** (2 commits on `dream/2026-09-08-memory`): `v3/@claude-flow/memory/src/consolidator.ts` — `selectKeeper`/`mergeGroup` helpers factored out of the original hash-pass logic and reused by a new embedding near-duplicate pass, looped to a fixed point (see Adversarial Critique below for why). New `ConsolidatorOptions.similarityThreshold` (default 0.95; `>= 1` disables the pass entirely). `v3/@claude-flow/memory/src/consolidator.test.ts` — 7 new tests. + +**Baseline vs. candidate, isolated via `git stash` of just the source file (test file kept):** +- First cut (single-round near-dup pass): baseline fails exactly the 2 tests requiring real cosine-merge behavior (`expected 0 to be 1`); candidate passes all 15. 13/15 pass on baseline (all pre-existing + 4 new guard/fallback tests that hold under both old and new code). +- After the convergence fix: re-ran the same isolation against the *first-cut* candidate specifically for the new 15-entry-cluster regression test — it fails (`expected 13 to be 14`, i.e. 2 leftover survivors instead of 1), reproducing the critic's exact finding; the fixed-point-loop version passes (16/16). + +**Full `@claude-flow/memory` package suite**: **472/473 passing** (466 pre-existing + 7 new; 1 pre-existing, unrelated, environmental failure — `auto-memory-bridge.test.ts`, a chmod-based read-only-file test that can't enforce permissions in this root-owned sandbox, confirmed identical with and without this diff and matching the architecture-reviewer's independently-measured pre-change baseline of 465/466). `tsc --noEmit`: zero errors. + +**Adversarial critique (STEP 10, independent subagent, no authoring context)** — verdict **CONFIRMED-WITH-CAVEATS**, full 14-point checklist run and answered with evidence (re-derived from the diff, not trusted from any prior claim): +- No test weakened (diff purely additive on the test file — confirmed via direct diff of the pre-existing 9 tests, byte-identical). +- No gold/expected data touched; no cherry-picking (ran the full suite itself: 471/472 on the first cut, matching what's claimed above pre-convergence-fix). +- No cost/latency/I/O regression — read `hnsw-index.ts`'s `search()`/`removePoint()` directly and confirmed pure in-memory graph traversal. +- Only one production call site (`controller-registry.ts:1021`, `nightlyLearner`) plus `MemoryService.getConsolidator()` — confirmed via repo-wide grep. **Disclosed, not hidden**: `getConsolidator()` does not override `similarityThreshold`, so the near-dup pass is **on by default at 0.95 for every existing `MemoryService` consumer**, not opt-in — a real behavior change for current users of `runAll()`/the background auto-run timer, not just new callers. Judged acceptable given the conservative default and the precedent set by 2026-09-03's MMR fix (also enabled by default), but explicitly flagged here rather than left implicit. +- No undocumented caching, no threshold gaming, no leaked expected answers, no security-boundary crossing (pure `Map`/`HNSWIndex` operations). +- Dimension-agnostic by inspection (no literal dimension anywhere in the new code path). +- **Found a real bug** (the convergence gap described above) — reported with a precise reproduction (15 pairwise-identical-embedding entries, `merged: 13` instead of 14, i.e. 2 survivors instead of 1) and fixed in this session per the report, with a new discriminating regression test added and confirmed to fail pre-fix / pass post-fix. + +## Darwin Results + +Real interface confirmed tonight (`ruvector harness darwin --help`): `darwin --execute`, JSON-genome-driven, requires an explicit `--execute` flag to authorize candidate execution. **Skipped, deliberately, with a reason distinct from most recent nights' skips**: unlike the last several accepted candidates (binary invariant/wiring fixes with no continuous parameter), tonight's `similarityThreshold` *is* a genuine continuous tunable with a real quality/false-positive-rate tradeoff — in principle Darwin-eligible. It was not run tonight because doing so honestly would require a labeled near-duplicate benchmark corpus (true-duplicate vs. true-distinct pairs with ground truth) that does not exist in this repo, and building one is scoped out tonight per STEP 0.6's budget protection (research depth over new evaluation infrastructure). The shipped default (0.95) is not arbitrary — it matches the orphaned domain-service's own already-chosen constant and CrewAI's closely-adjacent 0.85 gate — but a future `memory` DEEP night building a small labeled near-dup corpus and running bounded Darwin evolution over this exact parameter is a concrete, well-scoped follow-up (see Recommended Next Steps). + +## Flywheel Evidence + +Real interface confirmed tonight (`ruvector harness flywheel --help`): `flywheel verify ` / `flywheel gate `. No signed `@metaharness/flywheel` bundle exists for this candidate — the schema targets retrieval-policy candidates evaluated against an LLM-task corpus via paired bootstrap statistics; this is a deterministic code-correctness fix with no such corpus, consistent with every accepted dream-cycle night since 2026-08-18. Evidence retained as: 7 new deterministic tests (including the critic-driven convergence regression) + this gist + the linked issue + the two stash-isolated baseline-fails/candidate-passes comparisons (first-cut and convergence-fix) + the independent adversarial critique. + +Classified: **OBSERVATION** (byte-exact-hash-only dedup, `HNSWIndex` handle held but never queried for similarity — confirmed via direct read + repo-wide grep by two independent research roles) / **MEASUREMENT** (baseline fails exactly the discriminating tests both before and after the convergence fix; full-suite pass count 472/473, unchanged failure identity from the pre-existing baseline) / **INFERENCE** (the orphaned `MemoryDomainService` is not directly reusable — its `IMemoryRepository` abstraction requires real per-entry round-trips; the in-scope `HNSWIndex` is strictly cheaper) / **DECISION** (ship the minimal in-file wiring fix, not the domain-service swap; fix the convergence bug forward rather than merely disclosing it, since a working fix was cheap and available) / **REJECTION** (Darwin evolution of `similarityThreshold`, deferred — no labeled corpus exists; not attempted with a fabricated one). + +## Reward Hack Check + +No standalone generic reward-hack CLI reachable this session (`weight-eft` resolves to a bin name but is a LoRA-distillation tool per its own doctor description, not a diff/benchmark scanner — consistent with every recent night's note). Manual checklist, independently re-verified by the adversarial critic with fresh eyes: no test weakened (purely additive, confirmed via diff); no gold-label leakage (no LLM/gold-answer path in this code); no cherry-picking (full package suite reported both ways, honestly, including the one pre-existing unrelated failure); no seed manipulation (fully deterministic PRNG-seeded embeddings throughout); zero cost (no model calls anywhere); no undocumented caching. The one caveat disclosed rather than hidden: the near-dup pass is on-by-default for existing `MemoryService` consumers (see Evaluation above) — a real behavior change, not a hidden side effect, but worth a reviewer's explicit attention. + +## Security Review + +Not the primary focus, but directly relevant to data-retention correctness: this pass *deletes* entries judged near-duplicate, so an adversary who could influence embedding generation to force a coincidental high-cosine collision with a legitimate memory could, in principle, cause that memory to be silently dropped during consolidation. Mitigations already in place: the default threshold (0.95) is conservative and requires near-exact vector alignment, not merely "related" content; the pass only runs during consolidation (background timer or `nightlyLearner`), not on the synchronous write/store path, so it isn't reachable from a single malicious write in real time; and it operates purely on already-trusted, already-embedded internal state — no new network, filesystem, or credential surface is introduced (confirmed line-by-line by the adversarial critic). No fix required tonight; flagging the threshold-as-attack-surface framing for future `security`-DEEP consideration if Ruflo's embedding-generation path itself is ever exposed to less-trusted input. + +## Scan Findings: plugins + +Re-verified last night's finding (`PluginIntegrityVerifier` unwired, registry `checksum` fields placeholder, `requireVerification` enforced at the registry-listing level only) — **confirmed still accurate**, with new concrete artifacts found tonight: (1) `commands/plugins.ts`'s `--verify` flag (`default: true`, advertised to users) is completely dead — the captured `verify` variable is never read again, so `--verify` and `--no-verify` are byte-identical no-ops; (2) six more `PluginStoreConfig` fields (`requireSecurityAudit`, `minTrustLevel`, `trustedAuthors`, `blockedPlugins`, `requirePermissionPrompt`, `allowedPermissions`) are declared, defaulted, and never read anywhere; (3) `trust-anchors.json` ships a placeholder all-zero public key, so even wiring the verifier in today would validate nothing real. Competitor check: npm's own provenance/Sigstore attestation and VS Code Marketplace's install-time repository-signature verification both show install-time artifact verification is the industry norm for a plugin marketplace shaped like Ruflo's (Grade A sources, both). **Explicit novelty verdict: this is a refinement of a direction already surfaced 3+ times in the ledger** (#2254 2026-06-01, #2630 2026-07-11, last night 2026-09-07) — not filed as a new top-level issue; recorded here as supporting evidence for a future `security`-DEEP night, per STEP 1.1. + +## Scan Findings: automation + +`dream-cycle-backlog-guard.yml` (referenced in the ledger's own notes as addressing the dream-cycle-PR-backlog visibility gap) **does not exist on `main`** — it exists only inside still-open, still-draft PR #3205 (created 2026-09-05, zero comments, untouched since creation — 3 days stale as of tonight). Read directly from the PR diff: it is a **dashboard, not a fix** — a daily cron job that lists aging `dream/*` PRs into a GitHub Step Summary with a `::warning::` annotation past a configurable age threshold; its own header comment says so explicitly ("does not fix the append mechanism... adds the missing visibility"). No other open PR or issue addresses the underlying structural problem (ledger rows only reach `main` via PR merge). Two smaller, new findings in the hooks/automation surface: `commands/hooks.ts`'s `teammate-idle` handler collapses every exception (not just "no MCP server") into a bare success response, discarding the real error; `hooks/src/workers/session-hook.ts`'s per-worker session-start scan loop reports failure but discards the actual error message. Neither implemented tonight (low severity, narrow blast radius, and this cycle's budget was allocated to the memory DEEP surface per STEP 0.6). Primary-source check on evidence-durability patterns for scheduled pipelines found no A-grade source solving this exact case; the closest real precedent is the general "decouple durable-write from review-gate" pattern (transactional-outbox-style), which would mean giving STEP 25 a durability path independent of PR-merge — a concrete design direction for a future `automation` DEEP night, not something implemented anywhere in this repo yet. + +## Additional candidates surfaced but not selected tonight (STEP 3.2) + +Scored under `0.25·Ruflo_fit + 0.20·testability + 0.20·measurability + 0.15·production_value + 0.10·novelty + 0.10·reviewability`; tonight's selected candidate scored 4.80 (deep-researcher rubric) / independently corroborated by the architecture-reviewer's separate patch-plan analysis: + +1. **Prefer `accessCount`/`lastAccessedAt` over pure recency in keeper-selection** (score 4.20) — both fields are already resident on every entry and unused by `dedup()`'s keeper-selection logic; a frequently-accessed-but-slightly-older duplicate can currently lose to a barely-newer, never-accessed one. Same call site, ~10-15 lines. Changes default merge semantics, so deliberately not bundled with tonight's already-two-commit candidate. +2. **Fix `dedup()`'s missing empty-`Set` cleanup for `namespaceIndex`/`tagIndex`** (score 3.95) — `sweepExpired()` already prunes empty `Set`s; `dedup()` doesn't, an unbounded-growth gap in long-lived processes. Trivial (2-4 lines), but a separate conceptual change from tonight's candidate. +3. **Skip redundant per-entry `HNSWIndex.removePoint()` calls when `compactHnsw()` immediately rebuilds the index anyway** (score 3.75) — `runAll()` always follows sweep+dedup with a full rebuild from `entries`, making per-entry removal bookkeeping during those phases provably wasted (and occasionally O(n) per call via `removePoint()`'s entry-point-rescan branch). Needs a synthetic adversarial fixture to demonstrate cleanly; deferred. +4. **Delete or explicitly quarantine the orphaned DDD memory layer** (`MemoryDomainService`/`MemoryApplicationService`/`HybridMemoryRepository`, ~700 LOC, zero tests) (score 2.65) — dead code, but `index.test.ts:43`'s explicit "must not be exported" assertion suggests deliberate quarantine rather than pure oversight; deletion vs. eventual wiring is a design decision, not a bug fix, and needs an ADR rather than a silent removal. +5. **`QLearningRouter`-adjacent findings, `SONAManager`/`PersistentSonaCoordinator` orphans** — carried over from last night's `intelligence` surface, not re-scored tonight (different DEEP surface); still open per that night's own recommendation. + +## SOTA Proof & Witness + +| Field | Value | +|---|---| +| Session commit | `a295c68703158377a6ce827738bf8f13b94bd695` | +| Report SHA-256 (pre-witness content) | `24117f596be1304943be8705b08d1d157bf5904919df355774a027b523ff6909` | +| Witness stamp | `f5eaa46114b6b67e8dcf01e38d73c61aabb9400f8fe6b662cc191b1dc9787827` | + +Verifier procedure: fetch `docs/dream-cycle/dream-gist-2026-09-08.md` from this branch, strip the witness table's filled values back to `PENDING`, SHA-256 the result, concatenate with the session commit above, SHA-256 again — result must equal the witness stamp. + +## Recommended Next Steps + +1. **Merge the linked draft PR** (human review required) — two commits: the initial wiring fix and a fix-forward addressing a real convergence bug an independent adversarial critic found and this session fixed before shipping, with a discriminating regression test proving both the original fix and the fix-forward. +2. **Build a labeled near-duplicate benchmark corpus** (true-duplicate vs. true-distinct embedding pairs with ground truth) so a future night can run real bounded Darwin evolution over `similarityThreshold` instead of shipping a precedent-matched-but-unvalidated default. +3. **Decide the fate of the orphaned DDD memory layer** (`MemoryDomainService` et al., candidate 4 above) — an explicit ADR (keep-quarantined vs. delete vs. eventually-wire-in) would resolve ~700 LOC of ambiguous status that two research roles both had to independently re-derive tonight. +4. **Security-DEEP follow-up on plugin supply-chain integrity** — now surfaced a 4th time (2026-06-01, 2026-07-11, 2026-09-07, tonight) with concrete new artifacts (dead `--verify` flag, placeholder trust anchor, 6 more dead config fields); per STEP 1.1, this direction should be either picked up as a small, reviewable wiring fix on a future `security` night, or explicitly deprioritized with a stated reason, rather than surfaced a 5th time with no decision. +5. **Merge backlog remains real**: as of tonight, `#3086/#3094/#3103/#3110/#3119/#3123/#3130` (2026-08-24 through 08-30) are still open/draft — `#3086` (2026-08-24) is now 15 days old with no meaningful follow-up beyond its own author comments, crossing this pipeline's own STALE threshold (>14 days, no follow-up) for the first time this cycle. `#3139/#3152/#3160` (08-31 through 09-02) are 6-8 days old, not yet stale. Three PRs have merged since 09-03 (`#3169`, `#3184`, `#3221`), showing the review process does work when engaged — the backlog-guard dashboard (PR #3205, still unmerged itself) would make this visible automatically but doesn't yet. diff --git a/scripts/ci-test-baseline.txt b/scripts/ci-test-baseline.txt index 1db6eed536..3f7ce73c82 100644 --- a/scripts/ci-test-baseline.txt +++ b/scripts/ci-test-baseline.txt @@ -92,6 +92,7 @@ v3/@claude-flow/hooks/src/__tests__/guidance-provider.test.ts v3/@claude-flow/hooks/src/__tests__/reasoningbank.test.ts v3/@claude-flow/memory/src/agentdb-backend.test.ts v3/@claude-flow/memory/src/agentdb-retrieval-guard.test.ts +v3/@claude-flow/memory/src/consolidator-embedding-benchmark.test.ts v3/@claude-flow/memory/src/consolidator.test.ts v3/@claude-flow/memory/src/controller-registry.test.ts v3/@claude-flow/memory/src/database-provider.test.ts @@ -121,3 +122,8 @@ v3/plugins/quantum-optimizer/tests/mcp-tools.test.ts # ruflo-x-gateway: node:test suite with its own deps (nostr-tools, ws, MCP SDK) — run via `npm test` in plugins/ruflo-x-gateway; root vitest cannot resolve them (same pattern as plugins/ruflo-adr) plugins/ruflo-x-gateway/test/gateway.test.mjs +plugins/ruflo-x-gateway/test/oauth.test.mjs + +# ruflo-chatgpt-federation: same pattern — plugin-local deps (jose, nostr-tools/pure) not resolvable from root vitest +plugins/ruflo-chatgpt-federation/test/oauth.test.mjs +plugins/ruflo-chatgpt-federation/test/publisher.test.mjs diff --git a/v3/@claude-flow/memory/src/consolidator-embedding-benchmark.test.ts b/v3/@claude-flow/memory/src/consolidator-embedding-benchmark.test.ts new file mode 100644 index 0000000000..21ed8f15f5 --- /dev/null +++ b/v3/@claude-flow/memory/src/consolidator-embedding-benchmark.test.ts @@ -0,0 +1,398 @@ +/** + * Phase 4.1 — embedding near-duplicate pass: precision/recall, latency, + * namespace-isolation, and determinism receipts (response to PR #3232 + * review: "the 0.95 threshold needs an empirical contract rather than a + * smoke fixture"). + * + * IMPORTANT — corpus provenance: this file does NOT use a real pinned + * embedding model. Both real ONNX backends this monorepo ships + * (`@huggingface/transformers` and the legacy `@xenova/transformers`) were + * attempted directly in this environment and both fail for reasons + * unrelated to this candidate: + * - `@huggingface/transformers@4.2.0`: `transformers.node.mjs` does + * `import { Tensor } from "onnxruntime-common"` against a CJS module, + * a real ESM/CJS interop break under this Node version — reproduces + * on an unmodified checkout, confirmed before writing this file. + * - `@xenova/transformers@2.17.0`: pulls in `sharp@0.32.6`, whose + * native binding (`sharp-linux-x64.node`) isn't present for this + * sandbox's platform. + * Neither is fixable within this PR's scope without touching unrelated, + * pre-existing native/module-resolution infrastructure this diff never + * touches. This package's own `__tests__/embeddings.test.ts` exercises + * only the `mock` provider — it never invokes a real transformer either, + * consistent with what was found here. + * + * What this file DOES provide instead: a deterministic synthetic corpus + * where pairwise cosine similarity is constructed EXACTLY (not measured + * after the fact) via `pairAtSimilarity()` below — for every pair the + * ground-truth similarity is known analytically, not estimated. That + * makes this a rigorous test of the near-dup pass's THRESHOLD MECHANICS + * (precision/recall as a function of `similarityThreshold`, exactly where + * the boundary is) — arguably a cleaner instrument for that specific + * question than a real corpus would be, since there is no measurement + * noise in the "true" similarity. It does NOT validate that 0.95 is the + * right cutoff for real-world semantic paraphrase detection with a real + * embedding model — that claim remains open pending the blocked ONNX + * path, and is called out explicitly, not implied. + */ + +import { describe, it, expect } from 'vitest'; +import { MemoryService } from './index.js'; +import { MemoryConsolidator } from './consolidator.js'; +import { createDefaultEntry } from './types.js'; + +/** Deterministic xorshift PRNG, seeded — matches consolidator.test.ts's randomVec. */ +function seededRng(seed: number): () => number { + let s = seed | 0 || 1; + return () => { + s ^= s << 13; + s ^= s >>> 17; + s ^= s << 5; + return (s | 0) / 2 ** 31; + }; +} + +function randomUnitVector(dim: number, seed: number): Float32Array { + const rng = seededRng(seed); + const v = new Float32Array(dim); + for (let i = 0; i < dim; i++) v[i] = rng(); + return normalize(v); +} + +function normalize(v: Float32Array): Float32Array { + let norm = 0; + for (let i = 0; i < v.length; i++) norm += v[i] * v[i]; + norm = Math.sqrt(norm) || 1; + const out = new Float32Array(v.length); + for (let i = 0; i < v.length; i++) out[i] = v[i] / norm; + return out; +} + +function dot(a: Float32Array, b: Float32Array): number { + let s = 0; + for (let i = 0; i < a.length; i++) s += a[i] * b[i]; + return s; +} + +/** + * Construct a unit vector `b` such that cosine(a, b) === targetSimilarity + * exactly (up to float precision), by mixing `a` with a vector orthogonal + * to it: b = s*a + sqrt(1-s^2)*orth, |a|=|orth|=1, a⊥orth. + */ +function vectorAtSimilarity(a: Float32Array, targetSimilarity: number, seed: number): Float32Array { + const dim = a.length; + // Random candidate, then Gram-Schmidt to make it orthogonal to `a`. + let orth = randomUnitVector(dim, seed); + const proj = dot(orth, a); + const raw = new Float32Array(dim); + for (let i = 0; i < dim; i++) raw[i] = orth[i] - proj * a[i]; + orth = normalize(raw); + + const s = targetSimilarity; + const c = Math.sqrt(Math.max(0, 1 - s * s)); + const out = new Float32Array(dim); + for (let i = 0; i < dim; i++) out[i] = s * a[i] + c * orth[i]; + return out; +} + +async function newService(dimensions = 32) { + const svc = new MemoryService({ dimensions, persistenceEnabled: false, snapshotInterval: 0 } as any); + await svc.initialize(); + return svc; +} + +interface Pair { + label: string; + targetSimilarity: number; + shouldMergeAtDefault: boolean; // ground truth given the shipped 0.95 default +} + +const DIM = 32; + +// Base anchor vectors, one per pair, all mutually near-orthogonal (distinct +// content clusters) so pairs don't cross-contaminate each other's matches. +function buildCorpus(): { anchors: Float32Array[]; pairs: Pair[] } { + const pairs: Pair[] = [ + { label: 'near-dup-0.999', targetSimilarity: 0.999, shouldMergeAtDefault: true }, + { label: 'near-dup-0.98', targetSimilarity: 0.98, shouldMergeAtDefault: true }, + { label: 'near-dup-0.96', targetSimilarity: 0.96, shouldMergeAtDefault: true }, + { label: 'boundary-0.955', targetSimilarity: 0.955, shouldMergeAtDefault: true }, + { label: 'hard-negative-0.94', targetSimilarity: 0.94, shouldMergeAtDefault: false }, + { label: 'hard-negative-0.90', targetSimilarity: 0.90, shouldMergeAtDefault: false }, + { label: 'hard-negative-0.85', targetSimilarity: 0.85, shouldMergeAtDefault: false }, + { label: 'easy-negative-0.30', targetSimilarity: 0.30, shouldMergeAtDefault: false }, + ]; + const anchors = pairs.map((_, i) => randomUnitVector(DIM, 1000 + i * 7)); + return { anchors, pairs }; +} + +describe('Phase 4.1 embedding benchmark — pairwise similarity is exact by construction', () => { + it('vectorAtSimilarity produces the requested cosine similarity to within 1e-5', () => { + const a = randomUnitVector(DIM, 42); + for (const target of [0.999, 0.96, 0.955, 0.9, 0.85, 0.3, 0, -0.5]) { + const b = vectorAtSimilarity(a, target, 99); + expect(dot(a, b)).toBeCloseTo(target, 5); + } + }); +}); + +describe('Phase 4.1 embedding benchmark — precision/recall across a threshold sweep', () => { + it('at the shipped default (0.95), every near-dup pair merges and every hard negative does not', async () => { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any); + const { anchors, pairs } = buildCorpus(); + + const ids: Record = {}; + for (let i = 0; i < pairs.length; i++) { + const a = createDefaultEntry({ key: `${pairs[i].label}-a`, content: `${pairs[i].label} canonical text ${i}` }); + a.embedding = anchors[i]; + await svc.store(a); + const b = createDefaultEntry({ key: `${pairs[i].label}-b`, content: `${pairs[i].label} variant text ${i}` }); + b.embedding = vectorAtSimilarity(anchors[i], pairs[i].targetSimilarity, 5000 + i); + await svc.store(b); + ids[pairs[i].label] = [a.id, b.id]; + } + + await consolidator.dedup('keep-newest'); + const adapter: any = svc.getAdapter(); + + let truePositives = 0; + let falseNegatives = 0; + let trueNegatives = 0; + let falsePositives = 0; + for (const pair of pairs) { + const [idA, idB] = ids[pair.label]; + const bothSurvive = adapter.entries.has(idA) && adapter.entries.has(idB); + const merged = !bothSurvive; + if (pair.shouldMergeAtDefault) { + if (merged) truePositives++; + else falseNegatives++; + } else { + if (merged) falsePositives++; + else trueNegatives++; + } + } + + // Ground-truth contract for the shipped default: clean separation + // around the 0.95 boundary in this corpus — no false merges, no + // false splits among the near-dup/hard-negative pairs constructed + // above. + expect(falsePositives).toBe(0); // no hard negative incorrectly merged + expect(falseNegatives).toBe(0); // no true near-dup left unmerged + expect(truePositives).toBe(pairs.filter((p) => p.shouldMergeAtDefault).length); + expect(trueNegatives).toBe(pairs.filter((p) => !p.shouldMergeAtDefault).length); + + await svc.close(); + }); + + it('reports precision/recall/false-merge-rate across a threshold sweep', async () => { + // Deliberately offset from every corpus pair's constructed similarity + // (0.999/0.98/0.96/0.955/0.94/0.90/0.85/0.30) by a margin large enough + // to absorb float32 precision jitter from HNSW's internal + // normalize-then-dot-product distance computation (empirically <1e-5 + // here, but a boundary value with ZERO margin — e.g. sweeping exactly + // 0.90 against a pair constructed at exactly 0.90 — is a real flake + // this file hit and fixed while writing it; the margin here is the fix). + const thresholds = [0.5, 0.8, 0.875, 0.92, 0.97]; + const rows: Array<{ + threshold: number; + precision: number; + recall: number; + falseMergeRate: number; + }> = []; + + for (const threshold of thresholds) { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any, { similarityThreshold: threshold }); + const { anchors, pairs: p2 } = buildCorpus(); + const ids: Record = {}; + for (let i = 0; i < p2.length; i++) { + const a = createDefaultEntry({ key: `${p2[i].label}-a`, content: `${p2[i].label} canonical text ${i}` }); + a.embedding = anchors[i]; + await svc.store(a); + const b = createDefaultEntry({ key: `${p2[i].label}-b`, content: `${p2[i].label} variant text ${i}` }); + b.embedding = vectorAtSimilarity(anchors[i], p2[i].targetSimilarity, 5000 + i); + await svc.store(b); + ids[p2[i].label] = [a.id, b.id]; + } + await consolidator.dedup('keep-newest'); + const adapter: any = svc.getAdapter(); + + // Ground truth here is "similarity >= threshold" (not the fixed + // 0.95-default contract used above), so precision/recall are + // measured against what SHOULD merge at each threshold under test. + let tp = 0; + let fp = 0; + let fn = 0; + let tn = 0; + for (const pair of p2) { + const [idA, idB] = ids[pair.label]; + const merged = !(adapter.entries.has(idA) && adapter.entries.has(idB)); + const shouldMerge = pair.targetSimilarity >= threshold; + if (shouldMerge && merged) tp++; + else if (shouldMerge && !merged) fn++; + else if (!shouldMerge && merged) fp++; + else tn++; + } + const precision = tp + fp === 0 ? 1 : tp / (tp + fp); + const recall = tp + fn === 0 ? 1 : tp / (tp + fn); + const falseMergeRate = fp + tn === 0 ? 0 : fp / (fp + tn); + rows.push({ threshold, precision, recall, falseMergeRate }); + + await svc.close(); + } + + // eslint-disable-next-line no-console + console.log('similarityThreshold sweep (synthetic exact-similarity corpus):', JSON.stringify(rows)); + + // Every threshold in this sweep gets perfect precision/recall on this + // corpus, because pair similarities were deliberately placed with a + // >=0.005 margin on either side of each candidate threshold. That is + // the corpus proving the near-dup pass's comparison (`similarity >= + // threshold`) is exact and monotonic, not that any one threshold is + // "best" — this sweep is a mechanics check, not a model-selection + // benchmark. + for (const row of rows) { + expect(row.precision).toBe(1); + expect(row.recall).toBe(1); + expect(row.falseMergeRate).toBe(0); + } + }); +}); + +describe('Phase 4.1 embedding benchmark — namespace isolation (pre-existing behavior, unchanged)', () => { + it('cross-namespace near-duplicates merge exactly as cross-namespace hash-exact duplicates already did', async () => { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any); + + // Hash-exact case: identical content, two namespaces. dedup()'s + // bucket-by-hash pass has never scoped by namespace (grep-confirmed + // on the pre-candidate code: `for (const entry of entries.values())` + // with no namespace filter) — this candidate does not change that. + const h1 = createDefaultEntry({ key: 'h', content: 'exact-duplicate-content', namespace: 'ns-a' }); + await svc.store(h1); + const h2 = createDefaultEntry({ key: 'h', content: 'exact-duplicate-content', namespace: 'ns-b' }); + await svc.store(h2); + + // Near-dup case: different content, identical embedding, two namespaces. + const anchor = randomUnitVector(DIM, 777); + const e1 = createDefaultEntry({ key: 'e', content: 'namespace-a-phrasing', namespace: 'ns-a' }); + e1.embedding = anchor; + await svc.store(e1); + const e2 = createDefaultEntry({ key: 'e', content: 'namespace-b-phrasing', namespace: 'ns-b' }); + e2.embedding = new Float32Array(anchor); + await svc.store(e2); + + const result = await consolidator.dedup('keep-newest'); + + const adapter: any = svc.getAdapter(); + // Both the hash-exact pair and the near-dup pair merge across the + // namespace boundary — documented, not silently changed. If Ruflo + // later decides dedup() should be namespace-scoped, that's a + // separate, pre-existing-behavior-affecting decision spanning both + // passes, not something introduced by this candidate. + const hashSurvivors = [h1.id, h2.id].filter((id) => adapter.entries.has(id)); + const embedSurvivors = [e1.id, e2.id].filter((id) => adapter.entries.has(id)); + expect(hashSurvivors.length).toBe(1); + expect(embedSurvivors.length).toBe(1); + expect(result.merged).toBe(2); + + await svc.close(); + }); +}); + +describe('Phase 4.1 embedding benchmark — determinism / replay', () => { + it('the same corpus produces byte-identical merge/group counts and survivor keys across 3 independent runs', async () => { + // Entry ids are randomly generated per store() call (mem__), + // so they are NOT expected to match across independent runs even when + // merge behavior is fully deterministic — comparing raw ids here would + // be testing the id generator, not dedup(). Compare by `key` instead, + // which this corpus constructs deterministically from `pairs`. + async function runOnce() { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any); + const { anchors, pairs } = buildCorpus(); + const idToKey = new Map(); + for (let i = 0; i < pairs.length; i++) { + const aKey = `${pairs[i].label}-a`; + const a = createDefaultEntry({ key: aKey, content: `${pairs[i].label} canonical text ${i}` }); + a.embedding = anchors[i]; + a.updatedAt = 1000 + i * 2; + await svc.store(a); + idToKey.set(a.id, aKey); + const bKey = `${pairs[i].label}-b`; + const b = createDefaultEntry({ key: bKey, content: `${pairs[i].label} variant text ${i}` }); + b.embedding = vectorAtSimilarity(anchors[i], pairs[i].targetSimilarity, 5000 + i); + b.updatedAt = 1000 + i * 2 + 1; // always the newer of the pair + await svc.store(b); + idToKey.set(b.id, bKey); + } + const result = await consolidator.dedup('keep-newest'); + const adapter: any = svc.getAdapter(); + const survivorKeys = [...idToKey.entries()] + .filter(([id]) => adapter.entries.has(id)) + .map(([, key]) => key) + .sort(); + await svc.close(); + return { merged: result.merged, groups: result.groups, survivorKeys }; + } + + // Sequential, not parallel — isolates each run's timing/state fully, + // removing any doubt about event-loop interleaving as a confound. + const r1 = await runOnce(); + const r2 = await runOnce(); + const r3 = await runOnce(); + expect(r2).toEqual(r1); + expect(r3).toEqual(r1); + }); +}); + +describe('Phase 4.1 embedding benchmark — latency/memory at realistic cardinality', () => { + it('dedup() at N=5000 (matching this repo\'s own HNSW benchmark convention) completes within a generous ceiling, baseline vs candidate', async () => { + const N = 5000; + const DUP_FRACTION = 0.1; // 10% of entries are near-duplicates of an earlier entry + + async function buildAndRun(similarityThreshold: number) { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any, { similarityThreshold }); + const anchors: Float32Array[] = []; + for (let i = 0; i < N; i++) { + const isDup = i > 0 && i % Math.round(1 / DUP_FRACTION) === 0; + let embedding: Float32Array; + if (isDup) { + const anchor = anchors[Math.floor(Math.random() * anchors.length)]; + embedding = vectorAtSimilarity(anchor, 0.985, i); + } else { + embedding = randomUnitVector(DIM, 20000 + i); + anchors.push(embedding); + } + const entry = createDefaultEntry({ key: `n-${i}`, content: `entry-${i}-${isDup ? 'dup' : 'unique'}` }); + entry.embedding = embedding; + await svc.store(entry); + } + + const memBefore = process.memoryUsage().heapUsed; + const t0 = Date.now(); + const result = await consolidator.dedup('keep-newest'); + const durationMs = Date.now() - t0; + const memAfter = process.memoryUsage().heapUsed; + + await svc.close(); + return { durationMs, heapDeltaMb: (memAfter - memBefore) / (1024 * 1024), merged: result.merged }; + } + + const baseline = await buildAndRun(1); // near-dup pass disabled — hash-only, matches pre-candidate behavior + const candidate = await buildAndRun(0.95); // shipped default + + // eslint-disable-next-line no-console + console.log('latency/memory @ N=5000:', JSON.stringify({ baseline, candidate })); + + // Generous ceilings — this is a regression guard against a pathological + // blowup (e.g., an accidental O(n^2) reintroduction), not a tight perf + // assertion; CI hardware varies. Real numbers are logged above for the + // reviewer regardless of pass/fail margin. + expect(candidate.durationMs).toBeLessThan(15_000); + expect(candidate.merged).toBeGreaterThan(0); // the injected near-dups were actually caught + expect(baseline.merged).toBe(0); // hash-only baseline: all content strings are unique, so 0 hash-exact merges + }, 30_000); +}); diff --git a/v3/@claude-flow/memory/src/consolidator.test.ts b/v3/@claude-flow/memory/src/consolidator.test.ts index c98cb6a11b..a3b85a22c2 100644 --- a/v3/@claude-flow/memory/src/consolidator.test.ts +++ b/v3/@claude-flow/memory/src/consolidator.test.ts @@ -17,6 +17,7 @@ import { describe, it, expect, vi } from 'vitest'; import { MemoryService } from './index.js'; import { MemoryConsolidator } from './consolidator.js'; +import { HNSWIndex } from './hnsw-index.js'; import { ControllerRegistry } from './controller-registry.js'; import { createDefaultEntry } from './types.js'; @@ -188,6 +189,174 @@ describe('Phase 4 — MemoryConsolidator.dedup', () => { }); }); +describe('Phase 4.1 — MemoryConsolidator.dedup embedding near-duplicates', () => { + it('merges different-content entries whose embeddings are near-identical (cosine >= threshold)', async () => { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any); + + const base = randomVec(8, 42); + + const a = createDefaultEntry({ key: 'a', content: 'paraphrase-one' }); + a.embedding = base; + a.updatedAt = 1000; + await svc.store(a); + + const b = createDefaultEntry({ + key: 'b', + content: 'paraphrase-two-totally-different-text', + }); + b.embedding = new Float32Array(base); // identical vector, different content/hash + b.updatedAt = 2000; + await svc.store(b); + + const result = await consolidator.dedup('keep-newest'); + expect(result.merged).toBe(1); + expect(result.groups).toBe(1); + + const adapter: any = svc.getAdapter(); + expect(adapter.entries.size).toBe(1); + expect(adapter.entries.has(b.id)).toBe(true); // newer of the pair survives + + await svc.close(); + }); + + it('does not merge entries whose embeddings fall below the similarity threshold', async () => { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any); + + const a = createDefaultEntry({ key: 'a', content: 'unrelated-one' }); + a.embedding = randomVec(8, 1); + await svc.store(a); + + const b = createDefaultEntry({ key: 'b', content: 'unrelated-two' }); + b.embedding = randomVec(8, 999); + await svc.store(b); + + const result = await consolidator.dedup(); + expect(result.merged).toBe(0); + expect(result.groups).toBe(0); + + const adapter: any = svc.getAdapter(); + expect(adapter.entries.size).toBe(2); + await svc.close(); + }); + + it('falls back to hash-only behavior for entries with no embedding', async () => { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any); + + const a = createDefaultEntry({ key: 'a', content: 'no-embedding-a' }); + await svc.store(a); + const b = createDefaultEntry({ key: 'b', content: 'no-embedding-b' }); + await svc.store(b); + + const result = await consolidator.dedup(); + expect(result.merged).toBe(0); + expect(result.groups).toBe(0); + await svc.close(); + }); + + it('respects a custom similarityThreshold option', async () => { + const svc = await newService(); + // A threshold below cosine's minimum (-1) means every pair is "similar". + const consolidator = new MemoryConsolidator(svc as any, { similarityThreshold: -1 }); + + const a = createDefaultEntry({ key: 'a', content: 'threshold-one' }); + a.embedding = randomVec(8, 11); + await svc.store(a); + const b = createDefaultEntry({ key: 'b', content: 'threshold-two' }); + b.embedding = randomVec(8, 22); + await svc.store(b); + + const result = await consolidator.dedup(); + expect(result.merged).toBe(1); + expect(result.groups).toBe(1); + await svc.close(); + }); + + it('disables the embedding pass entirely when similarityThreshold >= 1', async () => { + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any, { similarityThreshold: 1 }); + + const embedding = randomVec(8, 5); + const a = createDefaultEntry({ key: 'a', content: 'disabled-one' }); + a.embedding = embedding; + await svc.store(a); + const b = createDefaultEntry({ key: 'b', content: 'disabled-two' }); + b.embedding = new Float32Array(embedding); // identical vector, still not merged + await svc.store(b); + + const result = await consolidator.dedup(); + expect(result.merged).toBe(0); + expect(result.groups).toBe(0); + await svc.close(); + }); + + it('fully converges a near-duplicate cluster larger than the search neighborhood in one call', async () => { + // Adversarial-critique regression: a cluster wider than + // NEAR_DUP_SEARCH_K must still collapse to a single survivor from one + // dedup() call, not split into multiple leftover sub-group survivors. + const svc = await newService(); + const consolidator = new MemoryConsolidator(svc as any); + + const base = randomVec(8, 12345); + const n = 15; // > NEAR_DUP_SEARCH_K (8) + for (let i = 0; i < n; i++) { + const entry = createDefaultEntry({ key: `cluster-${i}`, content: `unique-content-${i}` }); + entry.embedding = new Float32Array(base); // pairwise cosine similarity 1.0 + entry.updatedAt = 1000 + i; + await svc.store(entry); + } + + const result = await consolidator.dedup('keep-newest'); + expect(result.merged).toBe(n - 1); + + const adapter: any = svc.getAdapter(); + expect(adapter.entries.size).toBe(1); + // The single survivor should be the newest of the cluster. + const survivor: any = [...adapter.entries.values()][0]; + expect(survivor.updatedAt).toBe(1000 + n - 1); + + await svc.close(); + }); + + it('skips the near-duplicate pass entirely for a non-cosine-metric index', async () => { + const index = new HNSWIndex({ dimensions: 8, metric: 'euclidean' }); + const embedding = randomVec(8, 7); + + const a = createDefaultEntry({ key: 'a', content: 'euclid-one' }); + a.embedding = embedding; + const b = createDefaultEntry({ key: 'b', content: 'euclid-two' }); + b.embedding = new Float32Array(embedding); // identical vector, different content + + await index.addPoint(a.id, a.embedding); + await index.addPoint(b.id, b.embedding); + + const entries = new Map([ + [a.id, a], + [b.id, b], + ]); + const namespaceIndex = new Map>([ + [a.namespace, new Set([a.id, b.id])], + ]); + const keyIndex = new Map([ + [`${a.namespace}:${a.key}`, a.id], + [`${b.namespace}:${b.key}`, b.id], + ]); + const tagIndex = new Map>(); + + const fakeService = { + getAdapter: () => ({ entries, namespaceIndex, keyIndex, tagIndex, index }), + }; + const consolidator = new MemoryConsolidator(fakeService as any); + + const result = await consolidator.dedup(); + expect(result.merged).toBe(0); + expect(result.groups).toBe(0); + expect(entries.size).toBe(2); + }); +}); + describe('Phase 4 — MemoryConsolidator.compactHnsw', () => { it('rebuilds the HNSW index so size matches entries.size after sweep', async () => { const svc = await newService(); diff --git a/v3/@claude-flow/memory/src/consolidator.ts b/v3/@claude-flow/memory/src/consolidator.ts index aba044cb93..abcc4f0fb4 100644 --- a/v3/@claude-flow/memory/src/consolidator.ts +++ b/v3/@claude-flow/memory/src/consolidator.ts @@ -36,8 +36,35 @@ export interface ConsolidatorOptions { dedupStrategy?: DedupStrategy; /** Used by `MemoryService` when scheduling automatic runs (ms). */ intervalMs?: number; + /** + * Cosine-similarity floor (0-1) above which two entries with embeddings + * are treated as near-duplicates by `dedup()`'s embedding pass, in + * addition to its byte-exact content-hash pass. Set to `1` (or above) to + * disable the embedding pass entirely and keep pre-Phase-4.1 hash-only + * behavior. Default 0.95 — conservative, matches the threshold already + * used by the (unwired) domain-layer consolidator for the same purpose. + */ + similarityThreshold?: number; } +const DEFAULT_SIMILARITY_THRESHOLD = 0.95; +/** Neighborhood size for the near-duplicate HNSW lookup — small and cheap. */ +const NEAR_DUP_SEARCH_K = 8; +/** + * Explicit `ef` (candidate-list size) for the near-duplicate HNSW search. + * Left unset, `HNSWIndex.search()` defaults `ef` to + * `Math.max(k, efConstruction)` — efConstruction defaults to 200, so an + * unset `ef` here means every dedup() search traverses a 200-candidate + * list even though duplicate detection only needs to find very-close + * neighbors (high similarity), not broad top-k recall. Measured impact at + * N=5000 (`consolidator-embedding-benchmark.test.ts`): ~9.8s with the + * inherited ef=200 default vs a small fixed ef here. 32 is a deliberate + * margin above NEAR_DUP_SEARCH_K (8) for HNSW's approximate-search + * headroom, not a re-derivation of efConstruction's separate + * index-build-quality concern. + */ +const NEAR_DUP_SEARCH_EF = 32; + export interface SweepResult { removed: number; remaining: number; @@ -131,7 +158,85 @@ export class MemoryConsolidator { } /** - * Collapse content-hash duplicates across all namespaces per `strategy`. + * Pick the keeper for a duplicate group per `strategy` and, for + * `merge-tags`, union the group's tag sets onto it. Shared by both the + * content-hash pass and the embedding near-duplicate pass in `dedup()`. + */ + private selectKeeper(bucket: MemoryEntry[], strategy: DedupStrategy): MemoryEntry { + const keeper = + strategy === 'keep-oldest' + ? bucket.reduce((acc, e) => (e.createdAt < acc.createdAt ? e : acc)) + : // keep-newest + merge-tags share this branch + bucket.reduce((acc, e) => (e.updatedAt > acc.updatedAt ? e : acc)); + return keeper; + } + + /** + * Drop every entry in `bucket` except the strategy-selected keeper, + * updating all adapter indexes (and the HNSW index, for entries that have + * an embedding) to match. Shared by both dedup passes. + */ + private async mergeGroup( + bucket: MemoryEntry[], + strategy: DedupStrategy, + ctx: { + entries: Map; + namespaceIndex: Map>; + keyIndex: Map; + tagIndex: Map>; + index: HNSWIndex; + } + ): Promise<{ keeperId: string; dropped: number }> { + const keeper = this.selectKeeper(bucket, strategy); + + if (strategy === 'merge-tags') { + const tagSet = new Set(keeper.tags); + for (const e of bucket) for (const t of e.tags) tagSet.add(t); + const oldTags = keeper.tags; + keeper.tags = [...tagSet]; + // Update tagIndex membership for newly-added tags + for (const t of keeper.tags) { + if (!oldTags.includes(t)) { + if (!ctx.tagIndex.has(t)) ctx.tagIndex.set(t, new Set()); + ctx.tagIndex.get(t)!.add(keeper.id); + } + } + } + + let dropped = 0; + for (const e of bucket) { + if (e.id === keeper.id) continue; + ctx.entries.delete(e.id); + ctx.namespaceIndex.get(e.namespace)?.delete(e.id); + const compositeKey = `${e.namespace}:${e.key}`; + if (ctx.keyIndex.get(compositeKey) === e.id) { + ctx.keyIndex.delete(compositeKey); + } + for (const tag of e.tags) ctx.tagIndex.get(tag)?.delete(e.id); + if (e.embedding) { + await ctx.index.removePoint(e.id); + } + dropped += 1; + } + return { keeperId: keeper.id, dropped }; + } + + /** + * Collapse duplicates across all namespaces per `strategy`, in two passes: + * + * 1. Byte-exact content-hash buckets (unchanged from pre-4.1 behavior). + * 2. Embedding near-duplicates: for entries that survive pass 1 and carry + * an `embedding`, query the already-populated `HNSWIndex` (the same + * index instance this method already holds a handle to for removal + * bookkeeping) for cosine-similarity neighbors above + * `opts.similarityThreshold` (default 0.95). This catches paraphrases + * and reformattings that byte-exact hashing structurally cannot, at no + * extra backend round-trip cost — the embedding and the index are both + * already resident in memory at this call site. + * + * Pass 2 only runs when the index's configured metric is `'cosine'` (its + * distances are `1 - similarity`) and the threshold is `< 1`; otherwise + * behavior is identical to hash-only dedup. * * - `keep-newest`: keep the entry with the highest `updatedAt`, drop the rest. * - `keep-oldest`: keep the entry with the lowest `createdAt`, drop the rest. @@ -144,10 +249,11 @@ export class MemoryConsolidator { const keyIndex: Map = adapter.keyIndex; const tagIndex: Map> = adapter.tagIndex; const index: HNSWIndex = adapter.index; + const ctx = { entries, namespaceIndex, keyIndex, tagIndex, index }; const effective = strategy ?? this.opts.dedupStrategy ?? 'keep-newest'; - // Bucket by content hash + // Pass 1: bucket by content hash (byte-exact duplicates) const buckets = new Map(); for (const entry of entries.values()) { const hash = createHash('sha256').update(entry.content).digest('hex'); @@ -161,49 +267,59 @@ export class MemoryConsolidator { for (const bucket of buckets.values()) { if (bucket.length <= 1) continue; dupGroups += 1; + const { dropped } = await this.mergeGroup(bucket, effective, ctx); + merged += dropped; + } - // Choose keeper - let keeper: MemoryEntry; - if (effective === 'keep-oldest') { - keeper = bucket.reduce((acc, e) => - e.createdAt < acc.createdAt ? e : acc - ); - } else { - // keep-newest + merge-tags share this branch - keeper = bucket.reduce((acc, e) => - e.updatedAt > acc.updatedAt ? e : acc - ); - } - - if (effective === 'merge-tags') { - const tagSet = new Set(keeper.tags); - for (const e of bucket) for (const t of e.tags) tagSet.add(t); - const oldTags = keeper.tags; - keeper.tags = [...tagSet]; - // Update tagIndex membership for newly-added tags - for (const t of keeper.tags) { - if (!oldTags.includes(t)) { - if (!tagIndex.has(t)) tagIndex.set(t, new Set()); - tagIndex.get(t)!.add(keeper.id); + // Pass 2: embedding near-duplicates among pass-1 survivors. + // + // Each round only groups an entry with what its single + // `NEAR_DUP_SEARCH_K`-wide HNSW query returns, so a duplicate cluster + // larger than that window can split into more than one surviving + // sub-group in a single round (each sub-group's keeper remains + // un-merged with the others). Looping to a fixed point — re-scanning + // until a full round produces zero merges — guarantees a single + // `dedup()` call fully converges regardless of cluster size, at the + // cost of one extra full round whenever a cluster does split (bounded: + // strictly fewer entries remain each round that merges anything, so + // this always terminates). `groups` therefore counts merge operations + // across all rounds, which can exceed the number of ultimate + // underlying duplicate clusters when one of them needed more than one + // round to fully collapse. + const threshold = this.opts.similarityThreshold ?? DEFAULT_SIMILARITY_THRESHOLD; + if (index.getConfig().metric === 'cosine' && threshold < 1) { + let roundMerged: number; + do { + roundMerged = 0; + const consumed = new Set(); + // Snapshot survivors up front — mutated further as groups merge. + for (const entry of [...entries.values()]) { + if (consumed.has(entry.id) || !entry.embedding || !entries.has(entry.id)) { + continue; } - } - } - // Drop everyone except the keeper - for (const e of bucket) { - if (e.id === keeper.id) continue; - entries.delete(e.id); - namespaceIndex.get(e.namespace)?.delete(e.id); - const compositeKey = `${e.namespace}:${e.key}`; - if (keyIndex.get(compositeKey) === e.id) { - keyIndex.delete(compositeKey); + const hits = await index.search(entry.embedding, NEAR_DUP_SEARCH_K, NEAR_DUP_SEARCH_EF); + const group: MemoryEntry[] = [entry]; + for (const hit of hits) { + if (hit.id === entry.id || consumed.has(hit.id)) continue; + const candidate = entries.get(hit.id); + if (!candidate || !candidate.embedding) continue; + const similarity = 1 - hit.distance; + if (similarity >= threshold) group.push(candidate); + } + + if (group.length <= 1) { + consumed.add(entry.id); + continue; + } + + dupGroups += 1; + const { dropped } = await this.mergeGroup(group, effective, ctx); + merged += dropped; + roundMerged += dropped; + for (const e of group) consumed.add(e.id); } - for (const tag of e.tags) tagIndex.get(tag)?.delete(e.id); - if (e.embedding) { - await index.removePoint(e.id); - } - merged += 1; - } + } while (roundMerged > 0); } return { merged, groups: dupGroups };