dream(memory): #3231 wire embedding near-dup detection into MemoryConsolidator.dedup() (evaluated, ACCEPT) (#3232)

* dream(memory): wire embedding near-duplicate detection into MemoryConsolidator.dedup()

MemoryConsolidator.dedup() only ever deduplicated entries via byte-exact
SHA-256 content hashing, even though every entry already carries a
computed .embedding and the adapter's HNSWIndex (already held in scope
for removal bookkeeping) is incrementally kept in sync. Paraphrases and
reformattings of the same memory were never caught.

Adds a second pass: for hash-pass survivors with an embedding, query the
already-populated HNSWIndex for cosine-similarity neighbors above a
configurable similarityThreshold (default 0.95, matching the unwired
domain-layer consolidator's own constant), and merge using the same
keeper-selection strategies (now factored into shared selectKeeper/
mergeGroup helpers). Guarded to cosine-metric indexes only; disabled via
similarityThreshold >= 1.

Evaluation: 471/472 passing in @claude-flow/memory (1 pre-existing,
unrelated, environmental failure); baseline (git-stash-isolated source)
fails exactly the 2 new discriminating tests and passes the rest,
confirming the fix is additive and non-regressive. tsc --noEmit clean.

Co-Authored-By: RuFlo <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG

* dream(memory): fix-forward — converge near-dup pass to a fixed point in one dedup() call

Independent adversarial critique (STEP 10) reproduced a real, bounded gap
in the prior commit: a near-duplicate cluster larger than
NEAR_DUP_SEARCH_K (8) split into multiple leftover sub-group survivors
within a single dedup() call, because each round's `consumed` bookkeeping
permanently excluded a group's keeper from further matching even though
it was still fully present in the index. 15 pairwise-identical-embedding
entries collapsed to 2 survivors (merged: 13) instead of 1 (merged: 14).

It self-healed across repeated runAll() calls (background timer /
nightlyLearner), so this was never a permanent-data-loss bug, but a
single manual dedup() call under-converged relative to its documented
"collapse duplicates" contract.

Fix: loop pass 2 to a fixed point (re-scan until a full round produces
zero merges) instead of a single scan. Always terminates — entries
strictly decrease each round that merges anything. `groups` now counts
merge operations across all rounds, which can exceed the number of
underlying duplicate clusters when one needed more than one round;
documented in the method's doc comment.

Added a discriminating regression test reproducing the critic's exact
15-entry scenario; confirmed it fails against the pre-fix single-round
code (merged: 13) and passes against this fix (merged: 14, single
survivor). Full package suite: 472/473 passing (1 pre-existing,
unrelated, environmental failure, unchanged from before this commit).
tsc --noEmit clean.

Co-Authored-By: RuFlo <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG

* dream(memory): 2026-09-08 research gist — embedding near-dup consolidation

Full SOTA report: 5-role parallel research fan-out, ledger check with
GitHub-verified fates for the last 14 rows (3 merged since 09-03, one PR
now crossing the 14-day stale threshold for the first time this cycle),
competitor comparison (CrewAI/Mem0/Zep/LangMem/Qdrant/SemDedup), plugins
and automation scan findings, adversarial critique with a real bug found
and fixed, and witness stamp.

No gist-creation MCP tool available in this session (GitHub MCP tools
cover issues/PRs/repos, not gists) — committing to docs/dream-cycle/
instead, matching every dream-cycle night since 2026-08-14.

Co-Authored-By: RuFlo <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG

* dream(memory): #3231 update LEDGER.md for 2026-09-08

Co-Authored-By: RuFlo <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG

* dream(memory): #3231 empirical receipts for dedup() near-dup pass (response to review)

Response to PR #3232's INCONCLUSIVE review, which asked for a threshold
contract rather than a smoke fixture: a contamination-checked embedding
corpus with pinned model, precision/recall including hard negatives and
cross-namespace cases, baseline vs. candidate latency/memory at realistic
cardinality, and deterministic replay + unchanged exact-dedup behavior.

Real pinned-model corpus: attempted directly against both ONNX backends
this monorepo ships. Both fail for reasons unrelated to this candidate,
reproduced on an unmodified checkout before writing anything:
- @huggingface/transformers@4.2.0: transformers.node.mjs does
  `import { Tensor } from "onnxruntime-common"` against a CJS module — a
  real ESM/CJS interop break under this sandbox's Node version.
- @xenova/transformers@2.17.0: pulls in sharp@0.32.6, whose native
  binding (sharp-linux-x64.node) isn't present for this platform — the
  same failure independently reproduced moments later by this package's
  own nightlyLearner test, which falls back to mock embeddings.
Neither is fixable within this PR's scope without touching unrelated,
pre-existing native/module-resolution infrastructure. Documented in
detail at the top of the new test file rather than silently worked
around or omitted.

What's delivered instead, in
v3/@claude-flow/memory/src/consolidator-embedding-benchmark.test.ts:
- A deterministic synthetic corpus where pairwise cosine similarity is
  constructed EXACTLY via vectorAtSimilarity() (Gram-Schmidt against a
  random orthogonal vector), not measured after the fact — a rigorous
  instrument for threshold mechanics specifically, disclosed as not
  validating real-world semantic accuracy.
- Precision/recall/false-merge-rate across a 5-point threshold sweep,
  every point at a deliberate margin from every corpus similarity value
  after an exact-boundary collision (sweep value == corpus value) proved
  float32-jitter-sensitive while writing this — documented as a finding,
  not silently patched around.
- Namespace isolation: proved cross-namespace near-dup merging matches
  cross-namespace hash-exact merging exactly (both pre-existing,
  unscoped-by-namespace behavior — this candidate doesn't change it).
- Determinism: 3 independent runs, compared by deterministic entry KEY
  (not raw id, which is randomly generated per store() and was a second
  bug this file's own first draft had to fix).
- Latency/memory at N=5000 (matching this repo's own HNSW-benchmark
  convention): initially measured 9.8s for the candidate pass, which
  traced to an unset `ef` search parameter defaulting to efConstruction
  (200) — every dedup() search traversed a 200-candidate list for a
  duplicate-detection task that only needs very-close neighbors. Fixed
  by passing an explicit NEAR_DUP_SEARCH_EF=32 in consolidator.ts,
  re-measured at 3.7s (~2.7x), precision/recall unchanged (still exact).
  Real numbers logged in the test output either way, not asserted away.

Full package suite: 478/479 passing (472 pre-existing + 7 new; same 1
pre-existing unrelated environmental failure as every prior commit on
this branch). tsc --noEmit clean.

Co-Authored-By: RuFlo <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG

* dream(memory): #3231 add new benchmark file to CI test-ratchet baseline

CI on a2e99cf failed with "1 unexpected failing file(s): +
v3/@claude-flow/memory/src/consolidator-embedding-benchmark.test.ts".

Root-caused via the vitest JSON report artifact (test-results-ubuntu-latest,
downloaded and parsed directly, not guessed): a suite-level collection
failure, "Cannot find package '@claude-flow/security' imported from
'.../v3/@claude-flow/memory/src/agentdb-retrieval-guard.ts'" — the same
unbuilt-sibling-package gap already accepted in scripts/ci-test-baseline.txt
for 9 other @claude-flow/memory test files, including consolidator.test.ts
itself (line 95). Every file in this package fails identically in this CI
checkout state; my new file just wasn't in the baseline yet because it's
new. Not a regression this PR introduced — confirmed by the failure being
a pre-import-time module-resolution error, not a test assertion in the
new file's own logic (which passes locally in a checkout with @claude-flow/
memory's dependencies actually built: see the 6/6 local run in the prior
commit's message).

Fix: add the new file to the baseline, alongside its 9 memory-package
siblings already there, matching this repo's own established convention
for this exact gap rather than working around it some other way.

Co-Authored-By: RuFlo <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG

* dream(memory): #3231 baseline 2 more federation/gateway plugin test files

CI on 20afdc8 failed (run 34905889339) after main was rebased in — 3
unexpected failing files, all pre-existing, unrelated to this PR's diff:

  plugins/ruflo-chatgpt-federation/test/oauth.test.mjs
    Cannot find package 'jose'
  plugins/ruflo-chatgpt-federation/test/publisher.test.mjs
    Cannot find package 'nostr-tools/pure'
  plugins/ruflo-x-gateway/test/oauth.test.mjs
    Cannot find package 'jose'

Verified via the vitest.json artifact (downloaded and parsed, not
guessed): all three are plugin-local-dependency collection failures,
identical in kind to the already-baselined sibling
plugins/ruflo-x-gateway/test/gateway.test.mjs ("node:test suite with its
own deps... root vitest cannot resolve them"). Confirmed these are new
on main itself (main's own ci-test-baseline.txt at the rebase point does
not list them either) and unrelated to v3/@claude-flow/memory — nothing
this PR touches.

Re-verified locally: re-ran the full @claude-flow/memory suite against
the rebased main (496/497 passing, same 1 pre-existing unrelated
failure as always) and recomputed the ratchet's unexpected-set directly
against the CI run's vitest.json artifact — 0 unexpected after this
change.

Co-Authored-By: RuFlo <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_01H4kfasS4XkYaSGYXFLSSEG

---------

Co-authored-by: Claude <noreply@anthropic.com>

scripts/ci-test-baseline.txt

@@ -92,6 +92,7 @@ v3/@claude-flow/hooks/src/__tests__/guidance-provider.test.ts
 v3/@claude-flow/hooks/src/__tests__/reasoningbank.test.ts
 v3/@claude-flow/memory/src/agentdb-backend.test.ts
 v3/@claude-flow/memory/src/agentdb-retrieval-guard.test.ts
+v3/@claude-flow/memory/src/consolidator-embedding-benchmark.test.ts
 v3/@claude-flow/memory/src/consolidator.test.ts
 v3/@claude-flow/memory/src/controller-registry.test.ts
 v3/@claude-flow/memory/src/database-provider.test.ts
@@ -121,3 +122,8 @@ v3/plugins/quantum-optimizer/tests/mcp-tools.test.ts
 
 # ruflo-x-gateway: node:test suite with its own deps (nostr-tools, ws, MCP SDK) — run via `npm test` in plugins/ruflo-x-gateway; root vitest cannot resolve them (same pattern as plugins/ruflo-adr)
 plugins/ruflo-x-gateway/test/gateway.test.mjs
+plugins/ruflo-x-gateway/test/oauth.test.mjs
+
+# ruflo-chatgpt-federation: same pattern — plugin-local deps (jose, nostr-tools/pure) not resolvable from root vitest
+plugins/ruflo-chatgpt-federation/test/oauth.test.mjs
+plugins/ruflo-chatgpt-federation/test/publisher.test.mjs