mirror of
https://github.com/ruvnet/ruflo.git
synced 2026-11-09 22:13:39 +09:00
dream(performance): #3093 wire productQuantizeDistance() into HNSW search path (evaluated, ACCEPT) (#3094)
* dream(performance): #3093 wire productQuantizeDistance() into HNSW search path Quantizer.productQuantize()/productQuantizeDistance() correctly implement codebook-aware PQ encoding and distance, but HNSWIndex.distance()/ distanceOptimized() never called productQuantizeDistance() - generic cosine/euclidean/dot/manhattan distance was computed directly on raw PQ centroid-index arrays, which is numerically meaningless. Zero callers of productQuantizeDistance() existed anywhere in v3/ before this change. Wires the dispatch in, guarded by a new isValidPQEncoding() check to avoid crashing/corrupting on vectors added before PQ codebook training completes (a separate, disclosed, pre-existing limitation not fully solved here - see hnsw-index.ts comments and docs/dream-cycle/dream-gist-2026-08-25.md). Measured on a deterministic synthetic corpus: recall@10 recovers from 0.097 (pre-fix) to 0.270 (post-fix), independently reproduced by an adversarial critic across 3 additional corpus shapes. Full evaluation receipt, hypothesis, and adversarial critique in the linked gist and issue #3093. Also backfills the 2026-08-24 ledger row (branch/issue/PR existed but the row was never appended - recovery per this pipeline's STEP 1 convention). Co-Authored-By: RuFlo <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_01Aor8qaXa8T3ZCGgRmjc4VG * dream(performance): append 2026-08-25 ledger row (#3093/#3094) Follow-up to a8b527105 now that the issue and PR numbers exist. Co-Authored-By: RuFlo <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_01Aor8qaXa8T3ZCGgRmjc4VG --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
rUvandClaude authored and GitHub committed
1 parent
0d03ce06a2
commit
2d6227005e
4 files changed
+323
-9
No files matched your search
@@ -159,8 +159,8 @@ finding" to an explicit recommendation: a future `automation`/`meta` DEEP
|
||||
night should treat "harden STEP 25's ledger-append against silent failure"
|
||||
as a first-class, high-priority candidate, not a background note.
|
||||
|
||||
| 2026-08-24 | swarm | weightedConsensus() trust weights now reach raft/byzantine/gossip vote tally (was architecturally unreachable — no field existed to carry them); [0,1] clamp preserves BFT safety margin | #3085 | #3086 | yes | ACCEPT | gossip flip: flat approved=false → weighted approved=true (0.905 vs 0.50); 226/226 tests, 0 regressions; adversarial critic CONFIRMED-WITH-CAVEATS | 08d44c58... | 08-16,17,18,19 MERGED (4/7); 08-14,15 correctly unmerged REJECT evidence; 08-20..23 confirmed real gap (no branch/PR, not a ledger-append failure) |
|
||||
| 2026-08-25 | performance | productQuantizeDistance() implemented but never dispatched in HNSW search path — wired in | #3093 | #3094 | yes | ACCEPT | see issue #3093 | n/a (backfill) | unmerged draft as of 09-05 |
|
||||
| 2026-08-24 | swarm | weightedConsensus() trust weights now reach raft/byzantine/gossip vote tally (was architecturally unreachable — no field existed to carry them); [0,1] clamp preserves BFT safety margin | #3085 | #3086 | yes | ACCEPT | gossip flip: flat approved=false → weighted approved=true (0.905 vs 0.50); 226/226 tests, 0 regressions; adversarial critic CONFIRMED-WITH-CAVEATS | 08d44c58... | MERGED 2026-09-14 |
|
||||
| 2026-08-25 | performance | HNSW product-quantization: productQuantizeDistance() correctly implemented but zero callers — generic distance ran on raw PQ centroid indices, numerically meaningless; now dispatched, with a disclosed residual isValidPQEncoding() plausibility-heuristic caveat | #3093 | #3094 | yes | ACCEPT | recall@10 0.097 (pre-fix) → 0.270 (post-fix) on deterministic synthetic corpus, ~2.8x; 461/462 tests, 0 regressions | 300da158... | awaiting final CI/merge as of 2026-09-14 |
|
||||
| 2026-08-26 | security | authorizeMcpTool() trusted unsigned caller identity (ASI07) — ADR-377 verifyInvocationToken bound into live chokepoint | #3102 | #3103 | yes | ACCEPT-with-caveats | see issue #3102 | n/a (backfill) | unmerged draft as of 09-05 |
|
||||
| 2026-08-27 | intelligence | distillLearning() EWC gate read 1 of 384 Fisher dims via length-collapsed getPenalty() call; rewired to computeConfidencePenalty()/updateFisherFromConfidences() | #3109 | #3110 | yes | ACCEPT-scoped | 152/152 tests (+4 new), baseline-fails/candidate-passes via stash isolation | 89946458... | unmerged draft as of 09-05 |
|
||||
| 2026-08-28 | memory | HybridBackend's dead `weights` field now drives real weighted-RRF fusion (was computed, never read by merge helpers) | #3118 | #3119 | yes | ACCEPT | see issue #3118 | n/a (backfill) | unmerged draft as of 09-05 |
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
# Performance SOTA Report — 2026-08-25
|
||||
|
||||
TL;DR: Tonight's performance deep-dive found that Ruflo's HNSW product-quantization (PQ) feature (`quantization: { type: 'product' }`) has been silently broken since it was written: `Quantizer.productQuantize()`/`productQuantizeDistance()` correctly implement codebook-aware PQ encoding and distance (`hnsw-index.ts:1209-1360`), but the actual search path (`distance()`/`distanceOptimized()`) never called `productQuantizeDistance()` — it ran generic cosine/euclidean/dot/manhattan distance directly on raw PQ centroid-index arrays, which is numerically meaningless (index 5 vs. index 200 has no relation to embedding-space proximity). This is the same "computed-but-never-wired" bug class this pipeline has repeatedly shipped (2026-08-18 hybridSearch, 2026-08-19 MessageBus retry, 2026-08-24 weightedConsensus). Fix: dispatch to `productQuantizeDistance()` when PQ is active and codebooks are trained, guarded by a new `isValidPQEncoding()` check to avoid crashing/corrupting on vectors added before codebook training completes (a separate, disclosed, pre-existing limitation this patch does not fully solve). Measured on a deterministic synthetic clustered corpus: recall@10 recovers from 0.097 (pre-fix) to 0.270 (post-fix) — a real, reproducible ~2.8x improvement — while remaining honest that flat PQ trained globally on multi-modal data has a known ceiling well below "near-perfect," which this fix does not and should not claim to lift.
|
||||
|
||||
## What's New in 2026
|
||||
|
||||
| Finding | Source | Confidence |
|
||||
|---|---|---|
|
||||
| Two independent SIGMOD 2026 papers (Ada-ef, DARTH) converge on per-query *declarative-recall* adaptive `efSearch` as the fix for the exact static-ef recall/latency tradeoff that killed 2026-08-15's rejected candidate — strongest external evidence of the night, but scoped out of tonight's candidate for patch-size/risk reasons (see Recommended Next Steps) | arXiv:2512.06636 (SIGMOD 2026), arXiv:2505.19001 + github.com/MChatzakis/DARTH (SIGMOD 2026) | A |
|
||||
| CVE-2026-25536: `@modelcontextprotocol/sdk` cross-client data leak via `StreamableHTTPServerTransport` reuse (1.10.0-1.25.3, fixed 1.26.0) — checked against Ruflo, not exploitable here (stdio transport, and `ruflo/src/ruvocal` already pinned to the fixed version) | GHSA-345p-7cg4-v4c7, cross-checked SentinelOne/Tenable/GitLab | A |
|
||||
| OWASP Top 10 for Agentic Applications 2026 formalizes ASI07 "Insecure Inter-Agent Communication" — directly relevant to a dormant Ruflo control found tonight (see Scan Findings: security) | genai.owasp.org | A |
|
||||
| SWARM+ (arXiv:2603.19431) validates at 990-agent scale that federation-style consensus quorums must shrink as members go stale/unreachable — directly on-point for a new hive-mind scan finding tonight | arXiv:2603.19431 | B |
|
||||
| No production multi-agent framework (LangGraph, AutoGen/AG2, CrewAI, OpenAI Agents SDK) ships queue-aware/cost-aware model routing as a first-party feature — a genuine, well-documented open gap (layering convention, not a solved-but-unpublished capability), per InfraMIND (arXiv:2606.11440) | arXiv:2606.11440 + 4 official framework docs, checked 2026-08-25 | B |
|
||||
|
||||
## Ruflo Current Capability
|
||||
|
||||
`v3/@claude-flow/memory/src/hnsw-index.ts`: `Quantizer` supports `binary`/`scalar`/`product` quantization types. For `product`, `productQuantize()` (1209-1269) trains PQ codebooks via k-means and stores centroid indices; `productQuantizeDistance()` (1337-1360, pre-fix) correctly implements codebook-aware distance — but had **zero callers** anywhere in `v3/` (confirmed via repo-wide grep) prior to tonight. `addPoint()`/`search()` stored/queried via `this.quantizer.encode()` but then ran the generic `distance()` switch (cosine/euclidean/dot/manhattan) directly on the resulting centroid-index arrays, and even normalized those index arrays for the cosine fast-path — both operations meaningless on PQ codes. `type: 'product'` appeared in exactly one test fixture repo-wide and was exercised by zero recall/correctness tests before tonight.
|
||||
|
||||
Also confirmed, unrelated to tonight's fix: `binary`/`scalar` quantization have a related-but-distinct issue — `scalarQuantize()`'s output is `[min, range, ...levels]` (the prefix pollutes generic distance the same way PQ indices do), and `binaryQuantize()`'s packed-bit floats aren't cosine/Euclidean/dot-compatible either. **Disclosed, not fixed tonight** — kept out of scope to keep this patch one conceptual change (see Recommended Next Steps).
|
||||
|
||||
## Competitor Comparison
|
||||
|
||||
| Framework | Caching / KV-cache reuse | Adaptive/dynamic topology or scheduling | Queue-aware / cost-aware routing | Published perf benchmarks |
|
||||
|---|---|---|---|---|
|
||||
| LangGraph (v1.1.3) | Node-level output cache (`CachePolicy`, TTL) — not KV-cache | Conditional edges are task-logic-driven, not latency/cost-driven | None | Conflicting single-source claims (2.2x vs. 5.75x vs. losing to CrewAI) — B/C, not reproducible from a shared harness |
|
||||
| AutoGen/AG2 | `cache_seed` exact-match completion cache | `GroupChat` speaker selection is who-talks-next logic, not a perf scheduler | None (cost/token tracking exists, unused for routing) | "8-9x token-efficiency vs LangChain" (B, academic cross-framework study); no first-party latency benchmark |
|
||||
| CrewAI | Per-tool `cache_function` (developer-defined) | `Flows` branching is static, developer-defined | None — docs point to bolting on LiteLLM/Requesty | Vendor-adjacent "5.76x faster than LangGraph" contradicts the LangGraph row's own claim — neither independently reproducible (C) |
|
||||
| OpenAI Agents SDK | Platform-level prompt caching (0.1x read cost, ≥1024 tokens) — real and quantified (A) but API-layer, not SDK-specific | `Handoffs` are developer-authored, not adaptive | Load-balancing exists but is infra-level, invisible to orchestration logic | OpenHands' agent SDK measured 38.8% cache-hit rate vs. reference `codex` agent's 94.8% — a **2.7-3.4x cost inflation from one missing `prompt_cache_key` parameter** (B, concrete cautionary data point) |
|
||||
|
||||
The "None" pattern for queue-aware/cost-aware routing is a genuine open gap, not solved-but-unpublished: InfraMIND (arXiv:2606.11440) reports +7.6pp accuracy / 7x lower latency / 99.9% vs <50% SLO compliance when routing is made infra-aware. The gap exists because all four frameworks deliberately push load-balancing/cache-affinity/cost signals down to a gateway layer (LiteLLM, vLLM Router, provider backend) to stay provider-portable — a real design tradeoff, not an oversight. Real opportunity for Ruflo, but the OpenHands `prompt_cache_key` incident is a live warning: caching is trivial to silently break in a multi-agent context, so any cache-aware routing claim needs a hit-rate-regression benchmark per release, not just a feature flag.
|
||||
|
||||
## Hypothesis
|
||||
|
||||
> Given an HNSWIndex configured with `quantization: { type: 'product' }`, where `productQuantize()`/`productQuantizeDistance()` correctly implement codebook-aware PQ encoding and distance but are never invoked by the search path, when `distance()`/`distanceOptimized()` are patched to dispatch to `quantizer.productQuantizeDistance()` whenever product quantization is active and codebooks are trained (falling back to today's generic distance during the pre-training bootstrap window, unchanged), and vector normalization is skipped for product-quantized vectors, then recall@10 of a product-quantized HNSWIndex should recover from the current near-chance-level baseline to a level materially closer to the unquantized index's recall@10, subject to: (1) non-`'product'` quantization paths are byte-identical to today's; (2) existing HNSW/memory test suite remains green; (3) `getCompressionRatio()` is unchanged; (4) zero LLM/API cost.
|
||||
|
||||
Frozen before evaluation began; not modified after seeing results.
|
||||
|
||||
## Benchmarks
|
||||
|
||||
Deterministic, $0, zero-LLM vitest additions: `v3/@claude-flow/memory/src/hnsw-quantization.test.ts`. Synthetic clustered corpus (8 well-separated Gaussian-ish blobs, N=1500, dim=64, seeded xorshift32 PRNG for full reproducibility), 30 held-out queries, recall@10 against brute-force ground truth computed on the *original unquantized* vectors (never leaked into the candidate).
|
||||
|
||||
Disclosed for transparency: initial corpus parameters (N=400, `codebookSize=16`) were adjusted twice — N=400 put an unrepresentatively large fraction of the corpus inside the PQ pre-training bootstrap window; `codebookSize=16` was arbitrarily small. `codebookSize=256` is the code's actual documented default, not a tuned value. The pass-bar (`>= 0.25`) was set *after* measuring both baseline and candidate at the final N=1500/codebookSize=256 config, disclosed in the test file's comment with both real numbers.
|
||||
|
||||
Darwin (`metaharness-darwin evolve --bench <suite.json>`) confirmed out of scope again tonight: its real interface evolves routing/topology/prompt/memory/tool/tier/context/coordination genome parameters against an LLM-scored bench corpus — no analog for a scoped TS distance-dispatch fix. Same class of skip as 4 of the last 5 dream-cycle nights.
|
||||
|
||||
## Evaluation
|
||||
|
||||
**evaluated: accepted.**
|
||||
|
||||
Baseline captured via `git stash` on `hnsw-index.ts` only (test file kept in place):
|
||||
|
||||
| Config | Recall@10 |
|
||||
|---|---|
|
||||
| Unquantized index (sanity check on corpus/harness) | 1.000 |
|
||||
| Product-quantized, **pre-fix** (generic distance on raw PQ indices) | **0.097** |
|
||||
| Product-quantized, **post-fix** (dispatched to `productQuantizeDistance`) | **0.270** |
|
||||
|
||||
A ~2.8x recall@10 improvement, fully reproducible (all seeds fixed). A diagnostic run (not part of the committed suite) isolated PQ metric quality from HNSW graph-traversal quality: brute-force search using `productQuantizeDistance()` directly gave the *same* 0.270 as the full graph search, confirming the fix's ceiling is genuine PQ fidelity (flat PQ trained globally under-resolves fine-grained same-cluster ranking on multi-modal data — a real, known PQ characteristic, not a new bug), not an HNSW graph-construction artifact compounding on top of it.
|
||||
|
||||
Full package suite: 461/462 passing pre- and post-fix (the 1 failure, `auto-memory-bridge.test.ts`'s read-only-file error-handling test, is environmental — this sandbox runs as root, which bypasses the `chmod`-based read-only precondition the test depends on; confirmed unrelated to this diff by identical failure on baseline). `tsc --noEmit` shows zero new errors attributable to this diff (all remaining errors are pre-existing `@types/node`/environment noise unrelated to `hnsw-index.ts`'s logic).
|
||||
|
||||
**Independent adversarial critic** (fresh session, no authoring context) independently re-derived the 0.097→0.270 numbers via its own `git stash` isolation (confirmed exact match), re-ran the full suite, checked the `Uint8Array`→`Float32Array` signature change against every caller in `v3/` (found one, `@claude-flow/cli/__tests__/pq-validation.test.ts`, unaffected — 7/7 tests pass, `any`-typed accessor), and generalization-tested beyond the shipped corpus shape (128d/16 clusters, 16d/4 clusters, 64d/1 cluster — improvement direction and rough magnitude held in all three, pre-fix 0.068-0.088 → post-fix 0.264-0.296). **Verdict: CONFIRMED-WITH-CAVEATS.** The one real caveat: `isValidPQEncoding()` is a *plausibility heuristic, not a proof of provenance* — the critic built a reproduction where an all-zero vector added pre-training passes the guard and gets silently (not-crashing, not-correctly-falling-back) mis-dispatched as if it were a real centroid assignment. This is a narrower manifestation of the same disclosed pre-training-fallback limitation, not a new hole; closing it fully needs per-node encoding-provenance tracking (out of scope tonight). Disclosed in-code and here rather than treated as fully solved.
|
||||
|
||||
## Darwin Results
|
||||
|
||||
Skipped — scope mismatch (see Benchmarks). Consistent with 08-14, 08-15, 08-18, 08-19, 08-24.
|
||||
|
||||
## SOTA Proof & Witness
|
||||
|
||||
(Populated in the Witness section of tonight's issue/PR; see below.)
|
||||
|
||||
## Recommended Next Steps
|
||||
|
||||
1. **Merge this candidate** (human review required) — closes a real, zero-cost correctness bug in a documented compression feature, with a permanent deterministic regression test.
|
||||
2. **Follow up on 2026-08-15's rejected `efSearch` finding using tonight's Grade-A evidence** (Ada-ef/DARTH, SIGMOD 2026): per-query declarative-recall adaptive `efSearch`, materially different and better-evidenced than a `sqrt(N)` heuristic. Scoped out tonight for patch-size/risk (needs a calibration mechanism plus `deserialize()` persistence) — strongest lead of the night, deserves a dedicated future night.
|
||||
3. **Wire `verifyInvocationToken` into `AgentAuthorizationPropagator.checkToolCall`** (`v3/@claude-flow/security/src/mcp-caller-identity.ts`) — an ADR-377 Ed25519 caller-identity control dormant since written, freshly relevant given OWASP's 2026 ASI07 category. Small, deterministic, env-gated.
|
||||
4. **Fix `FederationHub.vote()`'s stale-vote/quorum-denominator mismatch** (`v3/@claude-flow/swarm/src/federation-hub.ts:700-728`) — a departed swarm's vote is never purged while the quorum denominator shrinks. Independent of 08-24's weightedConsensus fix, anchored by SWARM+ (arXiv:2603.19431).
|
||||
5. **Harden or extend `scalar`/`binary` quantization distance dispatch** — same bug class as tonight's fix, disclosed but not fixed.
|
||||
@@ -270,8 +270,11 @@ export class HNSWIndex extends EventEmitter {
|
||||
? this.quantizer.encode(vector)
|
||||
: vector;
|
||||
|
||||
// Pre-normalize vector for O(1) cosine similarity
|
||||
const normalizedVector = this.config.metric === 'cosine'
|
||||
// Pre-normalize vector for O(1) cosine similarity.
|
||||
// Skipped for product quantization: storedVector holds PQ centroid
|
||||
// indices, not embedding coordinates, so normalizing it is meaningless
|
||||
// (and productQuantizeDistance() below never consults normalizedVector).
|
||||
const normalizedVector = this.config.metric === 'cosine' && !this.isProductQuantized()
|
||||
? this.normalizeVector(storedVector)
|
||||
: null;
|
||||
|
||||
@@ -335,8 +338,9 @@ export class HNSWIndex extends EventEmitter {
|
||||
? this.quantizer.encode(query)
|
||||
: query;
|
||||
|
||||
// Pre-normalize query for O(1) cosine similarity
|
||||
const normalizedQuery = this.config.metric === 'cosine'
|
||||
// Pre-normalize query for O(1) cosine similarity (see addPoint() for why
|
||||
// this is skipped under product quantization).
|
||||
const normalizedQuery = this.config.metric === 'cosine' && !this.isProductQuantized()
|
||||
? this.normalizeVector(queryVector)
|
||||
: null;
|
||||
|
||||
@@ -993,7 +997,49 @@ export class HNSWIndex extends EventEmitter {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* True when this index uses product quantization with trained codebooks.
|
||||
* Stored vectors are then PQ centroid indices, not embedding coordinates —
|
||||
* generic metric distances (cosine/euclidean/dot/manhattan) computed
|
||||
* directly on those indices are meaningless (index 5 vs index 200 has no
|
||||
* relation to embedding-space proximity). Falls back to false during the
|
||||
* pre-training bootstrap window (see Quantizer.productQuantize()), where
|
||||
* stored vectors are still sub-vector-mean floats and generic distance,
|
||||
* while imprecise, is not nonsensical — matching prior behavior exactly.
|
||||
*/
|
||||
private isProductQuantized(): boolean {
|
||||
return (
|
||||
this.quantizer !== null &&
|
||||
this.config.quantization?.type === 'product' &&
|
||||
this.quantizer.isPQTrained
|
||||
);
|
||||
}
|
||||
|
||||
private distance(a: Float32Array, b: Float32Array): number {
|
||||
// Pre-existing, disclosed limitation (not introduced by this fix): any
|
||||
// vector added before PQ codebook training completes is stored under a
|
||||
// sub-vector-mean fallback encoding (see Quantizer.productQuantize())
|
||||
// and is never re-encoded once training finishes — the original raw
|
||||
// vector isn't retained. isValidPQEncoding() is a *plausibility*
|
||||
// heuristic, not a proof of provenance: it structurally guards the
|
||||
// crash/out-of-bounds case (fallback values outside codebook range, or
|
||||
// non-integer), but a fallback encoding whose sub-vector means
|
||||
// coincidentally land on small non-negative integers (most concretely,
|
||||
// an all-zero or otherwise degenerate vector added pre-training) can
|
||||
// still pass the check and be silently mis-dispatched to
|
||||
// productQuantizeDistance() as if it were a real centroid assignment.
|
||||
// Closing this fully would need per-node encoding-provenance tracking
|
||||
// (e.g. a sentinel/flag threaded through HNSWNode and its persistence
|
||||
// format), out of scope for this patch — flagged as a known residual
|
||||
// risk, not treated as fully solved.
|
||||
if (
|
||||
this.isProductQuantized() &&
|
||||
this.quantizer!.isValidPQEncoding(a) &&
|
||||
this.quantizer!.isValidPQEncoding(b)
|
||||
) {
|
||||
return this.quantizer!.productQuantizeDistance(a, b);
|
||||
}
|
||||
|
||||
switch (this.config.metric) {
|
||||
case 'cosine':
|
||||
return this.cosineDistance(a, b);
|
||||
@@ -1312,14 +1358,17 @@ class Quantizer {
|
||||
* Compute approximate squared Euclidean distance between two PQ-encoded
|
||||
* vectors using their centroid indices and the shared codebooks.
|
||||
*
|
||||
* @param encoded1 - Centroid indices for vector 1 (length = numSubquantizers)
|
||||
* @param encoded1 - Centroid indices for vector 1 (length = numSubquantizers).
|
||||
* Typed Float32Array to match productQuantize()'s actual return type —
|
||||
* indices are small integers (0..codebookSize-1) exactly representable
|
||||
* in float32.
|
||||
* @param encoded2 - Centroid indices for vector 2 (length = numSubquantizers)
|
||||
* @param codebooks - Trained codebooks from trainProductQuantizer()
|
||||
* @returns Approximate squared Euclidean distance
|
||||
*/
|
||||
productQuantizeDistance(
|
||||
encoded1: Uint8Array,
|
||||
encoded2: Uint8Array,
|
||||
encoded1: Float32Array,
|
||||
encoded2: Float32Array,
|
||||
codebooks?: number[][][],
|
||||
): number {
|
||||
const cb = codebooks || this.codebooks;
|
||||
@@ -1412,6 +1461,23 @@ class Quantizer {
|
||||
getCodebooks(): number[][][] | null {
|
||||
return this.codebooks;
|
||||
}
|
||||
|
||||
/**
|
||||
* True when `encoded` looks like a real trained-codebook PQ encoding
|
||||
* (length matches numSubquantizers, every value is a non-negative integer
|
||||
* in range for its sub-quantizer's codebook). False for the pre-training
|
||||
* sub-vector-mean fallback (see productQuantize()), which cannot be
|
||||
* compared via productQuantizeDistance() without corrupting results or
|
||||
* indexing out of bounds.
|
||||
*/
|
||||
isValidPQEncoding(encoded: Float32Array): boolean {
|
||||
if (!this.codebooks || encoded.length !== this.codebooks.length) return false;
|
||||
for (let m = 0; m < encoded.length; m++) {
|
||||
const idx = encoded[m];
|
||||
if (!Number.isInteger(idx) || idx < 0 || idx >= this.codebooks[m].length) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
// ===== Persistence Helpers (ADR-125 Phase 3) =====
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
/**
|
||||
* Dream Cycle 2026-08-25 — HNSW product-quantization distance dispatch.
|
||||
*
|
||||
* `Quantizer.productQuantize()` / `productQuantizeDistance()` correctly
|
||||
* implement codebook-aware PQ encoding and distance, but nothing in the
|
||||
* search path (`distance()` / `distanceOptimized()`) ever called
|
||||
* `productQuantizeDistance()` — generic cosine/euclidean/dot/manhattan
|
||||
* distance was computed directly on raw PQ centroid-index arrays, which is
|
||||
* numerically meaningless (adjacent centroid indices are not adjacent in
|
||||
* embedding space). This test builds a synthetic clustered corpus, computes
|
||||
* brute-force ground-truth top-10 neighbors on the *unquantized* vectors,
|
||||
* and asserts recall@10 for a product-quantized index stays well above
|
||||
* chance level. On the pre-fix code this assertion fails (recall collapses
|
||||
* toward chance); post-fix it passes.
|
||||
*/
|
||||
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { HNSWIndex } from './hnsw-index.js';
|
||||
|
||||
function xorshift32(seed: number): () => number {
|
||||
let s = seed || 1;
|
||||
return () => {
|
||||
s ^= s << 13;
|
||||
s ^= s >>> 17;
|
||||
s ^= s << 5;
|
||||
return ((s >>> 0) / 4294967296);
|
||||
};
|
||||
}
|
||||
|
||||
/** Deterministic clustered corpus: `numClusters` well-separated Gaussian blobs. */
|
||||
function buildClusteredCorpus(
|
||||
n: number,
|
||||
dim: number,
|
||||
numClusters: number,
|
||||
seed: number,
|
||||
): Float32Array[] {
|
||||
const rand = xorshift32(seed);
|
||||
const centers: Float32Array[] = [];
|
||||
for (let c = 0; c < numClusters; c++) {
|
||||
const center = new Float32Array(dim);
|
||||
for (let d = 0; d < dim; d++) {
|
||||
// Spread centers far apart so clusters are well-separated.
|
||||
center[d] = (rand() - 0.5) * 20;
|
||||
}
|
||||
centers.push(center);
|
||||
}
|
||||
|
||||
const vectors: Float32Array[] = [];
|
||||
for (let i = 0; i < n; i++) {
|
||||
const center = centers[i % numClusters];
|
||||
const v = new Float32Array(dim);
|
||||
for (let d = 0; d < dim; d++) {
|
||||
v[d] = center[d] + (rand() - 0.5) * 1.5; // small intra-cluster noise
|
||||
}
|
||||
vectors.push(v);
|
||||
}
|
||||
return vectors;
|
||||
}
|
||||
|
||||
function bruteForceTopK(
|
||||
query: Float32Array,
|
||||
corpus: Float32Array[],
|
||||
ids: string[],
|
||||
k: number,
|
||||
): Set<string> {
|
||||
const scored = corpus.map((v, i) => {
|
||||
let sum = 0;
|
||||
for (let d = 0; d < v.length; d++) {
|
||||
const diff = v[d] - query[d];
|
||||
sum += diff * diff;
|
||||
}
|
||||
return { id: ids[i], dist: sum };
|
||||
});
|
||||
scored.sort((a, b) => a.dist - b.dist);
|
||||
return new Set(scored.slice(0, k).map((s) => s.id));
|
||||
}
|
||||
|
||||
async function recallAtK(config: {
|
||||
dim: number;
|
||||
n: number;
|
||||
numClusters: number;
|
||||
numQueries: number;
|
||||
k: number;
|
||||
quantization?: { type: 'product'; subquantizers: number; codebookSize: number };
|
||||
}): Promise<number> {
|
||||
const corpus = buildClusteredCorpus(config.n, config.dim, config.numClusters, 42);
|
||||
const ids = corpus.map((_, i) => `v-${i}`);
|
||||
|
||||
const index = new HNSWIndex({
|
||||
dimensions: config.dim,
|
||||
metric: 'euclidean',
|
||||
M: 16,
|
||||
efConstruction: 100,
|
||||
...(config.quantization ? { quantization: config.quantization } : {}),
|
||||
});
|
||||
|
||||
for (let i = 0; i < corpus.length; i++) {
|
||||
await index.addPoint(ids[i], corpus[i]);
|
||||
}
|
||||
|
||||
const rand = xorshift32(1337);
|
||||
let totalRecall = 0;
|
||||
for (let q = 0; q < config.numQueries; q++) {
|
||||
const queryIdx = Math.floor(rand() * config.n);
|
||||
const query = corpus[queryIdx];
|
||||
|
||||
const groundTruth = bruteForceTopK(query, corpus, ids, config.k);
|
||||
const results = await index.search(query, config.k, 200);
|
||||
const returned = new Set(results.map((r) => r.id));
|
||||
|
||||
let hits = 0;
|
||||
for (const id of returned) if (groundTruth.has(id)) hits++;
|
||||
totalRecall += hits / config.k;
|
||||
}
|
||||
|
||||
return totalRecall / config.numQueries;
|
||||
}
|
||||
|
||||
describe('HNSWIndex product quantization — distance dispatch (Dream Cycle 2026-08-25)', () => {
|
||||
const DIM = 64;
|
||||
// pqTrainingThreshold is 256; N well above it keeps the pre-training
|
||||
// bootstrap window (see isValidPQEncoding()) a small fraction of the
|
||||
// corpus, representative of real usage rather than a pathological
|
||||
// near-100%-cold-start case.
|
||||
const N = 1500;
|
||||
const NUM_CLUSTERS = 8;
|
||||
const NUM_QUERIES = 30;
|
||||
const K = 10;
|
||||
|
||||
it('unquantized baseline recovers near-perfect recall@10 (sanity check on the corpus/harness itself)', async () => {
|
||||
const recall = await recallAtK({ dim: DIM, n: N, numClusters: NUM_CLUSTERS, numQueries: NUM_QUERIES, k: K });
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`[dream-cycle] unquantized recall@10 = ${recall.toFixed(3)}`);
|
||||
expect(recall).toBeGreaterThan(0.85);
|
||||
});
|
||||
|
||||
it('product-quantized index recall@10 recovers materially vs. the pre-fix baseline', async () => {
|
||||
const recall = await recallAtK({
|
||||
dim: DIM,
|
||||
n: N,
|
||||
numClusters: NUM_CLUSTERS,
|
||||
numQueries: NUM_QUERIES,
|
||||
k: K,
|
||||
quantization: { type: 'product', subquantizers: 8, codebookSize: 256 },
|
||||
});
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(`[dream-cycle] product-quantized recall@10 = ${recall.toFixed(3)}`);
|
||||
// Measured on this exact deterministic corpus (git stash on hnsw-index.ts
|
||||
// only, test file kept in place):
|
||||
// pre-fix (generic distance on raw PQ centroid indices): recall@10 = 0.097
|
||||
// post-fix (productQuantizeDistance dispatched correctly): recall@10 = 0.270
|
||||
// Flat PQ trained globally on multi-modal (clustered) data is known to
|
||||
// under-resolve fine-grained same-cluster ranking — recall@10 in the
|
||||
// 0.2-0.3 range here reflects that real PQ characteristic, not a bug;
|
||||
// the assertion checks the fix's effect (recall recovers well above the
|
||||
// measured buggy baseline), not an absolute "good PQ" bar.
|
||||
expect(recall).toBeGreaterThanOrEqual(0.25);
|
||||
});
|
||||
|
||||
it('getCompressionRatio() is unaffected by the distance-dispatch fix', () => {
|
||||
const index = new HNSWIndex({
|
||||
dimensions: DIM,
|
||||
metric: 'euclidean',
|
||||
quantization: { type: 'product', subquantizers: 8, codebookSize: 16 },
|
||||
});
|
||||
// 8 subquantizers -> 8x compression, unchanged by this fix (only the
|
||||
// distance computation changed, not the stored representation).
|
||||
expect((index as unknown as { quantizer: { getCompressionRatio(): number } }).quantizer.getCompressionRatio()).toBe(8);
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user