ruflo-mods 0.3.2: live validation of toolHints/agentTrim/status, correct the agentTrim option text

Co-Authored-By: RuFlo <ruv@ruv.net>

plugins/ruflo-mods/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "ruflo-mods",
   "description": "ruflo as a Claude Code mod (function hooks, ADR-404): in-process prompt routing, edit learning signals, tighten-only tool checks from ruflo policy, the cost-tracker budget ladder, a mod trust gate for newly installed mods, and the $.ruflo noun other mods compose their status with. Opt-in and removable: the classic hook-handler hooks stay the default and take every event back whenever the mod is not loaded.",
-  "version": "0.3.1",
+  "version": "0.3.2",
   "author": {
     "name": "ruvnet",
     "url": "https://github.com/ruvnet"

plugins/ruflo-mods/hooks/agents/index.ts

@@ -8,11 +8,12 @@ const LEDGER = 'agentUse'
 
 /**
  * `agent.offer` (ADR-451 item 2): keeps agent types the project does not use
- * out of the model's listing and out of dispatch. Off unless `agentTrim` is
+ * out of the model's listing. Off unless `agentTrim` is
  * on. A type stays when it is built in, a pinned core role, named in
  * `agentTrimKeep`, used in the last 30 days (the `$.store` ledger, fed by
  * `agent.spawn`), or named in the current prompt. Tighten-only on what the
- * model sees; any failure offers the type (the hook is skipped).
+ * model sees (not a dispatch block: live, a hidden type still spawns by name; the listing is built before the first prompt in a headless session,
+ * so a prompt naming a type does not keep it there). Any failure offers the type (the hook is skipped).
  */
 export function registerAgents(on: On, state: ModState, options: ModOptions) {
   state.agentTrim.enabled = true

plugins/ruflo-mods/scripts/smoke.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# Structural + security smoke for ruflo-mods v0.3.1 (ADR-404, ADR-447).
+# Structural + security smoke for ruflo-mods v0.3.2 (ADR-404, ADR-447).
 # Static only: CI has no Claude Code, so the hooks module's behaviour is held
 # by v3/@claude-flow/cli/__tests__/mods/*.test.ts and, where function hooks are
 # on, by `claude plugin test plugins/ruflo-mods`.
@@ -12,9 +12,9 @@ ok()   { printf "PASS\n"; PASS=$((PASS+1)); }
 bad()  { printf "FAIL: %s\n" "$1"; FAIL=$((FAIL+1)); }
 HOOKS="$ROOT/hooks"
 
-step "1. plugin.json declares ruflo-mods 0.3.1"
+step "1. plugin.json declares ruflo-mods 0.3.2"
 grep -q '"name": "ruflo-mods"' "$ROOT/.claude-plugin/plugin.json" \
-  && grep -q '"version": "0.3.1"' "$ROOT/.claude-plugin/plugin.json" && ok || bad "name/version"
+  && grep -q '"version": "0.3.2"' "$ROOT/.claude-plugin/plugin.json" && ok || bad "name/version"
 
 step "2. hooks.json names exactly one module and no classic hook commands"
 grep -q '"modules": \["./register.ts"\]' "$HOOKS/hooks.json" && ! grep -q '"command"' "$HOOKS/hooks.json" \

scripts/live-ruflo-mods.sh

@@ -0,0 +1,126 @@
+#!/usr/bin/env bash
+# Live verification of ruflo-mods (toolHints, agentTrim, /ruflo-mods status) against REAL headless Claude Code sessions.
+#   scripts/live-ruflo-mods.sh [status hints trim ledger spawn | all]   (default: all)
+# Everything runs in a private mktemp scratch project; results land in $S/*.json and are printed.
+# Spend: haiku only (about $0.5 for `all`). Env: LIVE_MODEL, LIVE_MCP_CMD (the `ruflo` MCP server that owns the hinted tools).
+set -u
+REPO="$(cd "$(dirname "$0")/.." && pwd)"
+S="${LIVE_SCRATCH:-$(mktemp -d "${TMPDIR:-/tmp}/mods-live.XXXXXX")}"
+PHASES=("$@"); [ ${#PHASES[@]} -eq 0 ] && PHASES=(all)
+[ "${PHASES[0]}" = all ] && PHASES=(status hints trim ledger spawn)
+mkdir -p "$S"; [ -d "$S/.git" ] || (cd "$S" && git init -q)
+echo "scratch: $S"
+sed -n '/^# --- node driver ---$/,$p' "$0" | sed 1d >"$S/live.mjs"
+for p in "${PHASES[@]}"; do echo "== phase: $p"; node "$S/live.mjs" "$p" "$S" "$REPO" || echo "phase $p exited $?"; done
+echo "results: $S"
+exit 0
+# --- node driver ---
+import { spawn } from 'node:child_process'
+import { writeFileSync, readFileSync, mkdirSync, rmSync, readdirSync } from 'node:fs'
+import { join } from 'node:path'
+
+const [phase, S, REPO] = process.argv.slice(2)
+const PLUG = `${REPO}/plugins/ruflo-mods`
+const MODEL = process.env.LIVE_MODEL ?? 'haiku'
+const MCP = process.env.LIVE_MCP_CMD ?? 'npx -y @claude-flow/cli@latest mcp start'
+const out = (name, data) => writeFileSync(join(S, name), JSON.stringify(data, null, 2))
+const AGENTS = {
+  coder: { description: 'Writes code', prompt: 'You write code.' },
+  reviewer: { description: 'Reviews code', prompt: 'You review code.' },
+  'zebra-specialist': { description: 'Zebra stripes expert', prompt: 'You know zebras.' },
+  'quokka-analyst': { description: 'Quokka analytics expert', prompt: 'You know quokkas.' },
+  'yak-planner': { description: 'Yak shaving planner', prompt: 'You plan yaks.' },
+}
+const HINTS = {
+  memory_search: 'Searches AgentDB only; memory_search_unified also covers Claude memories and patterns.',
+  swarm_init: 'For coding work use topology hierarchical, maxAgents 6-8, strategy specialized.',
+}
+const settingsFile = options => {
+  const f = join(S, `settings-${Math.random().toString(36).slice(2, 8)}.json`)
+  const o = options ? { options } : undefined
+  writeFileSync(f, JSON.stringify(o ? { pluginConfigs: { 'ruflo-mods@inline': o, 'ruflo-mods': o } } : {}))
+  return f
+}
+const mcpFile = () => { const f = join(S, 'mcp.json'); writeFileSync(f, JSON.stringify({ mcpServers: { ruflo: { command: 'sh', args: ['-c', MCP], env: { CLAUDE_FLOW_MCP_TRANSPORT: 'stdio' } } } })); return f }
+
+// One headless process, several prompts (a `/ruflo-mods` prompt is answered by the mod itself, no model call).
+async function session(prompts, { options, mcp = false, agents = false, budget = '0.50', env = {} }) {
+  const args = ['-p', '--input-format', 'stream-json', '--output-format', 'stream-json', '--verbose', '--model', MODEL, '--plugin-dir', PLUG, '--settings', settingsFile(options),
+    '--setting-sources', 'local', '--max-budget-usd', budget, '--allowedTools', 'ToolSearch', '--disallowedTools', 'Bash,Read,Grep,Glob,WebFetch,WebSearch,Edit,Write,NotebookEdit', '--append-system-prompt', 'Answer exactly as asked, no commentary.']
+  if (mcp) args.push('--strict-mcp-config', '--mcp-config', mcpFile())
+  if (agents) args.push('--agents', JSON.stringify(AGENTS))
+  const p = spawn('claude', args, { cwd: S, stdio: ['pipe', 'pipe', 'pipe'], env: { ...process.env, ...env } })
+  let buf = '', onResult, init
+  p.stdout.on('data', d => {
+    buf += d
+    let i
+    while ((i = buf.indexOf('\n')) >= 0) {
+      const line = buf.slice(0, i); buf = buf.slice(i + 1)
+      try { const m = JSON.parse(line); if (m.type === 'system' && m.subtype === 'init') init = m; if (m.type === 'result' && onResult) onResult(m) } catch {}
+    }
+  })
+  const rows = []
+  for (const text of prompts) {
+    const done = new Promise(res => { onResult = res })
+    p.stdin.write(`${JSON.stringify({ type: 'user', message: { role: 'user', content: text } })}\n`)
+    const r = await Promise.race([done, new Promise(res => setTimeout(() => res(undefined), 120_000))])
+    rows.push({ prompt: text, reply: r?.result ?? '(timeout)', cost: r?.total_cost_usd })
+  }
+  p.stdin.end()
+  await new Promise(res => { const t = setTimeout(() => (p.kill(), res()), 15_000); p.on('close', () => (clearTimeout(t), res())) })
+  return { rows, init, cost: rows.at(-1)?.cost }
+}
+const show = (name, res) => { out(`${name}.json`, { ...res, init: { agents: res.init?.agents, plugins: res.init?.plugins } }); for (const r of res.rows) console.log(`[${name}] > ${r.prompt.slice(0, 90)}\n${r.reply}\n`); console.log(`[${name}] cost $${res.cost}`) }
+
+const LIST = 'List every subagent_type value your Agent tool accepts, comma-separated, nothing else.'
+const DESCR = 'Use ToolSearch with query "select:mcp__ruflo__memory_search,mcp__ruflo__swarm_init,mcp__ruflo__memory_list". Then print, for each of the three tools, its tool name followed by its full description text verbatim, nothing else.'
+
+async function status() { // no model call: the mod answers /ruflo-mods itself
+  for (const [name, options] of [['status-default', undefined], ['status-on', { toolHints: true, agentTrim: true }]]) show(name, await session(['/ruflo-mods'], { options, env: { CLAUDE_CODE_ENABLE_FUNCTION_HOOKS: '1' } }))
+}
+async function hints() {
+  show('hints-off', await session([DESCR, '/ruflo-mods'], { mcp: true }))
+  show('hints-on', await session([DESCR, '/ruflo-mods'], { options: { toolHints: true }, mcp: true }))
+}
+async function trim() {
+  show('trim-off', await session([LIST, '/ruflo-mods'], { agents: true }))
+  show('trim-on', await session([LIST, '/ruflo-mods'], { options: { agentTrim: true }, agents: true }))
+  show('trim-on-keep', await session([LIST, '/ruflo-mods'], { options: { agentTrim: true, agentTrimKeep: 'zebra-specialist' }, agents: true }))
+  show('trim-on-named', await session(['Is quokka-analyst available? Answer yes/no, then ' + LIST, '/ruflo-mods'], { options: { agentTrim: true }, agents: true }))
+}
+// The usage ledger is $.store of the inline plugin: ~/.claude/plugins/store/ruflo-mods_inline-<hash>.json. It is user-global, so the harness
+// snapshots every such file and restores it after each phase: a run must not change what a later real session hides.
+const STORE_DIR = join(process.env.HOME ?? '', '.claude/plugins/store')
+const storeFiles = () => readdirSync(STORE_DIR).filter(n => /^ruflo-mods_inline-.*\.json$/.test(n)).map(n => join(STORE_DIR, n))
+async function withLedger(content, run) {
+  const saved = new Map(storeFiles().map(f => [f, readFileSync(f)]))
+  if (!saved.size) throw new Error('no ruflo-mods inline store file yet: run the spawn phase first')
+  try {
+    for (const f of saved.keys()) writeFileSync(f, content)
+    return await run()
+  } finally { for (const [f, b] of saved) writeFileSync(f, b) }
+}
+const DAY = 86_400_000
+async function ledger() { // a recent spawn keeps a type, an old one does not, a corrupt file must not break the session or the core roles
+  const now = Date.now()
+  const cases = [
+    ['ledger-recent-and-stale', JSON.stringify({ agentUse: { 'quokka-analyst': now - DAY, 'yak-planner': now - 40 * DAY } })],
+    ['ledger-corrupt-text', '{ this is not json'],
+    ['ledger-wrong-shape', JSON.stringify({ agentUse: ['quokka-analyst'] })],
+    ['ledger-hostile-values', JSON.stringify({ agentUse: { 'quokka-analyst': 'now', 'zebra-specialist': -1 } })],
+  ]
+  for (const [name, content] of cases) show(name, await withLedger(content, () => session([LIST, '/ruflo-mods'], { options: { agentTrim: true }, agents: true })))
+}
+async function spawn_() { // dispatch of a hidden type: is it refused, and does the accepted spawn land in the ledger?
+  const SPAWN = 'Call the Agent tool once, in the foreground, with subagent_type "quokka-analyst", description "hi", prompt "reply with the single word pong". Then reply with only the word ACCEPTED if the call ran, or REFUSED: <error text> if it errored.'
+  const before = new Map(storeFiles().map(f => [f, readFileSync(f)]))
+  try {
+    for (const f of before.keys()) writeFileSync(f, '{}')
+    show('spawn-hidden-type', await session([SPAWN, '/ruflo-mods'], { options: { agentTrim: true }, agents: true, budget: '0.40' }))
+    for (const f of storeFiles()) console.log('[spawn] ledger after:', readFileSync(f, 'utf8').replace(/\s+/g, ' '))
+  } finally { for (const [f, b] of before) writeFileSync(f, b) }
+}
+const PHASE = { status, hints, trim, ledger, spawn: spawn_ }
+if (!PHASE[phase]) { console.error(`unknown phase ${phase}`); process.exit(2) }
+await PHASE[phase]()
+process.exit(0)

v3/docs/validation/ruflo-mods-live-2026-10.md

@@ -0,0 +1,63 @@
+# Live validation of ruflo-mods 0.3.1, 2026-10-05
+
+Method as in `agentdb-recall-live-2026-10.md`: a private `mktemp -d` project, one real headless `claude -p --input-format stream-json --plugin-dir plugins/ruflo-mods` process per scenario, model `haiku`, options set through `--settings pluginConfigs`, the session's own `/ruflo-mods` command read back in the same process. Claude Code 2.1.287, `@claude-flow/cli` (the `ruflo` MCP server that owns the hinted tools). Reproduce: `scripts/live-ruflo-mods.sh [status hints trim ledger spawn | all]`; total spend for this report about $0.4 (`/ruflo-mods` itself is answered by the mod, no model call).
+
+## Summary
+
+| # | Claim | Result |
+|---|---|---|
+| 1 | `toolHints` off by default does nothing; on adds hints | **Confirmed.** Off: 0 hints, `tool hints: off`. On: the hint appears under exactly the 2 hinted tools asked about, none under `memory_list` (not in the table), status `6 tool(s) described`. |
+| 2a | `agentTrim` off hides nothing | **Confirmed.** All 9 types listed (5 defined + 4 built-in), `agent trim: off`. |
+| 2b | on hides unused types, keeps core roles | **Confirmed.** `coder`, `reviewer` + built-ins kept; `zebra-specialist`, `quokka-analyst`, `yak-planner` hidden (`3 type(s) hidden`). |
+| 2c | `agentTrimKeep`, 30-day ledger | **Confirmed.** Keep list keeps `zebra-specialist` (2 hidden). Ledger with a 1-day-old `quokka-analyst` keeps it, a 40-day-old `yak-planner` is hidden (2 hidden). |
+| 2d | fails open | **Partly.** A corrupt, wrong-shape or hostile-valued ledger never crashed or hid a core role; but the result is an empty ledger, so trimming still applies (3 hidden). The `.catch` fail-open (a hook that throws) could not be provoked from outside; unit tests cover it. |
+| 2e | "named in the prompt" keeps a type; trim covers "dispatch" | **Not as documented (fixed in 0.3.2, text only).** See below. |
+| 3 | `/ruflo-mods` status | **Confirmed**, the counters track the features live. |
+
+## 1. toolHints
+
+Prompt (both runs): `Use ToolSearch with query "select:mcp__ruflo__memory_search,mcp__ruflo__swarm_init,mcp__ruflo__memory_list" ... print each tool name and its full description verbatim`.
+
+- **Off (default, no settings):** descriptions are the server's own. No `ruflo:` line in any of the three. `/ruflo-mods` -> `tool hints:  off (set the toolHints option)`.
+- **On:** observed, verbatim:
+  - `mcp__ruflo__memory_search` description ends `...Diversity can change result order.` then `ruflo: Searches AgentDB only; memory_search_unified also covers Claude memories and patterns.`
+  - `mcp__ruflo__swarm_init` ends `...spawn each separately.` then `ruflo: For coding work use topology hierarchical, maxAgents 6-8, strategy specialized.`
+  - `mcp__ruflo__memory_list`: unchanged, no hint.
+  - `/ruflo-mods` -> `tool hints:  6 tool(s) described` (the table's 6 tools are described when the deferred list is built, not only the 2 asked for).
+- Cost: $0.051 (off), $0.027 (on).
+
+## 2. agentTrim
+
+Five agent types defined with `--agents` (`coder`, `reviewer`, `zebra-specialist`, `quokka-analyst`, `yak-planner`); the question is `List every subagent_type value your Agent tool accepts`.
+
+| Run | Model's answer | `/ruflo-mods` |
+|---|---|---|
+| off (default) | claude, coder, Explore, general-purpose, Plan, quokka-analyst, reviewer, yak-planner, zebra-specialist | `agent trim:  off (set the agentTrim option)` |
+| on | claude, coder, Explore, general-purpose, Plan, reviewer (+ `fork` in one of the on-runs; a model-side listing variation, not seen in the others) | `3 type(s) hidden` |
+| on, `agentTrimKeep=zebra-specialist` | ... reviewer, zebra-specialist | `2 type(s) hidden` |
+| on, ledger `{quokka-analyst: now-1d, yak-planner: now-40d}` | ... quokka-analyst, reviewer | `2 type(s) hidden` |
+| on, ledger `{ this is not json` | claude, coder, Explore, general-purpose, Plan, reviewer | `3 type(s) hidden` |
+| on, ledger `{agentUse: ["quokka-analyst"]}` | same as above | `3 type(s) hidden` |
+| on, ledger `{quokka-analyst: "now", zebra-specialist: -1}` | same as above | `3 type(s) hidden` |
+
+The ledger is `$.store`: `~/.claude/plugins/store/ruflo-mods_inline-<hash>.json`, user-global, so the harness snapshots and restores it around each phase.
+
+### Two claims the live runs contradict (text corrected in 0.3.2; no behaviour changed)
+
+1. **A hidden type can still be dispatched.** With `agentTrim` on and `quokka-analyst` hidden, `Call the Agent tool ... subagent_type "quokka-analyst"` replied `ACCEPTED`, ran, and the ledger afterwards held `{"agentUse": {"quokka-analyst": 1791174198156}}`, so it is kept from then on. The option text said "listing and dispatch"; it is the listing only.
+2. **"Named in the prompt" does not keep a type in a headless session.** `Is quokka-analyst available? ... list` answered `No` and the type stayed hidden (3 hidden); a second experiment (`Say hi`, list, then a prompt naming it plus the list) hid it in both lists. The listing is built before the first prompt and not rebuilt, so `agent.offer` runs before `prompt.submit` can record the prompt. Not tested: an interactive session (it may rebuild the listing). Use `agentTrimKeep` to be sure.
+
+## 3. /ruflo-mods
+
+Default session: `owns: nothing (classic hooks keep every event)`, `routed: 0 prompt(s)`, `budget: off`, `tool hints: off`, `agent trim: off`, `guidance: off`. With both options on and no activity: `tool hints:  0 tool(s) described`, `agent trim:  0 type(s) hidden`. After the activity above the counts read 6 and 3 (see the tables), so the report reflects live behaviour. `owns: nothing` is expected here: the classic hook-handler is not installed in the scratch project.
+
+## 4. Not done
+
+- Interactive (TUI) sessions; only headless `-p` was run.
+- The throwing-hook `.catch` fail-open path (needs a fault-injection point in the engine).
+- Only haiku, one run per cell for `hints` and `status`; trim answers were repeated with one model-side variation (`fork`).
+- The console UI was not driven: nothing the console shows changed.
+
+## 5. Change shipped: ruflo-mods 0.3.2 (text only)
+
+`plugin.json` `agentTrim` description, the `agents/index.ts` doc comment and the smoke version string. Plugin tests 39 pass / 0 fail, `claude plugin validate` passes, smoke 11/11.