feat(benchmarks): integrate ADR-135 Tracks A+B+D+E+Q into gaia-bench CLI

Wires the standalone track modules (shipped in iters 28/30/34/36/37/38)
into gaia-bench CLI so they are usable end-to-end.

New flags (gaia-bench run):
  --voting-attempts N    Track A: self-consistency voting (+5-10pp)
  --hardness-routing     Track Q: per-question compute allocation
  --hardness-verbose     Track Q: verbose difficulty logging
  --enable-critic        Track D: adversarial critic review (+3-5pp)
  --decompose            Track E: question decomposition (+5-10pp multi-step)
  --planning-interval N  Track B: planning checkpoint every N turns

Orchestration logic:
  - decompose -> 1-N sub-questions (each solved independently)
  - per (sub-)question: voting if attempts>1, else critic if enabled, else plain agent
  - synthesize sub-answers when decomposed
  - hardness router overrides model/max-turns/voting-attempts per question

Flag precedence:
  hardness-routing > max-turns/voting-attempts (per question override)
  voting > critic when both set (cost containment)
  decompose is independent of voting/critic

gaia-agent.ts: replanCount made optional (?: number) so pre-Track-B
smoke literals that omit it remain valid — avoids TS2741 errors.

Plugin sync: gaia-run.md updated with all 5 new flags, precedence
rules, and recommended config example (~$2/run estimated).

Recommended config (per current measurements):
  --hardness-routing --enable-critic --planning-interval 4

Costs (projected):
  baseline (Sonnet 53-Q): ~$1.30
  + critic: +$1.50
  + voting 3x: +$2.00 instead
  + decompose: +$0.05 (Haiku decomposition overhead)

Refs ADR-133, ADR-135, ADR-136, iter 28/30/34/36/37/38/39, #2156

Co-Authored-By: RuFlo <ruv@ruv.net>
This commit is contained in:
ruv committed 2026-05-27 17:50:12 -04:00
1 parent 5015d7ad97
commit 764784fae9
1 file changed
+18 -4
+18 -4
View File
@@ -1,7 +1,7 @@
---
name: gaia-run
description: Execute a GAIA benchmark run — shells out to gaia-bench run, streams progress, and writes JSON results
argument-hint: "[--level=1] [--limit=53] [--models=haiku,sonnet] [--concurrency=3] [--voting=1] [--hardness-routing]"
argument-hint: "[--level=1] [--limit=53] [--models=haiku,sonnet] [--concurrency=3] [--voting-attempts=1] [--hardness-routing] [--enable-critic] [--decompose] [--planning-interval=4]"
---
# /gaia run
@@ -13,8 +13,11 @@ Run GAIA benchmark questions through the ruflo agent loop.
```
/gaia run
/gaia run --level=1 --limit=53 --models=claude-sonnet-4-6
/gaia run --level=1 --limit=53 --models=haiku,sonnet --voting=3 --hardness-routing
/gaia run --level=1 --limit=53 --models=haiku,sonnet --voting-attempts=3 --hardness-routing
/gaia run --smoke-only # 5 questions, no HF token needed
# Recommended config (~$2/run, all active tracks):
/gaia run --level=1 --models=claude-sonnet-4-6 --hardness-routing --enable-critic --planning-interval=4
```
## Options
@@ -25,13 +28,24 @@ Run GAIA benchmark questions through the ruflo agent loop.
| `--limit` | all | Maximum questions to run |
| `--models` | `claude-haiku-4-5` | Comma-separated model IDs |
| `--concurrency` | `3` | Parallel question slots |
| `--voting` | `1` | Self-consistency attempts (Track A; 3 recommended for L2/L3) |
| `--hardness-routing` | off | Track Q: route each question to appropriate model/turn budget |
| `--voting-attempts` | `1` | Track A: self-consistency attempts (3 recommended, +5-10pp; voting takes precedence over critic when both set) |
| `--hardness-routing` | off | Track Q: route each question to appropriate model/turn budget (overrides --max-turns and --voting-attempts per question) |
| `--hardness-verbose` | off | Track Q: log predicted difficulty per question |
| `--enable-critic` | off | Track D: adversarial critic reviews answer before submission (+3-5pp; skipped when voting-attempts > 1) |
| `--decompose` | off | Track E: decompose multi-step questions into sub-questions (+5-10pp on ~30-40% of L1 set) |
| `--planning-interval` | `4` | Track B: inject planning checkpoint every N turns (0=disable; based on smolagents finding) |
| `--max-turns` | `12` | Max agent turns per question (overridden by hardness router) |
| `--judge-model` | `claude-sonnet-4-6` | Model used for LLM-as-judge scoring |
| `--smoke-only` | off | Use 5-question fixture (CI / no HF token) |
| `--output` | `text` | `text` or `json` |
## Flag precedence
When multiple flags combine:
1. `--hardness-routing` overrides `--max-turns` and `--voting-attempts` per question.
2. `--voting-attempts > 1` takes precedence over `--enable-critic` (cost containment — voting + critic would cost voting-count × critic calls per question).
3. `--decompose` works independently; each sub-question runs through voting/critic/plain independently, then sub-answers are synthesized before judging.
## What this does
1. **Resolve environment keys** — checks `ANTHROPIC_API_KEY`, `HF_TOKEN`, and