ruflo/v3/docs/adr/ADR-097-federation-budget-circuit-breaker.md

ADR-097: Federation-wide Budget Circuit Breaker and Token Quotas

Status: Accepted — Partially Implemented (Phase 1 + Phase 3 consumer-side landed; Phases 2 and 4 deferred) Date: 2026-05-04 · Updated: 2026-05-09 Version: Phase 1 shipped in @claude-flow/plugin-agent-federation (v3.6.x); Phase 3 consumer in ruflo-cost-tracker@0.14.0 Supersedes: nothing Related: ADR-086 (Agent Federation), ADR-095 (Architectural gaps — G2 includes federation transport), issues #1723 and #1724 (#1724 closed as duplicate of #1723), commit 6f495369 (G2 Ed25519 signing in federation)

Context

Agent Federation (ADR-086) lets agents on one Ruflo node delegate tasks to peer nodes across trust boundaries. The implementation in @claude-flow/plugin-agent-federation already covers identity (Ed25519 keypairs), trust scoring, and federation_send for cross-node delegation. What it does not cover today:

• Recursive delegation loops. Node A → Node B → Node A → … If a peer wraps a received task and delegates it back, there is no hop counter to break the cycle. A pathological multi-node ring can run until process memory or the network gives out.
• Cost cascades. A 200-token task delegated to a peer can spawn a sub-swarm of five worker agents on the remote, each potentially calling expensive frontier models. The originator never sees the bill until the cost-tracker reconciles after the fact.
• No backpressure on hostile / faulty peers. A misbehaving peer that returns expensive responses cannot be muted automatically — the local trust score drifts but there's no mechanism that says "this peer is now too expensive, stop sending it work."
• No unified cost surface. The ruflo-cost-tracker plugin tracks local agent token usage but federation traffic is invisible to it.

The original issue (#1723, #1724 dup) frames this as a stability + enterprise-readiness gap. The fix is a federated circuit breaker layered on top of the existing federation protocol — no breaking changes to the wire format, just additive metadata.

Existing surface to build on

Component	Path	Today
Federation MCP tool	v3/@claude-flow/plugin-agent-federation/src/mcp-tools.ts (federation_send)	Sends a task to a peer; no cost / hop awareness
Federation node entity	v3/@claude-flow/plugin-agent-federation/src/domain/entities/federation-node.ts	trustScore, trustLevel, lastSeen; no state: SUSPENDED
Cost tracker plugin	plugins/ruflo-cost-tracker/	Tracks local model spend per agent / per session
Behavioral trust	ADR-086 §"Trust scoring"	Adjusts on protocol misbehavior; no cost-based decay yet

Decision

Ship four cohesive but independently-shippable parts. Each part is one iteration; the parts compose into the full circuit breaker.

Part 1 — Budget header on federation_send

Extend the federation_send MCP tool input schema with three optional metadata fields:

{
  // existing
  peerId: string,
  taskId: string,
  payload: unknown,
  // new (all optional, defaults preserve current behavior)
  budget?: {
    maxTokens?: number,   // hard cap on Σ tokens across the whole hop chain
    maxUsd?: number,      // hard cap on Σ USD spend; enforced via cost-tracker
  },
  maxHops?: number,        // default 8; 0 = no remote delegation allowed
}

These travel inside the federation envelope as a budget block alongside the payload. The receiving peer:

1. Decrements maxHops on receive. If it drops below 0, returns an error response (HOP_LIMIT_EXCEEDED) without invoking any agent. The originator gets the failure synchronously.
2. Subtracts from the budget as it spends. Each completed sub-task reports tokensUsed and usdSpent; the running total is checked against the cap before each subsequent action. Overshoot returns BUDGET_EXCEEDED and refuses further work.
3. Propagates the remaining budget if the peer itself federates onward. The downstream call inherits whatever budget remains, never more.

Budget defaults: when omitted, treat as Infinity for maxTokens / maxUsd and 8 for maxHops. The default hop limit alone closes the recursion-loop class without any caller change.

Part 2 — Peer state machine: ACTIVE / SUSPENDED / EVICTED

Extend federation-node.ts with a state field driven by the breaker:

State	When transitioned	What it means
ACTIVE	default; healthy	federation_send accepts deliveries to this peer
SUSPENDED	breaker tripped (cost threshold or repeated failures)	sends to this peer return PEER_SUSPENDED immediately; receives still accepted but ignored for trust accumulation
EVICTED	manual or post-grace-period escalation from SUSPENDED	peer removed from registry; new delivery errors are emitted as PEER_EVICTED

Transition triggers:

• ACTIVE → SUSPENDED when either:
  • Trailing 24h cost from this peer > configured threshold (peer.costSuspensionUsd, default $5.00)
  • Trailing 1h failure ratio > 50% across ≥10 sends
• SUSPENDED → ACTIVE after a configurable cooldown (default 30 min) AND a successful health probe
• SUSPENDED → EVICTED after 24h continuous suspension OR explicit federation_evict MCP call

Cooldown + auto-recovery prevents the breaker from being a one-way door — same shape as a typical hystrix-style breaker.

Part 3 — Cost-tracker integration

Wire the federation layer into ruflo-cost-tracker so federated spend appears in the same dashboards as local spend:

1. New event type federation_spend published to the cost-tracker bus on every federation_send completion: { peerId, taskId, tokensUsed, usdSpent, ts }.
2. Cost-tracker aggregates per-peer rolling windows (1h / 24h / 7d) and exposes them via the existing cost-report skill.
3. The breaker queries cost-tracker for the per-peer 24h sum to evaluate the suspension threshold.

This is one direction (federation → cost-tracker). Cost-tracker doesn't need to mutate federation state directly; the breaker pulls.

Part 4 — Doctor + observability

• ruflo doctor reports the current state of every known peer (ACTIVE/SUSPENDED/EVICTED) and the trailing-24h cost. A peer pinned in SUSPENDED for >1h shows up as a yellow warning.
• New MCP tool federation_breaker_status returns the same info programmatically for swarm coordinators that want to route around suspended peers.
• Structured log line on every state transition with {prevState, newState, reason, peerId} so post-incident triage doesn't need a debugger.

Implementation outline

Per the user directive, the implementation team owns this section. Each phase is one iteration to keep blast radius bounded:

Phase	Scope	Lands in
P1	Budget envelope + hop counter, no peer state changes	federation plugin + new tests
P2	Peer state machine (ACTIVE/SUSPENDED/EVICTED) + transition rules	federation-node.ts + tests
P3	Cost-tracker bus event + per-peer rolling aggregation	cost-tracker plugin
P4	Doctor + federation_breaker_status MCP tool	doctor command + mcp-tools

Test surface:

• Unit: budget arithmetic, hop counter underflow, state-transition table (ACTIVE→SUSPENDED→ACTIVE→EVICTED).
• Integration: a synthetic peer that intentionally exceeds budget should produce BUDGET_EXCEEDED on the originator AND transition to SUSPENDED after the threshold.
• Property: random hop chains must always terminate at maxHops ≤ 8.

Trade-offs

Decision	Alternative	Why this
Optional budget fields, default Infinity	Required budget on every federation_send	Backward compatible: existing code paths keep working. Callers opt in to limits.
Hop counter default 8	Default unlimited	8 is generous for legitimate multi-hop flows (typical ≤3) and cheap insurance against the recursion-loop class.
Peer state in federation plugin (not cost-tracker)	Centralized in cost-tracker	Federation owns peer identity + trust; the breaker is a federation concern. Cost-tracker provides numbers; federation decides actions.
Trailing-24h $5 default suspension threshold	Per-peer config only	Sane default avoids "gun, foot" for new installs. Override via plugin config.
EVICTED separate from SUSPENDED	Single INACTIVE state	Eviction is operationally permanent (admin removes the peer); suspension is a soft, auto-recovering state. They have different audit/log/recovery semantics.
Pull from cost-tracker for breaker decisions	Push spend events into federation	Federation already speaks events to cost-tracker; reverse direction would couple in both ways. Pull-on-decision is simpler and just-in-time.

Risks

1. Race between budget check and remote spend. The receiving peer checks budget before invoking the agent, but the agent's actual spend is reported after. A budget can be modestly exceeded by the last-call's actual cost > predicted cost. Mitigation: enforce on the predicted cost (max_tokens × per-token-rate), then refund unused on the running total. Worst case overshoot is one model call's worth.
2. Clock skew across peers. The 24h rolling window is computed locally per-peer, so divergent clocks don't change the threshold semantics. Trust-score timestamps (lastSeen) already rely on local clocks; same threat model.
3. Suspended peer recovery storm. If many peers transition SUSPENDED → ACTIVE simultaneously after cooldown, they could all retry pending sends at once. Mitigation: jitter the cooldown by ±10% per peer.
4. Cost-tracker plugin not installed. If the user runs federation without cost-tracker, USD-based breaker rules degrade to "always permitted." Token-count breaker still works (federation tracks tokens itself). Document this — cost-tracker is the recommended pair, not strictly required.

Implementation status (2026-05-09)

ADR-097 is functionally complete end-to-end. All five phases are landed: Phase 1 (budget envelope), Phase 2.a (state machine entity), Phase 2.b (breaker service + outbound short-circuit), Phase 3 (consumer-side cost aggregation + upstream reportSpend emission with SpendReporter interface and breaker fan-out), Phase 4 (operator surface — federation_breaker_status / federation_evict / federation_reactivate MCP tools + ruflo doctor --component federation). Integrators wire a SpendReporter implementation to push to their cost-tracker / Datadog / accounting backend; the included InMemorySpendReporter covers tests + reference impl.

Phase / Component	Status	Files	Commit(s)
Phase 1 — Budget envelope + hop counter on federation_send	Implemented	v3/@claude-flow/plugin-agent-federation/src/domain/value-objects/federation-budget.ts (new), mcp-tools.ts updated	7e1cc06df feat(federation): ADR-097 Phase 1 — budget envelope + hop counter (#1723)
Phase 2.a — Peer state machine value object + entity transitions	Implemented	domain/value-objects/federation-node-state.ts (new), domain/entities/federation-node.ts (state field + suspend/evict/reactivate), __tests__/unit/federation-node-state.test.ts (27 tests)	feat/adr-100-promote-097-phase2
Phase 2.b — Breaker service + outbound short-circuit	Implemented	application/federation-breaker-service.ts (new — pure evaluatePolicy + stateful FederationBreakerService with bounded per-peer buffer), application/federation-coordinator.ts (sendMessage gates on !peer.isActive with PEER_SUSPENDED/PEER_EVICTED constant errors), __tests__/unit/federation-breaker-service.test.ts (25 tests)	feat/adr-100-promote-097-phase2
Phase 3 consumer — Cost-tracker bus event + per-peer rolling aggregation	Implemented	plugins/ruflo-cost-tracker/scripts/federation.mjs, skills/cost-federation/SKILL.md	1c0804315 feat(cost-tracker): P6 — ADR-097 Phase 3 federation_spend consumer (v0.14.0)
Phase 3 upstream — Federation reportSpend() + SpendReporter interface + breaker fan-out + federation_report_spend MCP tool	Implemented	application/spend-reporter.ts (new — interface + InMemorySpendReporter reference impl), application/federation-coordinator.ts (new optional spendReporter + breakerService constructor integrations + reportSpend method), mcp-tools.ts (federation_report_spend), __tests__/unit/coordinator-spend-reporting.test.ts (10 tests)	feat/adr-100-promote-097-phase2
Phase 3 plugin wiring — federation plugin adopts budget integration + ADR-097 doc	Implemented	plugins/ruflo-federation/ (v0.2.0), docs/adrs/0001-federation-contract.md	b0168e4a5 feat(ruflo-federation): adopt plugin contract — 3-gate alignment + ADR-097 budget integration + smoke
Phase 4 — Operator surface: 3 MCP tools + doctor health-check	Implemented	mcp-tools.ts (federation_breaker_status / federation_evict / federation_reactivate), application/federation-coordinator.ts (getPeerStates / getPeerStateCounts / evictPeer / reactivatePeer), v3/@claude-flow/cli/src/commands/doctor.ts (checkFederationBreaker), __tests__/unit/federation-coordinator-breaker.test.ts (11 tests)	feat/adr-100-promote-097-phase2

Open questions resolved during implementation

Original question	Resolution
Is upstream federation_spend event emission required for Phase 3 consumer?	No — the consumer (scripts/federation.mjs) activates the moment the upstream emitter lands; it reads from the federation-spend namespace. Until Phase 2/3 emit events, the consumer runs against an empty namespace and reports zero spend.
Does Phase 1's default maxHops=8 close the recursion-loop class for all callers?	Yes — hop enforcement fires even when no explicit budget is passed (validateBudget(undefined) returns a default-bounded budget).

Deferred

All ADR-097 phases are landed. Future work is integrator-side wiring:

• Production SpendReporter adapter that persists to ruflo memory (namespace=federation-spend, key=fed-spend-<peerId>-<ts>) per the cost-tracker consumer convention. Reference implementation: InMemorySpendReporter (in-memory buffer, fine for tests, NOT for production). A real adapter is a thin shell around memory_store — left to the integrator since they own the credentials/quota policy.
• Auto-emission from federation_send — currently reportSpend() is an explicit caller-side call (correct since the federation layer doesn't own model pricing). A future enhancement could auto-emit a "send-completed" event with the budget enforcement's predicted cost as a placeholder, but the integrator overrides with actual cost on completion. Decision deferred until a concrete integration request justifies the additional state.

---

Acceptance criteria

The full implementation is done when:

• federation_send accepts and propagates budget + maxHops without breaking any existing test
• A synthetic recursive delegation chain (peer A → B → A → …) terminates at maxHops with HOP_LIMIT_EXCEEDED on the originator
• A peer that exceeds the cost threshold over 24h transitions to SUSPENDED, refuses sends for the cooldown window, and auto-recovers on a successful probe
• ruflo doctor shows peer states + trailing-24h cost
• federation_breaker_status MCP tool returns structured state per peer
• Cost-tracker cost-report shows federated spend grouped by peer
• New tests cover budget arithmetic, hop counter, state transitions, and cost-tracker bus events
• Full vitest suite stays green; no regressions in the 1917-test baseline