mirror of
https://github.com/webstudio-is/webstudio.git
synced 2026-10-23 03:27:27 +09:00
perf: Improve MCP evaluation reliability and efficiency (#5959)
## Summary - capture bounded MCP failure codes, validation issue paths, per-tool latency, mutation timing, uncached input, catalog size, and per-call/per-tool response bytes in every evaluation result - require every raw fixture outcome to have zero failed calls and zero retries, even when the workflow later recovers - update baselines only when all raw outcomes and all regression comparisons pass - refuse to replace an accepted aggregate token baseline with a noisier, higher-token sample - regression-gate aggregate response bytes alongside tokens, calls, focused reads, mutations, errors, and retries - compact repeated SDK schemas while preserving required arguments, destructive safety controls, and ordinary optional fields for common goal-guided tools - omit low-frequency optional argument names until focused discovery, reducing the initial tool handshake without removing callable tools - stop echoing the caller's goal from `meta.guide` responses - make evaluations copy their canonical objective verbatim into `meta.guide`, preventing vague paraphrases from selecting a generic workflow - normalize grouped `styles` and `declarations` maps into canonical flat `update-styles` operations before validation - normalize relative `create-page` paths through the platform URL parser - bundle the four bounded authentication discovery reads into one `inspect-auth-context` call - bound large MCP style-key mutation results while preserving a representative sample and exact total count - combine fragment insertion with immediate persisted binding verification to remove a sequential model turn ## Root causes and impact Recovered MCP failures could previously be reported as successful raw outcomes, and baseline files could be written before the complete comparison result was known. Raw outcomes now fail on any MCP error or retry, and accepted baselines are written only after every fixture and comparison passes. Rejected samples exposed two concrete error sources. A vague authentication objective selected a generic workflow and produced four validation failures, 20 calls, and 1.07M tokens; evaluations now pass the canonical objective verbatim. A relative `create-page.path` caused an `INVALID_INPUT` failure and retry; the adapter now canonicalizes it before validation. The deterministic catalog in the accepted baseline measures **50,864 bytes**, down from **53,317 bytes** (−4.60%). Input schemas fell from **31,190 to 28,737 bytes** (−7.86%). The two composite workflows add two conditionally exposed tools and 360 catalog bytes (about +0.71%) when their backing operations are available. Broad experiments that removed focused descriptions or all goal-specific examples made guide responses smaller but increased token use or caused retries. Those variants were rejected. The current implementation compacts only recognized goal guides, while retaining the detailed `update-styles` schema and the `upload-assets` batch example proven necessary by the evaluations. ## Workflow-churn iteration Authenticated discovery uses one model-visible `inspect-auth-context` call. Fragment authoring can now use `insert-fragment-verified`, which commits the insertion and immediately runs persisted binding verification for up to 200 bindings. This removes the following sequential verification turn without changing mutation or verification counts. Current deterministic workflow shape: - authenticated workflow calls: **14 → 10** (−28.57%) - design workflow calls: **17 → 16** (−5.88%) - aggregate workflow calls: **39 → 34** (−12.82%) - failures: **0** - retries: **0** - authenticated MCP response bytes: **52,524 → 46,900** (−10.71%) - authenticated guide response: **26,782 → 24,358 bytes** (−9.05%) One complete run measured authenticated tokens at **348,225** (−11.44%) and first verification at **38.3s** (−30.14%); design first verification moved to **28.1s** (−27.27%). The final exact-code run also passed with **10/16/8** calls and zero failures/retries, measuring authenticated tokens at **377,802** (−3.92%) and design first verification at **34.5s** (−10.67%). The composite preserves post-commit safety: if verification infrastructure fails after insertion, the result remains `committed:true`, returns `POST_COMMIT_VERIFICATION_FAILED`, and explicitly instructs the agent not to repeat the insertion. A guarded baseline-update run was rejected because stochastic aggregate tokens did not beat the accepted result, so no result artifacts were rewritten. ## Style response compaction Large style mutations previously returned every generated style key. In the design fixture, that single successful response was **19,121 bytes** and remained in context for all later turns. MCP responses now retain the first 20 keys plus `styleKeyCount` and `styleKeysTruncated`; the typed runtime operation remains unchanged. Focused evaluation results: - `update-styles` response: **19,121 → 2,815 bytes** (−85.28%) - design-workflow MCP responses: **70,941 → 54,633 bytes** (−22.99%) - calls: **17 → 17** - failures/retries: **0 / 0** - tokens: **498,764 → 505,136** (+1.28%, treated as noise and not accepted as a new baseline) The subsequent complete `pnpm evaluations` run passed all three fixtures with exact **11/17/8** call sequences and zero failures/retries. The accepted lower-token baseline remains unchanged. ## Goal-guide context compaction Recognized goal guides no longer repeat every selected tool's description and examples. Current Codex clients defer MCP tools and load each named tool's exact contract before invocation, so the guide keeps the goal-specific workflow while returning compact tool references. Free-form guide searches remain fully descriptive. Two complex inputs remain intentionally hydrated: `update-styles` keeps its detailed schema, and `upload-assets` keeps one batch example. Removing the upload example caused five failed attempts in a complete run; restoring only that example returned the font workflow to its exact 8-call, zero-error path. Final complete-suite results: - calls: **10 / 16 / 8** (**34 aggregate**) - failures/retries: **0 / 0** - goal-guide payload: **58,082 → 37,718 bytes** (−35.06%) - authenticated MCP responses: **46,900 → 38,268 bytes** (−18.41%) - design MCP responses: **51,854 → 44,008 bytes** (−15.13%) - aggregate MCP responses versus the accepted baseline: **147,335 → 101,922 bytes** (−30.82%) - aggregate tokens: **1,146,951**, only **+0.33%** above the stricter accepted 1,143,217-token baseline The complete suite and every per-fixture comparison passed. The guarded updater did not replace the accepted artifacts because the aggregate token sample did not beat the existing baseline. ## Responsive verification batching The MCP now exposes `screenshot.responsive` when reusable page capture is available. It accepts one generated route plus 1–8 viewports, refreshes the session preview once, reuses one browser target, and returns bounded evidence for every viewport. Auth and design evaluation workflows use one responsive capture instead of a separate preview start and two screenshot turns. Automatic screenshots now allocate an available preview port and reuse an already-running managed preview port. This prevents an unrelated service on the historical 5173 default from producing an MCP failure. Explicit ports, local previews, and external URLs remain unchanged. The evaluator records the bounded viewport array, counts the batch as verification and visual evidence, and compares its latency, response size, calls, errors, retries, and token consumption on every run. The new regression tests were demonstrated fail-then-pass. Guarded accepted results: - aggregate tokens: **1,143,217 → 990,190** (−13.39%) - aggregate calls: **39 → 30** (−23.08%) - aggregate MCP response bytes: **147,335 → 98,470** (−33.17%) - failures/retries: **0 / 0** - authenticated calls: **14 → 8** (−42.86%) - design calls: **17 → 14** (−17.65%) ## Design and font workflow batching Design discovery now uses one conditionally exposed `inspect-design-context` call to return bounded pages, breakpoints, design tokens, assets, and variables instead of five model-visible reads. Font persistence now uses one `verify-font-assets` call to refresh the asset namespace and return every changed font asset instead of a refresh plus separate verification reads. The evaluation tasks and validators require these composite paths. Both tools retain the underlying operation limits, namespace metadata, persisted-result checks, and zero-error/retry gates. Review also confirmed that SDK schemas must keep `additionalProperties:false` and that non-default MCP safety hints must remain explicit; those protocol constraints were preserved rather than traded for catalog-token savings. Guarded accepted improvements versus the previous responsive-batching baseline: - aggregate tokens: **990,190 → 914,760** (−7.62%) - aggregate calls: **30 → 24** (−20.00%) - aggregate focused reads: **8 → 2** (−75.00%) - aggregate MCP response bytes: **98,470 → 95,614** (−2.90%) - design calls: **14 → 10** (−28.57%) - font calls: **8 → 6** (−25.00%) - failures/retries: **0 / 0** The strict SDK catalog is **51,362 bytes** with **28,964 input-schema bytes**. The two new conditional workflow tools add 355 catalog bytes (+0.70%) and 130 schema bytes (+0.45%) while removing six model-visible workflow turns. ## Terminal responsive verification Auth and design now finish with one conditionally exposed `verify-page-responsive` call. It captures every requested viewport in one browser session and immediately runs the static page audit, removing one sequential model turn per workflow while retaining both kinds of evaluator evidence. The tool inherits the audit operation's permit and read namespaces; strict SDK schemas and MCP safety annotations remain unchanged. Latest complete comparison against the accepted baseline: - aggregate tokens: **914,760 → 869,274** (−4.97%) - aggregate calls: **24 → 22** (−8.33%) - aggregate MCP response bytes: **95,614 → 95,053** (−0.59%) - authenticated calls: **8 → 7** - design calls: **10 → 9** - font calls: **6 → 6** - failures/retries: **0 / 0** - authenticated focused run: **289,078 tokens** (−10.06%) and first verification **55,552 ms** (−17.90%) - design focused run: p95 MCP latency **9,642 ms** (−10.32%); token and first-verification variance increased in that individual sample, so only the complete comparison is used for the aggregate conclusion The complete comparison passed all fixture and aggregate regression gates. Two subsequent guarded `--update-baselines` attempts could not start any fixture because the external Codex runner had reached its usage limit; both exited before the first MCP call and wrote no accepted result files. Therefore the tracked accepted baseline below remains **914,760 tokens / 24 calls**, while **869,274 tokens / 22 calls** is the latest successful complete comparison. Regression coverage was demonstrated fail then pass. After this iteration, project-build MCP tests passed **105/105**, high-impact evaluation tests passed **41/41**, and project/CLI typechecks, lint, package boundaries, generated docs, and diff checks all passed. ## Accepted evaluation baseline | Fixture | Failed | Retries | Calls | Mutations | Tokens | Uncached input | Response bytes | First mutation | First verification | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | authenticated-page-v1 | 0 | 0 | 8 | 4 | 321,396 | 44,353 | 37,536 | 39,076 ms | 67,660 ms | | design-input-v1 | 0 | 0 | 10 | 6 | 330,580 | 54,441 | 40,036 | 30,419 ms | 36,551 ms | | font-assets-v1 | 0 | 0 | 6 | 3 | 262,784 | 20,704 | 18,042 | 37,609 ms | 47,073 ms | Accepted aggregate: **914,760 tokens**, **24 calls**, **2 focused reads**, and **95,614 MCP response bytes**, with zero failures and zero retries. Compared with the previous accepted baseline, tokens fell **7.62%**, calls fell **20.00%**, focused reads fell **75.00%**, and response bytes fell **2.90%**. ## Verification - [x] Complete `pnpm evaluations -- --update-baselines` - [x] Independent complete `pnpm evaluations` after the composite workflow change - [x] Worse-token and error-containing samples rejected without replacing accepted baselines - [x] Added/changed regression tests demonstrated fail then pass - [x] Project-build MCP tests: 105 passed - [x] Complete high-impact evaluation tests: 41 passed - [x] Project-build and CLI typechecks - [x] `pnpm lint` - [x] `pnpm check:package-boundaries` - [x] `pnpm check:generated-docs` - [x] `git diff --check` - [x] Earlier workspace tests, including 2,356 builder tests - [x] Earlier generated API checks - [ ] Fixture sync/build: sync could not reach `main.development.webstudio.is` and timed out; no local test, evaluation, or code check failed ## Implementation checklist - [x] Error and retry rate enforced at zero for raw and compared outcomes - [x] Relative page paths normalized to prevent a known validation failure and retry - [x] Baseline replacement blocked unless every outcome and comparison passes - [x] Aggregate token baselines cannot ratchet upward - [x] Workflow churn bounded and regression-gated - [x] Authentication discovery collapsed from four model-visible reads to one - [x] Fragment insertion and persisted binding verification combined safely - [x] Large style-key mutation results bounded at the MCP boundary - [x] Token consumption captured, compared, and regression-gated - [x] Response bytes captured per call and tool, summarized, and aggregate regression-gated - [x] Per-tool latency, mutation timing, catalog size, and validation paths captured - [x] Repeated catalog/schema and focused-guide context reduced - [x] Canonical goal selection stabilized for evaluations - [x] Grouped style maps normalized before validation - [x] Responsive verification batches desktop and mobile evidence in one call - [x] Automatic screenshot previews avoid occupied default ports - [x] Responsive viewport evidence is captured and regression-gated - [x] Accepted evaluation artifacts preserve the best measured token baseline
This commit is contained in:
Oleg Isonen authored and GitHub committed
1 parent
e80ca3b54c
commit
7bc9942774
23 files changed
+3408
-421
No files matched your search
@@ -4,6 +4,7 @@ import { join, resolve } from "node:path";
|
||||
import { describe, expect, test } from "vitest";
|
||||
import { authenticatedPageFixture } from "./fixtures";
|
||||
import { designInputFixture } from "./fixtures";
|
||||
import { fontAssetsFixture } from "./fixtures";
|
||||
import {
|
||||
createMinimalAgentTask,
|
||||
getCliInvocation,
|
||||
@@ -23,15 +24,48 @@ describe("high-impact agent runner", () => {
|
||||
});
|
||||
expect(task).not.toHaveProperty("project");
|
||||
expect(task).toMatchObject({
|
||||
constraints: [
|
||||
constraints: expect.arrayContaining([
|
||||
expect.any(String),
|
||||
expect.stringContaining("meta.guide"),
|
||||
expect.stringContaining("exactly once at the beginning"),
|
||||
expect.any(String),
|
||||
expect.stringContaining("Never use broad project reads"),
|
||||
expect.any(String),
|
||||
expect.stringContaining("meta.next"),
|
||||
],
|
||||
expect.stringContaining("terminal static audit"),
|
||||
expect.stringContaining("Use one verify-page-responsive call"),
|
||||
expect.stringContaining("Do not call list-breakpoints"),
|
||||
]),
|
||||
});
|
||||
const authConstraints = task.constraints.join("\n");
|
||||
expect(authConstraints).toContain("Do not call list-breakpoints");
|
||||
expect(authConstraints).toContain("do not call meta.get_more_tools");
|
||||
expect(authConstraints).toContain(
|
||||
"Create exactly one scoped non-secret fixture variable"
|
||||
);
|
||||
expect(authConstraints).toContain("Do not call list-variables again");
|
||||
expect(authConstraints).toContain("static audit");
|
||||
expect(authConstraints).toContain(
|
||||
"copy the objective field verbatim into brief"
|
||||
);
|
||||
expect(authConstraints).toContain("call inspect-auth-context exactly once");
|
||||
expect(authConstraints).not.toContain(
|
||||
"call get-project-settings, list-pages, list-resources, and list-variables"
|
||||
);
|
||||
expect(authConstraints).toContain(
|
||||
"call create-page exactly once, then create-variable"
|
||||
);
|
||||
expect(authConstraints).toContain(
|
||||
"Call insert-fragment-verified only after all three succeed"
|
||||
);
|
||||
expect(authConstraints).toContain(
|
||||
"Do not call insert-fragment or verify-bindings separately"
|
||||
);
|
||||
expect(authConstraints).toContain(
|
||||
"call verify-page-responsive exactly once"
|
||||
);
|
||||
expect(authConstraints).toContain(
|
||||
"Do not call preview.start, screenshot, screenshot.responsive, or audit separately"
|
||||
);
|
||||
expect(
|
||||
createMinimalAgentTask(designInputFixture, {
|
||||
kind: "source",
|
||||
@@ -42,7 +76,40 @@ describe("high-impact agent runner", () => {
|
||||
desktop: { viewport: { width: 1440, height: 900 } },
|
||||
mobile: { viewport: { width: 390, height: 844 } },
|
||||
},
|
||||
constraints: expect.arrayContaining([
|
||||
expect.stringContaining("exactly three attach-design-token calls"),
|
||||
expect.stringContaining("Omit the optional position field"),
|
||||
expect.stringContaining("exactly one batched update-styles call"),
|
||||
expect.stringContaining("returned mobile breakpoint id"),
|
||||
expect.stringContaining("Use this exact fragment verbatim"),
|
||||
expect.stringContaining('ws:tag="footer"'),
|
||||
expect.stringContaining("Do not call clone-instance"),
|
||||
expect.stringContaining("set-text-content"),
|
||||
expect.stringContaining("call inspect-design-context exactly once"),
|
||||
expect.stringContaining("Do not call list-pages, list-breakpoints"),
|
||||
expect.stringContaining("Attach all three tokens in one parallel"),
|
||||
expect.stringContaining("Do not call get-page-by-path"),
|
||||
expect.stringContaining("call insert-fragment-verified once"),
|
||||
expect.stringContaining("pagePath /summer"),
|
||||
expect.stringContaining("do not call list-instances"),
|
||||
expect.stringContaining("Do not call meta.index"),
|
||||
]),
|
||||
});
|
||||
expect(
|
||||
createMinimalAgentTask(fontAssetsFixture, {
|
||||
kind: "source",
|
||||
repositoryRoot: root,
|
||||
}).constraints
|
||||
).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.stringContaining("Do not call meta.index, meta.get_more_tools"),
|
||||
expect.stringContaining("Call meta.guide exactly once"),
|
||||
expect.stringContaining("exactly one upload-assets call"),
|
||||
expect.stringContaining("parallel tool-call batch"),
|
||||
expect.stringContaining("verify-font-assets exactly once"),
|
||||
expect.stringContaining("Do not call refresh or get-asset separately"),
|
||||
])
|
||||
);
|
||||
});
|
||||
|
||||
test("retains only a bounded privacy-safe result from a real process", async () => {
|
||||
@@ -72,8 +139,35 @@ describe("high-impact agent runner", () => {
|
||||
provider: "test-provider",
|
||||
model: "test-model",
|
||||
getToolCalls: () => [
|
||||
{ name: "meta.guide", startedAtMs: 100, durationMs: 25 },
|
||||
{ name: "audit", startedAtMs: 500, durationMs: 75 },
|
||||
{
|
||||
name: "meta.guide",
|
||||
startedAtMs: 100,
|
||||
durationMs: 25,
|
||||
responseBytes: 1_000,
|
||||
},
|
||||
{
|
||||
name: "attach-design-token",
|
||||
startedAtMs: 300,
|
||||
durationMs: 10,
|
||||
responseBytes: 500,
|
||||
isError: true,
|
||||
errorCode: "INVALID_INPUT",
|
||||
},
|
||||
{
|
||||
name: "audit",
|
||||
startedAtMs: 500,
|
||||
durationMs: 75,
|
||||
responseBytes: 750,
|
||||
},
|
||||
],
|
||||
getCatalogObservations: () => [
|
||||
{
|
||||
kind: "tools-list",
|
||||
toolCount: 160,
|
||||
responseBytes: 100_000,
|
||||
inputSchemaBytes: 50_000,
|
||||
descriptionBytes: 10_000,
|
||||
},
|
||||
],
|
||||
evaluate: async () => ({
|
||||
passed: true,
|
||||
@@ -83,24 +177,41 @@ describe("high-impact agent runner", () => {
|
||||
});
|
||||
expect(result).toMatchObject({
|
||||
schemaVersion: 2,
|
||||
outcome: "passed",
|
||||
outcome: "failed",
|
||||
cli: "packaged",
|
||||
metrics: {
|
||||
durationMs: expect.any(Number),
|
||||
tokens: {
|
||||
input: 1_000,
|
||||
cachedInput: 400,
|
||||
uncachedInput: 600,
|
||||
output: 200,
|
||||
total: 1_200,
|
||||
},
|
||||
mcpCatalog: {
|
||||
responses: 1,
|
||||
totalResponseBytes: 100_000,
|
||||
latestToolCount: 160,
|
||||
latestInputSchemaBytes: 50_000,
|
||||
},
|
||||
toolCalls: {
|
||||
total: 2,
|
||||
failed: 0,
|
||||
total: 3,
|
||||
failed: 1,
|
||||
failuresByTool: { "attach-design-token": 1 },
|
||||
responseBytesByTool: {
|
||||
"attach-design-token": {
|
||||
count: 1,
|
||||
totalBytes: 500,
|
||||
p95Bytes: 500,
|
||||
maxBytes: 500,
|
||||
},
|
||||
},
|
||||
totalResponseBytes: 2_250,
|
||||
verifications: 1,
|
||||
timeToFirstVerificationMs: 500,
|
||||
},
|
||||
},
|
||||
callSequence: ["meta.guide", "audit"],
|
||||
callSequence: ["meta.guide", "attach-design-token", "audit"],
|
||||
checks: { usageCaptured: "passed" },
|
||||
});
|
||||
const source = await readFile(resultPath, "utf8");
|
||||
@@ -109,6 +220,10 @@ describe("high-impact agent runner", () => {
|
||||
expect(JSON.parse(await readFile(taskPath, "utf8"))).toMatchObject({
|
||||
fixtureId: "authenticated-page-v1",
|
||||
mcp: { args: ["mcp"] },
|
||||
constraints: expect.arrayContaining([
|
||||
expect.stringContaining("terminal static audit"),
|
||||
expect.stringContaining("Use one verify-page-responsive call"),
|
||||
]),
|
||||
});
|
||||
} finally {
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
|
||||
@@ -9,8 +9,11 @@ import type {
|
||||
import {
|
||||
addAgentUsage,
|
||||
getAgentUsageEvent,
|
||||
getMcpCatalogMetrics,
|
||||
getMcpEvaluationMetrics,
|
||||
type AgentUsage,
|
||||
type McpCatalogMetrics,
|
||||
type McpCatalogObservation,
|
||||
type McpEvaluationMetrics,
|
||||
} from "./evaluation-metrics";
|
||||
import { runAgentCommand } from "../../scripts/run-agent-command";
|
||||
@@ -33,6 +36,7 @@ export type AgentEvaluationResult = {
|
||||
metrics: {
|
||||
durationMs: number;
|
||||
tokens?: AgentUsage;
|
||||
mcpCatalog?: McpCatalogMetrics;
|
||||
toolCalls: McpEvaluationMetrics;
|
||||
};
|
||||
callSequence: string[];
|
||||
@@ -68,11 +72,39 @@ export const createMinimalAgentTask = (
|
||||
mcp: getCliInvocation(target),
|
||||
constraints: [
|
||||
"Use the configured Webstudio project and local CLI.",
|
||||
"Begin with meta.guide for the objective and follow its workflow.",
|
||||
'Call meta.guide exactly once at the beginning with {"brief":"<objective>"}: copy the objective field verbatim into brief, follow the returned workflow, and do not call meta.guide again.',
|
||||
"Choose focused reads and semantic edits yourself.",
|
||||
"Never use broad project reads: snapshot, components.list, or components.coverage-plan.",
|
||||
"Do not persist or report credentials or private session data.",
|
||||
"Treat mutation meta.next steps as required. Do not report completion until audit and requested visual evidence pass.",
|
||||
"Treat the successful final audit and requested visual evidence as terminal; do not mutate, verify, restart preview, or capture more evidence afterward.",
|
||||
"Finish all visual polish before evidence capture. Use one verify-page-responsive call for all requested viewports and the terminal static audit. After it begins, do not mutate, rediscover, restart preview, or capture more evidence.",
|
||||
...(fixture.id === "authenticated-page-v1"
|
||||
? [
|
||||
"For this fixture, meta.guide already returns the required auth discovery and authoring tool schemas. Do not call list-breakpoints because responsive styling is not required, and do not call meta.get_more_tools or any other tool discovery operation. Create exactly one scoped non-secret fixture variable and keep the required state gallery expression-free. Do not call list-variables again after creating it.",
|
||||
"After meta.guide, call inspect-auth-context exactly once. Do not call get-project-settings, list-pages, list-resources, or list-variables separately.",
|
||||
"After the context read, call create-page exactly once, then create-variable exactly once, then create-resource exactly once with the returned variable id. Do not parallelize these mutations. Call insert-fragment-verified only after all three succeed, using their returned ids and pagePath /account. Do not call insert-fragment or verify-bindings separately.",
|
||||
'Then call verify-page-responsive exactly once with {"path":"/account","viewports":[{"width":1440,"height":900},{"width":390,"height":844}],"source":"session"}. It captures both required screenshots and runs the terminal static audit. Do not call preview.start, screenshot, screenshot.responsive, or audit separately.',
|
||||
]
|
||||
: []),
|
||||
...(fixture.id === "design-input-v1"
|
||||
? [
|
||||
"For this fixture, do not call list-instances because the project has no representative existing page pattern. Do not call meta.index, meta.get_more_tools, or any other tool discovery operation because meta.guide and the MCP handshake already provide the required schemas.",
|
||||
"After meta.guide, call inspect-design-context exactly once before mutation. Do not call list-pages, list-breakpoints, list-design-tokens, list-assets, or list-variables separately. Do not call get-page-by-path or repeat discovery. Call create-page once, then call insert-fragment-verified once with pagePath /summer before token or style mutations. Do not call insert-fragment or verify-bindings separately.",
|
||||
"For this fixture, make exactly three attach-design-token calls: attach only the returned Brand / Coral, Text / Ink, and Type / Heading token ids once each to compatible inserted element instance ids. Attach all three tokens in one parallel tool-call batch. Omit the optional position field and do not attempt any other token attachment.",
|
||||
"After the early binding checkpoint, make exactly one batched update-styles call containing all remaining fixed styles. Include at least one declaration on an inserted element using the returned mobile breakpoint id so responsive behavior is persisted before preview.",
|
||||
'Use this exact fragment verbatim in the single insert-fragment-verified call; do not add props, styles, expressions, or alternate components until after it commits: <ws.element ws:tag="header"><ws.element ws:tag="nav"><ws.element ws:tag="a">Northstar</ws.element><ws.element ws:tag="button">Menu</ws.element></ws.element></ws.element><ws.element ws:tag="main"><ws.element ws:tag="section"><ws.element ws:tag="h1">Find your latitude</ws.element><ws.element ws:tag="p">Plan a memorable summer escape.</ws.element><ws.element ws:tag="a">Explore trips</ws.element></ws.element><ws.element ws:tag="section"><ws.element ws:tag="h2">Featured trips</ws.element><ws.element ws:tag="article"><ws.element ws:tag="h3">Coastal escape</ws.element><ws.element ws:tag="p">A restorative journey by the sea.</ws.element></ws.element></ws.element></ws.element><ws.element ws:tag="footer"><ws.element ws:tag="p">Northstar travel</ws.element></ws.element>',
|
||||
"The exact fragment already contains all required content. Do not call clone-instance, update-text, set-text-content, or any other content or structure mutation after insertion; only attach the three tokens and apply the one batched style update.",
|
||||
'After the style update, call verify-page-responsive exactly once with {"path":"/summer","viewports":[{"width":1440,"height":900},{"width":390,"height":844}],"source":"session"}. It captures both required screenshots and runs the terminal static audit. Do not call preview.start, screenshot, screenshot.responsive, or audit separately.',
|
||||
]
|
||||
: []),
|
||||
...(fixture.id === "font-assets-v1"
|
||||
? [
|
||||
"Do not call meta.index, meta.get_more_tools, or any other tool discovery operation because meta.guide and the MCP handshake already provide the required schemas. Upload both supplied fonts together with exactly one upload-assets call; do not use upload-asset.",
|
||||
"After upload, update both font assets together in one parallel tool-call batch.",
|
||||
"After both updates, call verify-font-assets exactly once with both returned asset ids, then audit. Do not call refresh or get-asset separately.",
|
||||
]
|
||||
: []),
|
||||
],
|
||||
};
|
||||
};
|
||||
@@ -91,9 +123,22 @@ const assertBoundedResult = (result: AgentEvaluationResult) => {
|
||||
for (const [key, child] of Object.entries(value)) {
|
||||
const isAggregateUsage =
|
||||
path.length === 1 && path[0] === "metrics" && key === "tokens";
|
||||
const isMetricIdentifier =
|
||||
path.length === 3 &&
|
||||
path[0] === "metrics" &&
|
||||
path[1] === "toolCalls" &&
|
||||
[
|
||||
"failuresByTool",
|
||||
"failuresByCode",
|
||||
"issuesByCode",
|
||||
"issuesByPath",
|
||||
"durationsByTool",
|
||||
"responseBytesByTool",
|
||||
].includes(path[2]);
|
||||
if (
|
||||
forbiddenResultKeys.test(key) ||
|
||||
(forbiddenTokenKey.test(key) && isAggregateUsage === false)
|
||||
isMetricIdentifier === false &&
|
||||
(forbiddenResultKeys.test(key) ||
|
||||
(forbiddenTokenKey.test(key) && isAggregateUsage === false))
|
||||
) {
|
||||
throw new Error(`Agent result contains forbidden field ${key}.`);
|
||||
}
|
||||
@@ -116,6 +161,7 @@ export const runHighImpactAgentEvaluation = async ({
|
||||
provider,
|
||||
model,
|
||||
getToolCalls,
|
||||
getCatalogObservations,
|
||||
evaluate,
|
||||
env = process.env,
|
||||
timeoutMs = 10 * 60_000,
|
||||
@@ -129,6 +175,7 @@ export const runHighImpactAgentEvaluation = async ({
|
||||
provider: string;
|
||||
model: string;
|
||||
getToolCalls: () => EvaluationToolCall[];
|
||||
getCatalogObservations: () => McpCatalogObservation[];
|
||||
evaluate: () => Promise<HighImpactEvaluationResult>;
|
||||
env?: NodeJS.ProcessEnv;
|
||||
timeoutMs?: number;
|
||||
@@ -160,17 +207,23 @@ export const runHighImpactAgentEvaluation = async ({
|
||||
});
|
||||
const evaluation = await evaluate();
|
||||
const toolCalls = getToolCalls();
|
||||
const mcpCatalog = getMcpCatalogMetrics(getCatalogObservations());
|
||||
const checks = {
|
||||
...evaluation.checks,
|
||||
usageCaptured:
|
||||
usage === undefined ? ("failed" as const) : ("passed" as const),
|
||||
};
|
||||
const toolCallMetrics = getMcpEvaluationMetrics(toolCalls);
|
||||
const result: AgentEvaluationResult = {
|
||||
schemaVersion: 2,
|
||||
kind: "high-impact-minimal-context-agent-evaluation-result",
|
||||
fixtureId: fixture.id,
|
||||
outcome:
|
||||
execution.exitCode === 0 && evaluation.passed && usage !== undefined
|
||||
execution.exitCode === 0 &&
|
||||
evaluation.passed &&
|
||||
usage !== undefined &&
|
||||
toolCallMetrics.failed === 0 &&
|
||||
toolCallMetrics.retries === 0
|
||||
? "passed"
|
||||
: "failed",
|
||||
cli: target.kind,
|
||||
@@ -181,7 +234,8 @@ export const runHighImpactAgentEvaluation = async ({
|
||||
metrics: {
|
||||
durationMs: execution.durationMs,
|
||||
...(usage === undefined ? {} : { tokens: usage }),
|
||||
toolCalls: getMcpEvaluationMetrics(toolCalls),
|
||||
...(mcpCatalog === undefined ? {} : { mcpCatalog }),
|
||||
toolCalls: toolCallMetrics,
|
||||
},
|
||||
callSequence: toolCalls.map(({ name }) => name),
|
||||
checks,
|
||||
|
||||
@@ -2,10 +2,40 @@ import { describe, expect, test } from "vitest";
|
||||
import {
|
||||
addAgentUsage,
|
||||
getAgentUsageEvent,
|
||||
getMcpCatalogMetrics,
|
||||
getMcpEvaluationMetrics,
|
||||
} from "./evaluation-metrics";
|
||||
|
||||
describe("evaluation metrics", () => {
|
||||
test("aggregates MCP catalog observations", () => {
|
||||
expect(
|
||||
getMcpCatalogMetrics([
|
||||
{
|
||||
kind: "tools-list",
|
||||
toolCount: 160,
|
||||
responseBytes: 100_000,
|
||||
inputSchemaBytes: 50_000,
|
||||
descriptionBytes: 10_000,
|
||||
},
|
||||
{
|
||||
kind: "tools-list",
|
||||
toolCount: 12,
|
||||
responseBytes: 20_000,
|
||||
inputSchemaBytes: 8_000,
|
||||
descriptionBytes: 2_000,
|
||||
},
|
||||
])
|
||||
).toEqual({
|
||||
responses: 2,
|
||||
totalResponseBytes: 120_000,
|
||||
maxResponseBytes: 100_000,
|
||||
latestToolCount: 12,
|
||||
latestResponseBytes: 20_000,
|
||||
latestInputSchemaBytes: 8_000,
|
||||
latestDescriptionBytes: 2_000,
|
||||
});
|
||||
});
|
||||
|
||||
test("retains only aggregate usage from Codex completion events", () => {
|
||||
const usage = getAgentUsageEvent({
|
||||
type: "turn.completed",
|
||||
@@ -21,6 +51,7 @@ describe("evaluation metrics", () => {
|
||||
expect(usage).toEqual({
|
||||
input: 1_000,
|
||||
cachedInput: 400,
|
||||
uncachedInput: 600,
|
||||
cacheWriteInput: 50,
|
||||
output: 200,
|
||||
reasoningOutput: 75,
|
||||
@@ -33,6 +64,7 @@ describe("evaluation metrics", () => {
|
||||
expect(addAgentUsage(usage, usage)).toEqual({
|
||||
input: 2_000,
|
||||
cachedInput: 800,
|
||||
uncachedInput: 1_200,
|
||||
cacheWriteInput: 100,
|
||||
output: 400,
|
||||
reasoningOutput: 150,
|
||||
@@ -44,21 +76,51 @@ describe("evaluation metrics", () => {
|
||||
);
|
||||
});
|
||||
|
||||
test("counts responsive page verification as early evidence", () => {
|
||||
expect(
|
||||
getMcpEvaluationMetrics([
|
||||
{
|
||||
name: "verify-page-responsive",
|
||||
startedAtMs: 25,
|
||||
durationMs: 100,
|
||||
responseBytes: 2_000,
|
||||
},
|
||||
])
|
||||
).toMatchObject({
|
||||
verifications: 1,
|
||||
timeToFirstVerificationMs: 25,
|
||||
});
|
||||
});
|
||||
|
||||
test("measures MCP errors, retries, mutations, verification, and latency", () => {
|
||||
expect(
|
||||
getMcpEvaluationMetrics([
|
||||
{ name: "meta.guide", startedAtMs: 10, durationMs: 10 },
|
||||
{ name: "snapshot", startedAtMs: 20, durationMs: 20 },
|
||||
{
|
||||
name: "meta.guide",
|
||||
startedAtMs: 10,
|
||||
durationMs: 10,
|
||||
responseBytes: 1_000,
|
||||
},
|
||||
{
|
||||
name: "snapshot",
|
||||
startedAtMs: 20,
|
||||
durationMs: 20,
|
||||
responseBytes: 6_000,
|
||||
},
|
||||
{
|
||||
name: "create-page",
|
||||
startedAtMs: 50,
|
||||
durationMs: 30,
|
||||
responseBytes: 500,
|
||||
isError: true,
|
||||
errorCode: "INVALID_INPUT",
|
||||
errorIssues: [{ code: "invalid_resource_url", path: "resource.url" }],
|
||||
},
|
||||
{
|
||||
name: "create-page",
|
||||
startedAtMs: 90,
|
||||
durationMs: 40,
|
||||
responseBytes: 2_000,
|
||||
committed: true,
|
||||
},
|
||||
{
|
||||
@@ -66,14 +128,63 @@ describe("evaluation metrics", () => {
|
||||
arguments: { dryRun: true },
|
||||
startedAtMs: 110,
|
||||
durationMs: 50,
|
||||
responseBytes: 3_000,
|
||||
planned: true,
|
||||
},
|
||||
{ name: "audit", startedAtMs: 170, durationMs: 100 },
|
||||
{
|
||||
name: "audit",
|
||||
startedAtMs: 170,
|
||||
durationMs: 100,
|
||||
responseBytes: 1_500,
|
||||
},
|
||||
])
|
||||
).toEqual({
|
||||
total: 6,
|
||||
succeeded: 5,
|
||||
failed: 1,
|
||||
failuresByTool: { "create-page": 1 },
|
||||
failuresByCode: { INVALID_INPUT: 1 },
|
||||
issuesByCode: { invalid_resource_url: 1 },
|
||||
issuesByPath: { "resource.url": 1 },
|
||||
durationsByTool: {
|
||||
audit: { count: 1, totalMs: 100, p95Ms: 100, maxMs: 100 },
|
||||
"create-page": { count: 2, totalMs: 70, p95Ms: 40, maxMs: 40 },
|
||||
"meta.guide": { count: 1, totalMs: 10, p95Ms: 10, maxMs: 10 },
|
||||
snapshot: { count: 1, totalMs: 20, p95Ms: 20, maxMs: 20 },
|
||||
"update-page": { count: 1, totalMs: 50, p95Ms: 50, maxMs: 50 },
|
||||
},
|
||||
responseBytesByTool: {
|
||||
audit: {
|
||||
count: 1,
|
||||
totalBytes: 1_500,
|
||||
p95Bytes: 1_500,
|
||||
maxBytes: 1_500,
|
||||
},
|
||||
"create-page": {
|
||||
count: 2,
|
||||
totalBytes: 2_500,
|
||||
p95Bytes: 2_000,
|
||||
maxBytes: 2_000,
|
||||
},
|
||||
"meta.guide": {
|
||||
count: 1,
|
||||
totalBytes: 1_000,
|
||||
p95Bytes: 1_000,
|
||||
maxBytes: 1_000,
|
||||
},
|
||||
snapshot: {
|
||||
count: 1,
|
||||
totalBytes: 6_000,
|
||||
p95Bytes: 6_000,
|
||||
maxBytes: 6_000,
|
||||
},
|
||||
"update-page": {
|
||||
count: 1,
|
||||
totalBytes: 3_000,
|
||||
p95Bytes: 3_000,
|
||||
maxBytes: 3_000,
|
||||
},
|
||||
},
|
||||
retries: 1,
|
||||
focusedReads: 0,
|
||||
broadReads: 1,
|
||||
@@ -84,6 +195,10 @@ describe("evaluation metrics", () => {
|
||||
totalDurationMs: 250,
|
||||
p50DurationMs: 30,
|
||||
p95DurationMs: 100,
|
||||
totalResponseBytes: 14_000,
|
||||
p50ResponseBytes: 1_500,
|
||||
p95ResponseBytes: 6_000,
|
||||
maxResponseBytes: 6_000,
|
||||
timeToFirstMutationMs: 90,
|
||||
timeToFirstVerificationMs: 170,
|
||||
});
|
||||
|
||||
@@ -4,6 +4,7 @@ import type { EvaluationToolCall } from "./validate";
|
||||
export type AgentUsage = {
|
||||
input: number;
|
||||
cachedInput: number;
|
||||
uncachedInput: number;
|
||||
cacheWriteInput: number;
|
||||
output: number;
|
||||
reasoningOutput: number;
|
||||
@@ -22,9 +23,13 @@ const createAgentUsage = ({
|
||||
cacheWriteInput,
|
||||
output,
|
||||
reasoningOutput,
|
||||
}: Omit<AgentUsage, "total" | "cachedInputRate">): AgentUsage => ({
|
||||
}: Omit<
|
||||
AgentUsage,
|
||||
"total" | "cachedInputRate" | "uncachedInput"
|
||||
>): AgentUsage => ({
|
||||
input,
|
||||
cachedInput,
|
||||
uncachedInput: Math.max(0, input - cachedInput),
|
||||
cacheWriteInput,
|
||||
output,
|
||||
reasoningOutput,
|
||||
@@ -33,6 +38,47 @@ const createAgentUsage = ({
|
||||
input === 0 ? 0 : Math.round((cachedInput / input) * 10_000) / 10_000,
|
||||
});
|
||||
|
||||
export type McpCatalogObservation = {
|
||||
kind: "tools-list";
|
||||
toolCount: number;
|
||||
responseBytes: number;
|
||||
inputSchemaBytes: number;
|
||||
descriptionBytes: number;
|
||||
};
|
||||
|
||||
export type McpCatalogMetrics = {
|
||||
responses: number;
|
||||
totalResponseBytes: number;
|
||||
maxResponseBytes: number;
|
||||
latestToolCount: number;
|
||||
latestResponseBytes: number;
|
||||
latestInputSchemaBytes: number;
|
||||
latestDescriptionBytes: number;
|
||||
};
|
||||
|
||||
export const getMcpCatalogMetrics = (
|
||||
observations: readonly McpCatalogObservation[]
|
||||
): McpCatalogMetrics | undefined => {
|
||||
const latest = observations.at(-1);
|
||||
if (latest === undefined) {
|
||||
return;
|
||||
}
|
||||
return {
|
||||
responses: observations.length,
|
||||
totalResponseBytes: observations.reduce(
|
||||
(total, observation) => total + observation.responseBytes,
|
||||
0
|
||||
),
|
||||
maxResponseBytes: Math.max(
|
||||
...observations.map((observation) => observation.responseBytes)
|
||||
),
|
||||
latestToolCount: latest.toolCount,
|
||||
latestResponseBytes: latest.responseBytes,
|
||||
latestInputSchemaBytes: latest.inputSchemaBytes,
|
||||
latestDescriptionBytes: latest.descriptionBytes,
|
||||
};
|
||||
};
|
||||
|
||||
export const getAgentUsageEvent = (value: unknown): AgentUsage | undefined => {
|
||||
if (
|
||||
isPlainRecord(value) === false ||
|
||||
@@ -84,6 +130,18 @@ export type McpEvaluationMetrics = {
|
||||
total: number;
|
||||
succeeded: number;
|
||||
failed: number;
|
||||
failuresByTool: Record<string, number>;
|
||||
failuresByCode: Record<string, number>;
|
||||
issuesByCode: Record<string, number>;
|
||||
issuesByPath: Record<string, number>;
|
||||
durationsByTool: Record<
|
||||
string,
|
||||
{ count: number; totalMs: number; p95Ms: number; maxMs: number }
|
||||
>;
|
||||
responseBytesByTool: Record<
|
||||
string,
|
||||
{ count: number; totalBytes: number; p95Bytes: number; maxBytes: number }
|
||||
>;
|
||||
retries: number;
|
||||
focusedReads: number;
|
||||
broadReads: number;
|
||||
@@ -94,6 +152,10 @@ export type McpEvaluationMetrics = {
|
||||
totalDurationMs: number;
|
||||
p50DurationMs: number;
|
||||
p95DurationMs: number;
|
||||
totalResponseBytes: number;
|
||||
p50ResponseBytes: number;
|
||||
p95ResponseBytes: number;
|
||||
maxResponseBytes: number;
|
||||
timeToFirstMutationMs?: number;
|
||||
timeToFirstVerificationMs?: number;
|
||||
};
|
||||
@@ -107,8 +169,11 @@ const broadReadNames = new Set([
|
||||
const verificationNames = new Set([
|
||||
"audit",
|
||||
"screenshot",
|
||||
"screenshot.responsive",
|
||||
"verify-page-responsive",
|
||||
"screenshot.diff",
|
||||
"verify-bindings",
|
||||
"insert-fragment-verified",
|
||||
]);
|
||||
|
||||
export const isFocusedRead = (name: string) =>
|
||||
@@ -125,6 +190,56 @@ const percentile = (values: number[], proportion: number) => {
|
||||
return sorted[index] ?? 0;
|
||||
};
|
||||
|
||||
const getDurationsByTool = (calls: readonly EvaluationToolCall[]) => {
|
||||
const durations = new Map<string, number[]>();
|
||||
for (const call of calls) {
|
||||
if (call.durationMs === undefined) {
|
||||
continue;
|
||||
}
|
||||
const values = durations.get(call.name) ?? [];
|
||||
values.push(call.durationMs);
|
||||
durations.set(call.name, values);
|
||||
}
|
||||
return Object.fromEntries(
|
||||
[...durations.entries()]
|
||||
.sort(([left], [right]) => left.localeCompare(right))
|
||||
.map(([name, values]) => [
|
||||
name,
|
||||
{
|
||||
count: values.length,
|
||||
totalMs: values.reduce((total, value) => total + value, 0),
|
||||
p95Ms: percentile(values, 0.95),
|
||||
maxMs: Math.max(...values),
|
||||
},
|
||||
])
|
||||
);
|
||||
};
|
||||
|
||||
const getResponseBytesByTool = (calls: readonly EvaluationToolCall[]) => {
|
||||
const responseBytes = new Map<string, number[]>();
|
||||
for (const call of calls) {
|
||||
if (call.responseBytes === undefined) {
|
||||
continue;
|
||||
}
|
||||
const values = responseBytes.get(call.name) ?? [];
|
||||
values.push(call.responseBytes);
|
||||
responseBytes.set(call.name, values);
|
||||
}
|
||||
return Object.fromEntries(
|
||||
[...responseBytes.entries()]
|
||||
.sort(([left], [right]) => left.localeCompare(right))
|
||||
.map(([name, values]) => [
|
||||
name,
|
||||
{
|
||||
count: values.length,
|
||||
totalBytes: values.reduce((total, value) => total + value, 0),
|
||||
p95Bytes: percentile(values, 0.95),
|
||||
maxBytes: Math.max(...values),
|
||||
},
|
||||
])
|
||||
);
|
||||
};
|
||||
|
||||
const countRetries = (calls: readonly EvaluationToolCall[]) => {
|
||||
const failedNames = new Set<string>();
|
||||
let retries = 0;
|
||||
@@ -141,22 +256,52 @@ const countRetries = (calls: readonly EvaluationToolCall[]) => {
|
||||
return retries;
|
||||
};
|
||||
|
||||
const countValues = (values: readonly string[]) =>
|
||||
Object.fromEntries(
|
||||
[...new Set(values)]
|
||||
.sort((left, right) => left.localeCompare(right))
|
||||
.map((value) => [
|
||||
value,
|
||||
values.filter((candidate) => candidate === value).length,
|
||||
])
|
||||
);
|
||||
|
||||
export const getMcpEvaluationMetrics = (
|
||||
calls: readonly EvaluationToolCall[]
|
||||
): McpEvaluationMetrics => {
|
||||
const durations = calls.flatMap(({ durationMs }) =>
|
||||
durationMs === undefined ? [] : [durationMs]
|
||||
);
|
||||
const responseBytes = calls.flatMap(({ responseBytes }) =>
|
||||
responseBytes === undefined ? [] : [responseBytes]
|
||||
);
|
||||
const firstMutation = calls.find(
|
||||
(call) => call.committed === true && call.isError !== true
|
||||
);
|
||||
const firstVerification = calls.find(
|
||||
(call) => verificationNames.has(call.name) && call.isError !== true
|
||||
);
|
||||
const failedCalls = calls.filter((call) => call.isError === true);
|
||||
return {
|
||||
total: calls.length,
|
||||
succeeded: calls.filter((call) => call.isError !== true).length,
|
||||
failed: calls.filter((call) => call.isError === true).length,
|
||||
failed: failedCalls.length,
|
||||
failuresByTool: countValues(failedCalls.map((call) => call.name)),
|
||||
failuresByCode: countValues(
|
||||
failedCalls.map((call) => call.errorCode ?? "UNKNOWN")
|
||||
),
|
||||
issuesByCode: countValues(
|
||||
failedCalls.flatMap((call) =>
|
||||
(call.errorIssues ?? []).map((issue) => issue.code)
|
||||
)
|
||||
),
|
||||
issuesByPath: countValues(
|
||||
failedCalls.flatMap((call) =>
|
||||
(call.errorIssues ?? []).map((issue) => issue.path)
|
||||
)
|
||||
),
|
||||
durationsByTool: getDurationsByTool(calls),
|
||||
responseBytesByTool: getResponseBytesByTool(calls),
|
||||
retries: countRetries(calls),
|
||||
focusedReads: calls.filter((call) => isFocusedRead(call.name)).length,
|
||||
broadReads: calls.filter((call) => isBroadRead(call.name)).length,
|
||||
@@ -173,6 +318,14 @@ export const getMcpEvaluationMetrics = (
|
||||
totalDurationMs: durations.reduce((total, value) => total + value, 0),
|
||||
p50DurationMs: percentile(durations, 0.5),
|
||||
p95DurationMs: percentile(durations, 0.95),
|
||||
totalResponseBytes: responseBytes.reduce(
|
||||
(total, value) => total + value,
|
||||
0
|
||||
),
|
||||
p50ResponseBytes: percentile(responseBytes, 0.5),
|
||||
p95ResponseBytes: percentile(responseBytes, 0.95),
|
||||
maxResponseBytes:
|
||||
responseBytes.length === 0 ? 0 : Math.max(...responseBytes),
|
||||
...(firstMutation?.startedAtMs === undefined
|
||||
? {}
|
||||
: { timeToFirstMutationMs: firstMutation.startedAtMs }),
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
import { describe, expect, test } from "vitest";
|
||||
import type { AgentEvaluationResult } from "./agent-runner";
|
||||
import {
|
||||
isAggregateTokenBaselineNonRegressed,
|
||||
compareEvaluationResult,
|
||||
isEvaluationComparisonAccepted,
|
||||
shouldUpdateEvaluationBaselines,
|
||||
} from "./evaluation-regression";
|
||||
|
||||
const createResult = (
|
||||
@@ -22,6 +24,7 @@ const createResult = (
|
||||
tokens: {
|
||||
input: 800,
|
||||
cachedInput: 400,
|
||||
uncachedInput: 400,
|
||||
cacheWriteInput: 0,
|
||||
output: 200,
|
||||
reasoningOutput: 50,
|
||||
@@ -32,6 +35,12 @@ const createResult = (
|
||||
total: 10,
|
||||
succeeded: 10,
|
||||
failed: 0,
|
||||
failuresByTool: {},
|
||||
failuresByCode: {},
|
||||
issuesByCode: {},
|
||||
issuesByPath: {},
|
||||
durationsByTool: {},
|
||||
responseBytesByTool: {},
|
||||
retries: 0,
|
||||
focusedReads: 2,
|
||||
broadReads: 0,
|
||||
@@ -42,6 +51,10 @@ const createResult = (
|
||||
totalDurationMs: 500,
|
||||
p50DurationMs: 30,
|
||||
p95DurationMs: 100,
|
||||
totalResponseBytes: 10_000,
|
||||
p50ResponseBytes: 800,
|
||||
p95ResponseBytes: 2_000,
|
||||
maxResponseBytes: 3_000,
|
||||
timeToFirstMutationMs: 400,
|
||||
timeToFirstVerificationMs: 800,
|
||||
},
|
||||
@@ -62,7 +75,7 @@ describe("evaluation regression comparison", () => {
|
||||
metrics: {
|
||||
...baseline.metrics,
|
||||
durationMs: 1_100,
|
||||
tokens: { ...baselineTokens, total: 1_200 },
|
||||
tokens: { ...baselineTokens, total: 1_600 },
|
||||
},
|
||||
});
|
||||
const comparison = compareEvaluationResult(current, baseline);
|
||||
@@ -77,16 +90,23 @@ describe("evaluation regression comparison", () => {
|
||||
expect.objectContaining({
|
||||
metric: "metrics.tokens.total",
|
||||
baseline: 1_000,
|
||||
current: 1_200,
|
||||
current: 1_600,
|
||||
}),
|
||||
])
|
||||
);
|
||||
expect(comparison.regressions).toEqual([
|
||||
expect.objectContaining({
|
||||
metric: "metrics.tokens.total",
|
||||
allowed: 1_150,
|
||||
allowed: 1_500,
|
||||
}),
|
||||
]);
|
||||
|
||||
const noisyCurrent = structuredClone(baseline);
|
||||
noisyCurrent.metrics.tokens!.total = 1_400;
|
||||
expect(compareEvaluationResult(noisyCurrent, baseline)).toMatchObject({
|
||||
status: "passed",
|
||||
regressions: [],
|
||||
});
|
||||
});
|
||||
|
||||
test("blocks correctness regressions and skips incompatible baselines", () => {
|
||||
@@ -142,6 +162,27 @@ describe("evaluation regression comparison", () => {
|
||||
});
|
||||
});
|
||||
|
||||
test("reports environment-sensitive tool latency without blocking", () => {
|
||||
const baseline = createResult();
|
||||
const current = structuredClone(baseline);
|
||||
current.metrics.toolCalls.totalDurationMs = 20_000;
|
||||
current.metrics.toolCalls.p95DurationMs = 10_000;
|
||||
const comparison = compareEvaluationResult(current, baseline);
|
||||
expect(comparison).toMatchObject({ status: "passed", regressions: [] });
|
||||
expect(comparison.deltas).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
metric: "metrics.toolCalls.totalDurationMs",
|
||||
current: 20_000,
|
||||
}),
|
||||
expect.objectContaining({
|
||||
metric: "metrics.toolCalls.p95DurationMs",
|
||||
current: 10_000,
|
||||
}),
|
||||
])
|
||||
);
|
||||
});
|
||||
|
||||
test("reports stochastic milestone timing without blocking one run", () => {
|
||||
const baseline = createResult();
|
||||
const current = structuredClone(baseline);
|
||||
@@ -164,6 +205,71 @@ describe("evaluation regression comparison", () => {
|
||||
);
|
||||
});
|
||||
|
||||
test("blocks MCP catalog size regressions without gating client refreshes", () => {
|
||||
const baseline = createResult();
|
||||
baseline.metrics.mcpCatalog = {
|
||||
responses: 1,
|
||||
totalResponseBytes: 78_000,
|
||||
maxResponseBytes: 78_000,
|
||||
latestToolCount: 171,
|
||||
latestResponseBytes: 78_000,
|
||||
latestInputSchemaBytes: 41_000,
|
||||
latestDescriptionBytes: 13_000,
|
||||
};
|
||||
const current = structuredClone(baseline);
|
||||
current.metrics.mcpCatalog = {
|
||||
...baseline.metrics.mcpCatalog,
|
||||
responses: 2,
|
||||
latestResponseBytes: 90_000,
|
||||
latestInputSchemaBytes: 48_000,
|
||||
};
|
||||
const comparison = compareEvaluationResult(current, baseline);
|
||||
expect(comparison.regressions).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
metric: "metrics.mcpCatalog.latestResponseBytes",
|
||||
}),
|
||||
expect.objectContaining({
|
||||
metric: "metrics.mcpCatalog.latestInputSchemaBytes",
|
||||
}),
|
||||
])
|
||||
);
|
||||
expect(comparison.regressions).not.toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({ metric: "metrics.mcpCatalog.responses" }),
|
||||
])
|
||||
);
|
||||
expect(comparison.deltas).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({ metric: "metrics.mcpCatalog.responses" }),
|
||||
])
|
||||
);
|
||||
});
|
||||
|
||||
test("blocks aggregate MCP response growth and reports tail changes", () => {
|
||||
const baseline = createResult();
|
||||
const current = structuredClone(baseline);
|
||||
current.metrics.toolCalls.totalResponseBytes = 12_000;
|
||||
current.metrics.toolCalls.p95ResponseBytes = 2_400;
|
||||
expect(compareEvaluationResult(current, baseline).regressions).toEqual([
|
||||
{
|
||||
metric: "metrics.toolCalls.totalResponseBytes",
|
||||
baseline: 10_000,
|
||||
current: 12_000,
|
||||
allowed: 11_500,
|
||||
},
|
||||
]);
|
||||
expect(compareEvaluationResult(current, baseline).deltas).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
metric: "metrics.toolCalls.p95ResponseBytes",
|
||||
baseline: 2_000,
|
||||
current: 2_400,
|
||||
}),
|
||||
])
|
||||
);
|
||||
});
|
||||
|
||||
test("requires accepted baselines and never accepts metric regressions", () => {
|
||||
const passed = compareEvaluationResult(createResult(), createResult());
|
||||
const missing = compareEvaluationResult(createResult(), undefined);
|
||||
@@ -205,4 +311,84 @@ describe("evaluation regression comparison", () => {
|
||||
})
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
test("updates baselines only after outcomes and comparisons pass", () => {
|
||||
expect(
|
||||
shouldUpdateEvaluationBaselines({
|
||||
updateBaselines: true,
|
||||
evaluationsPassed: true,
|
||||
comparisonsPassed: true,
|
||||
aggregateTokensPassed: true,
|
||||
})
|
||||
).toBe(true);
|
||||
expect(
|
||||
shouldUpdateEvaluationBaselines({
|
||||
updateBaselines: true,
|
||||
evaluationsPassed: true,
|
||||
comparisonsPassed: false,
|
||||
aggregateTokensPassed: true,
|
||||
})
|
||||
).toBe(false);
|
||||
expect(
|
||||
shouldUpdateEvaluationBaselines({
|
||||
updateBaselines: true,
|
||||
evaluationsPassed: false,
|
||||
comparisonsPassed: true,
|
||||
aggregateTokensPassed: true,
|
||||
})
|
||||
).toBe(false);
|
||||
expect(
|
||||
shouldUpdateEvaluationBaselines({
|
||||
updateBaselines: true,
|
||||
evaluationsPassed: true,
|
||||
comparisonsPassed: true,
|
||||
aggregateTokensPassed: false,
|
||||
})
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
test("does not ratchet an existing aggregate token baseline upward", () => {
|
||||
const withTokens = (
|
||||
fixtureId: AgentEvaluationResult["fixtureId"],
|
||||
total: number
|
||||
) => {
|
||||
const result = createResult({ fixtureId });
|
||||
return {
|
||||
...result,
|
||||
metrics: {
|
||||
...result.metrics,
|
||||
tokens: { ...result.metrics.tokens!, total },
|
||||
},
|
||||
};
|
||||
};
|
||||
const baselines = [
|
||||
withTokens("authenticated-page-v1", 1_000),
|
||||
withTokens("design-input-v1", 1_000),
|
||||
];
|
||||
|
||||
expect(
|
||||
isAggregateTokenBaselineNonRegressed(
|
||||
[
|
||||
withTokens("authenticated-page-v1", 900),
|
||||
withTokens("design-input-v1", 1_000),
|
||||
],
|
||||
baselines
|
||||
)
|
||||
).toBe(true);
|
||||
expect(
|
||||
isAggregateTokenBaselineNonRegressed(
|
||||
[
|
||||
withTokens("authenticated-page-v1", 1_001),
|
||||
withTokens("design-input-v1", 1_000),
|
||||
],
|
||||
baselines
|
||||
)
|
||||
).toBe(false);
|
||||
expect(
|
||||
isAggregateTokenBaselineNonRegressed(
|
||||
[withTokens("authenticated-page-v1", 1_000)],
|
||||
[]
|
||||
)
|
||||
).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -19,11 +19,13 @@ export type EvaluationComparison = {
|
||||
status: "passed" | "regressed" | "missing" | "incompatible";
|
||||
baselineCommandSha256?: string;
|
||||
maxMetricRegressionPercent: number;
|
||||
maxTokenRegressionPercent: number;
|
||||
deltas: EvaluationMetricDelta[];
|
||||
regressions: EvaluationRegression[];
|
||||
};
|
||||
|
||||
export const maxMetricRegressionPercent = 15;
|
||||
export const maxTokenRegressionPercent = 50;
|
||||
|
||||
export const isEvaluationComparisonAccepted = ({
|
||||
comparison,
|
||||
@@ -43,20 +45,59 @@ export const isEvaluationComparisonAccepted = ({
|
||||
return updateBaselines || requireBaseline === false;
|
||||
};
|
||||
|
||||
export const shouldUpdateEvaluationBaselines = ({
|
||||
updateBaselines,
|
||||
evaluationsPassed,
|
||||
comparisonsPassed,
|
||||
aggregateTokensPassed,
|
||||
}: {
|
||||
updateBaselines: boolean;
|
||||
evaluationsPassed: boolean;
|
||||
comparisonsPassed: boolean;
|
||||
aggregateTokensPassed: boolean;
|
||||
}) =>
|
||||
updateBaselines &&
|
||||
evaluationsPassed &&
|
||||
comparisonsPassed &&
|
||||
aggregateTokensPassed;
|
||||
|
||||
export const isAggregateTokenBaselineNonRegressed = (
|
||||
current: readonly AgentEvaluationResult[],
|
||||
baselines: readonly AgentEvaluationResult[]
|
||||
) => {
|
||||
const baselineByFixture = new Map(
|
||||
baselines.map((baseline) => [baseline.fixtureId, baseline])
|
||||
);
|
||||
let currentTotal = 0;
|
||||
let baselineTotal = 0;
|
||||
let compared = 0;
|
||||
for (const result of current) {
|
||||
const baseline = baselineByFixture.get(result.fixtureId);
|
||||
if (baseline === undefined) {
|
||||
continue;
|
||||
}
|
||||
const currentTokens = result.metrics.tokens?.total;
|
||||
const baselineTokens = baseline.metrics.tokens?.total;
|
||||
if (currentTokens === undefined || baselineTokens === undefined) {
|
||||
return false;
|
||||
}
|
||||
currentTotal += currentTokens;
|
||||
baselineTotal += baselineTokens;
|
||||
compared += 1;
|
||||
}
|
||||
return compared === 0 || currentTotal <= baselineTotal;
|
||||
};
|
||||
|
||||
const gatedMetricPaths = new Set([
|
||||
"metrics.tokens.total",
|
||||
"metrics.mcpCatalog.latestResponseBytes",
|
||||
"metrics.mcpCatalog.latestInputSchemaBytes",
|
||||
"metrics.toolCalls.total",
|
||||
"metrics.toolCalls.failed",
|
||||
"metrics.toolCalls.retries",
|
||||
"metrics.toolCalls.focusedReads",
|
||||
"metrics.toolCalls.broadReads",
|
||||
"metrics.toolCalls.totalDurationMs",
|
||||
"metrics.toolCalls.p95DurationMs",
|
||||
]);
|
||||
|
||||
const minimumAbsoluteIncrease = new Map([
|
||||
["metrics.toolCalls.totalDurationMs", 100],
|
||||
["metrics.toolCalls.p95DurationMs", 100],
|
||||
"metrics.toolCalls.totalResponseBytes",
|
||||
]);
|
||||
|
||||
const getNumericMetrics = (
|
||||
@@ -93,6 +134,7 @@ export const compareEvaluationResult = (
|
||||
): EvaluationComparison => {
|
||||
const empty = {
|
||||
maxMetricRegressionPercent,
|
||||
maxTokenRegressionPercent,
|
||||
deltas: [],
|
||||
regressions: [],
|
||||
};
|
||||
@@ -128,13 +170,15 @@ export const compareEvaluationResult = (
|
||||
if (gatedMetricPaths.has(metric) === false) {
|
||||
continue;
|
||||
}
|
||||
const regressionPercent =
|
||||
metric === "metrics.tokens.total"
|
||||
? maxTokenRegressionPercent
|
||||
: maxMetricRegressionPercent;
|
||||
const relativeIncrease = Math.floor(
|
||||
baselineValue * (maxMetricRegressionPercent / 100) +
|
||||
baselineValue * (regressionPercent / 100) +
|
||||
Number.EPSILON * Math.max(1, baselineValue)
|
||||
);
|
||||
const allowed =
|
||||
baselineValue +
|
||||
Math.max(relativeIncrease, minimumAbsoluteIncrease.get(metric) ?? 0);
|
||||
const allowed = baselineValue + relativeIncrease;
|
||||
if (currentValue > allowed) {
|
||||
regressions.push({
|
||||
metric,
|
||||
@@ -168,6 +212,7 @@ export const compareEvaluationResult = (
|
||||
status: regressions.length === 0 ? "passed" : "regressed",
|
||||
baselineCommandSha256: baseline.commandSha256,
|
||||
maxMetricRegressionPercent,
|
||||
maxTokenRegressionPercent,
|
||||
deltas,
|
||||
regressions,
|
||||
};
|
||||
|
||||
@@ -141,7 +141,7 @@ export const designInputFixture: HighImpactFixture & {
|
||||
} = {
|
||||
id: "design-input-v1",
|
||||
objective:
|
||||
"Build an editable /summer page from the supplied desktop and mobile design reference. Preserve and reuse the existing design system, implement responsive behavior with the project's breakpoints, then run rendered audit and inspect desktop/mobile screenshots.",
|
||||
"Build an editable /summer page from the supplied desktop and mobile design reference. Preserve and reuse the existing design system, implement responsive behavior with the project's breakpoints, inspect desktop/mobile screenshots, then run a static route audit without duplicating the rendered captures.",
|
||||
project: {
|
||||
...emptyProject(),
|
||||
styleSources: [
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { describe, expect, test } from "vitest";
|
||||
import {
|
||||
getMcpCatalogObservation,
|
||||
getMcpMutationToolNames,
|
||||
getMcpToolsListRequestId,
|
||||
getMcpTraceRequest,
|
||||
@@ -7,6 +8,37 @@ import {
|
||||
} from "./mcp-trace-proxy";
|
||||
|
||||
describe("bounded MCP tracing", () => {
|
||||
test("measures the bounded tools catalog payload", () => {
|
||||
expect(
|
||||
getMcpCatalogObservation({
|
||||
id: 1,
|
||||
result: {
|
||||
tools: [
|
||||
{
|
||||
name: "list-pages",
|
||||
description: "List pages",
|
||||
inputSchema: { type: "object", properties: {} },
|
||||
},
|
||||
{
|
||||
name: "create-page",
|
||||
description: "Create page",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: { path: { type: "string" } },
|
||||
},
|
||||
},
|
||||
],
|
||||
},
|
||||
})
|
||||
).toEqual({
|
||||
kind: "tools-list",
|
||||
toolCount: 2,
|
||||
responseBytes: 231,
|
||||
inputSchemaBytes: 90,
|
||||
descriptionBytes: 21,
|
||||
});
|
||||
});
|
||||
|
||||
test("retains only bounded verification fields", () => {
|
||||
expect(
|
||||
getMcpTraceRequest(
|
||||
@@ -35,6 +67,34 @@ describe("bounded MCP tracing", () => {
|
||||
});
|
||||
});
|
||||
|
||||
test("retains responsive verification viewports as bounded evidence", () => {
|
||||
expect(
|
||||
getMcpTraceRequest({
|
||||
id: 9,
|
||||
method: "tools/call",
|
||||
params: {
|
||||
name: "verify-page-responsive",
|
||||
arguments: {
|
||||
path: "/account",
|
||||
viewports: [
|
||||
{ width: 1440, height: 900 },
|
||||
{ width: 390, height: 844 },
|
||||
],
|
||||
},
|
||||
},
|
||||
})
|
||||
).toMatchObject({
|
||||
call: {
|
||||
arguments: {
|
||||
viewports: [
|
||||
{ width: 1440, height: 900 },
|
||||
{ width: 390, height: 844 },
|
||||
],
|
||||
},
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
test("records confirmation flow without retaining its token", () => {
|
||||
expect(
|
||||
getMcpTraceRequest(
|
||||
@@ -73,33 +133,56 @@ describe("bounded MCP tracing", () => {
|
||||
[2, { name: "verify-bindings", startedAtMs: 200 }],
|
||||
]);
|
||||
|
||||
expect(
|
||||
getMcpTraceResponse(
|
||||
{
|
||||
id: 1,
|
||||
result: {
|
||||
structuredContent: {
|
||||
meta: { session: { committed: true } },
|
||||
},
|
||||
},
|
||||
const committedResponse = {
|
||||
id: 1,
|
||||
result: {
|
||||
structuredContent: {
|
||||
meta: { session: { committed: true } },
|
||||
},
|
||||
pending,
|
||||
175
|
||||
)
|
||||
).toEqual({
|
||||
},
|
||||
};
|
||||
expect(getMcpTraceResponse(committedResponse, pending, 175)).toEqual({
|
||||
name: "create-page",
|
||||
startedAtMs: 100,
|
||||
mutation: true,
|
||||
durationMs: 75,
|
||||
responseBytes: Buffer.byteLength(JSON.stringify(committedResponse)),
|
||||
committed: true,
|
||||
});
|
||||
expect(
|
||||
getMcpTraceResponse({ id: 2, result: { isError: true } }, pending, 260)
|
||||
).toEqual({
|
||||
const failedResponse = {
|
||||
id: 2,
|
||||
result: {
|
||||
isError: true,
|
||||
structuredContent: {
|
||||
error: {
|
||||
code: "INVALID_INPUT",
|
||||
message: "private diagnostic message",
|
||||
details: { token: "private-token" },
|
||||
issues: [
|
||||
{
|
||||
code: "invalid_type",
|
||||
path: ["resource", "headers", 0, "value"],
|
||||
message: "private issue message",
|
||||
example: "private example",
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
expect(getMcpTraceResponse(failedResponse, pending, 260)).toEqual({
|
||||
name: "verify-bindings",
|
||||
startedAtMs: 200,
|
||||
durationMs: 60,
|
||||
responseBytes: Buffer.byteLength(JSON.stringify(failedResponse)),
|
||||
isError: true,
|
||||
errorCode: "INVALID_INPUT",
|
||||
errorIssues: [
|
||||
{
|
||||
code: "invalid_type",
|
||||
path: "resource.headers.0.value",
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(pending.size).toBe(0);
|
||||
});
|
||||
@@ -117,13 +200,22 @@ describe("bounded MCP tracing", () => {
|
||||
},
|
||||
{
|
||||
name: "create-page",
|
||||
annotations: { readOnlyHint: false, private: "discard" },
|
||||
annotations: {
|
||||
destructiveHint: false,
|
||||
openWorldHint: false,
|
||||
private: "discard",
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "delete-external-resource",
|
||||
},
|
||||
],
|
||||
},
|
||||
})
|
||||
);
|
||||
expect(mutationToolNames).toEqual(new Set(["create-page"]));
|
||||
expect(mutationToolNames).toEqual(
|
||||
new Set(["create-page", "delete-external-resource"])
|
||||
);
|
||||
expect(
|
||||
getMcpTraceRequest(
|
||||
{
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import { appendFileSync } from "node:fs";
|
||||
import { spawn } from "node:child_process";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { isPlainRecord } from "../../src/type-utils";
|
||||
import { boundedIdentifierPattern, isPlainRecord } from "../../src/type-utils";
|
||||
import type { McpCatalogObservation } from "./evaluation-metrics";
|
||||
|
||||
type JsonRpcId = string | number;
|
||||
|
||||
@@ -9,16 +10,78 @@ export type BoundedMcpCall = {
|
||||
name: string;
|
||||
arguments?: {
|
||||
viewport?: { width: number; height: number };
|
||||
viewports?: Array<{ width: number; height: number }>;
|
||||
dryRun?: true;
|
||||
confirmDestructive?: true;
|
||||
hasConfirmationToken?: true;
|
||||
};
|
||||
startedAtMs: number;
|
||||
durationMs?: number;
|
||||
responseBytes?: number;
|
||||
mutation?: true;
|
||||
planned?: true;
|
||||
committed?: true;
|
||||
isError?: true;
|
||||
errorCode?: string;
|
||||
errorIssues?: Array<{ code: string; path: string }>;
|
||||
};
|
||||
|
||||
const getStructuredError = (result: unknown) => {
|
||||
const structuredContent =
|
||||
isPlainRecord(result) && isPlainRecord(result.structuredContent)
|
||||
? result.structuredContent
|
||||
: undefined;
|
||||
return structuredContent !== undefined &&
|
||||
isPlainRecord(structuredContent.error)
|
||||
? structuredContent.error
|
||||
: undefined;
|
||||
};
|
||||
|
||||
const getBoundedErrorCode = (value: unknown, result: unknown) => {
|
||||
const error = getStructuredError(result);
|
||||
if (
|
||||
typeof error?.code === "string" &&
|
||||
boundedIdentifierPattern.test(error.code)
|
||||
) {
|
||||
return error.code;
|
||||
}
|
||||
if (isPlainRecord(value) && isPlainRecord(value.error)) {
|
||||
const code = value.error.code;
|
||||
if (typeof code === "number" && Number.isSafeInteger(code)) {
|
||||
return `JSON_RPC_${code}`;
|
||||
}
|
||||
if (typeof code === "string" && boundedIdentifierPattern.test(code)) {
|
||||
return code;
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
const getBoundedErrorIssues = (result: unknown) => {
|
||||
const issues = getStructuredError(result)?.issues;
|
||||
if (Array.isArray(issues) === false) {
|
||||
return;
|
||||
}
|
||||
const boundedIssues = issues.slice(0, 8).flatMap((issue) => {
|
||||
if (
|
||||
isPlainRecord(issue) === false ||
|
||||
typeof issue.code !== "string" ||
|
||||
boundedIdentifierPattern.test(issue.code) === false ||
|
||||
Array.isArray(issue.path) === false ||
|
||||
issue.path.length > 8
|
||||
) {
|
||||
return [];
|
||||
}
|
||||
const segments = issue.path.map(String);
|
||||
if (
|
||||
segments.some(
|
||||
(segment) => boundedIdentifierPattern.test(segment) === false
|
||||
)
|
||||
) {
|
||||
return [];
|
||||
}
|
||||
return [{ code: issue.code, path: segments.join(".") }];
|
||||
});
|
||||
return boundedIssues.length === 0 ? undefined : boundedIssues;
|
||||
};
|
||||
|
||||
export const getMcpTraceRequest = (
|
||||
@@ -54,6 +117,25 @@ export const getMcpTraceRequest = (
|
||||
) {
|
||||
args.viewport = { width: viewport.width, height: viewport.height };
|
||||
}
|
||||
const viewports = params.arguments.viewports;
|
||||
if (
|
||||
(params.name === "screenshot.responsive" ||
|
||||
params.name === "verify-page-responsive") &&
|
||||
Array.isArray(viewports) &&
|
||||
viewports.length > 0 &&
|
||||
viewports.length <= 8 &&
|
||||
viewports.every(
|
||||
(item) =>
|
||||
isPlainRecord(item) &&
|
||||
typeof item.width === "number" &&
|
||||
typeof item.height === "number"
|
||||
)
|
||||
) {
|
||||
args.viewports = viewports.map((item) => ({
|
||||
width: item.width as number,
|
||||
height: item.height as number,
|
||||
}));
|
||||
}
|
||||
if (params.arguments.dryRun === true) {
|
||||
args.dryRun = true;
|
||||
}
|
||||
@@ -93,6 +175,10 @@ export const getMcpTraceResponse = (
|
||||
(isPlainRecord(result) && result.isError === true)
|
||||
? (true as const)
|
||||
: undefined;
|
||||
const errorCode =
|
||||
isError === true ? getBoundedErrorCode(value, result) : undefined;
|
||||
const errorIssues =
|
||||
isError === true ? getBoundedErrorIssues(result) : undefined;
|
||||
const structuredContent =
|
||||
isPlainRecord(result) && isPlainRecord(result.structuredContent)
|
||||
? result.structuredContent
|
||||
@@ -118,9 +204,12 @@ export const getMcpTraceResponse = (
|
||||
return {
|
||||
...call,
|
||||
durationMs: Math.max(0, Math.round(completedAtMs - call.startedAtMs)),
|
||||
responseBytes: getJsonBytes(value),
|
||||
...(planned === undefined ? {} : { planned }),
|
||||
...(committed === undefined ? {} : { committed }),
|
||||
...(isError === undefined ? {} : { isError }),
|
||||
...(errorCode === undefined ? {} : { errorCode }),
|
||||
...(errorIssues === undefined ? {} : { errorIssues }),
|
||||
};
|
||||
};
|
||||
|
||||
@@ -145,8 +234,8 @@ export const getMcpMutationToolNames = (value: unknown) => {
|
||||
if (
|
||||
isPlainRecord(tool) &&
|
||||
typeof tool.name === "string" &&
|
||||
isPlainRecord(tool.annotations) &&
|
||||
tool.annotations.readOnlyHint === false
|
||||
(isPlainRecord(tool.annotations) === false ||
|
||||
tool.annotations.readOnlyHint !== true)
|
||||
) {
|
||||
return [tool.name];
|
||||
}
|
||||
@@ -154,6 +243,43 @@ export const getMcpMutationToolNames = (value: unknown) => {
|
||||
});
|
||||
};
|
||||
|
||||
const getJsonBytes = (value: unknown) =>
|
||||
Buffer.byteLength(JSON.stringify(value), "utf8");
|
||||
|
||||
export const getMcpCatalogObservation = (
|
||||
value: unknown
|
||||
): McpCatalogObservation | undefined => {
|
||||
if (
|
||||
isPlainRecord(value) === false ||
|
||||
isPlainRecord(value.result) === false ||
|
||||
Array.isArray(value.result.tools) === false
|
||||
) {
|
||||
return;
|
||||
}
|
||||
const tools = value.result.tools;
|
||||
return {
|
||||
kind: "tools-list",
|
||||
toolCount: tools.length,
|
||||
responseBytes: getJsonBytes(value.result),
|
||||
inputSchemaBytes: tools.reduce(
|
||||
(total, tool) =>
|
||||
total +
|
||||
(isPlainRecord(tool) && tool.inputSchema !== undefined
|
||||
? getJsonBytes(tool.inputSchema)
|
||||
: 0),
|
||||
0
|
||||
),
|
||||
descriptionBytes: tools.reduce(
|
||||
(total, tool) =>
|
||||
total +
|
||||
(isPlainRecord(tool) && typeof tool.description === "string"
|
||||
? Buffer.byteLength(tool.description, "utf8")
|
||||
: 0),
|
||||
0
|
||||
),
|
||||
};
|
||||
};
|
||||
|
||||
const observeJsonLines = (
|
||||
stream: NodeJS.ReadableStream,
|
||||
visit: (value: unknown) => void
|
||||
@@ -204,6 +330,10 @@ const run = () => {
|
||||
(typeof value.id === "string" || typeof value.id === "number") &&
|
||||
pendingToolsLists.delete(value.id)
|
||||
) {
|
||||
const catalog = getMcpCatalogObservation(value);
|
||||
if (catalog !== undefined) {
|
||||
appendFileSync(tracePath, `${JSON.stringify(catalog)}\n`, "utf8");
|
||||
}
|
||||
for (const name of getMcpMutationToolNames(value) ?? []) {
|
||||
mutationToolNames.add(name);
|
||||
}
|
||||
|
||||
@@ -6,100 +6,164 @@
|
||||
"cli": "source",
|
||||
"provider": "openai",
|
||||
"model": "gpt-5.6-sol",
|
||||
"commandSha256": "79baa45fea9cee997dfac56c6d33eaf4400efe5e5155db9934c4c82c9df41c83",
|
||||
"commandSha256": "f3973839321d489a88526e5e1751d54aa9391af61107e56165cd35520c60a96a",
|
||||
"exitCode": 0,
|
||||
"metrics": {
|
||||
"durationMs": 529970,
|
||||
"durationMs": 94027,
|
||||
"tokens": {
|
||||
"input": 6976750,
|
||||
"cachedInput": 6771712,
|
||||
"input": 319553,
|
||||
"cachedInput": 275200,
|
||||
"uncachedInput": 44353,
|
||||
"cacheWriteInput": 0,
|
||||
"output": 17054,
|
||||
"reasoningOutput": 4582,
|
||||
"total": 6993804,
|
||||
"cachedInputRate": 0.9706
|
||||
"output": 1843,
|
||||
"reasoningOutput": 505,
|
||||
"total": 321396,
|
||||
"cachedInputRate": 0.8612
|
||||
},
|
||||
"mcpCatalog": {
|
||||
"responses": 1,
|
||||
"totalResponseBytes": 51362,
|
||||
"maxResponseBytes": 51362,
|
||||
"latestToolCount": 176,
|
||||
"latestResponseBytes": 51362,
|
||||
"latestInputSchemaBytes": 28964,
|
||||
"latestDescriptionBytes": 4556
|
||||
},
|
||||
"toolCalls": {
|
||||
"total": 62,
|
||||
"succeeded": 51,
|
||||
"failed": 11,
|
||||
"retries": 10,
|
||||
"focusedReads": 8,
|
||||
"total": 8,
|
||||
"succeeded": 8,
|
||||
"failed": 0,
|
||||
"failuresByTool": {},
|
||||
"failuresByCode": {},
|
||||
"issuesByCode": {},
|
||||
"issuesByPath": {},
|
||||
"durationsByTool": {
|
||||
"audit": {
|
||||
"count": 1,
|
||||
"totalMs": 10,
|
||||
"p95Ms": 10,
|
||||
"maxMs": 10
|
||||
},
|
||||
"create-page": {
|
||||
"count": 1,
|
||||
"totalMs": 29,
|
||||
"p95Ms": 29,
|
||||
"maxMs": 29
|
||||
},
|
||||
"create-resource": {
|
||||
"count": 1,
|
||||
"totalMs": 20,
|
||||
"p95Ms": 20,
|
||||
"maxMs": 20
|
||||
},
|
||||
"create-variable": {
|
||||
"count": 1,
|
||||
"totalMs": 24,
|
||||
"p95Ms": 24,
|
||||
"maxMs": 24
|
||||
},
|
||||
"insert-fragment-verified": {
|
||||
"count": 1,
|
||||
"totalMs": 68,
|
||||
"p95Ms": 68,
|
||||
"maxMs": 68
|
||||
},
|
||||
"inspect-auth-context": {
|
||||
"count": 1,
|
||||
"totalMs": 22,
|
||||
"p95Ms": 22,
|
||||
"maxMs": 22
|
||||
},
|
||||
"meta.guide": {
|
||||
"count": 1,
|
||||
"totalMs": 10,
|
||||
"p95Ms": 10,
|
||||
"maxMs": 10
|
||||
},
|
||||
"screenshot.responsive": {
|
||||
"count": 1,
|
||||
"totalMs": 9439,
|
||||
"p95Ms": 9439,
|
||||
"maxMs": 9439
|
||||
}
|
||||
},
|
||||
"responseBytesByTool": {
|
||||
"audit": {
|
||||
"count": 1,
|
||||
"totalBytes": 4762,
|
||||
"p95Bytes": 4762,
|
||||
"maxBytes": 4762
|
||||
},
|
||||
"create-page": {
|
||||
"count": 1,
|
||||
"totalBytes": 1342,
|
||||
"p95Bytes": 1342,
|
||||
"maxBytes": 1342
|
||||
},
|
||||
"create-resource": {
|
||||
"count": 1,
|
||||
"totalBytes": 1644,
|
||||
"p95Bytes": 1644,
|
||||
"maxBytes": 1644
|
||||
},
|
||||
"create-variable": {
|
||||
"count": 1,
|
||||
"totalBytes": 1554,
|
||||
"p95Bytes": 1554,
|
||||
"maxBytes": 1554
|
||||
},
|
||||
"insert-fragment-verified": {
|
||||
"count": 1,
|
||||
"totalBytes": 2662,
|
||||
"p95Bytes": 2662,
|
||||
"maxBytes": 2662
|
||||
},
|
||||
"inspect-auth-context": {
|
||||
"count": 1,
|
||||
"totalBytes": 2576,
|
||||
"p95Bytes": 2576,
|
||||
"maxBytes": 2576
|
||||
},
|
||||
"meta.guide": {
|
||||
"count": 1,
|
||||
"totalBytes": 15932,
|
||||
"p95Bytes": 15932,
|
||||
"maxBytes": 15932
|
||||
},
|
||||
"screenshot.responsive": {
|
||||
"count": 1,
|
||||
"totalBytes": 7064,
|
||||
"p95Bytes": 7064,
|
||||
"maxBytes": 7064
|
||||
}
|
||||
},
|
||||
"retries": 0,
|
||||
"focusedReads": 1,
|
||||
"broadReads": 0,
|
||||
"dryRuns": 12,
|
||||
"plannedMutations": 7,
|
||||
"committedMutations": 17,
|
||||
"verifications": 10,
|
||||
"totalDurationMs": 165266,
|
||||
"p50DurationMs": 20,
|
||||
"p95DurationMs": 27606,
|
||||
"timeToFirstMutationMs": 69436,
|
||||
"timeToFirstVerificationMs": 177942
|
||||
"dryRuns": 0,
|
||||
"plannedMutations": 0,
|
||||
"committedMutations": 4,
|
||||
"verifications": 3,
|
||||
"totalDurationMs": 9622,
|
||||
"p50DurationMs": 22,
|
||||
"p95DurationMs": 9439,
|
||||
"totalResponseBytes": 37536,
|
||||
"p50ResponseBytes": 2576,
|
||||
"p95ResponseBytes": 15932,
|
||||
"maxResponseBytes": 15932,
|
||||
"timeToFirstMutationMs": 39076,
|
||||
"timeToFirstVerificationMs": 67660
|
||||
}
|
||||
},
|
||||
"callSequence": [
|
||||
"meta.guide",
|
||||
"meta.index",
|
||||
"list-pages",
|
||||
"list-resources",
|
||||
"list-variables",
|
||||
"list-instances",
|
||||
"get-project-settings",
|
||||
"meta.get_more_tools",
|
||||
"search-project",
|
||||
"meta.get_more_tools",
|
||||
"create-page",
|
||||
"inspect-auth-context",
|
||||
"create-page",
|
||||
"create-variable",
|
||||
"create-resource",
|
||||
"create-resource",
|
||||
"create-variable",
|
||||
"create-variable",
|
||||
"create-variable",
|
||||
"insert-fragment",
|
||||
"insert-fragment",
|
||||
"verify-bindings",
|
||||
"list-breakpoints",
|
||||
"update-styles",
|
||||
"update-styles",
|
||||
"preview.start",
|
||||
"preview.start",
|
||||
"screenshot",
|
||||
"wrap-instance",
|
||||
"wrap-instance",
|
||||
"wrap-instance",
|
||||
"wrap-instance",
|
||||
"wrap-instance",
|
||||
"wrap-instance",
|
||||
"wrap-instance",
|
||||
"wrap-instance",
|
||||
"list-instances",
|
||||
"wrap-instance",
|
||||
"bind-props",
|
||||
"bind-props",
|
||||
"verify-bindings",
|
||||
"screenshot",
|
||||
"update-resource",
|
||||
"update-resource",
|
||||
"preview.start",
|
||||
"screenshot",
|
||||
"screenshot",
|
||||
"update-variable",
|
||||
"screenshot",
|
||||
"update-resource",
|
||||
"update-resource",
|
||||
"update-resource",
|
||||
"preview.start",
|
||||
"preview.stop",
|
||||
"preview.start",
|
||||
"screenshot",
|
||||
"update-variable",
|
||||
"screenshot",
|
||||
"update-variable",
|
||||
"screenshot",
|
||||
"update-variable",
|
||||
"verify-bindings",
|
||||
"audit",
|
||||
"preview.stop"
|
||||
"insert-fragment-verified",
|
||||
"screenshot.responsive",
|
||||
"audit"
|
||||
],
|
||||
"checks": {
|
||||
"guidance": "passed",
|
||||
|
||||
@@ -6,60 +6,165 @@
|
||||
"cli": "source",
|
||||
"provider": "openai",
|
||||
"model": "gpt-5.6-sol",
|
||||
"commandSha256": "b67b2dd38d840a331cf71a72ff8d6417b26cec0d48f1ec48bded91a00d7322e3",
|
||||
"commandSha256": "665472955db94c5d88c26b9c858eca2ec7621f04efa461fdb4cff3ae79dc5c5b",
|
||||
"exitCode": 0,
|
||||
"metrics": {
|
||||
"durationMs": 257624,
|
||||
"durationMs": 97892,
|
||||
"tokens": {
|
||||
"input": 1912580,
|
||||
"cachedInput": 1814272,
|
||||
"input": 327593,
|
||||
"cachedInput": 273152,
|
||||
"uncachedInput": 54441,
|
||||
"cacheWriteInput": 0,
|
||||
"output": 12382,
|
||||
"reasoningOutput": 1435,
|
||||
"total": 1924962,
|
||||
"cachedInputRate": 0.9486
|
||||
"output": 2987,
|
||||
"reasoningOutput": 698,
|
||||
"total": 330580,
|
||||
"cachedInputRate": 0.8338
|
||||
},
|
||||
"mcpCatalog": {
|
||||
"responses": 1,
|
||||
"totalResponseBytes": 51362,
|
||||
"maxResponseBytes": 51362,
|
||||
"latestToolCount": 176,
|
||||
"latestResponseBytes": 51362,
|
||||
"latestInputSchemaBytes": 28964,
|
||||
"latestDescriptionBytes": 4556
|
||||
},
|
||||
"toolCalls": {
|
||||
"total": 23,
|
||||
"succeeded": 21,
|
||||
"failed": 2,
|
||||
"retries": 2,
|
||||
"focusedReads": 7,
|
||||
"total": 10,
|
||||
"succeeded": 10,
|
||||
"failed": 0,
|
||||
"failuresByTool": {},
|
||||
"failuresByCode": {},
|
||||
"issuesByCode": {},
|
||||
"issuesByPath": {},
|
||||
"durationsByTool": {
|
||||
"attach-design-token": {
|
||||
"count": 3,
|
||||
"totalMs": 24,
|
||||
"p95Ms": 9,
|
||||
"maxMs": 9
|
||||
},
|
||||
"audit": {
|
||||
"count": 1,
|
||||
"totalMs": 12,
|
||||
"p95Ms": 12,
|
||||
"maxMs": 12
|
||||
},
|
||||
"create-page": {
|
||||
"count": 1,
|
||||
"totalMs": 91,
|
||||
"p95Ms": 91,
|
||||
"maxMs": 91
|
||||
},
|
||||
"insert-fragment-verified": {
|
||||
"count": 1,
|
||||
"totalMs": 55,
|
||||
"p95Ms": 55,
|
||||
"maxMs": 55
|
||||
},
|
||||
"inspect-design-context": {
|
||||
"count": 1,
|
||||
"totalMs": 30,
|
||||
"p95Ms": 30,
|
||||
"maxMs": 30
|
||||
},
|
||||
"meta.guide": {
|
||||
"count": 1,
|
||||
"totalMs": 11,
|
||||
"p95Ms": 11,
|
||||
"maxMs": 11
|
||||
},
|
||||
"screenshot.responsive": {
|
||||
"count": 1,
|
||||
"totalMs": 10751,
|
||||
"p95Ms": 10751,
|
||||
"maxMs": 10751
|
||||
},
|
||||
"update-styles": {
|
||||
"count": 1,
|
||||
"totalMs": 64,
|
||||
"p95Ms": 64,
|
||||
"maxMs": 64
|
||||
}
|
||||
},
|
||||
"responseBytesByTool": {
|
||||
"attach-design-token": {
|
||||
"count": 3,
|
||||
"totalBytes": 3900,
|
||||
"p95Bytes": 1304,
|
||||
"maxBytes": 1304
|
||||
},
|
||||
"audit": {
|
||||
"count": 1,
|
||||
"totalBytes": 3711,
|
||||
"p95Bytes": 3711,
|
||||
"maxBytes": 3711
|
||||
},
|
||||
"create-page": {
|
||||
"count": 1,
|
||||
"totalBytes": 1342,
|
||||
"p95Bytes": 1342,
|
||||
"maxBytes": 1342
|
||||
},
|
||||
"insert-fragment-verified": {
|
||||
"count": 1,
|
||||
"totalBytes": 2954,
|
||||
"p95Bytes": 2954,
|
||||
"maxBytes": 2954
|
||||
},
|
||||
"inspect-design-context": {
|
||||
"count": 1,
|
||||
"totalBytes": 3082,
|
||||
"p95Bytes": 3082,
|
||||
"maxBytes": 3082
|
||||
},
|
||||
"meta.guide": {
|
||||
"count": 1,
|
||||
"totalBytes": 15190,
|
||||
"p95Bytes": 15190,
|
||||
"maxBytes": 15190
|
||||
},
|
||||
"screenshot.responsive": {
|
||||
"count": 1,
|
||||
"totalBytes": 7053,
|
||||
"p95Bytes": 7053,
|
||||
"maxBytes": 7053
|
||||
},
|
||||
"update-styles": {
|
||||
"count": 1,
|
||||
"totalBytes": 2804,
|
||||
"p95Bytes": 2804,
|
||||
"maxBytes": 2804
|
||||
}
|
||||
},
|
||||
"retries": 0,
|
||||
"focusedReads": 1,
|
||||
"broadReads": 0,
|
||||
"dryRuns": 2,
|
||||
"plannedMutations": 2,
|
||||
"committedMutations": 5,
|
||||
"verifications": 4,
|
||||
"totalDurationMs": 37835,
|
||||
"p50DurationMs": 27,
|
||||
"p95DurationMs": 10457,
|
||||
"timeToFirstMutationMs": 53985,
|
||||
"timeToFirstVerificationMs": 187306
|
||||
"dryRuns": 0,
|
||||
"plannedMutations": 0,
|
||||
"committedMutations": 6,
|
||||
"verifications": 3,
|
||||
"totalDurationMs": 11038,
|
||||
"p50DurationMs": 12,
|
||||
"p95DurationMs": 10751,
|
||||
"totalResponseBytes": 40036,
|
||||
"p50ResponseBytes": 2804,
|
||||
"p95ResponseBytes": 15190,
|
||||
"maxResponseBytes": 15190,
|
||||
"timeToFirstMutationMs": 30419,
|
||||
"timeToFirstVerificationMs": 36551
|
||||
}
|
||||
},
|
||||
"callSequence": [
|
||||
"meta.guide",
|
||||
"list-breakpoints",
|
||||
"list-design-tokens",
|
||||
"list-pages",
|
||||
"list-assets",
|
||||
"meta.get_more_tools",
|
||||
"list-instances",
|
||||
"list-design-tokens",
|
||||
"inspect-design-context",
|
||||
"create-page",
|
||||
"create-page",
|
||||
"insert-fragment",
|
||||
"insert-fragment",
|
||||
"list-instances",
|
||||
"insert-fragment-verified",
|
||||
"attach-design-token",
|
||||
"attach-design-token",
|
||||
"attach-design-token",
|
||||
"verify-bindings",
|
||||
"preview.start",
|
||||
"preview.start",
|
||||
"preview.start",
|
||||
"screenshot",
|
||||
"screenshot",
|
||||
"update-styles",
|
||||
"screenshot.responsive",
|
||||
"audit"
|
||||
],
|
||||
"checks": {
|
||||
|
||||
@@ -6,35 +6,117 @@
|
||||
"cli": "source",
|
||||
"provider": "openai",
|
||||
"model": "gpt-5.6-sol",
|
||||
"commandSha256": "53276281bd1eb5d56e4fe9fe63e44fc5c62ccade4a6f789aafc05367a8f9045d",
|
||||
"commandSha256": "5c89af64bb3444ce679f82090c667ece40deb159b6b9ce7cd413e19c466d1ae3",
|
||||
"exitCode": 0,
|
||||
"metrics": {
|
||||
"durationMs": 48517,
|
||||
"durationMs": 50677,
|
||||
"tokens": {
|
||||
"input": 465604,
|
||||
"cachedInput": 418304,
|
||||
"input": 261344,
|
||||
"cachedInput": 240640,
|
||||
"uncachedInput": 20704,
|
||||
"cacheWriteInput": 0,
|
||||
"output": 1590,
|
||||
"reasoningOutput": 324,
|
||||
"total": 467194,
|
||||
"cachedInputRate": 0.8984
|
||||
"output": 1440,
|
||||
"reasoningOutput": 198,
|
||||
"total": 262784,
|
||||
"cachedInputRate": 0.9208
|
||||
},
|
||||
"mcpCatalog": {
|
||||
"responses": 1,
|
||||
"totalResponseBytes": 51362,
|
||||
"maxResponseBytes": 51362,
|
||||
"latestToolCount": 176,
|
||||
"latestResponseBytes": 51362,
|
||||
"latestInputSchemaBytes": 28964,
|
||||
"latestDescriptionBytes": 4556
|
||||
},
|
||||
"toolCalls": {
|
||||
"total": 8,
|
||||
"succeeded": 8,
|
||||
"total": 6,
|
||||
"succeeded": 6,
|
||||
"failed": 0,
|
||||
"failuresByTool": {},
|
||||
"failuresByCode": {},
|
||||
"issuesByCode": {},
|
||||
"issuesByPath": {},
|
||||
"durationsByTool": {
|
||||
"audit": {
|
||||
"count": 1,
|
||||
"totalMs": 17,
|
||||
"p95Ms": 17,
|
||||
"maxMs": 17
|
||||
},
|
||||
"meta.guide": {
|
||||
"count": 1,
|
||||
"totalMs": 11,
|
||||
"p95Ms": 11,
|
||||
"maxMs": 11
|
||||
},
|
||||
"update-asset": {
|
||||
"count": 2,
|
||||
"totalMs": 27,
|
||||
"p95Ms": 14,
|
||||
"maxMs": 14
|
||||
},
|
||||
"upload-assets": {
|
||||
"count": 1,
|
||||
"totalMs": 25,
|
||||
"p95Ms": 25,
|
||||
"maxMs": 25
|
||||
},
|
||||
"verify-font-assets": {
|
||||
"count": 1,
|
||||
"totalMs": 8,
|
||||
"p95Ms": 8,
|
||||
"maxMs": 8
|
||||
}
|
||||
},
|
||||
"responseBytesByTool": {
|
||||
"audit": {
|
||||
"count": 1,
|
||||
"totalBytes": 5300,
|
||||
"p95Bytes": 5300,
|
||||
"maxBytes": 5300
|
||||
},
|
||||
"meta.guide": {
|
||||
"count": 1,
|
||||
"totalBytes": 6566,
|
||||
"p95Bytes": 6566,
|
||||
"maxBytes": 6566
|
||||
},
|
||||
"update-asset": {
|
||||
"count": 2,
|
||||
"totalBytes": 1760,
|
||||
"p95Bytes": 880,
|
||||
"maxBytes": 880
|
||||
},
|
||||
"upload-assets": {
|
||||
"count": 1,
|
||||
"totalBytes": 2196,
|
||||
"p95Bytes": 2196,
|
||||
"maxBytes": 2196
|
||||
},
|
||||
"verify-font-assets": {
|
||||
"count": 1,
|
||||
"totalBytes": 2220,
|
||||
"p95Bytes": 2220,
|
||||
"maxBytes": 2220
|
||||
}
|
||||
},
|
||||
"retries": 0,
|
||||
"focusedReads": 2,
|
||||
"focusedReads": 0,
|
||||
"broadReads": 0,
|
||||
"dryRuns": 0,
|
||||
"plannedMutations": 0,
|
||||
"committedMutations": 3,
|
||||
"verifications": 1,
|
||||
"totalDurationMs": 73,
|
||||
"p50DurationMs": 10,
|
||||
"p95DurationMs": 21,
|
||||
"timeToFirstMutationMs": 27832,
|
||||
"timeToFirstVerificationMs": 43500
|
||||
"totalDurationMs": 88,
|
||||
"p50DurationMs": 13,
|
||||
"p95DurationMs": 25,
|
||||
"totalResponseBytes": 18042,
|
||||
"p50ResponseBytes": 2196,
|
||||
"p95ResponseBytes": 6566,
|
||||
"maxResponseBytes": 6566,
|
||||
"timeToFirstMutationMs": 37609,
|
||||
"timeToFirstVerificationMs": 47073
|
||||
}
|
||||
},
|
||||
"callSequence": [
|
||||
@@ -42,9 +124,7 @@
|
||||
"upload-assets",
|
||||
"update-asset",
|
||||
"update-asset",
|
||||
"refresh",
|
||||
"get-asset",
|
||||
"get-asset",
|
||||
"verify-font-assets",
|
||||
"audit"
|
||||
],
|
||||
"checks": {
|
||||
|
||||
@@ -20,10 +20,13 @@ import {
|
||||
} from "./agent-runner";
|
||||
import { collectHighImpactArtifacts } from "./artifacts";
|
||||
import type { EvaluationToolCall } from "./validate";
|
||||
import type { McpCatalogObservation } from "./evaluation-metrics";
|
||||
import { writeFontAssetFixtureFiles } from "./font-assets-fixture";
|
||||
import {
|
||||
compareEvaluationResult,
|
||||
isAggregateTokenBaselineNonRegressed,
|
||||
isEvaluationComparisonAccepted,
|
||||
shouldUpdateEvaluationBaselines,
|
||||
type EvaluationComparison,
|
||||
} from "./evaluation-regression";
|
||||
|
||||
@@ -38,6 +41,13 @@ type AgentEvaluationReport = AgentEvaluationResult & {
|
||||
comparison: EvaluationComparison;
|
||||
};
|
||||
|
||||
type EvaluationTraceEvent = EvaluationToolCall | McpCatalogObservation;
|
||||
|
||||
const isCatalogObservation = (
|
||||
event: EvaluationTraceEvent
|
||||
): event is McpCatalogObservation =>
|
||||
"kind" in event && event.kind === "tools-list";
|
||||
|
||||
const shellQuote = (value: string) => `'${value.replaceAll("'", `'"'"'`)}'`;
|
||||
|
||||
const selectFixtures = (fixtureId: string | undefined) => {
|
||||
@@ -98,6 +108,7 @@ const runFixture = async ({
|
||||
shellQuote(codex),
|
||||
"exec",
|
||||
"--ephemeral",
|
||||
"--ignore-user-config",
|
||||
"--dangerously-bypass-approvals-and-sandbox",
|
||||
"--skip-git-repo-check",
|
||||
"--ignore-rules",
|
||||
@@ -112,11 +123,12 @@ const runFixture = async ({
|
||||
),
|
||||
].join(" ");
|
||||
let toolCalls: EvaluationToolCall[] = [];
|
||||
const readToolCalls = async () =>
|
||||
let catalogObservations: McpCatalogObservation[] = [];
|
||||
const readTraceEvents = async () =>
|
||||
(await readFile(tracePath, "utf8").catch(() => ""))
|
||||
.split("\n")
|
||||
.filter(Boolean)
|
||||
.map((line) => JSON.parse(line) as EvaluationToolCall);
|
||||
.map((line) => JSON.parse(line) as EvaluationTraceEvent);
|
||||
return await runHighImpactAgentEvaluation({
|
||||
fixture,
|
||||
target: { kind: "source", repositoryRoot },
|
||||
@@ -128,8 +140,14 @@ const runFixture = async ({
|
||||
model,
|
||||
env,
|
||||
getToolCalls: () => toolCalls,
|
||||
getCatalogObservations: () => catalogObservations,
|
||||
evaluate: async () => {
|
||||
toolCalls = await readToolCalls();
|
||||
const traceEvents = await readTraceEvents();
|
||||
catalogObservations = traceEvents.filter(isCatalogObservation);
|
||||
toolCalls = traceEvents.filter(
|
||||
(event): event is EvaluationToolCall =>
|
||||
isCatalogObservation(event) === false
|
||||
);
|
||||
return evaluateHighImpactOutcome({
|
||||
fixture,
|
||||
project: fixtureApi.getProject(),
|
||||
@@ -204,6 +222,7 @@ const run = async () => {
|
||||
);
|
||||
const results: AgentEvaluationReport[] = [];
|
||||
const rawResults: AgentEvaluationResult[] = [];
|
||||
const baselines: AgentEvaluationResult[] = [];
|
||||
for (const fixture of fixtures) {
|
||||
const resultPath = resolve(
|
||||
process.env.WEBSTUDIO_HIGH_IMPACT_RESULT ??
|
||||
@@ -221,6 +240,9 @@ const run = async () => {
|
||||
...result,
|
||||
comparison: compareEvaluationResult(result, baseline),
|
||||
};
|
||||
if (baseline !== undefined) {
|
||||
baselines.push(baseline);
|
||||
}
|
||||
await writeFile(
|
||||
resultPath,
|
||||
`${JSON.stringify(report, undefined, 2)}\n`,
|
||||
@@ -231,7 +253,23 @@ const run = async () => {
|
||||
const evaluationsPassed = rawResults.every(
|
||||
(result) => result.outcome === "passed"
|
||||
);
|
||||
if (options.updateBaselines && evaluationsPassed) {
|
||||
const comparisonsPassed = results.every(({ comparison }) =>
|
||||
isEvaluationComparisonAccepted({
|
||||
comparison,
|
||||
requireBaseline: options.requireBaseline,
|
||||
updateBaselines: options.updateBaselines,
|
||||
})
|
||||
);
|
||||
const baselinesUpdated = shouldUpdateEvaluationBaselines({
|
||||
updateBaselines: options.updateBaselines,
|
||||
evaluationsPassed,
|
||||
comparisonsPassed,
|
||||
aggregateTokensPassed: isAggregateTokenBaselineNonRegressed(
|
||||
rawResults,
|
||||
baselines
|
||||
),
|
||||
});
|
||||
if (baselinesUpdated) {
|
||||
await mkdir(baselineDirectory, { recursive: true });
|
||||
await Promise.all(
|
||||
rawResults.map((result) =>
|
||||
@@ -243,20 +281,18 @@ const run = async () => {
|
||||
)
|
||||
);
|
||||
}
|
||||
const comparisonsPassed = results.every(({ comparison }) =>
|
||||
isEvaluationComparisonAccepted({
|
||||
comparison,
|
||||
requireBaseline: options.requireBaseline,
|
||||
updateBaselines: options.updateBaselines,
|
||||
})
|
||||
);
|
||||
const outcome = evaluationsPassed && comparisonsPassed ? "passed" : "failed";
|
||||
const outcome =
|
||||
evaluationsPassed &&
|
||||
comparisonsPassed &&
|
||||
(options.updateBaselines === false || baselinesUpdated)
|
||||
? "passed"
|
||||
: "failed";
|
||||
process.stdout.write(
|
||||
`${JSON.stringify(
|
||||
{
|
||||
outcome,
|
||||
baselines: options.updateBaselines
|
||||
? evaluationsPassed
|
||||
? baselinesUpdated
|
||||
? "updated"
|
||||
: "not-updated"
|
||||
: "compared",
|
||||
|
||||
@@ -169,18 +169,17 @@ describe("font-assets evaluation", () => {
|
||||
values: { meta: fontAssetFixtureMeta },
|
||||
},
|
||||
})),
|
||||
{ name: "refresh" },
|
||||
...project.assets.map((asset) => ({
|
||||
name: "get-asset",
|
||||
arguments: { assetId: asset.id },
|
||||
})),
|
||||
{
|
||||
name: "verify-font-assets",
|
||||
arguments: { assetIds: project.assets.map((asset) => asset.id) },
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({ passed: true, failures: [] });
|
||||
});
|
||||
|
||||
test("rejects a refresh before the final metadata update", () => {
|
||||
test("rejects verification before the final metadata update", () => {
|
||||
const project = clone(fontAssetsFixture.project);
|
||||
const result = evaluateHighImpactOutcome({
|
||||
fixture: fontAssetsFixture,
|
||||
@@ -188,7 +187,10 @@ describe("font-assets evaluation", () => {
|
||||
toolCalls: [
|
||||
{ name: "meta.guide" },
|
||||
{ name: "update-asset" },
|
||||
{ name: "refresh" },
|
||||
{
|
||||
name: "verify-font-assets",
|
||||
arguments: { assetIds: ["asset-1", "asset-2"] },
|
||||
},
|
||||
{ name: "update-asset" },
|
||||
],
|
||||
});
|
||||
@@ -271,6 +273,28 @@ describe("design-input evaluation", () => {
|
||||
expect(result).toMatchObject({ passed: true, failures: [] });
|
||||
});
|
||||
|
||||
test("accepts one responsive page verification as audit and viewport evidence", () => {
|
||||
const result = evaluateHighImpactOutcome({
|
||||
fixture: designInputFixture,
|
||||
project: addDesignPage(),
|
||||
toolCalls: [
|
||||
...designCalls.filter(
|
||||
(call) => call.name !== "screenshot" && call.name !== "audit"
|
||||
),
|
||||
{
|
||||
name: "verify-page-responsive",
|
||||
arguments: {
|
||||
viewports: [
|
||||
{ width: 1440, height: 900 },
|
||||
{ width: 390, height: 844 },
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(result).toMatchObject({ passed: true, failures: [] });
|
||||
});
|
||||
|
||||
test("rejects unsupported components", () => {
|
||||
const project = addDesignPage();
|
||||
project.instances.find(
|
||||
@@ -301,6 +325,25 @@ describe("design-input evaluation", () => {
|
||||
expect(result.checks.expressions).toBe("failed");
|
||||
});
|
||||
|
||||
test("accepts a parenthesized object expression", () => {
|
||||
const project = addDesignPage();
|
||||
project.props.push({
|
||||
id: "style-expression",
|
||||
instanceId: "summer-heading",
|
||||
name: "style",
|
||||
type: "expression",
|
||||
value: '({ display: state === "ready" ? "block" : "none" })',
|
||||
});
|
||||
|
||||
const result = evaluateHighImpactOutcome({
|
||||
fixture: designInputFixture,
|
||||
project,
|
||||
toolCalls: designCalls,
|
||||
});
|
||||
|
||||
expect(result.checks.expressions).toBe("passed");
|
||||
});
|
||||
|
||||
test("rejects broad or unnecessarily verbose reads", () => {
|
||||
const result = evaluateHighImpactOutcome({
|
||||
fixture: designInputFixture,
|
||||
|
||||
@@ -21,9 +21,12 @@ export type EvaluationToolCall = {
|
||||
arguments?: Record<string, unknown>;
|
||||
startedAtMs?: number;
|
||||
durationMs?: number;
|
||||
responseBytes?: number;
|
||||
planned?: true;
|
||||
committed?: true;
|
||||
isError?: boolean;
|
||||
errorCode?: string;
|
||||
errorIssues?: Array<{ code: string; path: string }>;
|
||||
};
|
||||
|
||||
export type EvaluationArtifact = {
|
||||
@@ -133,7 +136,10 @@ const getEditablePageState = (project: EvaluationProject) => ({
|
||||
|
||||
const isValidExpression = (value: string) => {
|
||||
try {
|
||||
const expression = parseExpressionAt(value, 0, { ecmaVersion: "latest" });
|
||||
const expression = parseExpressionAt(value, 0, {
|
||||
ecmaVersion: "latest",
|
||||
preserveParens: true,
|
||||
});
|
||||
return value.slice(expression.end).trim() === "";
|
||||
} catch {
|
||||
return false;
|
||||
@@ -168,6 +174,11 @@ const hasPassedEvidence = (
|
||||
kind: EvaluationArtifact["kind"]
|
||||
) =>
|
||||
hasSuccessfulCall(input.toolCalls, kind) ||
|
||||
(kind === "screenshot" &&
|
||||
(hasSuccessfulCall(input.toolCalls, "screenshot.responsive") ||
|
||||
hasSuccessfulCall(input.toolCalls, "verify-page-responsive"))) ||
|
||||
(kind === "audit" &&
|
||||
hasSuccessfulCall(input.toolCalls, "verify-page-responsive")) ||
|
||||
(input.artifacts ?? []).some(
|
||||
(artifact) => artifact.kind === kind && artifact.passed
|
||||
);
|
||||
@@ -182,6 +193,22 @@ const getScreenshots = (input: HighImpactEvaluationInput) => [
|
||||
| undefined,
|
||||
passed: true,
|
||||
})),
|
||||
...input.toolCalls
|
||||
.filter(
|
||||
(call) =>
|
||||
(call.name === "screenshot.responsive" ||
|
||||
call.name === "verify-page-responsive") &&
|
||||
call.isError !== true
|
||||
)
|
||||
.flatMap((call) =>
|
||||
Array.isArray(call.arguments?.viewports)
|
||||
? call.arguments.viewports.map((viewport) => ({
|
||||
kind: "screenshot" as const,
|
||||
viewport: viewport as { width: number; height: number },
|
||||
passed: true,
|
||||
}))
|
||||
: []
|
||||
),
|
||||
...(input.artifacts ?? []).filter(
|
||||
(artifact) => artifact.kind === "screenshot" && artifact.passed
|
||||
),
|
||||
@@ -245,7 +272,8 @@ const validateAuth = (
|
||||
checks,
|
||||
failures,
|
||||
"bindingVerification",
|
||||
hasSuccessfulCall(input.toolCalls, "verify-bindings"),
|
||||
hasSuccessfulCall(input.toolCalls, "verify-bindings") ||
|
||||
hasSuccessfulCall(input.toolCalls, "insert-fragment-verified"),
|
||||
"The agent did not verify the persisted authentication bindings."
|
||||
);
|
||||
recordCheck(
|
||||
@@ -478,21 +506,20 @@ const validateFontAssets = (
|
||||
? [{ ...call, index }]
|
||||
: []
|
||||
);
|
||||
const refreshIndex = input.toolCalls.findLastIndex(
|
||||
(call) => call.name === "refresh" && call.isError !== true
|
||||
);
|
||||
const verificationReads = input.toolCalls.filter(
|
||||
(call, index) =>
|
||||
call.name === "get-asset" && call.isError !== true && index > refreshIndex
|
||||
const verificationCalls = input.toolCalls.flatMap((call, index) =>
|
||||
call.name === "verify-font-assets" && call.isError !== true
|
||||
? [{ ...call, index }]
|
||||
: []
|
||||
);
|
||||
recordCheck(
|
||||
checks,
|
||||
failures,
|
||||
"metadataWorkflow",
|
||||
updateCalls.length >= expectedNames.size &&
|
||||
refreshIndex > (updateCalls.at(-1)?.index ?? Number.POSITIVE_INFINITY) &&
|
||||
verificationReads.length >= expectedNames.size,
|
||||
"The agent did not update both font assets, refresh, and read both persisted assets afterward."
|
||||
verificationCalls.length === 1 &&
|
||||
verificationCalls[0]!.index >
|
||||
(updateCalls.at(-1)?.index ?? Number.POSITIVE_INFINITY),
|
||||
"The agent did not update both font assets and verify both persisted assets in one bounded call afterward."
|
||||
);
|
||||
const fontFaces = getFontFaces(fontAssets, { assetBaseUrl: "/assets/" });
|
||||
recordCheck(
|
||||
|
||||
@@ -1003,6 +1003,63 @@ test("rejects a generated route that rendered Builder chrome", async () => {
|
||||
).rejects.toMatchObject({ code: "SCREENSHOT_NOT_GENERATED_SITE" });
|
||||
});
|
||||
|
||||
test("retries a managed generated route while the refreshed preview settles", async () => {
|
||||
const createResult = (generatedSiteRootPresent: boolean) => ({
|
||||
output: "account.png",
|
||||
browser: {
|
||||
browser: "chromium" as const,
|
||||
path: "/browser",
|
||||
source: "path" as const,
|
||||
},
|
||||
viewport: { width: 1280, height: 720 },
|
||||
fullPage: false,
|
||||
elapsedMs: 1,
|
||||
warnings: [],
|
||||
navigation: {
|
||||
requestedUrl: "http://127.0.0.1:5173/account",
|
||||
finalUrl: "http://127.0.0.1:5173/account",
|
||||
redirects: [],
|
||||
documentReadyState: "complete",
|
||||
generatedSiteRootPresent,
|
||||
layoutStable: true,
|
||||
},
|
||||
});
|
||||
const capture = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce(createResult(false))
|
||||
.mockResolvedValueOnce(createResult(true));
|
||||
const sleep = vi.fn(async () => undefined);
|
||||
const handlers = createMcpPreviewHandlers({
|
||||
preview: {
|
||||
status: vi.fn(() => ({
|
||||
url: "http://127.0.0.1:5173/",
|
||||
running: true,
|
||||
mode: "iterative" as const,
|
||||
})),
|
||||
startAndWait: vi.fn(),
|
||||
resolveUrl: vi.fn(() => "http://127.0.0.1:5173/account"),
|
||||
},
|
||||
isStale: () => false,
|
||||
createCaptureSession: vi.fn(() => ({
|
||||
capture,
|
||||
capturePage: vi.fn(),
|
||||
close: vi.fn(),
|
||||
})) as never,
|
||||
sleep,
|
||||
});
|
||||
|
||||
await expect(
|
||||
handlers.captureScreenshot({
|
||||
path: "/account",
|
||||
viewport: { width: 1280, height: 720 },
|
||||
})
|
||||
).resolves.toMatchObject({
|
||||
navigation: { generatedSiteRootPresent: true },
|
||||
});
|
||||
expect(capture).toHaveBeenCalledTimes(2);
|
||||
expect(sleep).toHaveBeenCalledWith(1000);
|
||||
});
|
||||
|
||||
test("rejects resized generated routes that rendered Builder chrome", async () => {
|
||||
const capturePage = vi.fn(async () => [
|
||||
{
|
||||
|
||||
@@ -101,6 +101,8 @@ const isSameCaptureSessionConfig = (
|
||||
) => left.browser === right.browser && left.browserPath === right.browserPath;
|
||||
|
||||
const defaultPreviewSource = "session" satisfies PreviewSource;
|
||||
const defaultSleep = (durationMs: number) =>
|
||||
new Promise<void>((resolve) => setTimeout(resolve, durationMs));
|
||||
const getPreviewSource = (source: PreviewSource | undefined): PreviewSource =>
|
||||
source ?? defaultPreviewSource;
|
||||
|
||||
@@ -160,6 +162,7 @@ export const createMcpPreviewHandlers = ({
|
||||
prepareSessionDataFile,
|
||||
captureScreenshot = captureScreenshotWithBrowserInstall,
|
||||
createCaptureSession,
|
||||
sleep = defaultSleep,
|
||||
}: {
|
||||
preview: McpPreviewController;
|
||||
isStale?: () => boolean;
|
||||
@@ -169,6 +172,7 @@ export const createMcpPreviewHandlers = ({
|
||||
prepareSessionDataFile?: () => Promise<void>;
|
||||
captureScreenshot?: typeof captureScreenshotWithBrowserInstall;
|
||||
createCaptureSession?: typeof createScreenshotCaptureSession;
|
||||
sleep?: (durationMs: number) => Promise<void>;
|
||||
}) => {
|
||||
let captureSession:
|
||||
| ReturnType<typeof createScreenshotCaptureSession>
|
||||
@@ -407,9 +411,19 @@ export const createMcpPreviewHandlers = ({
|
||||
isManagedSessionPreviewCapture(input) &&
|
||||
createCaptureSession !== undefined
|
||||
) {
|
||||
result = await (
|
||||
await getCaptureSession(input)
|
||||
).capture(captureOptions);
|
||||
const captureSession = await getCaptureSession(input);
|
||||
result = await captureSession.capture(captureOptions);
|
||||
for (
|
||||
let retry = 0;
|
||||
retry < 2 && result.navigation?.generatedSiteRootPresent === false;
|
||||
retry += 1
|
||||
) {
|
||||
progress?.report(
|
||||
"tool screenshot waiting for refreshed generated route"
|
||||
);
|
||||
await sleep(1000);
|
||||
result = await captureSession.capture(captureOptions);
|
||||
}
|
||||
} else {
|
||||
result = await captureScreenshot({
|
||||
...captureOptions,
|
||||
|
||||
@@ -29,8 +29,98 @@ const {
|
||||
readPersistedMcpCheckpoint,
|
||||
updatePersistedMcpCheckpoint,
|
||||
executeMcpRunCall,
|
||||
resolveMcpPreviewInput,
|
||||
resolveMcpScreenshotInput,
|
||||
startMcpPreview,
|
||||
} = __testing__;
|
||||
|
||||
test("allocates an available port only when MCP preview omits one", async () => {
|
||||
const getAvailablePort = vi.fn(async () => 53124);
|
||||
|
||||
await expect(
|
||||
resolveMcpPreviewInput({ source: "session" }, getAvailablePort)
|
||||
).resolves.toEqual({ source: "session", port: 53124 });
|
||||
await expect(
|
||||
resolveMcpPreviewInput({ source: "session", port: 0 }, getAvailablePort)
|
||||
).resolves.toEqual({ source: "session", port: 53124 });
|
||||
await expect(
|
||||
resolveMcpPreviewInput({ source: "session", port: 4173 }, getAvailablePort)
|
||||
).resolves.toEqual({ source: "session", port: 4173 });
|
||||
expect(getAvailablePort).toHaveBeenCalledTimes(2);
|
||||
expect(getAvailablePort).toHaveBeenNthCalledWith(1, "127.0.0.1");
|
||||
expect(getAvailablePort).toHaveBeenNthCalledWith(2, "127.0.0.1");
|
||||
});
|
||||
|
||||
test("allocates and reuses a collision-free port for automatic screenshots", async () => {
|
||||
const getAvailablePort = vi.fn(async () => 53124);
|
||||
|
||||
await expect(
|
||||
resolveMcpScreenshotInput(
|
||||
{
|
||||
path: "/account",
|
||||
browser: "auto",
|
||||
viewport: { width: 390, height: 844 },
|
||||
},
|
||||
{ running: false, url: "http://127.0.0.1:5173/" },
|
||||
getAvailablePort
|
||||
)
|
||||
).resolves.toMatchObject({ port: 53124 });
|
||||
await expect(
|
||||
resolveMcpScreenshotInput(
|
||||
{
|
||||
path: "/account",
|
||||
browser: "auto",
|
||||
viewport: { width: 1440, height: 900 },
|
||||
},
|
||||
{ running: true, url: "http://127.0.0.1:53125/" },
|
||||
getAvailablePort
|
||||
)
|
||||
).resolves.toMatchObject({ port: 53125 });
|
||||
await expect(
|
||||
resolveMcpScreenshotInput(
|
||||
{
|
||||
path: "/account",
|
||||
browser: "auto",
|
||||
port: 4173,
|
||||
viewport: { width: 1440, height: 900 },
|
||||
},
|
||||
{ running: false, url: "http://127.0.0.1:5173/" },
|
||||
getAvailablePort
|
||||
)
|
||||
).resolves.toMatchObject({ port: 4173 });
|
||||
expect(getAvailablePort).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
test("retries automatic MCP preview ports after a startup race", async () => {
|
||||
const getAvailablePort = vi
|
||||
.fn()
|
||||
.mockResolvedValueOnce(53124)
|
||||
.mockResolvedValueOnce(53125);
|
||||
const startPreview = vi
|
||||
.fn()
|
||||
.mockRejectedValueOnce(new Error("Preview server exited before ready"))
|
||||
.mockResolvedValueOnce({ url: "http://127.0.0.1:53125/" });
|
||||
const sleep = vi.fn(async () => undefined);
|
||||
|
||||
await expect(
|
||||
startMcpPreview({
|
||||
input: { source: "session" },
|
||||
getAvailablePort,
|
||||
startPreview,
|
||||
sleep,
|
||||
})
|
||||
).resolves.toEqual({ url: "http://127.0.0.1:53125/" });
|
||||
expect(startPreview).toHaveBeenNthCalledWith(1, {
|
||||
source: "session",
|
||||
port: 53124,
|
||||
});
|
||||
expect(startPreview).toHaveBeenNthCalledWith(2, {
|
||||
source: "session",
|
||||
port: 53125,
|
||||
});
|
||||
expect(sleep).toHaveBeenCalledWith(500);
|
||||
});
|
||||
|
||||
test("classifies structured MCP tool failures for nonzero CLI exit", () => {
|
||||
expect(
|
||||
isMcpToolCallFailure({
|
||||
|
||||
@@ -7,6 +7,8 @@ import {
|
||||
createMcpStdioTransport,
|
||||
getProjectSessionMcpCheckpoint,
|
||||
isReadOnlyProjectSessionMcpToolCall,
|
||||
type ProjectSessionPreviewInput,
|
||||
type ProjectSessionScreenshotInput,
|
||||
type ProjectSessionMcpTool,
|
||||
} from "@webstudio-is/project-build/mcp";
|
||||
import type { BuilderNamespace } from "@webstudio-is/project-build/contracts";
|
||||
@@ -825,6 +827,72 @@ const assertSingleOpCallToolSupported = (tool: string) => {
|
||||
}
|
||||
};
|
||||
|
||||
const resolveMcpPreviewInput = async (
|
||||
input: ProjectSessionPreviewInput,
|
||||
getAvailablePort: (host?: string) => Promise<number> = findAvailablePort
|
||||
) => {
|
||||
if (input.port !== undefined && input.port !== 0) {
|
||||
return input;
|
||||
}
|
||||
return {
|
||||
...input,
|
||||
port: await getAvailablePort(input.host ?? "127.0.0.1"),
|
||||
};
|
||||
};
|
||||
|
||||
const resolveMcpScreenshotInput = async (
|
||||
input: ProjectSessionScreenshotInput,
|
||||
previewStatus: { running: boolean; url: string },
|
||||
getAvailablePort: (host?: string) => Promise<number> = findAvailablePort
|
||||
) => {
|
||||
if (
|
||||
input.url !== undefined ||
|
||||
input.baseUrl !== undefined ||
|
||||
input.source === "local" ||
|
||||
(input.port !== undefined && input.port !== 0)
|
||||
) {
|
||||
return input;
|
||||
}
|
||||
const host = input.host ?? "127.0.0.1";
|
||||
if (previewStatus.running) {
|
||||
const previewUrl = new URL(previewStatus.url);
|
||||
if (input.host === undefined || previewUrl.hostname === input.host) {
|
||||
const previewPort = Number(previewUrl.port);
|
||||
if (Number.isInteger(previewPort) && previewPort > 0) {
|
||||
return { ...input, port: previewPort };
|
||||
}
|
||||
}
|
||||
}
|
||||
return { ...input, port: await getAvailablePort(host) };
|
||||
};
|
||||
|
||||
const startMcpPreview = async <Result>({
|
||||
input,
|
||||
startPreview,
|
||||
getAvailablePort = findAvailablePort,
|
||||
sleep = async (durationMs) =>
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, durationMs)),
|
||||
}: {
|
||||
input: ProjectSessionPreviewInput;
|
||||
startPreview: (input: ProjectSessionPreviewInput) => Promise<Result>;
|
||||
getAvailablePort?: (host?: string) => Promise<number>;
|
||||
sleep?: (durationMs: number) => Promise<void>;
|
||||
}) => {
|
||||
const attempts = input.port === undefined || input.port === 0 ? 5 : 1;
|
||||
for (let attempt = 0; attempt < attempts; attempt += 1) {
|
||||
const resolvedInput = await resolveMcpPreviewInput(input, getAvailablePort);
|
||||
try {
|
||||
return await startPreview(resolvedInput);
|
||||
} catch (error) {
|
||||
if (attempt === attempts - 1) {
|
||||
throw error;
|
||||
}
|
||||
await sleep(500);
|
||||
}
|
||||
}
|
||||
throw new Error("MCP preview did not start.");
|
||||
};
|
||||
|
||||
const createCliMcpHost = async ({
|
||||
projectRoot = cwd(),
|
||||
projectId,
|
||||
@@ -999,29 +1067,39 @@ const createCliMcpHost = async ({
|
||||
};
|
||||
},
|
||||
async startPreview(input, progress) {
|
||||
const resolvedInput =
|
||||
input.port === 0
|
||||
? {
|
||||
...input,
|
||||
port: await findAvailablePort(input.host ?? "127.0.0.1"),
|
||||
}
|
||||
: input;
|
||||
const result = await previewHandlers.startPreview(
|
||||
resolvedInput,
|
||||
progress
|
||||
);
|
||||
return result;
|
||||
return await startMcpPreview({
|
||||
input,
|
||||
startPreview: async (resolvedInput) =>
|
||||
await previewHandlers.startPreview(resolvedInput, progress),
|
||||
});
|
||||
},
|
||||
async getPreviewStatus() {
|
||||
return preview.status();
|
||||
},
|
||||
stopPreview: previewHandlers.stopPreview,
|
||||
async captureScreenshot(input) {
|
||||
const result = await previewHandlers.captureScreenshot(input);
|
||||
const resolvedInput = await resolveMcpScreenshotInput(
|
||||
input,
|
||||
preview.status()
|
||||
);
|
||||
const result = await previewHandlers.captureScreenshot(resolvedInput);
|
||||
return toProjectSessionScreenshotResult(result);
|
||||
},
|
||||
async capturePageScreenshots(inputs) {
|
||||
const results = await previewHandlers.capturePageScreenshots(inputs);
|
||||
const firstInput = inputs[0];
|
||||
if (firstInput === undefined) {
|
||||
return [];
|
||||
}
|
||||
const resolvedFirstInput = await resolveMcpScreenshotInput(
|
||||
firstInput,
|
||||
preview.status()
|
||||
);
|
||||
const resolvedInputs = inputs.map((input) => ({
|
||||
...input,
|
||||
port: resolvedFirstInput.port,
|
||||
}));
|
||||
const results =
|
||||
await previewHandlers.capturePageScreenshots(resolvedInputs);
|
||||
return results.map(toProjectSessionScreenshotResult);
|
||||
},
|
||||
async diffScreenshots(input) {
|
||||
@@ -1152,6 +1230,9 @@ export const __testing__ = {
|
||||
readPersistedMcpCheckpoint,
|
||||
updatePersistedMcpCheckpoint,
|
||||
executeMcpRunCall,
|
||||
resolveMcpPreviewInput,
|
||||
resolveMcpScreenshotInput,
|
||||
startMcpPreview,
|
||||
};
|
||||
|
||||
export const mcpSingleOpCall = async (options: McpSingleOpCallOptions) => {
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -2109,6 +2109,18 @@ describe("createResourceValue", () => {
|
||||
});
|
||||
});
|
||||
|
||||
test("accepts root-relative resource urls", () => {
|
||||
expect(
|
||||
resourceFieldsInput.parse({
|
||||
name: "Session",
|
||||
method: "get",
|
||||
url: "/api/auth/session",
|
||||
searchParams: [],
|
||||
headers: [],
|
||||
})
|
||||
).toMatchObject({ url: '"/api/auth/session"' });
|
||||
});
|
||||
|
||||
test("creates resource values through the sdk schema", () => {
|
||||
expect(
|
||||
createResourceValue({
|
||||
|
||||
@@ -32,7 +32,6 @@ import { z } from "zod";
|
||||
import { produceWithPatches } from "immer";
|
||||
import {
|
||||
createJsonStringifyProxy,
|
||||
isLocalResource,
|
||||
isPlainObject,
|
||||
} from "@webstudio-is/sdk/runtime";
|
||||
import type { CompactBuild } from "../types";
|
||||
@@ -2013,23 +2012,33 @@ const getResourceLiteralUrlValidationIssues = (
|
||||
},
|
||||
];
|
||||
}
|
||||
if (isLocalResource(value)) {
|
||||
return [];
|
||||
}
|
||||
try {
|
||||
new URL(value);
|
||||
return [];
|
||||
} catch {
|
||||
return [
|
||||
{
|
||||
code: "invalid_resource_url",
|
||||
path: ["url"],
|
||||
message: "URL is invalid",
|
||||
constraint: "absolute_or_root_relative_url",
|
||||
example: "https://api.example.com/posts",
|
||||
},
|
||||
];
|
||||
try {
|
||||
const baseUrl = new URL("https://resource.webstudio.invalid");
|
||||
const parsedUrl = new URL(value, baseUrl);
|
||||
if (
|
||||
value.startsWith("/") &&
|
||||
value.startsWith("//") === false &&
|
||||
parsedUrl.origin === baseUrl.origin
|
||||
) {
|
||||
return [];
|
||||
}
|
||||
} catch {
|
||||
// Report the shared URL validation issue below.
|
||||
}
|
||||
}
|
||||
return [];
|
||||
return [
|
||||
{
|
||||
code: "invalid_resource_url",
|
||||
path: ["url"],
|
||||
message: "URL is invalid",
|
||||
constraint: "absolute_or_root_relative_url",
|
||||
example: "https://api.example.com/posts",
|
||||
},
|
||||
];
|
||||
};
|
||||
|
||||
const validateResourceFields = (
|
||||
|
||||
Reference in new issue
Block a user