webstudio

admin/webstudio

0 Fork 0

mirror of https://github.com/webstudio-is/webstudio.git synced 2026-11-09 14:03:33 +09:00

Commit Graph

Author	SHA1	Message	Date
Oleg Isonen	3d0a0657aa	feat: Support async resource dependencies (#6030 ) ## Summary Make `Resource<Document>` the lazy document-resolution abstraction shared by Content Engine graphs, generated sites, and Builder previews. Expressions use one async computation interface: referenced resource documents resolve first, independent requests run concurrently, and unused resources stay unloaded. Ref #6029 Ref #6031 ## Implementation checklist - [x] Define one resource contract for synchronous or asynchronous document resolution. - [x] Resolve explicit roots lazily with memoization, bounded concurrency, cancellation, and structured errors. - [x] Preserve cycle, duplicate, and missing-resource diagnostics. - [x] Extract resource dependencies with the existing expression AST. - [x] Discover consumed resource roots from the current page tree and page metadata. - [x] Generate a lazy request graph for published page data, including independent requests. - [x] Resolve dependency documents before evaluating dependent request expressions. - [x] Integrate dependency-first loading with Builder preview caching and invalidation. - [x] Expose safe Resource bindings in the editor and reject self/indirect cycles. - [x] Adapt REST, GraphQL, Assets, date, and sitemap requests through the same Resource contract. - [x] Cover laziness, chains, shared dependencies, concurrency, invalidation, cancellation, failures, and cycles with fail/pass tests. - [x] Review documentation impact and document dependent Resources in the Data variables guide. - [x] Track collection-item and component-parameter-dependent resources as follow-up #6031. - [x] Regenerate and review affected fixtures. - [x] Resolve action-resource request dependencies when an action is invoked. - [x] Reproduce and fix cached form submissions and multi-root dependency-cycle hangs. - [x] Document uncached form actions and regenerate affected fixtures. - [x] Preserve single and multi-selected content until a cut successfully writes to the clipboard. - [x] Preserve boolean expression values in the search-exclusion checkbox. - [x] Cancel cut deletion if project data changes while preparing or writing the clipboard; cover content/property edits and project switches for single and multi-selection, and document the retry behavior. - [x] Validate pending cuts after the final await, immediately before deletion, and reproduce the queued-update race for single and multi-selection. ## Technical direction The persisted, revisioned `DocumentGraph` remains the Content Engine artifact. `Resource<Document>` is the execution abstraction for obtaining a document; its resolver evaluates only the dependency closure reachable from requested roots. Static and remote documents therefore differ by Resource implementation, not by expression value type. Generated pages always emit a `ResourceRequestGraph`. Each node owns its output name, dependencies, and request factory. Roots come from resource variables actually referenced by the current page's metadata, props, actions, or expression children. Unused and off-page resources remain unresolved. The loader still accepts legacy request maps for compatibility. Builder preview uses the same page-root discovery as published generation. Its existing cache remains the lifecycle boundary: cached dependency documents unlock dependent requests, while invalidation removes downstream requests until their inputs resolve again. The separate canvas frame recomputes its local plan from the synchronized cache so rendered expressions receive those documents. Action requests and their dependencies resolve when a form is submitted. Each submission sends a new action request, bypassing Webstudio's resource cache. Dependency requests still use their configured cache lifetime. ## Review simplifications - Consolidated page resource-root discovery in the SDK and reused it in Builder. - Scoped root and action discovery to the current page tree. - Removed the redundant immediate-resource helper; `Resource.resolve` already accepts `Document \| Promise<Document>`. - Moved output names onto graph nodes and removed the duplicate graph-level map. - Removed unused planner output and corrected transitive cycle discovery to include only Resource data sources. - Regenerated fixtures so independent resources also use lazy graph execution. - Reused one abort-aware concurrency limiter for document sessions and Resource resolution. - Collapsed duplicate generator action scans and removed redundant generated-resource bookkeeping. - Unified legacy request maps and request graphs through one resolution path. - Shared the 20-request concurrency limit between SDK, Builder preview, and the server batch boundary. - Preserved dependency failure paths and original causes, including the first cancellation reason. - Removed invalid-request logging that could expose Resource headers or request bodies. - Made editor cycle detection alias-safe by comparing underlying Resource IDs. - Memoized page settings values to prevent repeated async validation renders. ## Verification Latest local verification for `bbca6395e3`: - Both queued-update regression cases failed before the fix and passed afterward. Validation now runs synchronously immediately before deletion, after the final await, in both cut paths. - Passed: 255 copy/paste and command tests, Builder typecheck, lint, package boundaries, formatting, and diff checks. - Reviewed documentation: the existing copy/paste safety description remains accurate. Fixture inputs and outputs are unaffected; no fixture regeneration was needed. Agent evaluations were not run because separate approval is required. - CI for the new commit is pending. Previous local verification for `ffbc75b152`: - Six regression cases failed before the fix and passed afterward: content edits, property edits, and project switches during a pending cut, with both single and multiple selections. - Passed: 253 copy/paste and command tests, Builder typecheck, lint, package boundaries, source formatting, and diff checks. - An unchanged content-block test file initially failed to import in the browser. Its isolated rerun and the full 253-test rerun passed without code changes. - The guard conservatively treats any project-data change as a reason to keep the clipboard copy without deleting content. It shows a retry message. Copy/paste documentation now describes this behavior. - Fixture inputs and outputs are unaffected; fixtures were not regenerated. Agent evaluations were not run because separate approval is required. CI for the new commit is pending. Previous local verification for `3eb580697c`: - Reproduced both regressions with failing tests before fixing them: premature deletion during cut and false search exclusion appearing checked. - Passed: 203 copy/paste and page-settings tests, Builder typecheck, lint, package boundaries, formatting, and diff checks. - Cut coverage includes keyboard and programmatic entry points, single and multiple selections, delayed successful writes, rejected writes, and disabled copying. - Reviewed documentation impact: these fixes restore existing behavior; no documentation update is needed. Only Builder UI and clipboard files changed; fixture inputs and outputs are unaffected, so fixtures were not regenerated for this commit. - CI status for this older commit was recorded as pending at handoff. Agent evaluations were not run; they require separate approval. Previous local verification for `858b348ad`, rebased onto `cf606ca4c`: - Regression tests failed before each fix and passed afterward: repeated submissions in both generated route templates, plus three multi-root/branching cycle cases. - Passed: 590 SDK tests, 1,129 CLI tests, 72 focused Builder tests, SDK/CLI/Builder typechecks, lint, package boundaries, and fixture link/sync/build. - Initial concurrent runs hit timeouts. The complete CLI suite passed with one worker; the focused Builder suites passed without file parallelism. - Documentation and generated fixture changes reviewed. Agent evaluations were not run; they require separate approval. - CI for `858b348ad` passed: Main (including all six E2E shards), Visual Regression, Lint PR, and Check submodules. Earlier verification (before this update): TDD fail/pass evidence includes resolver creation, AST dependency discovery, published graph reachability, dependency-gated request generation, Builder dependency-first planning, Resource editor cycle filtering, page-tree root scoping, off-page action isolation, transitive Resource-only cycle discovery, abort/timeout precedence, server batch limits, and the rebased SSG fixture consuming its lazy Resource. - `pnpm checks` passed after rebasing onto `ce84ddcf12`; it covered workspace tests, typechecks, lint, package boundaries, generated API/docs validation, and fixture link/sync/build. - Earlier focused checks passed: Builder typecheck, 12 page-settings/pages unit tests, lint, formatting, and `git diff --check`. - Local generated-resource E2E passed for HTTP, GraphQL, and current-date resources (3 tests). - The page-actions and Marketplace page-settings persistence E2Es both pass locally. - CI for `6b8b08a70` passed: Main workflow (including all six Builder E2E shards), Visual Regression, Lint PR, and Check submodules. - Agent evaluations were not run because they require separate explicit approval. ## Current behavior and limitations - A failed dependency blocks its dependents and reports the resource path and cause. - One published resolution call is the memoization boundary; Builder maps dependency readiness onto its versioned request-cache lifecycle. - Scoped collection-item and component-parameter resources remain in #6031. - Builder skips cyclic dependency paths without blocking independent resources.	2026-09-22 21:04:42 +02:00
Oleg Isonen	55920c57c4	fix: Show all Markdown and Assets query diagnostics (#6308 ) Ref #6305 ## Outcome Show actionable Markdown, MDX, and Assets query problems in the editor, query Diagnostics, API, and MCP. The current implementation substantially improves source diagnostics, query error aggregation, query scoping, exact paths, MCP feedback, and loading states. The exhaustive audit below also records what is not complete yet. ## Completion rule An error state is complete only when all applicable checks pass: - [x] The editor uses the production validator and shows every issue for the open file. - [x] Query execution uses the same validation and returns every issue for every file that matches the query, not only the returned page. - [ ] MCP can reproduce the state through a real tool call and returns the same code, severity, message, file, and location. - [x] Diagnostics shows fatal errors and nonfatal warnings in the correct place. - [x] Exact file is always present; line and column are present whenever the parser can determine them. - [x] Multiple issues in the same file, multiple files, and multiple validation phases are returned together. - [ ] A regression test proves failure without the behavior and success with it. ## Cross-surface contract ### Required validator parity - [x] Editor, query execution, and MCP file operations invoke the same exported production file-validation pipeline for byte decoding, source/frontmatter, Markdown/MDX, Content Block semantics, and references. Copied schemas, adapter-only checks, or reimplementations do not count as parity. - [x] Query builder, query execution, `validate-asset-query`, and `preview-asset-query` invoke the same exported structural and catalog-aware semantic query validator with the same project snapshot and context. - [x] Every surface preserves the same complete, deterministically ordered diagnostic array: severity, code, message, phase/scope, asset ID, file path, line/column/range, query path, and fatal-versus-warning/result behavior. UI, API, and MCP serializers must not drop or remap fields. - [ ] A table-driven contract suite calls the real editor, query, API, and MCP entry points—including MCP upload, update, download, validate, and preview—and compares full normalized arrays for multiple errors, multiple warnings, multiple files, and pre-pagination matches. - [x] Markdown and MDX source syntax/frontmatter use validateTextAssetSource in editor, query compilation, and MCP file tools. - [x] Query request execution and MCP local parsing use assetQueryRequest. - [x] Fatal query setup errors retain all detected semantic errors and accompanying warnings. - [x] Nonfatal query warnings retain a stable code and exact query path. - [x] Source diagnostics retain severity, phase, code, asset ID, path, and available source position. - [x] Query Diagnostics excludes diagnostics produced only by unrelated published queries. - [x] Query, Preview, and Diagnostics tabs show loading states. - [x] Authenticated diagnostics can show a sanitized internal message; public/pre-auth errors remain redacted. - [x] Share byte-level UTF-8 validation instead of decoding separately in query and MCP. - [x] Make the query editor and MCP call one literal shared query-validation function before execution. - [x] Share cross-file reference validation across editor, query, and MCP file operations. - [x] Share contextual Content Block/template validation across all applicable surfaces. - [x] Preserve an invalid stored query in the editor and explain it instead of replacing it with defaults. - [x] Define and enforce whether “matches the query” includes every pre-pagination match; current source validation follows returned/paginated IDs. - [x] Continue through source, reference, and runtime phases where safe so one phase does not hide later issues. - [x] Map every user-triggerable graph/storage/runtime failure to a structured public error instead of INTERNAL_ERROR. - [ ] Label candidate-file warnings separately from returned-file diagnostics. ## File transport and encoding - [x] Empty Markdown and MDX are accepted. - [x] UTF-8 BOM and CRLF are accepted. - [x] Whole Markdown/MDX file over 1 MiB is detected. - [x] Missing or unavailable selected content is reported. - [x] Invalid UTF-8 frontmatter returns the same structured diagnostic in query and MCP. - [x] Invalid UTF-8 body returns the same structured diagnostic in query and MCP. - [x] MCP download must use fatal UTF-8 decoding rather than replacement characters. - [x] Validate bytes before committing an upload, or return the committed Asset ID and explicit cleanup state when post-commit feedback fails. - [x] Reject filename/format mismatches instead of allowing either hint to bypass Markdown/MDX validation. - [x] Storage status, missing body, invalid metadata, revision mismatch, format mismatch, short read, and excess read each return structured file context. - [x] Content changing during validation has a clear retryable conflict diagnostic. ## Frontmatter - [x] No frontmatter. - [x] Empty frontmatter. - [x] Opening delimiter without a closing delimiter. - [x] Frontmatter over 64 KiB. - [x] Invalid YAML grammar. - [x] Multiple YAML parser errors. - [x] Duplicate keys. - [x] YAML aliases. - [x] Unsupported YAML tags. - [x] Scalar or array root instead of an object. - [x] Non-JSON values, NaN, Infinity, and unsafe integers. - [x] Nesting deeper than 8. - [x] More than 256 fields/array entries. - [x] One string over 16 KiB. - [x] Normalized indexed properties over 64 KiB. - [x] Frontmatter warnings remain nonfatal and visible. - [x] Collect all post-parse normalization/limit errors; this phase currently stops at the first. - [ ] Prove every state above through real MCP upload, update, download, query, and Diagnostics paths. ## Markdown body - [x] Treat ordinary CommonMark/GFM as permissive; malformed-looking Markdown and raw HTML are not automatically syntax errors. - [x] Preserve and validate the body when frontmatter is invalid. - [x] Detect the 1 MiB body/file limit. - [x] Handle body extraction and bounded excerpts. - [x] Return a shared structured invalid-UTF-8 body error from every byte-reading surface. - [ ] Distinguish author errors from internal excerpt-limit/configuration errors. - [ ] Prove body-size and decoding states through real MCP and Diagnostics. ## MDX grammar and safety - [x] Malformed MDX/JSX grammar. - [x] Imports. - [x] Exports. - [x] Executable flow expressions. - [x] Executable text expressions. - [x] JSX expression attributes. - [x] JSX spread attributes. - [x] Arbitrary components. - [x] HTML-looking JSX outside ws.element. - [x] Unsupported/raw MDX node types. - [x] Duplicate attributes and normalized alias collisions. - [x] Event-handler attributes. - [x] srcdoc, srcset, and dangerouslySetInnerHTML. - [x] Unsafe URL protocols in link/source/action-like props. - [x] Forbidden tags: base, link, meta, noscript, script, style, template, title. - [x] Conflicting, empty, or non-static ws:name and ws:tag. - [x] More than 20,000 nodes. - [x] Nesting deeper than 100. - [x] More than 4,000 props. - [x] Multiple recoverable unsafe-mdx warnings are collected. - [ ] Collect multiple fatal grammar/structural errors when parser recovery can do so; unrecoverable parsing currently stops at the first fatal error. - [x] Preserve sibling expression warnings when unsupported JSX is also present; {first()}, <Card />, {second()} currently reports only the Card warning. - [ ] Prove every state above through real MCP and Diagnostics, not only validator unit tests. ## Content Block semantics - [x] Unresolved template warning exists. - [x] Unknown template prop warning exists. - [x] Incompatible template prop warning exists. - [x] Design-only template prop warning exists. - [x] Stale template prop warning exists. - [x] Invalid materialized HTML content model error exists. - [x] Run these checks for an MDX file even when no active editor Content Block root has materialized it. - [x] Run applicable semantic checks in query Diagnostics. - [x] Run applicable semantic checks for MCP upload as well as contextual update/download. - [x] Return all Content Block frontmatter diagnostics; one path still throws only diagnostics[0]. - [ ] Add real MCP and UI coverage for every semantic state. ## Cross-file references - [x] Non-string $ref marker. - [x] Invalid reference URL. - [x] Invalid fragment encoding. - [x] Invalid fragment. - [x] Invalid JSON Pointer. - [x] Missing referenced document. - [x] Missing referenced JSON path. - [x] Missing referenced local Asset warning. - [x] Unsupported target representation. - [x] Reference cycle. - [x] Duplicate document ID or URL. - [x] Missing graph source or target. - [x] Root not found. - [x] Duplicate/unused/unresolved reference. - [x] Invalid resolved value. - [x] Content identity conflict. - [x] Target revision mismatch. - [x] Representation selector required/invalid. - [ ] Diagnose {$ref: value, extraKey: value} as an invalid marker or explicitly document it as ordinary data. - [x] Collect all bad references and targets instead of stopping at the first graph failure. - [x] Validate references from the editor and MCP file tools, not only query compilation. - [x] Map graph compilation/resolution, Markdown-document, and JSON-document errors to exact structured Diagnostics rows. - [x] Prevent an unrelated file's graph error or duplicate canonical URL from blocking an exact-file query. - [ ] Prove missing targets, cycles, bad representations, and bad JSON paths through real MCP. ## Query request and structure - [x] Invalid JSON, invalid request UTF-8, empty/non-object request, and request over 512 KiB. - [x] Missing query and unknown request/query keys. - [x] Invalid result mode. - [x] Malformed where/all/any/filter shape. - [x] More than 32 filters or children. - [x] Filter nesting deeper than 8. - [x] Empty field path or segment. - [x] Field path deeper than 9. - [x] Bare properties path. - [x] Unknown or nested standard field. - [x] Missing or unknown operator. - [x] Non-JSON operator value. - [x] Invalid in value or more than 1,000 in values. - [x] Non-boolean exists/isEmpty value. - [x] More than 8 sort entries. - [x] Invalid sort field or direction. - [x] first/last without a sort. - [x] Invalid output mode or includeMetadata. - [x] Duplicate output fields or more than 256 fields. - [x] No selected output. - [x] Negative, fractional, or over-1,000 limit. - [x] Negative, fractional, or over-1,000 offset. - [x] Invalid content mode. - [x] Invalid full/markdown-body-ref maxBytes outside 1..1 MiB. - [x] Invalid range offset/length or range over 256 KiB. - [x] Empty or over-255-character index revision. - [x] Multiple simultaneous schema issues retain exact query paths. - [x] Make content option objects strict; unknown keys are currently stripped. - [x] Show an invalid persisted resource body rather than silently loading the default query. - [x] Aggregate unknown-key errors with every other structural issue; MCP currently stops at the first unsupported key. - [x] Add field-value type validation, for example reject size gt "large" and id contains {}. - [x] Warn about unobserved dynamic fields selected only as output. - [ ] Prove the full structural matrix through the MCP server/API end to end; the current broad matrix tests the local adapter. ## Query semantic states - [x] INCOMPATIBLE_OPERATOR is fatal for standard fields. - [x] UNOBSERVED_FIELD is a nonfatal warning. - [x] INCOMPATIBLE_OBSERVED_TYPES is a nonfatal warning. - [x] STRUCTURED_SORT_ORDER is a nonfatal warning. - [x] Every incompatible standard-field operator is collected. - [x] Fatal semantic errors retain nonfatal warnings. - [x] Successful warnings appear with exact query paths. - [ ] Reproduce every catalog-dependent semantic state through real MCP. - [x] Test repeated warnings at distinct query paths without accidental deduplication. ## Query execution and hydration - [x] Stale index revision. - [x] result one matching multiple files, including matchedCount. - [x] Candidate document limit of 1,000. - [x] Result payload limit of 16 MiB. - [x] Hydrated file count limit of 20. - [x] Per-file hydration limit of 1 MiB. - [x] Total hydration limit of 2 MiB. - [x] Range limit of 256 KiB. - [x] Binary content requested as text. - [x] Invalid UTF-8 during hydration. - [x] Missing, stale, or conflicting content identity. - [x] Canonical metadata/byte-count mismatch. - [x] Storage returning more bytes than requested. - [x] Missing content reader. - [x] Selected content not returned. - [x] Runtime Asset URL unavailable. - [x] Document graph document-count/per-file/total limits. - [x] Classify candidate-document and result-byte limits as CONTENT_LIMIT_EXCEEDED instead of INVALID_REQUEST. - [x] Give unavailable URL, missing reader/content, storage, selection, and assembly failures stable codes and exact asset/file details instead of INTERNAL_ERROR. - [x] Never silently turn an exact-file unavailable-content failure into an empty successful result. - [x] Add direct regression tests for duplicate identity, excess bytes, and canonical byte mismatch. - [ ] Reproduce every runtime/hydration state through real MCP. ## Scoping and aggregation - [x] Errors and warnings from multiple loaded result files are returned together. - [x] Nonfatal errors do not suppress successful query results. - [x] Unrelated published-query diagnostics do not replace current-query diagnostics. - [x] File path and available line/column are shown. - [x] Exact nested query paths are shown. - [x] Validate and report every pre-pagination matching file if that is the product definition. - [ ] Explain candidate metadata warnings when invalid metadata prevents deciding whether a file matches. - [ ] Stop including METADATA_PREPARATION_FAILED globally unless the UI labels its candidate scope. - [x] Aggregate reference errors instead of stopping at the first. - [x] Aggregate across validation phases where safe. - [x] Add a matrix for same-file multiple errors, multiple files, outside-page errors, filter-only candidate files, and unrelated queries. ## Transport, authorization, cancellation, and internal failures - [x] UNAUTHORIZED and FORBIDDEN. - [x] REQUEST_CANCELLED in query HTTP paths. - [x] Sanitized authenticated internal message. - [x] Generic unauthenticated/public internal message. - [x] Preserve structured causes for document-load HTTP status/body/metadata failures. - [x] Preserve structured causes for object-storage read failures. - [x] Preserve structured causes for graph selection/assembly failures. - [ ] Test MCP/API cancellation directly. - [ ] Reproduce authorization, cancellation, storage, and sanitized-internal states through real MCP where safe. ## MCP reproduction - [x] Real MCP reproduction: multiple Markdown YAML errors. - [x] Real MCP reproduction: multiple simultaneous structural query errors. - [x] MCP update returns Markdown and MDX source diagnostics. - [x] MCP upload returns Markdown and MDX source diagnostics. - [x] MCP download returns Markdown and MDX source diagnostics. - [x] Temporary remote fixtures are removed after reproduction. - [ ] Real MCP reproduction for every frontmatter code. - [ ] Real MCP reproduction for every MDX fatal/warning class. - [ ] Real MCP reproduction for every Content Block semantic diagnostic. - [ ] Real MCP reproduction for every reference/graph error. - [ ] Real MCP reproduction for every semantic query warning/error. - [ ] Real MCP reproduction for every hydration/runtime/limit error. - [ ] Real MCP reproduction for auth, cancellation, storage, and internal errors where safe. - [ ] A durable table-driven test proves code/severity/message/path parity between editor, query, Diagnostics, and MCP. - [x] Preserve stable top-level error identity and structured details through MCP instead of flattening known errors to MCP_TOOL_FAILED. - [x] Prove failed post-commit MCP feedback reports whether a mutation committed and returns enough identity for cleanup. ## Real MCP recheck and follow-up fixes (2026-09-03) Ran the branch CLI against the PR preview backend. The first run exposed that the saved CLI origin pointed to a different backend, so those results were discarded. Every MCP result below came from the PR preview. All temporary remote and local fixtures were removed afterward. The failures found in that run were then fixed and covered by regression tests. - [x] Exact broken Markdown query returns FRONTMATTER_INVALID with severity, filename, message, line, and column. - [x] Exact MDX query returns unsafe-mdx with severity, filename, message, line, and column. - [x] A no-extension query returns warnings from both matching Markdown and MDX files. - [x] A Markdown-only query excludes the unrelated MDX warning. - [x] Two fatal MDX files inside the returned page are both reported; matching warnings remain attached. - [x] Structural query validation returns three independent errors with exact paths in one call. - [x] Semantic query validation returns three fatal operator errors and two warnings with exact paths in one call. - [x] MCP upload, update, download, and exact-file query return multiple source diagnostics for the same MDX source. - [x] Query preview reports a missing Asset reference as ASSET_NOT_FOUND with the source file and reference. - [x] All temporary MCP fixtures were deleted and the saved CLI connection was restored. - [x] Fixed after recheck: validate every matching file before pagination, including broken files outside the returned page. - [x] Fixed after recheck: unrelated graph failures no longer block an exact-file query. - [x] Fixed after recheck: update, download, editor, and query paths use shared reference validation. - [x] Fixed after recheck: graph failures map to structured diagnostics with the exact file. - [x] Fixed after recheck: shared fatal UTF-8 validation runs before upload writes and preserves structured error identity. - [x] Fixed after recheck: MCP rejects filename/format mismatches. - [x] Fixed after recheck: MDX validation keeps sibling and nested warnings around unsupported components. - [x] Fixed after recheck: MCP preserves stable top-level error codes and structured details. - [x] Fixed after recheck: query and MCP serializers preserve full normalized diagnostic fields. - [x] Fixed after recheck: unknown keys are aggregated with all other structural query errors. - [x] Fixed after recheck: shared query validation covers literal types, output-only fields, and strict content options. - [x] Fixed after recheck: unavailable selected content returns an exact-file diagnostic instead of a silent empty result. ## Tests and delivery - [x] Full Builder suite: 2,707 tests. - [x] Content Engine suite: 445 tests. - [x] Asset uploader suite: 211 tests. - [x] Project runtime/MCP targeted suite: 128 tests. - [x] CLI/MCP suite: 1,051 tests. - [x] Relevant package typechecks, lint, generated docs, and diff checks. - [x] Documentation impact reviewed and generated Content Engine/MCP reference updated. - [x] Add the missing cross-surface error matrix described above. - [ ] Demonstrate red/green behavior for every new matrix case. - [ ] Run the complete agent evaluation suite after explicit approval. - [ ] Re-run all checks after the remaining checklist is implemented. ## Compatibility and risk Normal query result and expression-binding shapes remain unchanged. Diagnostics add optional fields. Existing public/pre-auth error redaction remains intact. The remaining unchecked items are broader exhaustive real-MCP coverage, parser-recovery limits, candidate-warning labeling, cancellation coverage, and agent evaluations.	2026-09-04 16:00:28 +02:00
Oleg Isonen	8718feb465	test: simplify browser testing infrastructure (#6271 ) ## Summary Unifies frontend testing around three intentional boundaries: - Vitest Node projects for pure and server tests - Vitest Browser Mode with Playwright/Chromium for DOM and component tests - Playwright Test for Builder end-to-end workflows The Builder no longer uses jsdom. Ordinary E2E workflows use an automatically managed isolated `page` fixture; explicit contexts remain only for simultaneous identities and suite-scoped setup. ## Technical choices - Split Builder tests into one Node project and independent browser projects so each Chromium/Vite process has a bounded workload. - Added shared Browser Mode configuration for the affected SDK React packages. - Preserve the Builder's production top-window bridge ownership; Browser Mode uses explicit test-only installation seams. - Centralized Playwright context options and kept action/navigation timeouts bounded. - Validate Playwright filters before Docker, migrations, browser installation, or Builder builds start. - Preserve the first failed E2E attempt's report, traces, screenshots, and videos when CI retries a shard. - Split static checks, typechecks, CLI tests, core package tests, browser package tests, and Builder projects into parallel CI jobs with five-minute hard ceilings. - Distribute Builder E2E tests across ten isolated shards with two workers; parallel suites create unique per-worker setup data. - Reuse color contrast DOM/canvas samples so the exhaustive theme sweep takes about four seconds instead of timing out at 30 seconds. - The removed cost is jsdom behavior drift, oversized serial check jobs, and unnecessary E2E startup for empty selections. Real browser startup and E2E environment setup remain. ## Todo - [x] Split Builder pure/server tests from browser/component tests. - [x] Migrate Builder DOM tests from jsdom to Vitest Browser Mode. - [x] Migrate affected SDK React DOM tests to the shared Browser Mode setup. - [x] Remove jsdom annotations, dependencies, and compatibility setup. - [x] Replace ordinary manual E2E contexts with the isolated `page` fixture. - [x] Keep explicit contexts only for concurrent identities and `beforeAll` setup. - [x] Preserve setup authentication isolation from workflow pages. - [x] Add bounded Playwright action and navigation timeouts. - [x] Preserve first-attempt E2E failure evidence across CI retry. - [x] Reject empty E2E selections before expensive environment setup. - [x] Parallelize repository checks into bounded CI shards. - [x] Isolate parallel E2E setup data across workers. - [x] Remove the color contrast sweep bottleneck. - [x] Update E2E contributor documentation. - [x] Review product documentation impact; no user-facing product docs are affected. - [x] Run the full repository tests and typechecks. - [x] Run formatting, lint, package-boundary, generated-file, and fixture checks. - [x] Complete final correctness and simplification review. ## Verification - `pnpm --filter @webstudio-is/builder test` - `pnpm -r typecheck` - `pnpm lint` - `pnpm check:package-boundaries` - `pnpm check:generated-api` - `pnpm check:generated-docs` - `pnpm fixtures` - Design-system: 222 tests pass locally. - Builder: all seven Vitest projects pass locally. - The formerly slow E2E shard passes locally in 1m55s. - Full database-backed Builder E2E execution passes in CI. - All CI jobs pass; the slowest completed in 4m58s. - No fixture inputs/outputs or generated files changed. Ref #6270	2026-08-31 17:17:28 +02:00

Author

SHA1

Message

Date

Oleg Isonen

3d0a0657aa

feat: Support async resource dependencies (#6030 )

## Summary

Make `Resource<Document>` the lazy document-resolution abstraction
shared by Content Engine graphs, generated sites, and Builder previews.
Expressions use one async computation interface: referenced resource
documents resolve first, independent requests run concurrently, and
unused resources stay unloaded.

Ref #6029
Ref #6031

## Implementation checklist

- [x] Define one resource contract for synchronous or asynchronous
document resolution.
- [x] Resolve explicit roots lazily with memoization, bounded
concurrency, cancellation, and structured errors.
- [x] Preserve cycle, duplicate, and missing-resource diagnostics.
- [x] Extract resource dependencies with the existing expression AST.
- [x] Discover consumed resource roots from the current page tree and
page metadata.
- [x] Generate a lazy request graph for published page data, including
independent requests.
- [x] Resolve dependency documents before evaluating dependent request
expressions.
- [x] Integrate dependency-first loading with Builder preview caching
and invalidation.
- [x] Expose safe Resource bindings in the editor and reject
self/indirect cycles.
- [x] Adapt REST, GraphQL, Assets, date, and sitemap requests through
the same Resource contract.
- [x] Cover laziness, chains, shared dependencies, concurrency,
invalidation, cancellation, failures, and cycles with fail/pass tests.
- [x] Review documentation impact and document dependent Resources in
the Data variables guide.
- [x] Track collection-item and component-parameter-dependent resources
as follow-up #6031.
- [x] Regenerate and review affected fixtures.
- [x] Resolve action-resource request dependencies when an action is
invoked.
- [x] Reproduce and fix cached form submissions and multi-root
dependency-cycle hangs.
- [x] Document uncached form actions and regenerate affected fixtures.
- [x] Preserve single and multi-selected content until a cut
successfully writes to the clipboard.
- [x] Preserve boolean expression values in the search-exclusion
checkbox.
- [x] Cancel cut deletion if project data changes while preparing or
writing the clipboard; cover content/property edits and project switches
for single and multi-selection, and document the retry behavior.

- [x] Validate pending cuts after the final await, immediately before
deletion, and reproduce the queued-update race for single and
multi-selection.

## Technical direction

The persisted, revisioned `DocumentGraph` remains the Content Engine
artifact. `Resource<Document>` is the execution abstraction for
obtaining a document; its resolver evaluates only the dependency closure
reachable from requested roots. Static and remote documents therefore
differ by Resource implementation, not by expression value type.

Generated pages always emit a `ResourceRequestGraph`. Each node owns its
output name, dependencies, and request factory. Roots come from resource
variables actually referenced by the current page's metadata, props,
actions, or expression children. Unused and off-page resources remain
unresolved. The loader still accepts legacy request maps for
compatibility.

Builder preview uses the same page-root discovery as published
generation. Its existing cache remains the lifecycle boundary: cached
dependency documents unlock dependent requests, while invalidation
removes downstream requests until their inputs resolve again. The
separate canvas frame recomputes its local plan from the synchronized
cache so rendered expressions receive those documents.

Action requests and their dependencies resolve when a form is submitted.
Each submission sends a new action request, bypassing Webstudio's
resource cache. Dependency requests still use their configured cache
lifetime.

## Review simplifications

- Consolidated page resource-root discovery in the SDK and reused it in
Builder.
- Scoped root and action discovery to the current page tree.
- Removed the redundant immediate-resource helper; `Resource.resolve`
already accepts `Document | Promise<Document>`.
- Moved output names onto graph nodes and removed the duplicate
graph-level map.
- Removed unused planner output and corrected transitive cycle discovery
to include only Resource data sources.
- Regenerated fixtures so independent resources also use lazy graph
execution.
- Reused one abort-aware concurrency limiter for document sessions and
Resource resolution.
- Collapsed duplicate generator action scans and removed redundant
generated-resource bookkeeping.
- Unified legacy request maps and request graphs through one resolution
path.
- Shared the 20-request concurrency limit between SDK, Builder preview,
and the server batch boundary.
- Preserved dependency failure paths and original causes, including the
first cancellation reason.
- Removed invalid-request logging that could expose Resource headers or
request bodies.
- Made editor cycle detection alias-safe by comparing underlying
Resource IDs.
- Memoized page settings values to prevent repeated async validation
renders.

## Verification

Latest local verification for `bbca6395e3`:

- Both queued-update regression cases failed before the fix and passed
afterward. Validation now runs synchronously immediately before
deletion, after the final await, in both cut paths.
- Passed: 255 copy/paste and command tests, Builder typecheck, lint,
package boundaries, formatting, and diff checks.
- Reviewed documentation: the existing copy/paste safety description
remains accurate. Fixture inputs and outputs are unaffected; no fixture
regeneration was needed. Agent evaluations were not run because separate
approval is required.
- CI for the new commit is pending.

Previous local verification for `ffbc75b152`:

- Six regression cases failed before the fix and passed afterward:
content edits, property edits, and project switches during a pending
cut, with both single and multiple selections.
- Passed: 253 copy/paste and command tests, Builder typecheck, lint,
package boundaries, source formatting, and diff checks.
- An unchanged content-block test file initially failed to import in the
browser. Its isolated rerun and the full 253-test rerun passed without
code changes.
- The guard conservatively treats any project-data change as a reason to
keep the clipboard copy without deleting content. It shows a retry
message. Copy/paste documentation now describes this behavior.
- Fixture inputs and outputs are unaffected; fixtures were not
regenerated. Agent evaluations were not run because separate approval is
required. CI for the new commit is pending.

Previous local verification for `3eb580697c`:

- Reproduced both regressions with failing tests before fixing them:
premature deletion during cut and false search exclusion appearing
checked.
- Passed: 203 copy/paste and page-settings tests, Builder typecheck,
lint, package boundaries, formatting, and diff checks.
- Cut coverage includes keyboard and programmatic entry points, single
and multiple selections, delayed successful writes, rejected writes, and
disabled copying.
- Reviewed documentation impact: these fixes restore existing behavior;
no documentation update is needed. Only Builder UI and clipboard files
changed; fixture inputs and outputs are unaffected, so fixtures were not
regenerated for this commit.
- CI status for this older commit was recorded as pending at handoff.
Agent evaluations were not run; they require separate approval.

Previous local verification for `858b348ad`, rebased onto `cf606ca4c`:

- Regression tests failed before each fix and passed afterward: repeated
submissions in both generated route templates, plus three
multi-root/branching cycle cases.
- Passed: 590 SDK tests, 1,129 CLI tests, 72 focused Builder tests,
SDK/CLI/Builder typechecks, lint, package boundaries, and fixture
link/sync/build.
- Initial concurrent runs hit timeouts. The complete CLI suite passed
with one worker; the focused Builder suites passed without file
parallelism.
- Documentation and generated fixture changes reviewed. Agent
evaluations were not run; they require separate approval.
- CI for `858b348ad` passed: Main (including all six E2E shards), Visual
Regression, Lint PR, and Check submodules.

Earlier verification (before this update):

TDD fail/pass evidence includes resolver creation, AST dependency
discovery, published graph reachability, dependency-gated request
generation, Builder dependency-first planning, Resource editor cycle
filtering, page-tree root scoping, off-page action isolation, transitive
Resource-only cycle discovery, abort/timeout precedence, server batch
limits, and the rebased SSG fixture consuming its lazy Resource.

- `pnpm checks` passed after rebasing onto `ce84ddcf12`; it covered
workspace tests, typechecks, lint, package boundaries, generated
API/docs validation, and fixture link/sync/build.
- Earlier focused checks passed: Builder typecheck, 12
page-settings/pages unit tests, lint, formatting, and `git diff
--check`.
- Local generated-resource E2E passed for HTTP, GraphQL, and
current-date resources (3 tests).
- The page-actions and Marketplace page-settings persistence E2Es both
pass locally.
- CI for `6b8b08a70` passed: Main workflow (including all six Builder
E2E shards), Visual Regression, Lint PR, and Check submodules.
- Agent evaluations were not run because they require separate explicit
approval.

## Current behavior and limitations

- A failed dependency blocks its dependents and reports the resource
path and cause.
- One published resolution call is the memoization boundary; Builder
maps dependency readiness onto its versioned request-cache lifecycle.
- Scoped collection-item and component-parameter resources remain in
#6031.
- Builder skips cyclic dependency paths without blocking independent
resources.

2026-09-22 21:04:42 +02:00

Oleg Isonen

55920c57c4

fix: Show all Markdown and Assets query diagnostics (#6308 )

Ref #6305

## Outcome

Show actionable Markdown, MDX, and Assets query problems in the editor,
query Diagnostics, API, and MCP.

The current implementation substantially improves source diagnostics,
query error aggregation, query scoping, exact paths, MCP feedback, and
loading states. The exhaustive audit below also records what is not
complete yet.

## Completion rule

An error state is complete only when all applicable checks pass:

- [x] The editor uses the production validator and shows every issue for
the open file.
- [x] Query execution uses the same validation and returns every issue
for every file that matches the query, not only the returned page.
- [ ] MCP can reproduce the state through a real tool call and returns
the same code, severity, message, file, and location.
- [x] Diagnostics shows fatal errors and nonfatal warnings in the
correct place.
- [x] Exact file is always present; line and column are present whenever
the parser can determine them.
- [x] Multiple issues in the same file, multiple files, and multiple
validation phases are returned together.
- [ ] A regression test proves failure without the behavior and success
with it.

## Cross-surface contract

### Required validator parity

- [x] Editor, query execution, and MCP file operations invoke the same
exported production file-validation pipeline for byte decoding,
source/frontmatter, Markdown/MDX, Content Block semantics, and
references. Copied schemas, adapter-only checks, or reimplementations do
not count as parity.
- [x] Query builder, query execution, `validate-asset-query`, and
`preview-asset-query` invoke the same exported structural and
catalog-aware semantic query validator with the same project snapshot
and context.
- [x] Every surface preserves the same complete, deterministically
ordered diagnostic array: severity, code, message, phase/scope, asset
ID, file path, line/column/range, query path, and
fatal-versus-warning/result behavior. UI, API, and MCP serializers must
not drop or remap fields.
- [ ] A table-driven contract suite calls the real editor, query, API,
and MCP entry points—including MCP upload, update, download, validate,
and preview—and compares full normalized arrays for multiple errors,
multiple warnings, multiple files, and pre-pagination matches.

- [x] Markdown and MDX source syntax/frontmatter use
validateTextAssetSource in editor, query compilation, and MCP file
tools.
- [x] Query request execution and MCP local parsing use
assetQueryRequest.
- [x] Fatal query setup errors retain all detected semantic errors and
accompanying warnings.
- [x] Nonfatal query warnings retain a stable code and exact query path.
- [x] Source diagnostics retain severity, phase, code, asset ID, path,
and available source position.
- [x] Query Diagnostics excludes diagnostics produced only by unrelated
published queries.
- [x] Query, Preview, and Diagnostics tabs show loading states.
- [x] Authenticated diagnostics can show a sanitized internal message;
public/pre-auth errors remain redacted.
- [x] Share byte-level UTF-8 validation instead of decoding separately
in query and MCP.
- [x] Make the query editor and MCP call one literal shared
query-validation function before execution.
- [x] Share cross-file reference validation across editor, query, and
MCP file operations.
- [x] Share contextual Content Block/template validation across all
applicable surfaces.
- [x] Preserve an invalid stored query in the editor and explain it
instead of replacing it with defaults.
- [x] Define and enforce whether “matches the query” includes every
pre-pagination match; current source validation follows
returned/paginated IDs.
- [x] Continue through source, reference, and runtime phases where safe
so one phase does not hide later issues.
- [x] Map every user-triggerable graph/storage/runtime failure to a
structured public error instead of INTERNAL_ERROR.
- [ ] Label candidate-file warnings separately from returned-file
diagnostics.

## File transport and encoding

- [x] Empty Markdown and MDX are accepted.
- [x] UTF-8 BOM and CRLF are accepted.
- [x] Whole Markdown/MDX file over 1 MiB is detected.
- [x] Missing or unavailable selected content is reported.
- [x] Invalid UTF-8 frontmatter returns the same structured diagnostic
in query and MCP.
- [x] Invalid UTF-8 body returns the same structured diagnostic in query
and MCP.
- [x] MCP download must use fatal UTF-8 decoding rather than replacement
characters.
- [x] Validate bytes before committing an upload, or return the
committed Asset ID and explicit cleanup state when post-commit feedback
fails.
- [x] Reject filename/format mismatches instead of allowing either hint
to bypass Markdown/MDX validation.
- [x] Storage status, missing body, invalid metadata, revision mismatch,
format mismatch, short read, and excess read each return structured file
context.
- [x] Content changing during validation has a clear retryable conflict
diagnostic.

## Frontmatter

- [x] No frontmatter.
- [x] Empty frontmatter.
- [x] Opening delimiter without a closing delimiter.
- [x] Frontmatter over 64 KiB.
- [x] Invalid YAML grammar.
- [x] Multiple YAML parser errors.
- [x] Duplicate keys.
- [x] YAML aliases.
- [x] Unsupported YAML tags.
- [x] Scalar or array root instead of an object.
- [x] Non-JSON values, NaN, Infinity, and unsafe integers.
- [x] Nesting deeper than 8.
- [x] More than 256 fields/array entries.
- [x] One string over 16 KiB.
- [x] Normalized indexed properties over 64 KiB.
- [x] Frontmatter warnings remain nonfatal and visible.
- [x] Collect all post-parse normalization/limit errors; this phase
currently stops at the first.
- [ ] Prove every state above through real MCP upload, update, download,
query, and Diagnostics paths.

## Markdown body

- [x] Treat ordinary CommonMark/GFM as permissive; malformed-looking
Markdown and raw HTML are not automatically syntax errors.
- [x] Preserve and validate the body when frontmatter is invalid.
- [x] Detect the 1 MiB body/file limit.
- [x] Handle body extraction and bounded excerpts.
- [x] Return a shared structured invalid-UTF-8 body error from every
byte-reading surface.
- [ ] Distinguish author errors from internal
excerpt-limit/configuration errors.
- [ ] Prove body-size and decoding states through real MCP and
Diagnostics.

## MDX grammar and safety

- [x] Malformed MDX/JSX grammar.
- [x] Imports.
- [x] Exports.
- [x] Executable flow expressions.
- [x] Executable text expressions.
- [x] JSX expression attributes.
- [x] JSX spread attributes.
- [x] Arbitrary components.
- [x] HTML-looking JSX outside ws.element.
- [x] Unsupported/raw MDX node types.
- [x] Duplicate attributes and normalized alias collisions.
- [x] Event-handler attributes.
- [x] srcdoc, srcset, and dangerouslySetInnerHTML.
- [x] Unsafe URL protocols in link/source/action-like props.
- [x] Forbidden tags: base, link, meta, noscript, script, style,
template, title.
- [x] Conflicting, empty, or non-static ws:name and ws:tag.
- [x] More than 20,000 nodes.
- [x] Nesting deeper than 100.
- [x] More than 4,000 props.
- [x] Multiple recoverable unsafe-mdx warnings are collected.
- [ ] Collect multiple fatal grammar/structural errors when parser
recovery can do so; unrecoverable parsing currently stops at the first
fatal error.
- [x] Preserve sibling expression warnings when unsupported JSX is also
present; {first()}, <Card />, {second()} currently reports only the Card
warning.
- [ ] Prove every state above through real MCP and Diagnostics, not only
validator unit tests.

## Content Block semantics

- [x] Unresolved template warning exists.
- [x] Unknown template prop warning exists.
- [x] Incompatible template prop warning exists.
- [x] Design-only template prop warning exists.
- [x] Stale template prop warning exists.
- [x] Invalid materialized HTML content model error exists.
- [x] Run these checks for an MDX file even when no active editor
Content Block root has materialized it.
- [x] Run applicable semantic checks in query Diagnostics.
- [x] Run applicable semantic checks for MCP upload as well as
contextual update/download.
- [x] Return all Content Block frontmatter diagnostics; one path still
throws only diagnostics[0].
- [ ] Add real MCP and UI coverage for every semantic state.

## Cross-file references

- [x] Non-string $ref marker.
- [x] Invalid reference URL.
- [x] Invalid fragment encoding.
- [x] Invalid fragment.
- [x] Invalid JSON Pointer.
- [x] Missing referenced document.
- [x] Missing referenced JSON path.
- [x] Missing referenced local Asset warning.
- [x] Unsupported target representation.
- [x] Reference cycle.
- [x] Duplicate document ID or URL.
- [x] Missing graph source or target.
- [x] Root not found.
- [x] Duplicate/unused/unresolved reference.
- [x] Invalid resolved value.
- [x] Content identity conflict.
- [x] Target revision mismatch.
- [x] Representation selector required/invalid.
- [ ] Diagnose {$ref: value, extraKey: value} as an invalid marker or
explicitly document it as ordinary data.
- [x] Collect all bad references and targets instead of stopping at the
first graph failure.
- [x] Validate references from the editor and MCP file tools, not only
query compilation.
- [x] Map graph compilation/resolution, Markdown-document, and
JSON-document errors to exact structured Diagnostics rows.
- [x] Prevent an unrelated file's graph error or duplicate canonical URL
from blocking an exact-file query.
- [ ] Prove missing targets, cycles, bad representations, and bad JSON
paths through real MCP.

## Query request and structure

- [x] Invalid JSON, invalid request UTF-8, empty/non-object request, and
request over 512 KiB.
- [x] Missing query and unknown request/query keys.
- [x] Invalid result mode.
- [x] Malformed where/all/any/filter shape.
- [x] More than 32 filters or children.
- [x] Filter nesting deeper than 8.
- [x] Empty field path or segment.
- [x] Field path deeper than 9.
- [x] Bare properties path.
- [x] Unknown or nested standard field.
- [x] Missing or unknown operator.
- [x] Non-JSON operator value.
- [x] Invalid in value or more than 1,000 in values.
- [x] Non-boolean exists/isEmpty value.
- [x] More than 8 sort entries.
- [x] Invalid sort field or direction.
- [x] first/last without a sort.
- [x] Invalid output mode or includeMetadata.
- [x] Duplicate output fields or more than 256 fields.
- [x] No selected output.
- [x] Negative, fractional, or over-1,000 limit.
- [x] Negative, fractional, or over-1,000 offset.
- [x] Invalid content mode.
- [x] Invalid full/markdown-body-ref maxBytes outside 1..1 MiB.
- [x] Invalid range offset/length or range over 256 KiB.
- [x] Empty or over-255-character index revision.
- [x] Multiple simultaneous schema issues retain exact query paths.
- [x] Make content option objects strict; unknown keys are currently
stripped.
- [x] Show an invalid persisted resource body rather than silently
loading the default query.
- [x] Aggregate unknown-key errors with every other structural issue;
MCP currently stops at the first unsupported key.
- [x] Add field-value type validation, for example reject size gt
"large" and id contains {}.
- [x] Warn about unobserved dynamic fields selected only as output.
- [ ] Prove the full structural matrix through the MCP server/API end to
end; the current broad matrix tests the local adapter.

## Query semantic states

- [x] INCOMPATIBLE_OPERATOR is fatal for standard fields.
- [x] UNOBSERVED_FIELD is a nonfatal warning.
- [x] INCOMPATIBLE_OBSERVED_TYPES is a nonfatal warning.
- [x] STRUCTURED_SORT_ORDER is a nonfatal warning.
- [x] Every incompatible standard-field operator is collected.
- [x] Fatal semantic errors retain nonfatal warnings.
- [x] Successful warnings appear with exact query paths.
- [ ] Reproduce every catalog-dependent semantic state through real MCP.
- [x] Test repeated warnings at distinct query paths without accidental
deduplication.

## Query execution and hydration

- [x] Stale index revision.
- [x] result one matching multiple files, including matchedCount.
- [x] Candidate document limit of 1,000.
- [x] Result payload limit of 16 MiB.
- [x] Hydrated file count limit of 20.
- [x] Per-file hydration limit of 1 MiB.
- [x] Total hydration limit of 2 MiB.
- [x] Range limit of 256 KiB.
- [x] Binary content requested as text.
- [x] Invalid UTF-8 during hydration.
- [x] Missing, stale, or conflicting content identity.
- [x] Canonical metadata/byte-count mismatch.
- [x] Storage returning more bytes than requested.
- [x] Missing content reader.
- [x] Selected content not returned.
- [x] Runtime Asset URL unavailable.
- [x] Document graph document-count/per-file/total limits.
- [x] Classify candidate-document and result-byte limits as
CONTENT_LIMIT_EXCEEDED instead of INVALID_REQUEST.
- [x] Give unavailable URL, missing reader/content, storage, selection,
and assembly failures stable codes and exact asset/file details instead
of INTERNAL_ERROR.
- [x] Never silently turn an exact-file unavailable-content failure into
an empty successful result.
- [x] Add direct regression tests for duplicate identity, excess bytes,
and canonical byte mismatch.
- [ ] Reproduce every runtime/hydration state through real MCP.

## Scoping and aggregation

- [x] Errors and warnings from multiple loaded result files are returned
together.
- [x] Nonfatal errors do not suppress successful query results.
- [x] Unrelated published-query diagnostics do not replace current-query
diagnostics.
- [x] File path and available line/column are shown.
- [x] Exact nested query paths are shown.
- [x] Validate and report every pre-pagination matching file if that is
the product definition.
- [ ] Explain candidate metadata warnings when invalid metadata prevents
deciding whether a file matches.
- [ ] Stop including METADATA_PREPARATION_FAILED globally unless the UI
labels its candidate scope.
- [x] Aggregate reference errors instead of stopping at the first.
- [x] Aggregate across validation phases where safe.
- [x] Add a matrix for same-file multiple errors, multiple files,
outside-page errors, filter-only candidate files, and unrelated queries.

## Transport, authorization, cancellation, and internal failures

- [x] UNAUTHORIZED and FORBIDDEN.
- [x] REQUEST_CANCELLED in query HTTP paths.
- [x] Sanitized authenticated internal message.
- [x] Generic unauthenticated/public internal message.
- [x] Preserve structured causes for document-load HTTP
status/body/metadata failures.
- [x] Preserve structured causes for object-storage read failures.
- [x] Preserve structured causes for graph selection/assembly failures.
- [ ] Test MCP/API cancellation directly.
- [ ] Reproduce authorization, cancellation, storage, and
sanitized-internal states through real MCP where safe.

## MCP reproduction

- [x] Real MCP reproduction: multiple Markdown YAML errors.
- [x] Real MCP reproduction: multiple simultaneous structural query
errors.
- [x] MCP update returns Markdown and MDX source diagnostics.
- [x] MCP upload returns Markdown and MDX source diagnostics.
- [x] MCP download returns Markdown and MDX source diagnostics.
- [x] Temporary remote fixtures are removed after reproduction.
- [ ] Real MCP reproduction for every frontmatter code.
- [ ] Real MCP reproduction for every MDX fatal/warning class.
- [ ] Real MCP reproduction for every Content Block semantic diagnostic.
- [ ] Real MCP reproduction for every reference/graph error.
- [ ] Real MCP reproduction for every semantic query warning/error.
- [ ] Real MCP reproduction for every hydration/runtime/limit error.
- [ ] Real MCP reproduction for auth, cancellation, storage, and
internal errors where safe.
- [ ] A durable table-driven test proves code/severity/message/path
parity between editor, query, Diagnostics, and MCP.
- [x] Preserve stable top-level error identity and structured details
through MCP instead of flattening known errors to MCP_TOOL_FAILED.
- [x] Prove failed post-commit MCP feedback reports whether a mutation
committed and returns enough identity for cleanup.

## Real MCP recheck and follow-up fixes (2026-09-03)

Ran the branch CLI against the PR preview backend. The first run exposed
that the saved CLI origin pointed to a different backend, so those
results were discarded. Every MCP result below came from the PR preview.
All temporary remote and local fixtures were removed afterward. The
failures found in that run were then fixed and covered by regression
tests.

- [x] Exact broken Markdown query returns FRONTMATTER_INVALID with
severity, filename, message, line, and column.
- [x] Exact MDX query returns unsafe-mdx with severity, filename,
message, line, and column.
- [x] A no-extension query returns warnings from both matching Markdown
and MDX files.
- [x] A Markdown-only query excludes the unrelated MDX warning.
- [x] Two fatal MDX files inside the returned page are both reported;
matching warnings remain attached.
- [x] Structural query validation returns three independent errors with
exact paths in one call.
- [x] Semantic query validation returns three fatal operator errors and
two warnings with exact paths in one call.
- [x] MCP upload, update, download, and exact-file query return multiple
source diagnostics for the same MDX source.
- [x] Query preview reports a missing Asset reference as ASSET_NOT_FOUND
with the source file and reference.
- [x] All temporary MCP fixtures were deleted and the saved CLI
connection was restored.
- [x] Fixed after recheck: validate every matching file before
pagination, including broken files outside the returned page.
- [x] Fixed after recheck: unrelated graph failures no longer block an
exact-file query.
- [x] Fixed after recheck: update, download, editor, and query paths use
shared reference validation.
- [x] Fixed after recheck: graph failures map to structured diagnostics
with the exact file.
- [x] Fixed after recheck: shared fatal UTF-8 validation runs before
upload writes and preserves structured error identity.
- [x] Fixed after recheck: MCP rejects filename/format mismatches.
- [x] Fixed after recheck: MDX validation keeps sibling and nested
warnings around unsupported components.
- [x] Fixed after recheck: MCP preserves stable top-level error codes
and structured details.
- [x] Fixed after recheck: query and MCP serializers preserve full
normalized diagnostic fields.
- [x] Fixed after recheck: unknown keys are aggregated with all other
structural query errors.
- [x] Fixed after recheck: shared query validation covers literal types,
output-only fields, and strict content options.
- [x] Fixed after recheck: unavailable selected content returns an
exact-file diagnostic instead of a silent empty result.

## Tests and delivery

- [x] Full Builder suite: 2,707 tests.
- [x] Content Engine suite: 445 tests.
- [x] Asset uploader suite: 211 tests.
- [x] Project runtime/MCP targeted suite: 128 tests.
- [x] CLI/MCP suite: 1,051 tests.
- [x] Relevant package typechecks, lint, generated docs, and diff
checks.
- [x] Documentation impact reviewed and generated Content Engine/MCP
reference updated.
- [x] Add the missing cross-surface error matrix described above.
- [ ] Demonstrate red/green behavior for every new matrix case.
- [ ] Run the complete agent evaluation suite after explicit approval.
- [ ] Re-run all checks after the remaining checklist is implemented.

## Compatibility and risk

Normal query result and expression-binding shapes remain unchanged.
Diagnostics add optional fields. Existing public/pre-auth error
redaction remains intact. The remaining unchecked items are broader
exhaustive real-MCP coverage, parser-recovery limits, candidate-warning
labeling, cancellation coverage, and agent evaluations.

2026-09-04 16:00:28 +02:00

Oleg Isonen

8718feb465

test: simplify browser testing infrastructure (#6271 )

## Summary

Unifies frontend testing around three intentional boundaries:

- Vitest Node projects for pure and server tests
- Vitest Browser Mode with Playwright/Chromium for DOM and component
tests
- Playwright Test for Builder end-to-end workflows

The Builder no longer uses jsdom. Ordinary E2E workflows use an
automatically managed isolated `page` fixture; explicit contexts remain
only for simultaneous identities and suite-scoped setup.

## Technical choices

- Split Builder tests into one Node project and independent browser
projects so each Chromium/Vite process has a bounded workload.
- Added shared Browser Mode configuration for the affected SDK React
packages.
- Preserve the Builder's production top-window bridge ownership; Browser
Mode uses explicit test-only installation seams.
- Centralized Playwright context options and kept action/navigation
timeouts bounded.
- Validate Playwright filters before Docker, migrations, browser
installation, or Builder builds start.
- Preserve the first failed E2E attempt's report, traces, screenshots,
and videos when CI retries a shard.
- Split static checks, typechecks, CLI tests, core package tests,
browser package tests, and Builder projects into parallel CI jobs with
five-minute hard ceilings.
- Distribute Builder E2E tests across ten isolated shards with two
workers; parallel suites create unique per-worker setup data.
- Reuse color contrast DOM/canvas samples so the exhaustive theme sweep
takes about four seconds instead of timing out at 30 seconds.
- The removed cost is jsdom behavior drift, oversized serial check jobs,
and unnecessary E2E startup for empty selections. Real browser startup
and E2E environment setup remain.

## Todo

- [x] Split Builder pure/server tests from browser/component tests.
- [x] Migrate Builder DOM tests from jsdom to Vitest Browser Mode.
- [x] Migrate affected SDK React DOM tests to the shared Browser Mode
setup.
- [x] Remove jsdom annotations, dependencies, and compatibility setup.
- [x] Replace ordinary manual E2E contexts with the isolated `page`
fixture.
- [x] Keep explicit contexts only for concurrent identities and
`beforeAll` setup.
- [x] Preserve setup authentication isolation from workflow pages.
- [x] Add bounded Playwright action and navigation timeouts.
- [x] Preserve first-attempt E2E failure evidence across CI retry.
- [x] Reject empty E2E selections before expensive environment setup.
- [x] Parallelize repository checks into bounded CI shards.
- [x] Isolate parallel E2E setup data across workers.
- [x] Remove the color contrast sweep bottleneck.
- [x] Update E2E contributor documentation.
- [x] Review product documentation impact; no user-facing product docs
are affected.
- [x] Run the full repository tests and typechecks.
- [x] Run formatting, lint, package-boundary, generated-file, and
fixture checks.
- [x] Complete final correctness and simplification review.

## Verification

- `pnpm --filter @webstudio-is/builder test`
- `pnpm -r typecheck`
- `pnpm lint`
- `pnpm check:package-boundaries`
- `pnpm check:generated-api`
- `pnpm check:generated-docs`
- `pnpm fixtures`
- Design-system: 222 tests pass locally.
- Builder: all seven Vitest projects pass locally.
- The formerly slow E2E shard passes locally in 1m55s.
- Full database-backed Builder E2E execution passes in CI.
- All CI jobs pass; the slowest completed in 4m58s.
- No fixture inputs/outputs or generated files changed.

Ref #6270

2026-08-31 17:17:28 +02:00