mirror of
https://github.com/webstudio-is/webstudio.git
synced 2026-11-10 22:44:24 +09:00
codex/headers-settings-docs
55
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Oleg Isonen
|
f467e415e1 | refactor: share published route parameter conversion | ||
|
Oleg Isonen
|
d78e770d82 |
perf: Keep article indexing out of publish API requests (#6427)
Ref #6416 The publish workflow was doing too much work in one short-lived API request. This moves content indexing into the build runner and keeps article bodies out of hosted/SSR sync. ## What changed - The build-bundle API can return the lightweight project bundle without preparing a full content index. - The CLI downloads required assets directly from storage and builds the index locally for SSG. Asset downloads remain bounded to 10 concurrent requests. - Hosted/SSR sync skips Markdown and MDX article downloads and does not compile an article index. It still downloads regular assets needed by the build. - Markdown body queries use stored frontmatter and fetch only the requested article at runtime. MDX still needs source downloads and compilation for SSG. - Public MCP and content-engine docs explain the Markdown/MDX tradeoff. This removes article parsing and article-body transfer from the tRPC request. SSG still downloads article sources from storage and compiles MDX in the build runner. Hosted runtime article reads remain on demand. ## Checklist - [x] Keep the tRPC bundle request limited to project metadata and necessary records. - [x] Prepare SSG content indexes in the build runner; keep hosted article bodies out of sync. - [x] Cover no-body query preparation, direct asset reads, and SSG-versus-hosted sync behavior with regression tests. - [x] Update public CLI, MCP, and content-engine guidance. - [x] Regenerate and verify fixtures. - [x] Run workspace tests, typechecks, lint, formatting, package-boundary, generated API/docs, and fixture-size checks. - [ ] Verify the deployed publish flow has comfortable margin below the hosted API timeout. ## Verification - `pnpm -r --workspace-concurrency=1 test`: passed, including all 3,082 Builder tests and 1,150 CLI tests. - `pnpm -r typecheck`, `pnpm lint`, `pnpm check:package-boundaries`, `pnpm check:generated-api`, `pnpm check:generated-docs`, `pnpm check:fixture-sizes`, and `pnpm exec oxfmt --check`: passed. - The new hosted article-download regression test was confirmed to fail with the old behavior and pass after the fix. - A deployed end-to-end publish timing check is still needed, so this PR remains draft. |
||
|
Oleg Isonen
|
9845517029 |
fix: hydrate published Markdown assets (#6422)
Migrated from #6421 to the short .staging branch name. |
||
|
Oleg Isonen
|
3d0a0657aa |
feat: Support async resource dependencies (#6030)
## Summary Make `Resource<Document>` the lazy document-resolution abstraction shared by Content Engine graphs, generated sites, and Builder previews. Expressions use one async computation interface: referenced resource documents resolve first, independent requests run concurrently, and unused resources stay unloaded. Ref #6029 Ref #6031 ## Implementation checklist - [x] Define one resource contract for synchronous or asynchronous document resolution. - [x] Resolve explicit roots lazily with memoization, bounded concurrency, cancellation, and structured errors. - [x] Preserve cycle, duplicate, and missing-resource diagnostics. - [x] Extract resource dependencies with the existing expression AST. - [x] Discover consumed resource roots from the current page tree and page metadata. - [x] Generate a lazy request graph for published page data, including independent requests. - [x] Resolve dependency documents before evaluating dependent request expressions. - [x] Integrate dependency-first loading with Builder preview caching and invalidation. - [x] Expose safe Resource bindings in the editor and reject self/indirect cycles. - [x] Adapt REST, GraphQL, Assets, date, and sitemap requests through the same Resource contract. - [x] Cover laziness, chains, shared dependencies, concurrency, invalidation, cancellation, failures, and cycles with fail/pass tests. - [x] Review documentation impact and document dependent Resources in the Data variables guide. - [x] Track collection-item and component-parameter-dependent resources as follow-up #6031. - [x] Regenerate and review affected fixtures. - [x] Resolve action-resource request dependencies when an action is invoked. - [x] Reproduce and fix cached form submissions and multi-root dependency-cycle hangs. - [x] Document uncached form actions and regenerate affected fixtures. - [x] Preserve single and multi-selected content until a cut successfully writes to the clipboard. - [x] Preserve boolean expression values in the search-exclusion checkbox. - [x] Cancel cut deletion if project data changes while preparing or writing the clipboard; cover content/property edits and project switches for single and multi-selection, and document the retry behavior. - [x] Validate pending cuts after the final await, immediately before deletion, and reproduce the queued-update race for single and multi-selection. ## Technical direction The persisted, revisioned `DocumentGraph` remains the Content Engine artifact. `Resource<Document>` is the execution abstraction for obtaining a document; its resolver evaluates only the dependency closure reachable from requested roots. Static and remote documents therefore differ by Resource implementation, not by expression value type. Generated pages always emit a `ResourceRequestGraph`. Each node owns its output name, dependencies, and request factory. Roots come from resource variables actually referenced by the current page's metadata, props, actions, or expression children. Unused and off-page resources remain unresolved. The loader still accepts legacy request maps for compatibility. Builder preview uses the same page-root discovery as published generation. Its existing cache remains the lifecycle boundary: cached dependency documents unlock dependent requests, while invalidation removes downstream requests until their inputs resolve again. The separate canvas frame recomputes its local plan from the synchronized cache so rendered expressions receive those documents. Action requests and their dependencies resolve when a form is submitted. Each submission sends a new action request, bypassing Webstudio's resource cache. Dependency requests still use their configured cache lifetime. ## Review simplifications - Consolidated page resource-root discovery in the SDK and reused it in Builder. - Scoped root and action discovery to the current page tree. - Removed the redundant immediate-resource helper; `Resource.resolve` already accepts `Document | Promise<Document>`. - Moved output names onto graph nodes and removed the duplicate graph-level map. - Removed unused planner output and corrected transitive cycle discovery to include only Resource data sources. - Regenerated fixtures so independent resources also use lazy graph execution. - Reused one abort-aware concurrency limiter for document sessions and Resource resolution. - Collapsed duplicate generator action scans and removed redundant generated-resource bookkeeping. - Unified legacy request maps and request graphs through one resolution path. - Shared the 20-request concurrency limit between SDK, Builder preview, and the server batch boundary. - Preserved dependency failure paths and original causes, including the first cancellation reason. - Removed invalid-request logging that could expose Resource headers or request bodies. - Made editor cycle detection alias-safe by comparing underlying Resource IDs. - Memoized page settings values to prevent repeated async validation renders. ## Verification Latest local verification for `bbca6395e3`: - Both queued-update regression cases failed before the fix and passed afterward. Validation now runs synchronously immediately before deletion, after the final await, in both cut paths. - Passed: 255 copy/paste and command tests, Builder typecheck, lint, package boundaries, formatting, and diff checks. - Reviewed documentation: the existing copy/paste safety description remains accurate. Fixture inputs and outputs are unaffected; no fixture regeneration was needed. Agent evaluations were not run because separate approval is required. - CI for the new commit is pending. Previous local verification for `ffbc75b152`: - Six regression cases failed before the fix and passed afterward: content edits, property edits, and project switches during a pending cut, with both single and multiple selections. - Passed: 253 copy/paste and command tests, Builder typecheck, lint, package boundaries, source formatting, and diff checks. - An unchanged content-block test file initially failed to import in the browser. Its isolated rerun and the full 253-test rerun passed without code changes. - The guard conservatively treats any project-data change as a reason to keep the clipboard copy without deleting content. It shows a retry message. Copy/paste documentation now describes this behavior. - Fixture inputs and outputs are unaffected; fixtures were not regenerated. Agent evaluations were not run because separate approval is required. CI for the new commit is pending. Previous local verification for `3eb580697c`: - Reproduced both regressions with failing tests before fixing them: premature deletion during cut and false search exclusion appearing checked. - Passed: 203 copy/paste and page-settings tests, Builder typecheck, lint, package boundaries, formatting, and diff checks. - Cut coverage includes keyboard and programmatic entry points, single and multiple selections, delayed successful writes, rejected writes, and disabled copying. - Reviewed documentation impact: these fixes restore existing behavior; no documentation update is needed. Only Builder UI and clipboard files changed; fixture inputs and outputs are unaffected, so fixtures were not regenerated for this commit. - CI status for this older commit was recorded as pending at handoff. Agent evaluations were not run; they require separate approval. Previous local verification for `858b348ad`, rebased onto `cf606ca4c`: - Regression tests failed before each fix and passed afterward: repeated submissions in both generated route templates, plus three multi-root/branching cycle cases. - Passed: 590 SDK tests, 1,129 CLI tests, 72 focused Builder tests, SDK/CLI/Builder typechecks, lint, package boundaries, and fixture link/sync/build. - Initial concurrent runs hit timeouts. The complete CLI suite passed with one worker; the focused Builder suites passed without file parallelism. - Documentation and generated fixture changes reviewed. Agent evaluations were not run; they require separate approval. - CI for `858b348ad` passed: Main (including all six E2E shards), Visual Regression, Lint PR, and Check submodules. Earlier verification (before this update): TDD fail/pass evidence includes resolver creation, AST dependency discovery, published graph reachability, dependency-gated request generation, Builder dependency-first planning, Resource editor cycle filtering, page-tree root scoping, off-page action isolation, transitive Resource-only cycle discovery, abort/timeout precedence, server batch limits, and the rebased SSG fixture consuming its lazy Resource. - `pnpm checks` passed after rebasing onto `ce84ddcf12`; it covered workspace tests, typechecks, lint, package boundaries, generated API/docs validation, and fixture link/sync/build. - Earlier focused checks passed: Builder typecheck, 12 page-settings/pages unit tests, lint, formatting, and `git diff --check`. - Local generated-resource E2E passed for HTTP, GraphQL, and current-date resources (3 tests). - The page-actions and Marketplace page-settings persistence E2Es both pass locally. - CI for `6b8b08a70` passed: Main workflow (including all six Builder E2E shards), Visual Regression, Lint PR, and Check submodules. - Agent evaluations were not run because they require separate explicit approval. ## Current behavior and limitations - A failed dependency blocks its dependents and reports the resource path and cause. - One published resolution call is the memoization boundary; Builder maps dependency readiness onto its versioned request-cache lifecycle. - Scoped collection-item and component-parameter resources remain in #6031. - Builder skips cyclic dependency paths without blocking independent resources. |
||
|
Oleg Isonen
|
2a389b878c |
fix: load embedded MDX documents in previews (#6415)
## What changed Published MDX Content Blocks could build with document bindings but fail at runtime when the content index did not include the MDX source text. Dynamic article routes then returned an asset-query 500 or crashed during SSR. This change: - hydrates missing local MDX source text during CLI prebuild; - resolves document-graph references from embedded content before making an HTTP request; - adds a regression test for embedded document loading. ## Verification - published runtime tests passed (17 tests) - prebuild tests passed (69 tests) - content-engine and CLI typechecks passed - exact imported marketplace project reproduced locally and verified: SSG generated 10 pages, `/posts/designing-with-intent` returned HTTP 200 with article content, and no `ReferenceError` or asset-query 500 occurred |
||
|
Oleg Isonen
|
18d171212f |
feat: Resolve asset paths in HTML embeds (#6409)
## Description - Resolve root-relative script, stylesheet, image, and other asset attributes in HTML Embed markup. - Use the same asset-path map in Builder Canvas and generated published sites. - Keep SaaS asset proxy URLs deployment-relative instead of hard-coding the `p-….apps.webstudio.is` project origin. - Preserve canonical `/cgi/asset/` versus `/cgi/image/` routing and `?format=raw` behavior in Canvas. ## Regression coverage Added coverage for SaaS prebuild URLs, the real Builder asset-map-to-HTML-Embed integration, every supported asset attribute, HTML character references, and responsive-image `srcset` edge cases. ## Verification - [x] CLI prebuild suite: 69 passed - [x] HTML Embed suites: 63 passed - [x] Full Builder browser suite: 2,646 passed - [x] Asset URL map tests: 2 passed - [x] CLI, Builder, React SDK, and project-build typechecks - [x] Lint and package-boundary checks - [x] Previous PR CI: 29 successful, 1 skipped ## Code Review - [x] Self-reviewed - [x] Added regression and integration tests |
||
|
Oleg Isonen
|
f23007457d |
fix(content-engine): normalize asset path filters (#6389)
## Description Fixes Assets path filters so authors can use visible folder and file names while canonical stored paths remain URL-encoded. - normalizes path filter operands segment by segment without double encoding - preserves already encoded filters and literal string matching behavior - covers equality, inequality, membership, contains, starts-with, and ends-with operators with spaces and URL-significant characters - verifies local/Preview database execution and published runtime return the same result - documents visible path input in the generated Content Engine reference Closes #6388 ## Steps for reproduction 1. Create an Assets folder named `Tutorial posts`. 2. Add an MDX file inside it. 3. Create an Assets system resource with `path starts with "Tutorial posts/"`. 4. Preview the query and expect the MDX file to match. 5. Change the value to `Tutorial%20posts/` and expect the same result for backward compatibility. ## Regression proof Before the implementation, the focused reproduction failed because the matcher returned no items: expected `["alpha"]`, received `[]`. With the fix, the expanded path-operator regression suite passes. ## Verification - [x] Content Engine test suite: 47 files, 577 tests passed - [x] Content Engine typecheck - [x] CLI typecheck - [x] Repository lint with warnings denied - [x] Formatting check for every changed source/test file - [x] Content Engine documentation test - [x] Content Engine and MCP documentation generation is idempotent - [x] Webstudio Features fixture generation is idempotent and typechecks - [x] No agent evaluations were run ## Documentation impact Updated the authoritative CLI Content Engine reference source and regenerated the Builder Content Engine and MCP documentation. The new guidance tells users to enter visible path names and notes that existing encoded filters remain supported. ## Code Review - [ ] hi @kof, I need you to do - conceptual review (architecture, feature-correctness) - detailed review (read every line) - test it on preview ## Before requesting a review - [x] made a self-review - [x] added inline comments where behavior is not obvious ## Before merging - [ ] tested on preview environment (preview dev login: 0000) - [ ] updated test cases document (not needed; automated regression and authoritative user docs were updated) - [x] added tests - [x] no new environment variables |
||
|
Oleg Isonen
|
5d34b77fac |
feat: Improve collection entry authoring (#6361)
Collection authors and editors get clearer field help, direct entry navigation, and actionable validation instead of warnings that point at the wrong place. Closes #6360 ## Implementation - [x] Keep resettable help open after a click while the label remains hovered - [x] Stop repeating the property label inside resettable help - [x] Store JSON Schema field descriptions and show them in entry forms - [x] Add an optional dynamic collection entry page for explicit canvas navigation - [x] Rename entry file actions to **Edit file** - [x] Add **Open on canvas** to entry menus and Entry settings - [x] Put the entry filename in the Entry settings title - [x] Make collection validation banners explain the problem and open the first affected entry - [x] Remove false output-only `UNOBSERVED_FIELD` warnings while keeping filter and sort diagnostics - [x] Encourage useful field descriptions in general MCP collection guidance - [x] Update Content Engine documentation - [x] Confirm no database migration is required - [x] Regenerate the affected asset-query runtime fixture - [ ] Visually review the affected Storybook states in light and dark themes - [ ] Run the complete local agent evaluation suite - [x] Run broader package and Builder checks - [x] Complete final diff review ## Technical notes - `entryPageId` lives in `collection.json` under `x-webstudio`; no database schema changes are needed. - Entry creation still leaves the current canvas unchanged. Canvas navigation only happens after the user selects **Open on canvas**. - An entry page must contain exactly one URL parameter. The entry filename supplies that parameter, so the page can identify the selected entry. - Output projection fields are schemaless and may be produced after reference resolution, so catalog absence is no longer reported there. Filter and sort fields still report catalog diagnostics. - The structured-query runtime change affects the generated `webstudio-features` fixture, which is regenerated in this PR. ## Verification - 561 Content Engine tests - 2,413 Project Build tests - 253 Design System tests - 1,794 Builder tests - 67 focused Builder regression tests after the final navigation change - 11 ResettableLabel browser tests - 128 focused MCP tests - Builder, Content Engine, Design System, and Project Build typechecks - Repository lint and package-boundary checks - Generated documentation checks - Full workspace fixture build with `pnpm -r fixtures:build` - Formatting and diff checks - New regression tests were demonstrated failing before their fixes and passing afterward ## Documentation impact - [x] Updated the Content Engine guide for field descriptions, dynamic entry pages, actionable validation, **Edit file**, and **Open on canvas** - [x] Updated MCP guidance so collection fields include editor-facing descriptions |
||
|
Oleg Isonen
|
55920c57c4 |
fix: Show all Markdown and Assets query diagnostics (#6308)
Ref #6305 ## Outcome Show actionable Markdown, MDX, and Assets query problems in the editor, query Diagnostics, API, and MCP. The current implementation substantially improves source diagnostics, query error aggregation, query scoping, exact paths, MCP feedback, and loading states. The exhaustive audit below also records what is not complete yet. ## Completion rule An error state is complete only when all applicable checks pass: - [x] The editor uses the production validator and shows every issue for the open file. - [x] Query execution uses the same validation and returns every issue for every file that matches the query, not only the returned page. - [ ] MCP can reproduce the state through a real tool call and returns the same code, severity, message, file, and location. - [x] Diagnostics shows fatal errors and nonfatal warnings in the correct place. - [x] Exact file is always present; line and column are present whenever the parser can determine them. - [x] Multiple issues in the same file, multiple files, and multiple validation phases are returned together. - [ ] A regression test proves failure without the behavior and success with it. ## Cross-surface contract ### Required validator parity - [x] Editor, query execution, and MCP file operations invoke the same exported production file-validation pipeline for byte decoding, source/frontmatter, Markdown/MDX, Content Block semantics, and references. Copied schemas, adapter-only checks, or reimplementations do not count as parity. - [x] Query builder, query execution, `validate-asset-query`, and `preview-asset-query` invoke the same exported structural and catalog-aware semantic query validator with the same project snapshot and context. - [x] Every surface preserves the same complete, deterministically ordered diagnostic array: severity, code, message, phase/scope, asset ID, file path, line/column/range, query path, and fatal-versus-warning/result behavior. UI, API, and MCP serializers must not drop or remap fields. - [ ] A table-driven contract suite calls the real editor, query, API, and MCP entry points—including MCP upload, update, download, validate, and preview—and compares full normalized arrays for multiple errors, multiple warnings, multiple files, and pre-pagination matches. - [x] Markdown and MDX source syntax/frontmatter use validateTextAssetSource in editor, query compilation, and MCP file tools. - [x] Query request execution and MCP local parsing use assetQueryRequest. - [x] Fatal query setup errors retain all detected semantic errors and accompanying warnings. - [x] Nonfatal query warnings retain a stable code and exact query path. - [x] Source diagnostics retain severity, phase, code, asset ID, path, and available source position. - [x] Query Diagnostics excludes diagnostics produced only by unrelated published queries. - [x] Query, Preview, and Diagnostics tabs show loading states. - [x] Authenticated diagnostics can show a sanitized internal message; public/pre-auth errors remain redacted. - [x] Share byte-level UTF-8 validation instead of decoding separately in query and MCP. - [x] Make the query editor and MCP call one literal shared query-validation function before execution. - [x] Share cross-file reference validation across editor, query, and MCP file operations. - [x] Share contextual Content Block/template validation across all applicable surfaces. - [x] Preserve an invalid stored query in the editor and explain it instead of replacing it with defaults. - [x] Define and enforce whether “matches the query” includes every pre-pagination match; current source validation follows returned/paginated IDs. - [x] Continue through source, reference, and runtime phases where safe so one phase does not hide later issues. - [x] Map every user-triggerable graph/storage/runtime failure to a structured public error instead of INTERNAL_ERROR. - [ ] Label candidate-file warnings separately from returned-file diagnostics. ## File transport and encoding - [x] Empty Markdown and MDX are accepted. - [x] UTF-8 BOM and CRLF are accepted. - [x] Whole Markdown/MDX file over 1 MiB is detected. - [x] Missing or unavailable selected content is reported. - [x] Invalid UTF-8 frontmatter returns the same structured diagnostic in query and MCP. - [x] Invalid UTF-8 body returns the same structured diagnostic in query and MCP. - [x] MCP download must use fatal UTF-8 decoding rather than replacement characters. - [x] Validate bytes before committing an upload, or return the committed Asset ID and explicit cleanup state when post-commit feedback fails. - [x] Reject filename/format mismatches instead of allowing either hint to bypass Markdown/MDX validation. - [x] Storage status, missing body, invalid metadata, revision mismatch, format mismatch, short read, and excess read each return structured file context. - [x] Content changing during validation has a clear retryable conflict diagnostic. ## Frontmatter - [x] No frontmatter. - [x] Empty frontmatter. - [x] Opening delimiter without a closing delimiter. - [x] Frontmatter over 64 KiB. - [x] Invalid YAML grammar. - [x] Multiple YAML parser errors. - [x] Duplicate keys. - [x] YAML aliases. - [x] Unsupported YAML tags. - [x] Scalar or array root instead of an object. - [x] Non-JSON values, NaN, Infinity, and unsafe integers. - [x] Nesting deeper than 8. - [x] More than 256 fields/array entries. - [x] One string over 16 KiB. - [x] Normalized indexed properties over 64 KiB. - [x] Frontmatter warnings remain nonfatal and visible. - [x] Collect all post-parse normalization/limit errors; this phase currently stops at the first. - [ ] Prove every state above through real MCP upload, update, download, query, and Diagnostics paths. ## Markdown body - [x] Treat ordinary CommonMark/GFM as permissive; malformed-looking Markdown and raw HTML are not automatically syntax errors. - [x] Preserve and validate the body when frontmatter is invalid. - [x] Detect the 1 MiB body/file limit. - [x] Handle body extraction and bounded excerpts. - [x] Return a shared structured invalid-UTF-8 body error from every byte-reading surface. - [ ] Distinguish author errors from internal excerpt-limit/configuration errors. - [ ] Prove body-size and decoding states through real MCP and Diagnostics. ## MDX grammar and safety - [x] Malformed MDX/JSX grammar. - [x] Imports. - [x] Exports. - [x] Executable flow expressions. - [x] Executable text expressions. - [x] JSX expression attributes. - [x] JSX spread attributes. - [x] Arbitrary components. - [x] HTML-looking JSX outside ws.element. - [x] Unsupported/raw MDX node types. - [x] Duplicate attributes and normalized alias collisions. - [x] Event-handler attributes. - [x] srcdoc, srcset, and dangerouslySetInnerHTML. - [x] Unsafe URL protocols in link/source/action-like props. - [x] Forbidden tags: base, link, meta, noscript, script, style, template, title. - [x] Conflicting, empty, or non-static ws:name and ws:tag. - [x] More than 20,000 nodes. - [x] Nesting deeper than 100. - [x] More than 4,000 props. - [x] Multiple recoverable unsafe-mdx warnings are collected. - [ ] Collect multiple fatal grammar/structural errors when parser recovery can do so; unrecoverable parsing currently stops at the first fatal error. - [x] Preserve sibling expression warnings when unsupported JSX is also present; {first()}, <Card />, {second()} currently reports only the Card warning. - [ ] Prove every state above through real MCP and Diagnostics, not only validator unit tests. ## Content Block semantics - [x] Unresolved template warning exists. - [x] Unknown template prop warning exists. - [x] Incompatible template prop warning exists. - [x] Design-only template prop warning exists. - [x] Stale template prop warning exists. - [x] Invalid materialized HTML content model error exists. - [x] Run these checks for an MDX file even when no active editor Content Block root has materialized it. - [x] Run applicable semantic checks in query Diagnostics. - [x] Run applicable semantic checks for MCP upload as well as contextual update/download. - [x] Return all Content Block frontmatter diagnostics; one path still throws only diagnostics[0]. - [ ] Add real MCP and UI coverage for every semantic state. ## Cross-file references - [x] Non-string $ref marker. - [x] Invalid reference URL. - [x] Invalid fragment encoding. - [x] Invalid fragment. - [x] Invalid JSON Pointer. - [x] Missing referenced document. - [x] Missing referenced JSON path. - [x] Missing referenced local Asset warning. - [x] Unsupported target representation. - [x] Reference cycle. - [x] Duplicate document ID or URL. - [x] Missing graph source or target. - [x] Root not found. - [x] Duplicate/unused/unresolved reference. - [x] Invalid resolved value. - [x] Content identity conflict. - [x] Target revision mismatch. - [x] Representation selector required/invalid. - [ ] Diagnose {$ref: value, extraKey: value} as an invalid marker or explicitly document it as ordinary data. - [x] Collect all bad references and targets instead of stopping at the first graph failure. - [x] Validate references from the editor and MCP file tools, not only query compilation. - [x] Map graph compilation/resolution, Markdown-document, and JSON-document errors to exact structured Diagnostics rows. - [x] Prevent an unrelated file's graph error or duplicate canonical URL from blocking an exact-file query. - [ ] Prove missing targets, cycles, bad representations, and bad JSON paths through real MCP. ## Query request and structure - [x] Invalid JSON, invalid request UTF-8, empty/non-object request, and request over 512 KiB. - [x] Missing query and unknown request/query keys. - [x] Invalid result mode. - [x] Malformed where/all/any/filter shape. - [x] More than 32 filters or children. - [x] Filter nesting deeper than 8. - [x] Empty field path or segment. - [x] Field path deeper than 9. - [x] Bare properties path. - [x] Unknown or nested standard field. - [x] Missing or unknown operator. - [x] Non-JSON operator value. - [x] Invalid in value or more than 1,000 in values. - [x] Non-boolean exists/isEmpty value. - [x] More than 8 sort entries. - [x] Invalid sort field or direction. - [x] first/last without a sort. - [x] Invalid output mode or includeMetadata. - [x] Duplicate output fields or more than 256 fields. - [x] No selected output. - [x] Negative, fractional, or over-1,000 limit. - [x] Negative, fractional, or over-1,000 offset. - [x] Invalid content mode. - [x] Invalid full/markdown-body-ref maxBytes outside 1..1 MiB. - [x] Invalid range offset/length or range over 256 KiB. - [x] Empty or over-255-character index revision. - [x] Multiple simultaneous schema issues retain exact query paths. - [x] Make content option objects strict; unknown keys are currently stripped. - [x] Show an invalid persisted resource body rather than silently loading the default query. - [x] Aggregate unknown-key errors with every other structural issue; MCP currently stops at the first unsupported key. - [x] Add field-value type validation, for example reject size gt "large" and id contains {}. - [x] Warn about unobserved dynamic fields selected only as output. - [ ] Prove the full structural matrix through the MCP server/API end to end; the current broad matrix tests the local adapter. ## Query semantic states - [x] INCOMPATIBLE_OPERATOR is fatal for standard fields. - [x] UNOBSERVED_FIELD is a nonfatal warning. - [x] INCOMPATIBLE_OBSERVED_TYPES is a nonfatal warning. - [x] STRUCTURED_SORT_ORDER is a nonfatal warning. - [x] Every incompatible standard-field operator is collected. - [x] Fatal semantic errors retain nonfatal warnings. - [x] Successful warnings appear with exact query paths. - [ ] Reproduce every catalog-dependent semantic state through real MCP. - [x] Test repeated warnings at distinct query paths without accidental deduplication. ## Query execution and hydration - [x] Stale index revision. - [x] result one matching multiple files, including matchedCount. - [x] Candidate document limit of 1,000. - [x] Result payload limit of 16 MiB. - [x] Hydrated file count limit of 20. - [x] Per-file hydration limit of 1 MiB. - [x] Total hydration limit of 2 MiB. - [x] Range limit of 256 KiB. - [x] Binary content requested as text. - [x] Invalid UTF-8 during hydration. - [x] Missing, stale, or conflicting content identity. - [x] Canonical metadata/byte-count mismatch. - [x] Storage returning more bytes than requested. - [x] Missing content reader. - [x] Selected content not returned. - [x] Runtime Asset URL unavailable. - [x] Document graph document-count/per-file/total limits. - [x] Classify candidate-document and result-byte limits as CONTENT_LIMIT_EXCEEDED instead of INVALID_REQUEST. - [x] Give unavailable URL, missing reader/content, storage, selection, and assembly failures stable codes and exact asset/file details instead of INTERNAL_ERROR. - [x] Never silently turn an exact-file unavailable-content failure into an empty successful result. - [x] Add direct regression tests for duplicate identity, excess bytes, and canonical byte mismatch. - [ ] Reproduce every runtime/hydration state through real MCP. ## Scoping and aggregation - [x] Errors and warnings from multiple loaded result files are returned together. - [x] Nonfatal errors do not suppress successful query results. - [x] Unrelated published-query diagnostics do not replace current-query diagnostics. - [x] File path and available line/column are shown. - [x] Exact nested query paths are shown. - [x] Validate and report every pre-pagination matching file if that is the product definition. - [ ] Explain candidate metadata warnings when invalid metadata prevents deciding whether a file matches. - [ ] Stop including METADATA_PREPARATION_FAILED globally unless the UI labels its candidate scope. - [x] Aggregate reference errors instead of stopping at the first. - [x] Aggregate across validation phases where safe. - [x] Add a matrix for same-file multiple errors, multiple files, outside-page errors, filter-only candidate files, and unrelated queries. ## Transport, authorization, cancellation, and internal failures - [x] UNAUTHORIZED and FORBIDDEN. - [x] REQUEST_CANCELLED in query HTTP paths. - [x] Sanitized authenticated internal message. - [x] Generic unauthenticated/public internal message. - [x] Preserve structured causes for document-load HTTP status/body/metadata failures. - [x] Preserve structured causes for object-storage read failures. - [x] Preserve structured causes for graph selection/assembly failures. - [ ] Test MCP/API cancellation directly. - [ ] Reproduce authorization, cancellation, storage, and sanitized-internal states through real MCP where safe. ## MCP reproduction - [x] Real MCP reproduction: multiple Markdown YAML errors. - [x] Real MCP reproduction: multiple simultaneous structural query errors. - [x] MCP update returns Markdown and MDX source diagnostics. - [x] MCP upload returns Markdown and MDX source diagnostics. - [x] MCP download returns Markdown and MDX source diagnostics. - [x] Temporary remote fixtures are removed after reproduction. - [ ] Real MCP reproduction for every frontmatter code. - [ ] Real MCP reproduction for every MDX fatal/warning class. - [ ] Real MCP reproduction for every Content Block semantic diagnostic. - [ ] Real MCP reproduction for every reference/graph error. - [ ] Real MCP reproduction for every semantic query warning/error. - [ ] Real MCP reproduction for every hydration/runtime/limit error. - [ ] Real MCP reproduction for auth, cancellation, storage, and internal errors where safe. - [ ] A durable table-driven test proves code/severity/message/path parity between editor, query, Diagnostics, and MCP. - [x] Preserve stable top-level error identity and structured details through MCP instead of flattening known errors to MCP_TOOL_FAILED. - [x] Prove failed post-commit MCP feedback reports whether a mutation committed and returns enough identity for cleanup. ## Real MCP recheck and follow-up fixes (2026-09-03) Ran the branch CLI against the PR preview backend. The first run exposed that the saved CLI origin pointed to a different backend, so those results were discarded. Every MCP result below came from the PR preview. All temporary remote and local fixtures were removed afterward. The failures found in that run were then fixed and covered by regression tests. - [x] Exact broken Markdown query returns FRONTMATTER_INVALID with severity, filename, message, line, and column. - [x] Exact MDX query returns unsafe-mdx with severity, filename, message, line, and column. - [x] A no-extension query returns warnings from both matching Markdown and MDX files. - [x] A Markdown-only query excludes the unrelated MDX warning. - [x] Two fatal MDX files inside the returned page are both reported; matching warnings remain attached. - [x] Structural query validation returns three independent errors with exact paths in one call. - [x] Semantic query validation returns three fatal operator errors and two warnings with exact paths in one call. - [x] MCP upload, update, download, and exact-file query return multiple source diagnostics for the same MDX source. - [x] Query preview reports a missing Asset reference as ASSET_NOT_FOUND with the source file and reference. - [x] All temporary MCP fixtures were deleted and the saved CLI connection was restored. - [x] Fixed after recheck: validate every matching file before pagination, including broken files outside the returned page. - [x] Fixed after recheck: unrelated graph failures no longer block an exact-file query. - [x] Fixed after recheck: update, download, editor, and query paths use shared reference validation. - [x] Fixed after recheck: graph failures map to structured diagnostics with the exact file. - [x] Fixed after recheck: shared fatal UTF-8 validation runs before upload writes and preserves structured error identity. - [x] Fixed after recheck: MCP rejects filename/format mismatches. - [x] Fixed after recheck: MDX validation keeps sibling and nested warnings around unsupported components. - [x] Fixed after recheck: MCP preserves stable top-level error codes and structured details. - [x] Fixed after recheck: query and MCP serializers preserve full normalized diagnostic fields. - [x] Fixed after recheck: unknown keys are aggregated with all other structural query errors. - [x] Fixed after recheck: shared query validation covers literal types, output-only fields, and strict content options. - [x] Fixed after recheck: unavailable selected content returns an exact-file diagnostic instead of a silent empty result. ## Tests and delivery - [x] Full Builder suite: 2,707 tests. - [x] Content Engine suite: 445 tests. - [x] Asset uploader suite: 211 tests. - [x] Project runtime/MCP targeted suite: 128 tests. - [x] CLI/MCP suite: 1,051 tests. - [x] Relevant package typechecks, lint, generated docs, and diff checks. - [x] Documentation impact reviewed and generated Content Engine/MCP reference updated. - [x] Add the missing cross-surface error matrix described above. - [ ] Demonstrate red/green behavior for every new matrix case. - [ ] Run the complete agent evaluation suite after explicit approval. - [ ] Re-run all checks after the remaining checklist is implemented. ## Compatibility and risk Normal query result and expression-binding shapes remain unchanged. Diagnostics add optional fields. Existing public/pre-auth error redaction remains intact. The remaining unchecked items are broader exhaustive real-MCP coverage, parser-recovery limits, candidate-warning labeling, cancellation coverage, and agent evaluations. |
||
|
Oleg Isonen
|
f9681dc30f |
feat: MDX-backed Content Blocks (#6234)
## Summary Supersede #6182 with a clean implementation of MDX-backed Content Blocks on top of the unchanged `main` Content Block editing model. A connected `.mdx` Asset changes where authored children are loaded and saved; it must not change how Content mode inserts, edits, selects, moves, copies, or deletes instances. The previous implementation remains available in `agent/mdx-content-block-6124` and `agent/mdx-content-block-6124-review`. Reuse its independently tested parsers, converters, schemas, Asset repository, template resolution, publication, MCP, and UI work where those pieces fit the corrected architecture. Do not port its per-operation storage routing, projected editing stores, MDX undo/history, or atomic multi-root machinery. Ref #6124 Ref #6182 Closes #6261 ## Locked behavior - `.md` keeps its existing Markdown behavior; `.mdx` enables MDX. - Standard Markdown structures are stored as Markdown. JSX is used only for structures Markdown cannot represent losslessly. - Generic elements use `<ws.element ws:tag="…">`; template instances use `<ws.element ws:name="…">`. - Invalid, unsafe, or unsupported MDX never prevents editing the file or publishing the project. Builder and publication use the same recovery result: recoverable nodes are skipped with source-located warnings, while an unrecoverable document renders no authored file content in either environment. - Connecting a source never serializes existing Content Block children into the file. **Connect** removes ordinary persisted children, preserves Templates, sets `src`, and uses file content. - Disconnecting resets `src`, discards derived instances, leaves the Content Block empty, and never changes the Asset. - While connected, normal Content Block mutations update in-memory instances synchronously. Asset persistence happens afterward and never blocks rendering. - Genuine external revision conflicts use the standard reload-required flow. Rapid local edits are ordered/coalesced and must never conflict with each other. - Repeated scopes use Content Block ID plus Collection item identity. - Multiple Content Blocks may intentionally share one MDX Asset. They share one revisioned document session and edits propagate to every connected occurrence. - Missing templates show warnings/placeholders only in Builder and are skipped on published sites. ## Reuse strategy - [x] Audit all 62 original implementation commits and 5 later review/fix commits - [x] Reuse the shared Content Engine Markdown/MDX AST, parser, serializer, diagnostics, reference discovery, and tests - [x] Reuse SDK source schemas, instance-name utilities, unique template naming, and source-integrity validation - [x] Reuse template resolution/materialization and authored-tree conversion after removing storage-operation coupling - [x] Reuse revisioned Asset content repository, authorization, limits, and browser/server transport - [x] Reuse publication, document-graph, CLI, MCP, source-control UI, frontmatter UI, and E2E work where contracts remain valid - [x] Reuse tests as behavioral specifications; rewrite tests that assert the rejected projection/routing architecture - [x] Do not port per-operation MDX routing, alternate projected mutation stores, semantic reconciliation after each command, MDX Builder history/undo, atomic multi-file rollback, or canvas loading/saving UI ## Old-branch porting map — do not rewrite Before implementing any checklist batch, inspect the named implementation and tests on `agent/mdx-content-block-6124-review`. Move the implementation and its tests first; change only the boundary that conflicts with the rebuilt architecture. Do not create a second implementation under a new name. When a test encodes the rejected projected-store architecture, retain its behavioral assertion and replace only its setup. ### Already ported - `parseMarkdownAst`, `createMdxDocumentFromMarkdown`, `parseMdxDocument`, `serializeMdxDocument`, `preferMarkdownSyntax`, `discoverMdxBodyAssetReferences`, `discoverMdxAssetReferences`, and `previewMarkdownToMdxConversion` - Asset-reference, byte-stream, document-graph, Markdown, and MDX helpers included in commit `76bc388ba9` - Tests: the corresponding `packages/content-engine/src/**/*.test.ts` files already on this branch ### Port with their existing tests | Area | Functions to move or adapt | Tests to move with them | | --- | --- | --- | | Source contract and integrity | `parseContentBlockSourceProp`, `getContentBlockSource`, `isEqualContentBlockSource`, `formatContentBlockSourceIntegrityIssue`, `getContentBlockSourceIntegrityIssues`, `allocateUniqueContentBlockTemplateName` | `packages/sdk/src/content-block.test.ts`, `packages/sdk/src/schema/content-block.test.ts` | | Existing instance/name utilities | `findChildReferenceIndex`, `findParentInstanceReference`, `getInstanceName` | `packages/sdk/src/instances-utils.test.tsx` | | Template naming in mutations | `assignUniqueBlockTemplateNamesMutable`, `findBlockTemplateNameCollision`, `getSourceBackedBlockTemplateContext`, `getBlockTemplateNameConfirmation`, `getBlockTemplateInsertionIndex` | `packages/project-build/src/runtime/block.test.ts` | | Content-mode property permissions | `getContentModePropEligibility`; preserve its distinction between editable, design-only, incompatible, and unknown props | `packages/project-build/src/runtime/content-mode-permissions.test.ts` | | Template resolution | `resolveMdxTemplates` | `packages/project-build/src/runtime/mdx-template-resolution.test.ts` | | Template materialization | `createMdxScopeIdGenerator`, `materializeMdxTemplates` | `packages/project-build/src/runtime/mdx-materialization.test.ts` | | MDX → instances and instances → MDX | `isMaterializedMdxAuthoredContentRoot`, `preserveMdxAuthoredContentIds`, `materializeMdxAuthoredContent`, `reconcileMdxAuthoredContent`, `omitTransientEmptyParagraphs`, `serializeMdxTemplateInsertion` | `packages/project-build/src/runtime/mdx-authored-content.test.ts`; retain the pure serialization cases from `mdx-storage-adapter.test.ts` | | Revisioned Asset I/O | `createAssetContentRevision`, `readAssetContentBytes`, the package-owned `createHttpAssetContentRepository`, the thin Builder `createBuilderHttpAssetContentRepository`, `createAssetContentBridge`, `initAssetContentBridge`, `getAssetContentBridge`; keep one repository implementation rather than copying the old Builder duplicate | `packages/asset-uploader/src/content-repository.test.ts`, `apps/builder/app/builder/shared/assets/mdx-content-repository.test.ts`, `apps/builder/app/shared/asset-content-bridge.client.test.ts` | | Source lifecycle planning | `createContentBlockBodyRemoval`, `prepareContentBlockConnect`, `prepareContentBlockDisconnect`, `prepareContentBlockSwitch`, `ContentBlockSourceRevisionConflictError` | `packages/project-build/src/runtime/content-block-source-lifecycle.test.ts` | | Per-Asset revision/session behavior | `getContentBlockSessionSource`, `isContentBlockSessionSourceCommitted`; reuse the loading, authorization, revision, queue, conflict, retry, cancellation, and source-replacement internals of `createMdxAssetEditingSession`, but key the rebuilt session by Asset rather than Content Block/render root | Port the applicable cases from `packages/project-build/src/runtime/mdx-asset-session.test.ts`; replace root-projection setup with in-memory snapshot setup | | Template migrations | `validateMdxTemplateMigrationConfirmation`, `planMdxTemplateMigration` | `packages/project-build/src/runtime/mdx-template-migration.test.ts` | | Publication | `createPublishedMdxMaterializationCache`, `materializePublishedMdx`, `getPublishedMdxContentDatabaseMaxBytes`, `getDynamicPublishedMdxSourceBlockIds`, `hasDynamicPublishedMdxSources`, `createPublishedBuildContentCompilationPlan`, `resolvePublishedMdxDependencyClosure`, `resolvePublishedMdxAssetCandidates` | `packages/project-build/src/runtime/published-mdx.test.ts`, `packages/project-build/src/content-database.test.ts` | | Build/import integrity | `assertContentBlockSourceIntegrity` and the Content Block cases in `getBuildIntegrityIssues`/`formatBuildIntegrityIssue` | `packages/project-build/src/build-integrity.test.ts` | | MDX paste/MCP insertion | `insertMdxText` and its input/result schemas; retain parsing, confirmation, template-resolution, and insertion-index logic, then invoke the normal mutation path | `packages/project-build/src/runtime/mdx-paste.test.ts` | | Builder source control | `createContentBlockSourceController`, `createBuilderContentBlockSourceController` and the existing source-control components | `content-block-source-controller.test.ts`, `content-block-source.test.tsx` | | Diagnostics | `formatContentBlockDiagnostic`, `deduplicateContentBlockDiagnostics`, `takeNewContentBlockDiagnostics` | Port only their focused cases from `apps/builder/app/shared/content-block-content.test.ts` | ### Explicitly do not port - `projectContentStorageChanges`, `createContentStorageProjection`, `createContentStorageSelectorProjection`, `executeContentStorageMutation`, `executeContentStorageTextReplacement`, `executeContentStoragePropMutation`, and `executeContentStorageStructuralMutation` - `applyMdxContentStorageChanges` and `prepareMdxContentStorageWrites`; replace them with one post-mutation snapshot serializer built from the reusable pure `reconcileMdxAuthoredContent` converter - `createContentBlockApplicationOperations` and the persistence-plan wrappers that route individual editing commands - Builder `$runtime*` projected stores, materialized mutation functions, alternate selector/tree helpers, pending projected-change state, and MDX-specific mutation sequences from `content-block-content.ts` - Canvas loading/saving/open-file presentation state, MDX history/undo, and atomic rollback machinery ## Implementation checklist ### 1. Preserve the `main` editing architecture - [x] Keep normal synchronous insert, text, Enter, slash menu, property, delete, move, wrap, copy/paste, selection, and focus mutations unchanged - [x] Materialize connected content through the same runtime instance interfaces used by ordinary Content Blocks - [x] Keep connected authored children in memory and exclude them from persisted build data - [x] Route persistence only after a synchronous mutation has updated the canvas - [x] Never wait for Asset loading, conversion, or saving before rendering an interaction - [x] Add no MDX-specific implementation to individual editing commands ### 2. Shared Markdown and MDX conversion - [x] Parse Markdown and the Markdown portion of MDX through one shared AST and plugin configuration - [x] Implement one shared MDX-to-instances converter and one shared instances-to-MDX converter - [x] Serialize headings, paragraphs, links, images, lists, tables, task lists, blockquotes, thematic breaks, code, emphasis, strong, strikethrough, and other supported GFM structures as Markdown - [x] Serialize only non-lossless/custom structures as Webstudio JSX - [x] Preserve nested Markdown in MDX elements, frontmatter, comments, tight/loose list semantics, code whitespace, reference links, and Asset references - [x] Omit transient empty draft paragraphs instead of emitting `<ws.element ws:tag="p" />` - [x] Preserve unsupported source text and authored metadata whenever it cannot be converted safely - [x] Add deterministic round-trip and canonical-output coverage for every supported Markdown structure ### 3. Invalid and unsafe MDX - [x] Preserve file source exactly; never silently repair or rewrite invalid source - [x] Ignore imports, exports, expressions, executable JavaScript, invalid properties, and other unsupported nodes when they can be isolated - [x] Continue rendering valid siblings and report every skipped subtree once with filename, line, and column - [x] Use the same recovery pipeline in Builder and publication so both render the same authored result for the current file revision - [x] When structural syntax prevents recovery, render no authored file content in Builder or publication instead of retaining a stale materialization - [x] Skip an unrecoverable Content Block on published sites and continue the build with a warning - [x] Keep invalid and unsafe files editable and reconnectable - [x] Enforce UTF-8, byte-size, depth, node-count, property-count, and generated-output limits with source-located diagnostics where possible ### 4. File formats and Asset creation - [x] Keep `.md` parsing and raw-HTML behavior unchanged - [x] Support creating, opening, editing, deleting, and recreating `.mdx` files through the normal text-file UI - [x] Open `.mdx` in the same editor experience as `.md` - [x] Always create an independent empty MDX Asset even when identical empty files exist - [x] Allow renaming `.md` to `.mdx` without converting or rewriting its contents; after rename, handle invalid contents through the normal MDX diagnostics and recovery pipeline - [x] Update an already-open MDX editor immediately after canvas persistence - [x] Bound read/write buffering and verify the returned Asset identity and revision ### 5. Source contracts and lifecycle - [x] Support direct Asset bindings and dynamic expression bindings - [x] Keep source mutation controls available only in Design mode; show the connected filename and **Open** action read-only in Content mode - [x] Reject duplicate source props, unsupported prop types, missing Assets, non-MDX Assets, and unauthorized Assets - [x] Analyze selected file diagnostics before confirmation without modifying the block or file; MDX syntax diagnostics do not block connection - [x] For a non-empty block, show **Abort** and **Connect** and explain that ordinary children will be replaced - [x] On connect, remove every non-template child, preserve Templates, set `src`, and materialize file content - [x] Replace disconnect handoff with normal `src` reset; discard derived content and leave the block empty - [x] Remove disconnect confirmation and conversion; resetting a repeated binding leaves every occurrence empty - [x] Switching sources validates/loads the target before replacing the usable binding and leaves both files unchanged - [x] Revalidate Asset revision and project version when confirmation is submitted - [x] Make repeated connect/disconnect operations idempotent - [x] Prevent disposed, stale, or out-of-order loads from replacing the newest source - [x] Keep loading, saving, conflict, empty-file, and open-file actions off canvas ### 6. Asynchronous persistence and conflicts - [x] Snapshot the current in-memory authored root after normal mutations - [x] Serialize/coalesce rapid edits per resolved Asset and always prepare against the latest local snapshot - [x] Allow independent Assets to save concurrently without scope leakage - [x] Use one revisioned document session and one ordered save queue per Asset, even when multiple Content Blocks or render scopes reference it - [x] After a shared Asset changes, rematerialize every connected occurrence without persisting their generated instances to project data - [x] Pin the resolved Asset and exact loaded revision for each write - [x] Recheck authorization immediately before storage - [x] Ensure local queued edits never produce stale-revision conflicts with each other - [x] On a genuine external revision conflict, preserve local content, stop writes, and enter the standard reload-required state - [x] Keep failed non-conflicting writes retryable without discarding local content - [x] Finish already accepted persistence writes after a Content Block unmounts or switches source - [x] Cancel obsolete loads, materializations, and UI updates after unmount, reload, or source switch - [x] Dispose project-scoped roots, queues, parse caches, failure state, generations, and session subscriptions when the Asset session is replaced - [x] Invalidate Asset resources and synchronize open file editors after successful writes - [x] Keep multi-root persistence ordered, serial, and non-atomic; report partial completion without rollback - [x] Do not add Asset-history undo/redo; native text-editor undo remains local to the open editor ### 7. Content mode parity - [x] Empty connected blocks remain selectable and expose the normal Content-mode `+` affordance - [x] First insertion renders immediately - [x] Enter creates and focuses the next paragraph immediately - [x] Slash commands work in newly created empty paragraphs - [x] Deleting one selected instance deletes exactly that instance - [x] Mouse and keyboard moves use the same operation and permissions - [x] Preserve sibling reorder, nested reparenting, and moves between editable Content Blocks where `main` permits them - [x] Preserve content-editable props such as Link `href` - [x] Preserve copy, paste, cut, multi-cut, wrapping, unwrapping, grid insertion, selection, Navigator order, and canvas order - [x] Cover mutation routing centrally, retain the full unconnected Builder suite, and exercise representative connected interactions in real Builder E2E ### 8. Template names and permissions - [x] Resolve only exact user-facing names from the direct flat Templates list - [x] Enforce unique names within each Templates list - [x] Reject duplicate names during rename with inline validation - [x] Allocate unique names during insert, paste, duplicate, move, wrap, and unwrap - [x] Apply valid template properties using the same permissions as Content mode - [x] Preserve and warn about ignored design-only, invalid, or stale properties - [x] Preserve template shell provenance without serializing expanded template internals - [x] Generate deterministic materialized IDs from source identity, render scope, and authored usage path so focus/selection survives saves - [x] Warn before renaming or deleting referenced templates; files remain unchanged - [x] Provide shared AST-based rename/remove migration operations for MCP-assisted multi-file cleanup ### 9. Scope, frontmatter, and diagnostics - [x] Scope materialized content by Content Block ID plus exact Collection-item/render context - [x] Prevent repeated instances and dynamic candidates from leaking generated IDs, template resolution, or diagnostics across scopes while sharing the underlying Asset source and revision where applicable - [x] Remove frontmatter editing from Settings; edit the exact YAML through the MDX Asset file editor - [x] Preserve the exact frontmatter source byte-for-byte during canvas body saves - [x] Route canvas body changes and complete-file editor saves through one revisioned per-Asset coordinator - [x] Show a source warning indicator in Settings and precise source-located diagnostics in the Asset editor - [x] Deduplicate repeated diagnostics without losing distinct locations - [x] Keep unresolved-template placeholders selectable and accessible only in Builder ### 10. Asset references, transfer, and integrity - [x] Discover and rewrite Asset references in frontmatter, Markdown links/images, and MDX properties - [x] Preserve Asset references while editing unrelated content - [x] Include referenced Asset and template revisions in dependency/cache identity - [x] Route referenced Assets through published URL rewriting - [x] Validate Content Block source integrity during project import, fragment/page transfer, copy/paste, build integrity, and pre-publish audit - [x] Allow multiple Content Blocks and dynamic render scopes to reference the same MDX Asset without treating the connection as an integrity error - [x] Preserve Templates when copying a connected Content Block - [x] Preserve direct and dynamic Content Block source bindings when copying or pasting within the same project - [x] When copying only one rendered Collection item, resolve that item’s current MDX Asset and replace the copied Content Block’s dynamic source with a direct binding to that same Asset - [x] When copying an entire Collection, preserve the Collection, its instances, and its dynamic source bindings - [x] Preserve the complete MDX source and frontmatter by reference; copying a connected block must not rewrite the shared file - [x] During cross-project transfer, copy every discoverable distinct bound MDX Asset and rewrite transferred bindings to the destination Assets - [x] If some cross-project dynamic candidates cannot be discovered or copied, continue with successful copies, preserve what can be represented safely, and warn about skipped candidates - [x] Revalidate source revisions and the destination project before applying a cross-project transfer - [x] If cross-project Asset creation or project insertion fails, leave the source project unchanged and clean up any newly reserved destination Asset - [x] Use the same transfer operation through Builder and MCP ### 11. Dynamic sources and publication - [x] Resolve dynamic bindings once per render/save scope - [x] Require finite, discoverable candidate sets for publication and load no unrelated MDX - [x] Warn and skip unbounded, missing, unreadable, malformed, or unsafe sources without failing the complete build - [x] Materialize only rendered Content Blocks; ignore Templates definitions and `src` on non-Content-Block instances - [x] Skip unresolved template subtrees while keeping valid siblings - [x] Include selected template styles, fonts, resources, unresolved names, and referenced Asset revisions in generated dependency output - [x] Keep draft-only MDX Assets out of publication retention ### 12. Builder, MCP, CLI, and shared operations - [x] Use the same converters, source lifecycle, revision checks, persistence service, diagnostics, and template migrations in Builder and MCP - [x] Keep frontmatter editing file-based; retain whole-frontmatter runtime operations only where compatibility requires them - [x] Avoid Builder-only or MCP-only mutation/conversion implementations - [x] Regenerate runtime/API contracts from their source definitions - [x] Update CLI Content Engine reference and generated MCP documentation ### 13. Verification and delivery - [x] Add regression tests that reproduce failures from the previous implementation, including double deletion and blocked rendering - [x] Cover two Content Blocks sharing one Asset: edits from either block serialize once, update both canvases, preserve scope-specific Templates, and cannot overwrite a newer local snapshot - [x] Verify new tests fail against the broken implementation and pass with the corrected behavior - [x] Run SDK, Content Engine, Asset Uploader, Project Build, Builder, React SDK, CLI, Protocol, and HTTP Client affected suites and typechecks - [x] Run real Builder E2E for source lifecycle, live file refresh, empty insertion, Enter/slash focus, Link props, delete, keyboard reorder, switching, disconnect, conflicts, unresolved templates, dynamic sources, and repeated scopes - [x] Regenerate shared project fixture bundles and verify a second generation leaves the tree clean - [x] Run lint, package boundaries, `pnpm oxfmt --check`, and `git diff --check` - [x] Review and update authoritative `docs/` content and navigation - [ ] Run the complete agent evaluation suite — not run because it requires separate explicit approval ### 14. Content-mode parity and disconnect handoff - [x] Allow keyboard reordering in Content mode through the normal move command while keeping the editable Content Block boundary - [x] Let already accepted Asset writes drain independently before resetting the source; disconnect does not load, flush, or convert the file - [x] Remove MDX-to-instance copying from disconnect - [x] Reset `src`, remove the derived tree, and leave the ordinary Content Block empty - [x] Preserve automatic text-editor focus after inserting editable content from the template menu - [x] Remove Builder frontmatter-save UI and report complete-file save failures through the Asset editor - [x] Add regression coverage for keyboard movement, disconnect teardown, source switching, insertion focus, and paragraph deletion focus restoration - [x] Run the complete six-scenario MDX Content Block Builder E2E suite ### 15. Final architecture review fixes - [x] Keep one canonical ordered pending document per Asset so shared Content Block occurrences cannot conflict with one another - [x] Merge ordered local edits from different occurrences while preserving text changes, insertions, deletions, and sibling reordering; keep external revisions on the strict reload-required path - [x] Centralize typed render-scope parsing, repeated-occurrence detection, scoped variable lookup, and dynamic Asset resolution across Canvas, Settings, source lifecycle, and copy/paste - [x] Capture the owning Asset session when a save is accepted so normal unmount/source-switch teardown cannot redirect or lose it - [x] Dispose project-owned roots, queues, caches, generations, failure state, and subscriptions when replacing the project Asset session - [x] Add regression coverage for same-Asset edits from two occurrences, same-node last-write ordering, structural reorder, insertion preservation, scoped source resolution, and project cleanup - [x] Re-run affected full unit suites, typechecks, production build, static checks, and all six MDX Content Block Builder E2E scenarios ### 16. Final persistence and transfer hardening - [x] Drain every accepted per-Asset edit before project-session teardown while retaining the owning authorization context - [x] Drain canvas serialization queues and revisioned Asset writes before pre-publish audit and publication - [x] Block publication on a queued serialization failure or Asset revision conflict - [x] Copy actual Asset bytes during cross-project paste, preserve MDX reference names, rewrite direct and Collection-backed Asset IDs, and clean up newly created Assets when insertion fails - [x] Include every resolved dynamic MDX source when copying an entire Collection and warn while preserving successful copies when individual Assets cannot be transferred - [x] Keep MDX editor save failures visible with retry and route conflicts through the standard reload-required flow - [x] Give keyboard nested reparenting the same editable Content Block boundary used by pointer movement - [x] Add regression coverage and rerun the complete Content Engine and Builder unit suites plus type, lint, formatting, package-boundary, and diff checks ## Current rebuild status - The connected Content Block now owns one complete MDX document: frontmatter feeds designed-shell bindings and the explicit Body outlet owns the authored MDX body. - Builder and published output resolve the same frontmatter document graph, including nested document references and Asset references. - Direct frontmatter text/property edits use the existing ordered per-Asset session; referenced-document values remain file-owned. - Repeated Collection occurrences keep isolated runtime shell/Body identities and occurrence-specific document values. - The local blog article layout has been upgraded to the document/Body model and its bindings validate cleanly. - Full affected package suites and typechecks pass; every new regression was demonstrated fail/pass. - `pnpm lint`, package boundaries, `pnpm oxfmt --check`, generated API/docs checks, fixture builds, and `git diff --check` pass. - Full fixture synchronization is blocked by the development API rejecting this local CLI version before fixture generation; existing fixture bundles build successfully. - Complete agent evaluations remain pending explicit approval. ### 17. Treat Content Block source as a property - [x] Declare Content Block `src` in component metadata as a Design-mode initial property labeled **Source** - [x] Add the short property description: “Create an .mdx file in Assets, then select or bind it here.” - [x] Accept only a direct Asset or an expression resolving to an Asset ID; reject URL/string sources, missing Assets, and non-MDX Assets - [x] Extract and reuse the Image-style Asset picker and binding presentation without duplicating source-control UI - [x] Render Source through the normal properties section and remove the separate source-section placement and special `src` filtering - [x] Route Source changes through Content Block connect, switch, and disconnect lifecycle operations instead of generic prop updates - [x] Keep connect confirmation, Open, diagnostics, and persistence state attached to Source; remove frontmatter and disconnect UI - [x] Add regression coverage for direct Assets, dynamic Asset bindings, invalid resolved values, lifecycle operations, and Design/Content mode visibility - [x] Re-review authoritative `docs/` content for the Source-property UI change and update it if required ### 18. Source property UX - [x] Show one full-width bindable Connect .mdx file button while disconnected - [x] Show equal-width filename and Open buttons while connected - [x] Switch files from the filename button through the source lifecycle - [x] Override PropertyLabel reset and Alt-click with the disconnect lifecycle - [x] Preserve confirmation, pending, failure, repeated-scope, and dynamic-source behavior - [x] Add regression coverage for connect, switch, binding, reset, and disconnect - [x] Update Content Block documentation - [x] Run affected tests, typecheck, lint, formatting, package boundaries, and diff checks ### 19. Reduced file-owned frontmatter follow-up This checklist supersedes the earlier structured-frontmatter and disconnect-copy behavior. Frontmatter remains file-owned and is edited only in the MDX Asset editor. #### A. Remove structured frontmatter UI - [x] Remove the JSON frontmatter editor and unused Builder controller/state - [x] Keep source mutation Design-mode-only; show connected Source read-only in Content mode and allow authorized editors to open the complete file - [x] Retain runtime frontmatter APIs only where compatibility requires them #### B. Preserve source and separate validation - [x] Preserve the exact frontmatter envelope during every canvas body save without parsing/stringifying YAML - [x] Preserve comments, ordering, quoting, whitespace, BOM, newline style, delimiters, and invalid but closed YAML - [x] Keep the existing unsupported-node/property MDX recovery; do not add an ad hoc partial parser - [x] Parse body and frontmatter independently so either valid half remains usable - [x] Keep bounded Asset-query frontmatter extraction unchanged - [x] Treat structurally malformed MDX as having no safe body tree #### C. Share Asset state with the raw editor - [x] Route canvas and complete-file changes through one ordered, non-blocking per-Asset coordinator - [x] Update canvas and clean raw editors immediately from shared in-memory source - [x] Reject stale dirty-editor saves, preserve unsaved text, and require reload; never force-overwrite - [x] Drain accepted writes after source replacement/disconnect and report background failures globally - [x] Dispose sessions only after they have no consumers and no queued work #### D. Connect, replace, and disconnect - [x] Load and diagnose before connect; block only unreadable/unauthorized/non-MDX Assets - [x] Show **Abort / Connect** for non-empty blocks, including diagnostics and the recoverable-content warning - [x] Set `src` and delete persisted children through one runtime mutation - [x] Switch to a loaded source without waiting for the previous Asset's network save - [x] Remove disconnect confirmation and conversion - [x] Reset `src` immediately, discard the derived tree, leave the block empty, and preserve the Asset - [x] Resetting a bound source leaves every repeated occurrence empty #### E. Dynamic and shared sources - [x] Keep one source/revision state per Asset and a separate derived instances tree per occurrence - [x] Resolve Templates independently per Content Block and preserve Builder-only missing-template placeholders - [x] Resolve bound `src` from the selected Collection occurrence - [x] Make **Open** use the occurrence's resolved Asset; update the filename and disable Open when unresolved - [x] Add direct/bound tests including Collection item A opening `a.mdx` and item B opening `b.mdx` #### F. Diagnostics, MCP, and publication - [x] Show an accessible warning icon beside Source with the first diagnostic, line/column, and remaining count - [x] Show precise YAML/MDX diagnostics in the Asset editor without repetitive canvas/toast UI - [x] Reuse parser/diagnostic/revision contracts in Builder, build, publication, and MCP - [x] Keep MCP frontmatter editing file-based and return structured diagnostics for complete-source writes - [x] Emit build warnings; published sites skip invalid/unsupported content without diagnostic UI #### G. Verification and documentation - [x] Add fail/pass regressions for source preservation, independent validation, stale raw saves, lifecycle, dynamic Open, shared Assets, and cleanup - [x] Run affected tests, typechecks, lint, formatting, package boundaries, fixtures, and diff checks - [x] Open Markdown and MDX editors at twice the default dialog size with the right-side preview visible initially - [x] Document canvas body editing, file-owned frontmatter, Content-mode Asset access, and cloning an MDX Asset for a new post - [x] Perform a final correctness, data-loss, DRY, simplification, accessibility, and performance review - [x] Keep this checklist synchronized with implementation status ### 20. Content-mode component properties - [x] Show a connected Content Block Source read-only in Content mode with an active **Open** action - [x] Keep source switching, binding, reset, and disconnect available only in Design mode - [x] Expose HTML Embed code and preserve Code Text's code-editor metadata on authored Text Content - [x] Audit every registered component and expose authored payload while keeping design, structure, and behavior controls in Design mode - [x] Keep media dimensions and Code Text theme Design-only while preserving adapter-owned intrinsic Image metadata during MDX round trips - [x] Add exact component-permission and regression coverage - [x] Update authoritative component documentation - [x] Run affected full suites, typechecks, formatting, lint, package boundaries, and generated contract/documentation checks ### 21. Designed document shell and Body outlet - [x] Add one structural **Content Block Body** outlet and keep legacy Content Blocks without it compatible - [x] Materialize and persist only `document.body` children through the Body outlet while preserving the surrounding designed shell and Templates - [x] Expose the connected document frontmatter through a Content Block `document` parameter for shell text and property bindings - [x] Write direct frontmatter-bound text and properties back through the same ordered per-Asset session without blocking canvas mutations - [x] Resolve document `$ref` dependencies and Asset references consistently in Builder and published output - [x] Keep referenced-document values read-only on canvas and direct editors to the referenced file - [x] Isolate repeated Collection occurrences while retaining occurrence-specific source, frontmatter, shell, and Body content - [x] Render the same designed shell, frontmatter values, and Body substitution in generated/published output - [x] Upgrade the local blog article layout to bind title, excerpt, dates, author data, and images from frontmatter and render MDX through Body - [x] Add fail/pass regressions for nested Body discovery, lifecycle, permissions, repeated values, document references, and generated shell rendering - [x] Update authoritative Content Block documentation - [x] Run affected tests and typechecks, formatting, lint, package boundaries, generated API/docs checks, and fixture builds ### 22. Final nested references and frontmatter editing - [x] Keep direct frontmatter-bound text and property controls editable in Content mode without exposing or replacing their bindings - [x] Keep referenced-document values read-only and preserve legacy instance-property name normalization - [x] Render direct frontmatter text bindings through the plain-text editing path - [x] Resolve nested legacy and structured Asset references from referenced Markdown/MDX frontmatter - [x] Preserve missing Asset references, emit structured `ASSET_NOT_FOUND` diagnostics, and keep them out of the document graph - [x] Add fail/pass regressions for the real Settings control and root/nested missing Asset-reference cases - [x] Fix exact Content Block selection after the Body node appears in Navigator - [x] Run full affected suites/typechecks, all six MDX Content Block E2E scenarios, deterministic fixture builds, generated checks, lint, formatting, package boundaries, and diff checks ### Before merge - [x] Resolve #6261: Nested Asset references in referenced Markdown frontmatter work consistently in Assets queries, previews, publication, and static export ### 23. Explicit writable expression bindings - [x] Add an explicit `readwrite` mode to expression bindings; existing expressions remain read-only by default - [x] Accept `readwrite` only for direct static member paths and centralize their parsing and validation - [x] Require the expression root to provide a writable data source; initially support the connected Content Block document frontmatter - [x] Update Content Block permissions, canvas text editing, and property controls to require a valid writable binding - [x] Persist direct frontmatter edits through the existing ordered per-Asset session - [x] Create new direct frontmatter bindings with `readwrite` mode and migrate existing compatible bindings - [x] Preserve writable mode through schemas, serialization, loading, copy/paste, transfer, and export - [x] Add fail/pass regressions for valid paths, derived expressions, unsupported sources, permissions, persistence, and compatibility - [x] Review authoritative `docs/` content and update it if the persisted binding contract affects user-facing documentation - [x] Reject unsafe frontmatter paths and reject `readwrite` bindings outside a connected Content Block document - [x] Preserve expression binding mode through the public HTTP client and document the MCP authoring contract - [x] Limit frontmatter value write-back to Content mode so Design mode retains normal binding replacement/reset behavior - [x] Keep bindings inside Templates and across nested Content Block ownership boundaries read-only - [x] Upgrade compatible legacy bindings even when the Content Block is disconnected during migration - [x] Run affected tests, typechecks, formatting, lint, package boundaries, fixture checks, and `git diff --check` Verification: full SDK (567), Project Build (2,277), and Builder (2,695) suites pass, along with affected Expression, Migration, Template, HTTP Client, and MCP suites; typechecks, lint, formatting, package boundaries, and diff checks pass. Generated contracts are refreshed. Fixture synchronization remains blocked before generation because the development API rejects this local CLI version as incompatible. The checked-in bundles pass the current Protocol suite and every fixture build with the current bundle version. |
||
|
Oleg Isonen
|
a2e01a8ebe |
fix: Export configured Webhook Form methods (#6192)
## Summary
- Add an optional metadata contract for copying allowlisted resource
fields into generated component props.
- Pass action resources into page component generation.
- Configure Webhook Form's resource-bound `action` prop to generate the
form's `method` prop.
- Preserve explicitly authored props over generated resource props.
- Add generic generator, schema safety, rendered static-export
regression coverage, and updated generated fixture output.
## Root cause
Builder stores a Webhook Form's selected method on its action resource,
not as an ordinary instance prop. Component generation serialized the
resource-bound action identifier but had no generic contract for copying
safe resource fields into component props. The generated component
therefore received `action={"action"}` without the configured method,
and plain HTML rendering omitted the `method` attribute.
## Implementation
Resource prop metadata can now declare same-named fields using
`generatedProps`:
```ts
generatedProps: ["method"]
```
The component generator resolves those declarations generically; it
contains no Webhook Form component checks. The schema currently
allowlists only `method`, so resource URLs, headers, bodies, and other
potentially sensitive configuration cannot be generated as component
props. An explicit instance prop with the same name takes precedence.
Dynamic routed Webhook Forms retain their existing browser-to-server
POST behavior. This change preserves configured resource metadata in
generated component output and does not add static Webhook submission
support.
## Todo
- [x] Reproduce the missing method from an existing resource-bound
Webhook Form.
- [x] Demonstrate the generator regression test fails against the
previous output.
- [x] Define a typed, allowlisted generated-props contract for
resources.
- [x] Apply generated props generically without component-name checks.
- [x] Preserve explicit instance props.
- [x] Verify a real SSG prerender contains `method="post"`.
- [x] Verify unsafe resource fields such as `url` cannot be generated.
- [x] Give the build/prerender regression test the established 30-second
timeout.
- [x] Regenerate and verify the affected Webstudio Features fixture.
- [x] Run affected lint, typechecks, unit suites, builds, and CLI
prebuild tests.
- [x] Review documentation impact; no update is needed because
configuration and static-site support constraints are unchanged.
## Verification
- Regression test before implementation: failed because resource-bound
forms contained `action` but no generated `method`.
- `@webstudio-is/sdk`: 517 tests passed; focused schema test passed
after the rename.
- `@webstudio-is/sdk-components-react`: 126 passed, 1 skipped.
- `@webstudio-is/react-sdk`: 140 tests passed; focused generator test
passed after the rename.
- CLI prebuild: 59 tests passed; focused rendered static HTML test
passed in 2.7 seconds with a 30-second test timeout.
- Webstudio Features fixture: regeneration is idempotent; typecheck and
production build passed.
- Typechecks: SDK, SDK components, React SDK, and CLI.
- Targeted formatting and `oxlint --deny-warnings`.
- React SDK and CLI builds.
- `git diff --check origin/main...HEAD`.
- Agent evaluations were not run because approval was not requested.
Closes #6191
|
||
|
Oleg Isonen
|
2df130171f |
feat: Resolve asset references as structured metadata (#6184)
## Summary - add exact `$ref` asset values that resolve to Asset Manager metadata - keep plain local asset paths returning URL strings for backward compatibility - derive reference metadata from the canonical asset object so newly added asset fields flow through automatically - let Assets query output fields select only the metadata returned, including nested arrays - carry the same resolved shape through preview, generated preview, published runtime, and static export - document the syntax and accessible image binding in Content Engine and MCP docs ## Design The source document declares only the relationship: ```yaml featureImage: $ref: ./assets/hero.png ``` The Assets query controls projection with fields such as `properties.featureImage.src` and `properties.featureImage.description`. Selecting `properties.featureImage` returns the canonical Asset Manager asset metadata together with the resolved `src`. The Content Engine runtime contract accepts the canonical asset object instead of maintaining a second metadata field list; `url` becomes `src`, and internal `contentRef` is not exposed. Plain strings retain their existing URL-string result. Existing document `$ref` behavior is preserved by resolving structured asset metadata only when the target is a known non-document asset. ## Verification - [x] Content Engine full test suite: 298 tests - [x] SDK full test suite: 517 tests - [x] Asset repository tests: 39 tests - [x] CLI prebuild tests: 58 tests - [x] Content Engine, SDK, Asset Uploader, CLI, and Project Build typechecks - [x] Repository lint - [x] Content Engine and MCP documentation generation and tests - [x] Generated API check - [x] Local fixture build and generated fixture update - [x] New behavior tests demonstrated fail before/pass after the implementation - [ ] Complete fixture sync (development API currently rejects this branch's newer CLI protocol; local fixture build passes) - [ ] Agent evaluations (not run; require explicit approval) ## Checklist - [x] Local structured references return field-selected metadata - [x] Newly added canonical asset fields flow through without a Content Engine whitelist change - [x] Plain-path queries remain backward compatible - [x] Nested object and array projection - [x] External URL strings remain usable - [x] Missing or null descriptions allow `??` fallback to the title - [x] Expanded metadata counts toward query result-size limits - [x] Documentation impact reviewed and Content Engine and MCP docs updated - [ ] Missing asset targets return a non-failing unresolved value and dedicated diagnostic - [ ] Apply the resource and Image binding changes to the separate Webstudio blog project ## Known limitation A `$ref` whose asset target no longer exists is not yet converted into an unresolved asset value with a dedicated diagnostic. This PR intentionally does not claim that acceptance criterion yet. Ref #6179 |
||
|
Oleg Isonen
|
0760d6cfec |
fix: upload videos from Builder and MCP (#6189)
## Summary Video uploads failed because file detection returned `video` while the canonical Asset schema accepted only `font | image | file`. The mismatch was introduced when video detection was added in #5539; the recent JSON work only exposed the adjacent contract gap and did not cause it. This makes `video` a first-class `Asset["type"]` across Builder, SDK, storage formatting, project bundles, HTTP uploads, and MCP. There is no separate upload-only type and no conversion from `video` to `file`. ## Implementation - Add the canonical `video` Asset variant with width/height metadata. - Make Builder UI reservations and optimistic assets preserve `video`. - Make MCP and HTTP upload schemas derive their enum from the canonical SDK `assetType`. - Preserve video metadata through S3 and filesystem upload paths. - Include canonical video assets in project/canvas bundles. - Keep legacy video records without dimension metadata readable as generic `file` assets. - Fall back to a labeled file preview when the browser cannot decode a video format such as AVI; browser-supported formats continue to render video thumbnails. - Regenerate runtime operation contracts and sync fixtures from the PR deployment. ## Contract protection - Assert every detected upload category parses as a canonical Asset type. - Exercise Builder reservations for every value in `assetType.options`. - Assert MCP's upload enum exactly equals `assetType.options`. - Exercise direct HTTP video upload metadata and URL construction. - Cover canonical video formatting, legacy fallback, preview assets, and canvas serialization. - Verify a video decode error replaces the blank media element with a visible format-labeled fallback. - Store canonical WebM and MP4 records with dimensions in the `webstudio-features` fixture and verify generated runtime metadata. ## Checklist - [x] Make video a canonical Asset type - [x] Verify Builder UI upload flow - [x] Verify MCP and direct HTTP upload contracts - [x] Add fail/pass regression coverage for the broken paths - [x] Add an Assets panel fallback for browser-unsupported video formats - [x] Regenerate runtime contracts - [x] Sync and build fixtures against the PR deployment - [x] Run affected package tests, typechecks, lint, package-boundary, generated API, and generated docs checks - [x] Review documentation impact — existing Assets documentation already lists supported video formats and upload behavior; no user-facing documentation change is required ## Verification - Builder: 2,543 tests passed - Builder asset thumbnail regression: 14 tests passed - Project build/MCP runtime: 2,111 tests passed - CLI: 999 tests passed - SDK: 516 tests passed - Asset uploader: 199 tests passed - Protocol: 92 tests passed - HTTP client: 38 tests passed - Typechecks passed for Builder, SDK, asset uploader, project build, protocol, HTTP client, and CLI - Lint and package-boundary checks passed - Generated API and documentation checks passed - Fixture sync completed against `fix-video-up-be4d8a0.development.webstudio.is` - All fixture builds passed after synchronization Closes #6186 |
||
|
Oleg Isonen
|
5a57234c75 |
revert: Store Content Block content in MDX assets
Revert the accidental merge of #6125 so the implementation can continue in draft PR #6181. |
||
|
Oleg Isonen
|
86b6310fff | feat: Store Content Block content in MDX assets | ||
|
Oleg Isonen
|
42f5291168 |
fix: Reconcile MCP commits and Cloudflare scaffold types (#6131)
## Summary - confirm ambiguous project commits by retrying the exact transaction ID before treating a version conflict as a rejection - share the confirmation path between normal mutations and restore-point commits - emit an explicit sitemap entry type so an empty sitemap remains type-safe - generate and publish TypeScript declarations for `@webstudio-is/wsauth` - regenerate affected CLI fixtures ## Root cause The project API already treats the latest transaction ID as idempotent, but ProjectSession interpreted the first `CONFLICT` response as a rejected write. If the original transaction had committed, the subsequent refresh exposed the requested values while the tool still reported failure. The Cloudflare scaffold had two independent type-contract gaps: an empty generated sitemap inferred `never[]`, and the wsauth declaration build inherited a shared output directory, leaving its published package without the declaration files referenced by its exports. ## Technical choices - Retry only a version conflict, once, with the same base version and transaction ID. A confirmed retry returns the committed server version; persistent conflicts continue through the existing refresh/retry-safe flow. - Use the same confirmation helper for regular patch and restore-point transports. - Keep the generated sitemap module standalone by emitting its concrete entry type inline. - Make the wsauth declaration config extend its package config and publish nested `lib/types` output. - Preserve existing API payloads and server idempotency behavior; no migration is required. ## Todo - [x] Reproduce and cover ambiguous committed mutations - [x] Reuse conflict confirmation for restore-point commits - [x] Cover empty sitemap typechecking - [x] Generate and inspect the wsauth package declarations - [x] Regenerate affected fixtures - [x] Verify the Cloudflare fixture typecheck and production build - [x] Review documentation impact — no user-facing documentation changes are required ## Verification - `pnpm --filter @webstudio-is/project-build test` — 2,092 tests passed - `pnpm --filter @webstudio-is/project-build typecheck` - `pnpm --filter webstudio test` — 972 tests passed - `pnpm --filter webstudio typecheck` - `pnpm --filter @webstudio-is/wsauth test` - `pnpm --filter @webstudio-is/wsauth typecheck` - `pnpm --filter @webstudio-is/wsauth build && pnpm --filter @webstudio-is/wsauth dts` - `npm pack --dry-run --json` in `packages/wsauth` — both public declaration entrypoints included - `pnpm -r fixtures:build` - `pnpm --filter webstudio-cloudflare-template typecheck` - `pnpm --filter webstudio-cloudflare-template build` - focused formatting, lint, and `git diff --check` Agent evaluations were not run because they require separate explicit approval. Closes #6129 Closes #6130 |
||
|
Oleg Isonen
|
2ec78c183d |
feat: Add single-result modes to Assets queries (#6040)
## Summary - Add `many`, `one`, `first`, and `last` result modes to Assets queries while keeping `many` as the backward-compatible default. - Return direct items for single-result modes, validate deterministic ordering, and report duplicate `one` matches as a structured `MULTIPLE_RESULTS` error. - Update Builder controls, preview/published runtime handling, SDK resource encoding, MCP/CLI guidance, generated contracts, and fixtures. ## Implementation TODOs - [x] Add result-mode schemas and core execution behavior. - [x] Select single results before content hydration and preserve deterministic ID tie-breaking. - [x] Add structured duplicate-match transport errors. - [x] Update Builder controls and hide/reset pagination for single-result modes. - [x] Keep existing settings first, then Output and Result, without a duplicate Return label. - [x] Preserve legacy saved resources with an implicit `many` default. - [x] Update preview, published runtime, SDK, protocol, MCP, and CLI behavior. - [x] Regenerate runtime contracts, server metadata, CLI docs, and fixtures. - [x] Review documentation impact and update the authoritative MCP/CLI guidance. - [x] Add regression coverage and verify the new tests fail without the implementation. - [ ] Run the agent evaluation suite (not run; requires explicit user approval). ## Compatibility and behavior - Existing queries without `result` continue to behave as `many` and retain the ID-keyed collection shape. - `one` returns an item or `null`, but fails when more than one asset matches. - `first` and `last` require explicit sorting; the existing asset-ID tie-breaker makes equal sort values deterministic. - Single-result modes ignore and do not persist pagination controls. Materialized collection queries remain collection-only; single-result queries execute dynamically. ## Verification - Content engine typecheck and full test suite (291 tests). - Query builder, query-builder React, SDK, protocol, asset uploader, project build, Builder, and CLI targeted typechecks/tests. - Root lint. - MCP documentation tests, typecheck, sync check, and generated-doc checks. - Generated API contract checks. - Fixture generation pipeline. - Final focused content-engine regression run (53 tests). - Final DRY review: affected typechecks, focused tests, lint, package boundaries, generated checks, and fixtures. Closes #6039 |
||
|
Oleg Isonen
|
c44ef705cf |
perf: Speed up Builder resource loading (#6011)
## Problem Assets-backed Builder pages can take 6–7 seconds to become usable even when their Assets resources only request metadata. On the production-derived Blog project, a single Assets query measured 8,703 ms and an immediate repeat measured 8,110 ms. Four sibling Assets resources repeated the same authorization, development-build load/parse, compilation, canonical-asset preparation, runtime metadata load, and document work. ## Architecture The loader now treats remote resources holistically while preserving provider boundaries: - The client scheduler is resource-provider neutral. It versions logical requests, frees capacity when obsolete requests are replaced, ignores stale results, and aborts physical batches only when every member is obsolete. - Ordinary Assets requests in one server resource-loader call use a deferred Assets provider and execute as one repository batch. Independent HTTP, GraphQL, and other resources start immediately instead of waiting for Assets preparation. - Arbitrary remote responses are **not** coalesced or persistently cached. Their auth headers, cookies, upstream freshness, and `Vary` semantics remain isolated. - The Assets provider authorizes and creates its API context once per batch, then loads only `props`, `dataSources`, and `resources` from the development Build. - Queries covered by the saved Build compilation plan reuse that exact plan/artifact, preserving published byte limits, truncation, and pinned-revision behavior. Uncovered unpinned queries use a deterministic literal union and safely fall back to their existing individual path if shared preparation cannot preserve behavior. - Request-scoped document loading uses npm's byte-bounded `lru-cache` so sibling queries do not reread the same referenced author/content files. - Sibling dynamic queries with the same filter, sort, output, and content shape stitch into one document-graph resolution while keeping filter values, pagination windows, results, and failures independent. Aggregate document/byte-limit failures fall back to isolated execution. - Provider-neutral batching lives in the SDK runtime, and project asset-query preview orchestration lives in the shared project package so Builder and MCP paths can reuse the same behavior. - Cancellation flows from the client batch through the SDK/server/build/query path, with checkpoints between expensive phases. - Runtime asset metadata remains conservatively artifact-wide because URL/size filters, structured references, markdown hydration, materialized rows, and saved plans have different reachability rules. To make 1,000-asset artifacts safe, PostgREST `.in(...)` lookups are deduplicated and split into 100-ID chunks with bounded concurrency. ## Implementation checklist - [x] Trace the production Blog resource path and identify repeated work - [x] Copy the production Blog project into an ignored disposable local project - [x] Record production and local baselines - [x] Add provider-neutral client scheduling, stale-result protection, and cancellation - [x] Batch ordinary Assets queries without coupling or caching HTTP/GraphQL responses - [x] Load only Build fields required to create the saved content compilation plan - [x] Preserve saved-plan, truncation, fallback, diagnostics, and revision semantics - [x] Share Assets artifacts, runtime metadata, and revision-keyed document reads per request - [x] Stitch compatible query shapes into one graph resolution with isolated fallback - [x] Extract reusable SDK batching and project preview orchestration for MCP/runtime consumers - [x] Bound large runtime asset metadata lookups for the 1,000-asset project limit - [x] Add regression coverage and demonstrate new tests failing before their fixes - [x] Run focused and broad tests, type checks, lint, format, package-boundary, and diff checks - [x] Review documentation impact; no authoritative `docs/` content changes are required - [x] Regenerate and verify the affected asset-query runtime fixture; no production data is changed - [x] Address all actionable PR review comments - [x] Rerun the final parity-preserving implementation against the copied Blog project ## Measurements Production trace before the change: - Updates Assets query: 8,703 ms - Immediate identical repeat: 8,110 ms - Query artifact: 9,344 bytes / 7 documents - Saved source database: 42,086 bytes / 28 documents Copied-project local baseline: - Builder navigation until resource loading: 5,111 ms - Exact four-resource batches: 1,448 / 797 / 1,377 / 774 / 3,018 ms During implementation, before the final saved-plan parity tightening: - Exact four-resource batch: 174 ms first run, then 51–80 ms - Actual Builder resource-loader requests: 234–246 ms Final parity-preserving copied-project rerun: - Preflight: 10 ms - Exact four-resource batch: 435 ms cold, then 119 / 97 / 90 / 79 ms warm - Every response returned HTTP 200 with the same 16,884-byte payload The final cold batch is 70% faster than the best copied-project cold baseline, and the fastest warm batch is 90% faster than the best warm baseline. The production trace remains the authoritative production baseline; these before/after comparisons use the same disposable local project and captured request payload. ## Verification Passed: - Full package suites: Builder (212 files / 2,496 tests), asset-uploader (22 / 193), content-engine (40 / 274), project (7 / 84), SDK (29 / 508), and project-build (82 / 2,070) - Final focused regressions: Builder preview/API/resource integration (3 files / 42 tests), SDK resource loader (23 tests), and document-graph resolution (6 tests) - Builder, asset-uploader, content-engine, project, SDK, and project-build type checks - Fixture regeneration is deterministic; the generated vendor exactly matches the source runtime bundle - Fixture typecheck and production build - Changed-file Oxfmt, repository Oxlint, and package-boundary checks - `git diff --check` - `git diff --name-only HEAD --exit-code` New behavioral tests were demonstrated to fail before implementation and pass afterward, including Assets batch coordination, saved-plan wiring, truncation and pinned revisions, shared document reads, cancellation checkpoints, stale/replaced client requests, and large PostgREST ID filters. Agent evaluations were not run because they require separate explicit approval. ## Compatibility and remaining work The public resource result/error shape and on-demand diagnostics contract stay compatible. Generic HTTP/GraphQL caching semantics are unchanged. Compilation already in progress is not force-cancelled because safely interrupting shared compilation requires ownership/reference tracking; cancellation prevents later phases from starting. Closes #6003 |
||
|
Oleg Isonen
|
74d88a3fb2 |
fix: Resolve structured Asset paths in queries (#5995)
## Summary Resolve portable relative Asset paths stored in Markdown frontmatter and structured JSON properties to canonical runtime Asset URLs inside the content engine. The authored source remains unchanged. Asset resource consumers now receive canonical URLs without Image, CSS, JSON-LD, or other binding-specific transformations. ## Root cause Markdown body links already recorded relative Asset references for runtime rewriting, but structured values bypassed that pipeline. Browsers therefore resolved values such as `./assets/hero.png` against the page URL instead of the source document folder. ## Implementation - Recursively discover unique structured Asset references using the existing canonical path resolver and the platform URL parser. - Store optional `assetValueReferences` in content artifacts and runtime projections. - Include referenced Assets in publication/runtime dependency collection and validation. - Resolve values before filtering, sorting, and projection in normal and document-graph queries. - Rewrite materialized query items at runtime without mutating canonical documents or compiled rows. - Merge authored query parameters and fragments with canonical Asset URLs. - Keep queries whose filter/sort semantics depend on runtime Asset URLs out of compile-time materialization. - Add an explicit content-compilation cache generation so pre-feature artifacts are recompiled. - Refresh fixture bundle versions and the generated asset-query runtime. - Document relative references in Markdown frontmatter and JSON content. ## Checklist - [x] Add structured reference discovery for nested objects and arrays - [x] Support Markdown frontmatter and JSON document properties - [x] Preserve URL query strings and fragments - [x] Thread references through build/runtime artifacts and publication dependencies - [x] Resolve normal, materialized, and document-graph query results - [x] Apply resolved values to filtering, sorting, and projection - [x] Preserve compatibility with artifacts that omit the optional field - [x] Add fail/pass regression coverage - [x] Run package tests, typechecks, lint, and generated API checks - [x] Regenerate and build all fixtures - [x] Review documentation impact and update the Assets guide ## Verification - `pnpm --filter @webstudio-is/content-engine test` — 269 passed - `pnpm --filter @webstudio-is/content-engine typecheck` — passed - `pnpm --filter @webstudio-is/asset-uploader test` — 184 passed - `pnpm --filter @webstudio-is/asset-uploader typecheck` — passed - `pnpm --filter @webstudio-is/protocol test` — 88 passed - `pnpm lint` — passed - `pnpm check:generated-api` — passed; generated files clean - `pnpm --filter webstudio build:content-runtime && pnpm -r fixtures:build` — passed for all fixtures - Assets documentation — internal links resolve and `git diff --check` passes - Regression proof: neutralizing structured discovery/replacement caused 8 expected failures across source compilation, standard queries, document-graph queries, and materialized queries; restoring the implementation passed all tests. - `pnpm evaluations` — not run because explicit approval is required. ## Compatibility The artifact field is optional, so older artifacts continue to parse and execute. Existing absolute, protocol-relative, root-relative, fragment-only, query-only, external-scheme, malformed, missing, and ambiguous values remain unchanged. |
||
|
Oleg Isonen
|
5cefbc0320 |
perf: Speed up Builder resource loading (#5980)
## Summary Remove avoidable latency from Builder resource loading and make Assets resources use the content database efficiently. ## Why Builder currently applies a one-second debounce to every uncached resource request. Assets queries also perform server-side preparation work that should be reused or deferred, making pagination and route changes noticeably slower than expected. ## Implementation plan - [x] Dispatch the complete set of currently computable page resources without the one-second debounce. - [x] Preserve batching, deduplication, exact-request caching, invalidation, and bounded queue draining. - [x] Add regression tests for immediate dispatch, batching, deduplication, invalidation, and queue draining. - [x] Cache compiled Assets content databases by the relevant project/build and asset revision. - [x] Run pagination, filtering, sorting, and metadata selection against the cached database without rebuilding it for each request. - [x] Fetch only referenced document content that is required by the query and keep graph reads parallel. - [x] Avoid generating and returning full diagnostics artifacts for ordinary canvas resource loads; load detailed diagnostics on demand. - [x] Preserve distinct loading and confirmed-empty states for article and author resources. - [x] Add coverage for paginated overview and article navigation behavior. - [x] Deep-review the complete change and remove redundant plan construction, duplicated JSON comparison, path-based diagnostics inference, and synthetic canvas diagnostics. - [x] Make resource and diagnostics invalidation race-safe, including invalidation during in-flight requests. - [x] Run formatting, tests, lint, type checking, package-boundary checks, and affected fixtures. ## Verification - `pnpm oxfmt --check` - `pnpm lint` - `pnpm check:package-boundaries` - `pnpm --filter @webstudio-is/content-engine test` (262 tests) - `pnpm --filter @webstudio-is/asset-uploader test` (184 tests) - `pnpm --filter @webstudio-is/builder test` (2,483 tests) - Typechecks for Content Engine, Project Build, Asset Uploader, and Builder - `pnpm fixtures` (passed; no fixture diff) The new invalidation regression tests were verified to fail against the previous behavior and pass with the fix. Agent evaluations were not run because they require separate approval. ## Known limitations None identified. |
||
|
Oleg Isonen
|
f335340a54 |
fix: Correct content database sizing and Markdown references (#5972)
## Summary - Report only serialized content-database bytes that are actually included in the published bundle. - Replace the ambiguous `markdown-body` query mode with the strict `markdown-body-ref` mode. - Require every selected Markdown body to have a document-graph node; compilation now fails instead of embedding source bytes as a fallback. - Apply the same Markdown eligibility rule during compilation and runtime query execution, excluding non-Markdown files from reference-only databases and results. - Highlight database sizes in red only when documents were omitted because of the size limit. - Refresh Assets diagnostics after an Assets resource edit is confirmed by the server. - Update CLI/MCP guidance, examples, schemas, generated contracts, and fixtures to use the reference-only mode. ## Root cause Dynamic Markdown-body queries conservatively treated unrelated files, including remote images, as possible hydration candidates. The old mode could also fall back to embedding a Markdown body when graph resolution was unavailable, despite its intended storage-backed behavior. Compilation and runtime selection did not previously share the same content-mode eligibility check. Diagnostics displayed the unbounded pre-limit estimate instead of the serialized bundled size, did not carry the omission reason into query-preview diagnostics, and an early cached preview could leave the merged measurement stale until reload. ## Impact Content database diagnostics and publish warnings now reflect bundled database bytes rather than remote source-file sizes. Saved query filters trigger a post-sync diagnostics refresh. Markdown detail queries must use `content.mode: "markdown-body-ref"`; the old `markdown-body` value is rejected, missing graph support is an explicit compilation error, and non-Markdown assets cannot be selected or counted by a reference-only query. Ref #5966 ## Verification - Content engine: 39 files / 261 tests passed. - Asset uploader: 22 files / 183 tests passed. - SDK: 29 files / 505 tests passed. - Protocol: 9 files / 86 tests passed. - Project build: 82 files / 2,055 tests passed. - CLI: full suite previously passed (57 files / 933 tests); final prebuild verification passed 48 tests. - Focused Builder diagnostics tests: 4 files / 10 tests passed. - Typechecks passed for Builder, content engine, asset uploader, SDK, protocol, project build, and CLI. - Generated API contracts and docs are current. - Fixture build passed and is reproducible after the final review changes. - Formatting, lint, package-boundary checks, and diff checks passed. - New strict-reference, runtime-selection, compilation-selection, and size-highlight regression tests were demonstrated failing without their implementations and passing with them. - Agent evaluations were not run because this follow-up did not receive separate evaluation approval. |
||
|
Oleg Isonen
|
56a22fde8a |
feat: Add storage-backed document graph resolution (#5966)
## Summary Introduce a document graph architecture for Assets resources so structured documents can reference one another while keeping full document payloads on CDN-backed storage. The content database will retain the searchable index and dependency graph. At runtime, the server will select a root document, determine its complete dependency set before performing I/O, fetch each unique document concurrently, and assemble the final result server-side. ## Implementation todos ### Document graph foundation - [x] Add normalized internal reference contracts for whole documents, JSON paths, Markdown body, and Markdown frontmatter. - [x] Add an immutable graph abstraction with deterministic ordering, dependency traversal, revision validation, and cycle detection. - [x] Add a storage-independent resolver with dependency closure, bounded parallel loading, shared-content deduplication, representation selection, assembly hooks, cancellation, and structured errors. - [x] Add focused unit coverage for the isolated document-graph abstraction. - [x] Define and validate the external remote-document reference syntax. - [x] Add a pure graph compiler that resolves canonical document URLs into IDs and revisions and validates missing, duplicate, and cyclic dependencies. ### Build-time graph - [x] Add a storage-independent source-catalog compiler for bounded parallel reference discovery and deterministic graph compilation. - [x] Add a versioned deterministic document graph artifact with canonical serialization and integrity verification. - [x] Discover and normalize statically declared references while Assets resources are processed. - [x] Resolve document identities and revisions and report missing, stale, or cyclic dependencies with document context. - [x] Persist document nodes and dependency edges alongside searchable resource metadata. - [x] Include persisted graph data in artifact integrity and size validation. ### Document representations - [x] Add a standalone JSON document adapter for exact reference markers, compiler occurrences, JSON path selection, and immutable assembly. - [x] Add a standalone Markdown document adapter for bounded decoding, frontmatter references, immutable assembly, and source, body, and frontmatter selection. - [x] Parse fetched JSON and Markdown documents through format-specific adapters. - [x] Select complete documents, JSON fragments, Markdown body, and Markdown frontmatter through the shared reference model. - [x] Assemble nested and shared references into the final server-side value. ### Runtime resolution - [x] Enforce reachable-document, per-document byte, and aggregate-byte limits before parsing and assembly. - [x] Add a revision-keyed bounded source-cache wrapper with replayable bytes, stale-entry fallback, and in-flight deduplication. - [x] Add a storage-neutral loaded-source contract that rejects revision mismatches before parsing. - [x] Resolve graph-backed property paths before filtering, sorting, pagination, and projection; skip graph loading for unrelated queries. - [x] Add an injected HTTP/CDN document loader with revision metadata validation. - [x] Apply runtime payload limits, bounded concurrency, caching, and structured failure mapping. - [x] Ensure shared documents are fetched and parsed once per resolution. ### Product integration - [x] Integrate equivalent behavior in local preview, SSR, SSG, and published runtime paths. - [x] Preserve existing embedded Assets resource behavior without requiring migration. - [x] Add observability for graph loading, document fetching, caching, and resolution failures. - [x] Document the public reference and resolution behavior. ### Review hardening - [x] Enforce graph, artifact, cache, and published-storage identity consistency. - [x] Compile only graph sources reachable from selected build entries. - [x] Resolve graph-backed fields before query evaluation and avoid unnecessary graph loading. - [x] Consolidate duplicated graph schemas, utilities, and document content-type detection. - [x] Correct runtime documentation and use structural evaluation validation. ### Verification - [x] Keep a deterministic high-impact agent evaluation for Markdown-to-JSON references across the dynamic blog overview and article routes. - [x] Add a separate discoverability evaluation that must learn the reference workflow from MCP guidance without prompt-supplied syntax, queries, bindings, or call order. - [ ] Re-run the complete local agent evaluation suite after the discoverability fixture change and compare every fixture with its accepted baseline. - [x] Add integration coverage for CDN-backed hydration, parallel fetching, caching, and shared dependencies. - [x] Add integration coverage for missing references, cycles, revision changes, limits, and cancellation. - [x] Verify equivalent behavior across preview, SSR, SSG, and published deployments. - [x] Verify that existing embedded Assets resources retain their current behavior. ## Intended outcome - Storage-backed documents do not require their complete payloads to be embedded in the content database. - The server knows the complete statically discoverable dependency set before fetching document content. - Shared dependencies are fetched once per resolution and independent resources can be fetched in parallel. - JSON fragments and Markdown body/frontmatter representations use the same document-resolution system. - Reference and payload revisions cannot be mixed silently. - Existing Assets resources continue to work without migration. ## Verification results - `@webstudio-is/content-engine`: 37 files / 244 tests passed; typecheck passed. - `@webstudio-is/asset-uploader`: 22 files / 182 tests passed; typecheck passed. - CLI: 57 files / 925 tests passed, including generated SSR and SSG build/prerender coverage; typecheck passed. - Repository-wide `pnpm oxfmt --check` passed across 2,757 files. - Package-boundary validation passed. - Previous `pnpm evaluations` run: all 4 original high-impact fixtures passed. The complete 5-fixture rerun is pending explicit approval after adding the discoverability fixture. |
||
|
Oleg Isonen
|
3ccb46cec7 |
fix: restore Marketplace settings and optimize Assets publishing (#5964)
## Outcome Restore the Marketplace section for projects without existing marketplace metadata, make the complete review-submission workflow available over MCP, clarify shared Assets database diagnostics, and reduce published database size by materializing fully static Assets queries. ## Implementation ### Marketplace and MCP - Initialize the Marketplace form from a safe partial default instead of rendering nothing when metadata is unconfigured. - Keep review submission disabled until the complete metadata schema is valid. - Allow `get-marketplace-product` to return `null` and `update-marketplace-product` to create the first marketplace product through ProjectSession/MCP. - Add a confirmed `submit-marketplace-product` server operation that validates configured metadata before setting review status to `PENDING`. - Document the MCP workflow: upload the thumbnail, update metadata, publish production, then submit. - Correct the Marketplace thumbnail upload example to use the current MCP input field. ### Assets database - Show query size and merged published-database size separately in resource diagnostics. - Clarify that only the merged database counts toward the publish limit. - Allow the Assets query editor to scroll long query source. - Materialize fully static metadata-only Assets queries at publish time. - Store materialized results as compact field columns and rows keyed by a deterministic normalized-query hash. - Serve materialized rows through the existing Assets response contract without candidate documents or a runtime field catalog. - Keep route- or variable-dependent queries on the existing document-backed runtime path. - Exclude overview-only fields from dynamic detail documents so the merged database does not duplicate them. - Preserve byte-limit selection, truncation diagnostics, integrity verification, caching, and backward compatibility with existing artifacts. ### Release - Keep the release lockfile stable when preparing the CLI package. - Regenerate runtime contracts, server-only API metadata, and CLI documentation where required. ## Todo - [x] Restore the empty Marketplace settings state - [x] Prevent empty review submissions - [x] Support first-time Marketplace metadata creation over MCP - [x] Add MCP review submission - [x] Document the end-to-end MCP workflow - [x] Correct query and merged-database diagnostics - [x] Make long Assets query source scrollable - [x] Materialize static overview queries - [x] Preserve dynamic detail-query behavior - [x] Add regression coverage with fail/pass proof - [x] Run relevant package and integration checks ## Verification - [x] Content engine tests: 158 passed - [x] Asset uploader tests: 179 passed - [x] SDK tests: 505 passed - [x] CLI tests: 921 passed - [x] CLI prebuild integration tests: 46 passed - [x] Builder Assets resource and diagnostics tests: 14 passed - [x] Project runtime and ProjectSession tests: 133 passed - [x] Builder API router tests: 30 passed - [x] Protocol operation tests: 18 passed - [x] Content engine build and declaration generation - [x] Content engine, asset uploader, SDK, Builder, and CLI typechecks - [x] Repository lint and formatting checks - [x] Generated API and documentation checks - [x] New materialization tests demonstrated failing with the implementation disabled and passing after restoration ## Notes - The representative five-post overview fixture shrinks from 2,122 bytes to 1,197 bytes while returning the same sorted and paginated result. - Materialization is intentionally limited to static, metadata-only field queries without runtime asset fields. Unsupported or data-incompatible queries safely use the existing runtime database. - The complete `pnpm evaluations` agent suite was not run because repository policy requires separate explicit approval. - No database schema or migration changes are required. |
||
|
Oleg Isonen
|
efbc3098c4 |
feat: add queryable Assets resources (#5946)
Closes #5919 Ref #5920 ## Outcome Extend the existing Assets system resource with optional queries for Markdown, JSON, and standard file metadata. Unconfigured Assets resources retain the existing fetch-all behavior and activate none of the content-query infrastructure. Files are the authoring model. A standalone content engine converts an immutable snapshot of only the files required by configured queries into one bounded, deterministic runtime database. Builder preview, SSR, and SSG use the same query semantics without requiring a database service for a published site. ## Final architecture ```text Markdown / JSON files │ ▼ ContentSource adapter (FS or Webstudio Assets) │ immutable snapshot ▼ ContentCompilationPlan (union of configured query requirements) │ ▼ ContentCompiler (parse, normalize, discover fields, enforce budget) │ ▼ ContentArtifact (bounded, deterministic, immutable JSON) │ ▼ ContentDatabase (filter, OR groups, sort, paginate, project, read content) │ ├── Builder preview ├── SSR runtime └── SSG build only ``` The authoring/storage layer owns mutable files. The content engine consumes immutable snapshots and produces immutable, disposable artifacts. Query authoring is a separate OpenAPI-driven path: ```text Assets REST OpenAPI operation + standard request JSON Schema + dynamic field paths │ ▼ Generic OpenAPI adapter │ resolved schemas, controls, paths, and transport mapping ▼ Generic query-builder core and React renderer │ arbitrary provider-defined value ▼ Assets REST request adapter │ ▼ { query: value } ``` The selected standard OpenAPI operation is the only source of query UI semantics. Request parameters and JSON Schema describe the editable data; the generic renderer derives controls and owns layout. ### Activation and compilation - The compiler does not index every asset. It combines all reachable configured Assets-resource queries into one `ContentCompilationPlan`. - Static constraints such as folder, path, and file type select candidate files. Dynamic values such as a route-bound slug cannot reduce the build-time candidate set. - The plan contains the union of fields used by filtering, sorting, routing, projection, excerpts, and requested full content. - Documents shared by multiple resources are stored once in one deployment-wide runtime database. - Markdown/frontmatter, JSON properties, excerpts, and full bodies are prepared only when required. - No configured content query means no parsing, artifact, or shipped runtime dependency. Fresh generated apps need no content-runtime cleanup; incremental generation removes stale integration if a previously configured query is deleted. ### Runtime database limit - `CONTENT_DATABASE_MAX_BYTES` limits the complete serialized runtime database, not source files, images, or unrelated assets. - The default is `500 * 1024` bytes and is presented as 500 KiB. - Counted data includes indexed metadata, frontmatter/JSON properties, excerpts, the field catalog, serialization overhead, and full Markdown bodies only when requested. - Prepublish validation builds and serializes the exact database that will be attached to the immutable build, so it knows the unbounded size, bounded size, included documents, omitted documents, and affected resources. - Overflow never stores partial documents. Candidates are ordered by `createdAt` descending, canonical path ascending, and stable document ID ascending. Missing dates follow dated documents. A document that does not fit is skipped while smaller later documents are still considered. - Exceeding the limit warns but does not block publishing. - Builder preview can report exact bounded versus unbounded results when dynamic expressions have concrete preview values. Prepublish warnings remain conservative for future route values. ### Snapshot consistency The source contract guarantees a consistent compilation, while adapters provide the mechanism: - Webstudio Assets captures asset metadata and immutable content references. - Filesystem sources capture file identity and verify files did not change before accepting the artifact. - A changed source retries once, then fails clearly instead of producing a mixed-revision database. - This requires no universal snapshot service, source-tree copy, or new versioned-asset infrastructure. ### Package ownership - `@webstudio-is/content-engine`: runtime artifact schema, query execution, field catalog, compact runtime statistics. - `@webstudio-is/content-engine/compiler`: source snapshots, compilation plans, Markdown/JSON parsing, deterministic compilation, byte accounting, and compiler diagnostics. This entry exists to keep Remark/YAML/future MDX dependencies out of runtime bundles. - `@webstudio-is/query-builder`: resolves standard OpenAPI operation schemas into an internal provider-neutral editor model, validation, mutations, and source serialization. It owns no Assets, provider-authored UI contract, or fixed REST-query semantics. - `@webstudio-is/query-builder/react`: container-neutral schema renderer whose layout is renderer policy. It remains separate from the content engine. - SDK: Webstudio Assets/Resource domain models and mapping Webstudio expressions into generic content requirements. - Protocol: external Assets REST/OpenAPI transport contracts. - Asset uploader/repository: asset and folder mutations, generic metadata, immutable byte references, and storage access. Its hosted adapter supplies Webstudio Assets snapshots and parsed-metadata cache persistence to the content engine; parsing, artifact creation, and query semantics remain engine-owned. - Builder and generated-app integrations decide when to compile, cache, embed, load, or discard a database. ### Runtime lifecycle - Builder caches immutable `ContentDatabase` instances by source revision plus compilation-plan hash and coalesces concurrent compilation. Matching operations reuse the same instance; field discovery uses a dedicated minimal all-properties plan that excludes excerpts and bodies. - SSR attaches the validated bounded database to the build and includes the small runtime only when required. - SSG uses the database only while generating pages and omits it and the runtime from final output unless a future runtime-query feature requires them. - Local development compiles from the filesystem and uses identical query semantics. - Published pages query their local artifact; they do not call Builder REST endpoints. - Published artifacts need no external index store, invalidation, garbage collection, SaaS content proxy, new bucket, or new environment variable. ## Implemented foundation These existing parts remain useful and should be moved behind the final ownership boundaries rather than reimplemented. - [x] Upgrade the existing Assets system resource in place and preserve unconfigured fetch-all behavior. - [x] Implement typed filtering, nested boolean groups, deterministic sorting, pagination, projections, dynamic values, and content hydration. - [x] Add Markdown frontmatter, excerpt, JSON-property, and field-catalog extraction. - [x] Add reusable query-control implementations and a separately imported React UI as a foundation for the final OpenAPI-driven contract. - [x] Add Assets query authoring and dynamic field discovery as existing behavior to migrate into the Assets OpenAPI operation. - [x] Add synchronized visual and editable expression authoring. - [x] Unify Assets REST operations under `/rest/assets/*`. - [x] Define authenticated collection, item, content, upload, folder, query, field-catalog, and OpenAPI operations. - [x] Support Builder cookie authentication and public API token authentication. - [x] Add Builder preview, MCP/CLI operations, SSR execution, SSG path generation, and conditional activation. - [x] Add revision-keyed parsed metadata persistence for the hosted Builder. - [x] Add focused query, API, authorization, Builder, CLI, SSR, SSG, migration, and fixture coverage. ## Remaining implementation plan ### 1. Establish the content engine - [x] Rename the generic `@webstudio-is/asset-resource` implementation to `@webstudio-is/content-engine`. - [x] Move existing behavior instead of creating a parallel implementation. - [x] Keep the primary entry runtime-only and add only the compiler entry needed for bundle separation. - [x] Remove the old package after all consumers migrate. - [x] Ensure the engine has no Builder, SDK, asset-uploader, PostgreSQL, R2, or deployment-target dependency. ### 2. Define source snapshots and compilation plans - [x] Introduce `ContentSource.openSnapshot()` and a revision-bearing `ContentSourceSnapshot`. - [x] Include canonical path, stable document ID, content type, immutable content reference, size, and optional `createdAt`. - [x] Implement Webstudio Assets and filesystem snapshot guarantees using their adapter-specific mechanisms. - [x] Introduce `ContentCompilationPlan` for the union of reachable configured resource requirements. - [x] Separate statically selectable candidates from dynamic runtime predicates. - [x] Add mutation-during-compilation tests for replacement, deletion, folder movement, and filesystem changes. ### 3. Separate storage from content processing - [x] Keep upload, replacement, folders, byte storage, revisions, and generic metadata in the Assets repository/uploader. - [x] Move parsing, excerpts, field discovery, query execution, and artifact creation into the content engine. - [x] Remove content-index coordination from asset mutation business logic. - [x] Preserve original source `createdAt` across content replacement and expose it as optional queryable metadata. - [x] Keep hosted parsed-document persistence behind the content compiler's optional cache interface. ### 4. Build deterministic compilation - [x] Move the existing Markdown, frontmatter, JSON, canonicalization, hashing, document-building, and tests into the compiler. - [x] Keep frontmatter behind a dedicated module and continue using the existing Remark/Micromark pipeline. - [x] Parse only candidate files and only the tiers required by the compilation plan. - [x] Deduplicate documents and fields required by multiple resources. - [x] Define a versioned, deterministic `ContentArtifact` with integrity information. - [x] Keep verbose parse and omission diagnostics outside the runtime artifact. - [x] Reject incompatible artifact versions cleanly. ### 5. Enforce the deployment-wide database budget - [x] Replace the old asset-index limit with `CONTENT_DATABASE_MAX_BYTES`, defaulting to 500 KiB. - [x] Have applications read configuration and pass the byte limit into the compiler. - [x] Measure the exact serialized database, including selected bodies and overhead. - [x] Implement deterministic skip-and-continue packing of complete documents. - [x] Return compiler diagnostics with unbounded/bounded bytes, included/omitted counts, omitted identities, largest contributors, and affected resources. - [x] Expose only compact artifact statistics at runtime through `ContentDatabase.getStats()`. - [x] Add exact boundary, deterministic ordering, deduplication, and oversized-document tests. ### 6. Finalize the runtime database - [x] Define `ContentDatabase` over an immutable artifact. - [x] Preserve filtering, nested AND/OR groups, sorting, offset/limit, projection, field references, and content retrieval. - [x] Expose `getFieldCatalog()` and compact `getStats()`. - [x] Keep Markdown/YAML/compiler dependencies out of the runtime entry. - [x] Add execution guards and maximum-size runtime benchmarks. ### 7. Make query authoring OpenAPI-driven The selected standard OpenAPI operation is the complete query-authoring contract. The query builder resolves parameters and request-body JSON Schema, derives its internal editor model, and applies renderer-owned layout policy. It must not embed or fetch a second capabilities/UI language through `x-webstudio-query` or another side channel. - [x] Remove `x-webstudio-query` and all vendor query-definition metadata from the Assets OpenAPI operation. - [x] Resolve local OpenAPI references and derive the editable request value, defaults, validation, labels, choices, arrays, objects, and unions solely from standard operation parameters and request-body JSON Schema. - [x] Represent dynamic Assets field paths with standard JSON Schema vocabulary in the `queryAssets` request schema. - [x] Keep the normalized editor representation internal to query-builder; do not expose another provider-authored capabilities contract. - [x] Remove provider-controlled layout metadata such as `group`; make compact scalar packing and all other layout renderer policy. - [x] Keep Webstudio expression binding as a consumer/editor concern layered over schema values, not part of the OpenAPI contract. - [x] Preserve the existing Assets filter, sort, output selection, full-content/range selection, limit/offset, editable source, and configured-field behavior. - [x] Keep the Assets resource adapter responsible for serializing the edited query into its request body. - [x] Remove the obsolete Assets query-capabilities endpoint, URL, and UI-side capability parsing. - [x] Add a non-Assets OpenAPI proof with differently named parameters and no vendor metadata. - [x] Verify the generic packages contain no Assets imports, fixed Assets property contract, or provider-authored layout model. - [x] Update focused core, React, SDK, protocol/OpenAPI, Builder, and source round-trip tests. ### 8. Manage ContentDatabase lifecycle in Builder - [x] Cache databases by source revision plus normalized compilation-plan hash. - [x] Coalesce concurrent compilation requests. - [x] Reuse cached database instances across field discovery, autocomplete, query preview, and response preview when their normalized compilation plans match. - [x] Replace it when source revision or query requirements change. - [x] Preserve the optional revision-keyed hosted parsed-document cache without making it a source of truth. - [x] Show exact preview warnings when concrete values reveal missing bounded results. ### 9. Materialize and verify immutable builds - [x] Compile and serialize the exact deployment-wide database during prepublish validation. - [x] Attach that validated bounded database to the immutable build without rebuilding it from later asset state. - [x] Show non-blocking prepublish warnings with database bytes, omissions, and potentially affected resources. - [x] Avoid claiming exact future route matches when dynamic values are unknown. - [x] Include the database/runtime only when a reachable configured query requires them. - [x] Ensure detail resources requesting Markdown content include all statically eligible bodies needed by dynamic routes. ### 10. Integrate all targets - [x] SSR: load the build's bounded database and conditionally include the small runtime. - [x] SSG: query the same database during generation, then omit database/runtime output when no runtime query is needed. - [x] Local/CLI: compile from the filesystem source with the same semantics. - [x] Preserve overview pagination, alternative route fields, draft-as-user-data behavior, and full article rendering. - [x] Verify ordinary Assets and fresh query-free sites perform no parser, artifact, content-runtime dependency install, or cleanup work. ### 11. Correct API and package ownership - [x] Move external Assets REST/OpenAPI contracts to `@webstudio-is/protocol`. - [x] Keep Webstudio domain models and Assets system-resource configuration in the SDK. - [x] Keep both cookie and token authentication behavior on the unified Assets REST surface. - [x] Remove pass-through route wrappers/reexports that add no transport behavior. - [x] Keep published runtime queries local to the artifact rather than routing them through Builder REST. ### 12. Remove superseded deployment infrastructure - [x] Remove external deployment-index upload, fetch, invalidation, cleanup, and garbage-collection code. - [x] Remove the SaaS asset-origin adapter requirement; close webstudio-is/webstudio-saas#652 as superseded because it has no independent purpose. - [x] Remove index-bucket compatibility configuration and avoid adding Infisical variables. - [x] Remove obsolete external-index paths, package dependencies, routes, scripts, and tests. - [x] Consolidate unreleased intermediate migrations where safe and retain only schema required by the final hosted cache/API design. ### 13. Verification and release readiness - [x] Move and preserve existing tests while changing ownership; add new tests only for changed contracts and behavior. - [x] Run content-engine compiler/runtime tests, query-builder tests, SDK/protocol/API tests, Builder tests, CLI tests, and affected typechecks. - [x] Run formatting, generated-file idempotence, fixtures, clean Supabase-compatible migrations, E2E shards, and complete agent evaluations. - [x] Verify SSR and SSG conditional inclusion and query-free generated package manifests. - [x] Run the five-article MCP blog workflow: authoring, field discovery, overview, dynamic detail routes, full Markdown rendering, publishing, and published-site verification. - [x] Run a maximum-budget database test and a mutation-during-compilation test. - [x] Perform a final architecture, security, API compatibility, migration, bundle, correctness, and DRY review. - [x] Reconcile every checkbox and replace the verification section with results from the final architecture before release. ## Future MDX work MDX is intentionally separate from this release. - The compiler contract should allow a future build-only MDX parser. - MDX compilation must not enter the published runtime. - Embedded JSX must use an explicit compiled representation rather than runtime source evaluation. - Webstudio components, tokens, styles, and component allowlists remain Webstudio integration concerns. - The future MDX work should reuse `ContentArtifact` and `ContentDatabase` where practical. ## Superseded design The following earlier design choices are no longer the target architecture: - GROQ or GraphQL as the Assets query language. - One external or long-lived index object per resource/query. - A deployment index fetched by the Worker from SaaS/CDN infrastructure. - Lazy production reads of source Markdown bodies. - A separate index bucket, asset origin, invalidation service, or garbage collector. - A 4 MiB published index limit. - Rebuilding an old build's database from the current mutable asset set. - Bundling query infrastructure into projects without a configured content query. ## Verification baseline The embedded runtime-database architecture previously passed the checks below. The OpenAPI-driven query-authoring change additionally passed: - Query-builder core: 4 files / 19 tests. - Query-builder React: 1 file / 7 tests. - SDK Assets source/config: 1 file / 7 tests. - Protocol Assets OpenAPI: 1 file / 9 tests, including independent OpenAPI validation. - Builder Assets OpenAPI description: 1 file / 5 tests. - Query-builder, React, SDK, protocol, and Builder typechecks. - Package-boundary validation and formatting of every affected file. Local verification baseline: - Content engine: 15 files / 104 tests. - Asset uploader: 24 files / 175 tests. - Builder: 204 files / 2,423 tests. - CLI: 54 files / 866 tests. - Complete workspace typecheck, formatting, lint, package boundaries, generated fixtures, and fixture builds. - The complete migration history applied successfully to a fresh Supabase-compatible database, including every migration after the consolidated Assets migration. - Builder production build and full local E2E suite passed. - Complete agent evaluations passed: authenticated page (170,859 ms), design input (207,101 ms), and font assets (67,050 ms). - Final review additionally verified correct REST edit/view/build permits, cookie and public-token read paths, immutable snapshot retries, exact database budgeting, query-free dependency omission, and SSR/SSG conditional runtime inclusion. Final staging verification passed against the embedded runtime-database architecture: MCP uploaded and updated five Markdown articles, discovered dynamic frontmatter fields, previewed a five-item sorted overview and one full-body slug query, updated both Assets resources, found zero binding issues, rendered `/blog` and `/blog/dynamic-routes` at HTTP 200, published build version 266, and verified both published routes at HTTP 200 with the expected article metadata and Markdown body. The workflow also found and fixed offset-form asset timestamps failing canonical datetime validation and made MCP `preview.start({ port: 0 })` correctly select an available local port. ## Release constraints - No new SaaS content service, bucket, or environment variable. - No published-site database dependency. - No parser/compiler dependency in query-free sites or runtime-only bundles. - The validated content database is immutable build output. - Publishing warnings do not block deployment. - The selected standard OpenAPI operation and its request JSON Schema are the only source of query UI semantics. |
||
|
Oleg Isonen
|
b2a34f1713 |
feat: Fast MCP iterative previews (#5947)
## Summary
This PR documents a measured architecture plan for making the MCP
edit-to-screenshot loop fast enough and reliable enough for iterative
site authoring. It intentionally changes no runtime behavior yet.
The current implementation rebuilds a production preview after small
committed mutations. Testing an editable local clone of the production
Webstudio homepage also exposed a correctness gap: a path screenshot
called directly after a mutation can return a stale, byte-identical
image.
## Measured baseline
Repository fixture:
- committed text edit: **24-25 ms**
- post-edit regeneration, production build, and server restart: **7.9
s**
- first screenshot after restart and browser relaunch: **9.1 s**
- combined first post-edit screenshot: **17.1 s**
- unchanged warm screenshot: **1.1 s**
Production homepage, read-only:
- remote refresh: **3.1 s**
- complete production preview preparation/start: **16.0 s**
- cold homepage screenshot: **10.2 s**
- unchanged warm screenshot: **1.36 s**
- predicted post-edit screenshot from measured phases: **approximately
26.2 s**
Editable local production-homepage clone:
- retained **43 routes** and **266 homepage text nodes**
- committed heading mutation: **0.81-0.87 s**
- explicit post-edit preview generation/build/start: **11.6-12.6 s**
- first screenshot after explicit preview start: **9.8-11.8 s**
- complete mutation-to-visible-screenshot cycle: **22.9-25.2 s**
- complete edit, verify, screenshot, restore, and stop run: **52.7 s**
The changed screenshot visibly rendered the test heading, and the
restored screenshot matched the original byte for byte.
## Correctness finding
Calling a path screenshot directly after the committed mutation took
**20.1 s**, but both post-edit captures were byte-identical to the
baseline and rendered the old heading. Explicitly calling
`preview.start({ source: "session" })` made the edit visible.
The reported generated-build version also remained at 1 while the
project session advanced through versions 4 and 5. Current screenshot
output therefore cannot prove that it rendered the mutation it is meant
to verify.
## Proposed direction
- [ ] Introduce explicit iterative and production preview modes under
one controller contract.
- [ ] Keep a Vite development preview and browser alive for MCP
authoring screenshots.
- [ ] Preserve production preview for rendered audits and final
verification.
- [ ] Regenerate from the authoritative project session, wait for Vite
compilation, then normally reload the retained browser page.
- [ ] Make a direct post-mutation path screenshot refresh the latest
committed session without using HMR.
- [ ] Add a project-version freshness handshake so stale screenshots
fail.
- [ ] Serialize and coalesce regeneration after rapid mutations.
- [ ] Return preview mode, rendered project version, and complete phase
timings.
- [ ] Add lifecycle, integration, and local agent evaluation coverage.
## Performance target
Warm edit-to-screenshot latency should be at most **3 seconds p95**
across ten sequential small edits, without invoking `npm run build`,
`npm run start`, or launching another browser.
## Verification
- [x] Measured the source CLI against the authenticated repository
fixture.
- [x] Measured the production homepage without changing production.
- [x] Synchronized the production bundle into a disposable local Builder
project.
- [x] Committed and visually verified a real heading edit through local
MCP.
- [x] Restored the original heading and verified the restored screenshot
byte for byte.
- [x] Reproduced the stale direct-screenshot behavior.
- [x] Formatted the plan with `oxfmt`.
- [x] Verified the documentation diff with `git diff --check`.
The complete architecture, race handling, test matrix, acceptance
criteria, and non-goals are in
`apps/builder/docs/mcp-iterative-preview-plan.md`.
|
||
|
Oleg Isonen
|
d291e4b671 |
feat: Builder api runtime contracts guardrails for MCP (#5857)
This PR expands Webstudio MCP and CLI into a more complete agent authoring interface. Agents can discover project capabilities, edit real Webstudio content, use data/resources, verify visual output, and report useful issues when workflows fail. ## MCP User Capabilities - Create, update, duplicate, reorder, and delete pages. - Create pages from templates and manage reusable page templates. - Inspect project pages, instances, props, styles, assets, variables, resources, and metadata. - Insert Webstudio components and authored JSX fragments into pages. - Edit text, props, styles, breakpoints, design tokens, and CSS variables. - Update page metadata, SEO fields, redirects, marketplace metadata, and project settings. - Create and use static data variables. - Create and use dynamic resources fetched over HTTP. - Use REST, GraphQL, and built-in system resources in authored pages. - Bind data variables and resources to visible page content. - Create resource-backed pages such as sitemap XML, structured SEO data, and API-driven content sections. - Preview current project state without requiring a publish step. - Capture screenshots for visual verification. - Verify responsive designs across desktop, tablet, and mobile breakpoints. - Use screenshot diff and OCR-style checks to inspect rendered output. - Reset and refresh local MCP project state when needed. - Discover available CLI/MCP commands from generated manuals. - Use clearer examples for resources, data variables, JSX authoring, previewing, and visual checks. - Read marketplace product metadata without requiring a full project snapshot. - Get safer guidance when using low-level patch tools. - Get a ready-to-copy Discord report template when MCP/CLI behavior is confusing or broken. |
||
|
Oleg Isonen
|
b67524bc09 | fix: Fix published redirect generation (#5825) | ||
|
Oleg Isonen
|
c3d2fa7fa3 |
feat: Render current page state on links in preview mode (#5810)
Closes https://github.com/webstudio-is/webstudio/issues/5207 |
||
|
Oleg Isonen
|
c585918459 |
fix: Fix text binding rendering for non-text values (#5808)
Closes https://github.com/webstudio-is/webstudio/issues/2966 ## Description Fixes visitor-facing crashes when a text binding resolves to a non-text value, such as an object returned from a resource. Generated expression children now render through `renderText`, which preserves strings and numbers and renders other runtime values as empty text. This prevents React SSR from throwing `Objects are not valid as a React child` when resource data shape changes unexpectedly. ## Changes - Added `renderText(value)` runtime helper. - Wrapped generated expression children with `renderText(...)`. - Updated generated page imports to include `renderText`. - Added rendering tests for resource status/error fields and object-shaped bindings. ## Tests ```bash pnpm --filter @webstudio-is/react-sdk test -- context.test.tsx component-generator.test.tsx pnpm --filter @webstudio-is/react-sdk typecheck pnpm --filter ./packages/cli typecheck |
||
|
|
d45d1b4274 |
feat: add plain text page support (#5783)
## Description Adds support for plain text pages. Related to https://github.com/webstudio-is/webstudio/issues/4257 This PR introduces `text` as a supported document type, stores plain text page content in page metadata, and generates `text/plain` route handlers for Remix and React Router templates. It also limits HTML-specific Builder actions on text pages, excludes XML/text pages from the generated static sitemap, and adds SDK coverage for text document schema and sitemap behavior. ## Steps for reproduction 1. Create or edit a page in Builder. 2. Set the document type to `TEXT`. 3. Enter plain text content in page settings. 4. Build with a Remix or React Router template. 5. Open the generated text page URL, for example `/robots.txt`. 6. Expect the response body to contain the configured text content with `Content-Type: text/plain; charset=utf-8`. 7. In Builder, expect HTML insert/wrap/convert/tag actions to be unavailable for text pages. ## Code Review - [ ] hi @kof, I need you to do - conceptual review (architecture, feature-correctness) - detailed review (read every line) - test it on preview ## Before requesting a review - [x] made a self-review - [ ] added inline comments where things may be not obvious (the "why", not "what") ## Before merging - [x] tested locally and on preview environment (preview dev login: 0000) - [ ] updated [test cases](https://github.com/webstudio-is/webstudio/blob/main/apps/builder/docs/test-cases.md) document - [x] added tests - [ ] if any new env variables are added, added them to `.env` file --------- Co-authored-by: shreyanshkotak <sk7739@nyu.edu> Co-authored-by: Shrek <122366903+shreyanshkotak@users.noreply.github.com> |
||
|
Oleg Isonen
|
16fa5ab0e3 |
feat: Basic authentication (#5779)
Closes https://github.com/webstudio-is/webstudio/issues/3991 Adds HTTP Basic Authentication support for published pages and project-level `.wsauth` rules. This introduces a new `@webstudio-is/wsauth` package that owns the generic authentication format logic: parsing, validation, serialization, route matching, resource generation, and request authentication. The current supported method is `basic`, with the package structured so additional auth methods can be added later. Builder changes include: - Page Settings now has an `Authentication` section for per-page Basic Auth credentials. - Project Settings now has an `Authentication` section for editing `.wsauth` route rules. - Page Settings was split into section-specific files with focused tests. - Auth UI shows Pro badges/info tooltips but remains editable. - Publishing to custom domains is blocked on plans without `allowAuth` when either page auth or project auth is configured. - ProBadge now renders as a real pricing link and uses improved text alignment. Build/publish changes include: - CLI prebuild generates and merges `.wsauth` entries. - Published app templates enforce auth for HTML/XML routes. - Generated auth resources are shared between build output and runtime. |
||
|
Oleg Isonen
|
04188dbbf4 |
feat: CSS Grid UI (#5614)
## Description 1. What is this PR about (link the issue and add a short description) ## Steps for reproduction 1. click button 2. expect xyz ## Code Review - [ ] hi @kof, I need you to do - conceptual review (architecture, feature-correctness) - detailed review (read every line) - test it on preview ## Before requesting a review - [ ] made a self-review - [ ] added inline comments where things may be not obvious (the "why", not "what") ## Before merging - [ ] tested locally and on preview environment (preview dev login: 0000) - [ ] updated [test cases](https://github.com/webstudio-is/webstudio/blob/main/apps/builder/docs/test-cases.md) document - [ ] added tests - [ ] if any new env variables are added, added them to `.env` file |
||
|
Oleg Isonen
|
b0fe05611b |
feat: Collection improvements (#5571)
## Description - Provide a key variable next to collection item to access array index or object key - Add .length autocomplete for arrays - Add coercion rules when validating bindings, so you can bind a number to a text attribute or textContent without errors/warnings ## Steps for reproduction 1. click button 2. expect xyz ## Code Review - [ ] hi @kof, I need you to do - conceptual review (architecture, feature-correctness) - detailed review (read every line) - test it on preview ## Before requesting a review - [ ] made a self-review - [ ] added inline comments where things may be not obvious (the "why", not "what") ## Before merging - [ ] tested locally and on preview environment (preview dev login: 0000) - [ ] updated [test cases](https://github.com/webstudio-is/webstudio/blob/main/apps/builder/docs/test-cases.md) document - [ ] added tests - [ ] if any new env variables are added, added them to `.env` file |
||
|
Oleg Isonen
|
770d6c2fe3 |
feat: Allow Collection component to accept both - arrays and objects (#5570)
## Description
Now user can pass to collection an array like this (this worked before):
[{a: 1}] and also object like this {key: {a: 1}}.
What this means is that now you can just iterate over objects returned
from some api, you don't need arrays-only any more.
As an item, the value of each object key will be provied.
|
||
|
Oleg Isonen
|
33f720fb9f |
fix: Fix asset resource on published sites (#5565)
## Description 1. What is this PR about (link the issue and add a short description) ## Steps for reproduction 1. click button 2. expect xyz ## Code Review - [ ] hi @kof, I need you to do - conceptual review (architecture, feature-correctness) - detailed review (read every line) - test it on preview ## Before requesting a review - [ ] made a self-review - [ ] added inline comments where things may be not obvious (the "why", not "what") ## Before merging - [ ] tested locally and on preview environment (preview dev login: 0000) - [ ] updated [test cases](https://github.com/webstudio-is/webstudio/blob/main/apps/builder/docs/test-cases.md) document - [ ] added tests - [ ] if any new env variables are added, added them to `.env` file |
||
|
Bogdan Chadkin
|
8d14d1b5e3 |
refactor: generate modern color syntax (#5456)
Ref https://github.com/webstudio-is/webstudio/issues/3574#issuecomment-3529164146 Here added ColorValue to styles schema. toValue now generates modern syntax like rgb(0 0 0 / 0.5) for both color and rgb values. |
||
|
Bogdan Chadkin
|
1662637d92 |
fix: remove unnecessary border-width from preset styles (#5453)
Fixes https://github.com/webstudio-is/webstudio/issues/5451 We used this to not confuse users with "medium" value but it heavily bloats resulting css and does affect much UX. <img width="241" height="527" alt="image" src="https://github.com/user-attachments/assets/0be935a4-d563-4ae8-90e6-6c8bc7eedd11" /> |
||
|
Bogdan Chadkin
|
f558311897 |
feat: load data before saving resource variable (#5392)
Ref https://github.com/webstudio-is/webstudio/issues/3691 - new resource loader with queue to avoid bloating network with huge responses - data can be loaded for new variable before saving, easier to preview result https://github.com/user-attachments/assets/5c26c5a2-fd33-401b-a4d2-a321c34ca083 |
||
|
Bogdan Chadkin
|
cdb2a9581c |
feat: add search params list to resource UI (#5384)
Ref https://github.com/webstudio-is/webstudio/issues/3691 Editing search params separately from url can improve experience working with external CMS or databases like baserow. Also made headers list more compact and aligned with search params. Validation is simplified, empty pairs will be just ignored without blocking user with validation errors. https://github.com/user-attachments/assets/3102d2c8-a902-4e86-8011-d71346fb5491 |
||
|
Bogdan Chadkin
|
7981060826 |
fix: shorten fixture image to fix cloning on windows (#5284)
Fixes https://github.com/webstudio-is/webstudio/issues/5267 |
||
|
Bogdan Chadkin
|
c14238ed5e |
experimental: override some tags with components (#5210)
Ref https://github.com/webstudio-is/webstudio/issues/3632 Remix client routing should still work even if we go full html. Here added tag overrides to component generator so sdk could pass component replacements for some tags like - body -> Body - a -> Link - form -> RemixForm See example here https://github.com/webstudio-is/webstudio/pull/5210/files#diff-f2df0aea6543a33e57d770ff1791da886ab3dfd6dec37da4a41a611b9b13a19aR6 |
||
|
Ivan Starkov
|
0e5d4f739e |
experimental: Allow styling staggering and text animations (#5135)
## Description 1. What is this PR about (link the issue and add a short description) ## Steps for reproduction 1. click button 2. expect xyz ## Code Review - [ ] hi @kof, I need you to do - conceptual review (architecture, feature-correctness) - detailed review (read every line) - test it on preview ## Before requesting a review - [ ] made a self-review - [ ] added inline comments where things may be not obvious (the "why", not "what") ## Before merging - [ ] tested locally and on preview environment (preview dev login: 0000) - [ ] updated [test cases](https://github.com/webstudio-is/webstudio/blob/main/apps/builder/docs/test-cases.md) document - [ ] added tests - [ ] if any new env variables are added, added them to `.env` file |
||
|
Bogdan Chadkin
|
925ea10391 |
feat: avoid generating unused preset styles (#5121)
Now we have first-class support for html tags and able to figure out which tags are rendered so unused preset styles can be removed from websites. For example ``` <Box ws:tag="div"> <Box ws:tag="span"> </Box> </Box> ``` will render only div.w-box and span.w-box though there are more presets for for other tags like main, section etc. Before all tags were generated |
||
|
Ivan Starkov
|
acbb03827c |
experimental: Various Animation fixes (#5122)
## Description 1. What is this PR about (link the issue and add a short description) ## Steps for reproduction 1. click button 2. expect xyz ## Code Review - [ ] hi @kof, I need you to do - conceptual review (architecture, feature-correctness) - detailed review (read every line) - test it on preview ## Before requesting a review - [ ] made a self-review - [ ] added inline comments where things may be not obvious (the "why", not "what") ## Before merging - [ ] tested locally and on preview environment (preview dev login: 0000) - [ ] updated [test cases](https://github.com/webstudio-is/webstudio/blob/main/apps/builder/docs/test-cases.md) document - [ ] added tests - [ ] if any new env variables are added, added them to `.env` file |
||
|
Bogdan Chadkin
|
88e5119b02 |
feat: add project and date to published websites (#5114)
Added following attributes to html element - data-ws-project - data-ws-last-published This should help debugging issues and webstudio discoverability <img width="851" alt="image" src="https://github.com/user-attachments/assets/a4572070-a6da-40ec-88cd-308ab7f7d22f" /> |
||
|
Ivan Starkov
|
19fd9871dc |
experimental: Animate Text Split (#5048)
## Description New Split By option Animated Text 2nd example https://p-1ada14ed-beb3-4910-8af1-dacaa550310b-dot-anim-txt.staging.webstudio.is/?authToken=1e621fc4-dfb0-4189-a9b0-4fe2e189ec93&mode=preview https://animations.wstd.work/text-animations <img width="258" alt="image" src="https://github.com/user-attachments/assets/b007ec9a-11fd-4a20-969b-a8451f599eeb" /> See 2nd animated per word  ## Steps for reproduction 1. click button 2. expect xyz ## Code Review - [ ] hi @kof, I need you to do - conceptual review (architecture, feature-correctness) - detailed review (read every line) - test it on preview ## Before requesting a review - [ ] made a self-review - [ ] added inline comments where things may be not obvious (the "why", not "what") ## Before merging - [ ] tested locally and on preview environment (preview dev login: 0000) - [ ] updated [test cases](https://github.com/webstudio-is/webstudio/blob/main/apps/builder/docs/test-cases.md) document - [ ] added tests - [ ] if any new env variables are added, added them to `.env` file |
||
|
Ivan Starkov
|
35d0d7c8ac |
experimental: Breakpoint support for the Animation component (#5012)
## Description https://animations.wstd.work/ Breakpoints Example See on narrow screen animation is disabled. <img width="699" alt="image" src="https://github.com/user-attachments/assets/f464fdf3-f57a-4cf7-91a1-beb355f8a5bd" /> ## Steps for reproduction 1. click button 2. expect xyz ## Code Review - [ ] hi @kof, I need you to do - conceptual review (architecture, feature-correctness) - detailed review (read every line) - test it on preview ## Before requesting a review - [ ] made a self-review - [ ] added inline comments where things may be not obvious (the "why", not "what") ## Before merging - [ ] tested locally and on preview environment (preview dev login: 0000) - [ ] updated [test cases](https://github.com/webstudio-is/webstudio/blob/main/apps/builder/docs/test-cases.md) document - [ ] added tests - [ ] if any new env variables are added, added them to `.env` file |
||
|
Bogdan Chadkin
|
8b526dc7e9 |
refactor: store webhook form action as resource (#4983)
Ref https://github.com/webstudio-is/webstudio/issues/4093 Here's refactoring to unlock resource UI in webhook form Need to test in builder and on published site 1. Create webhook form in main and try to update it on this branch. So legacy text prop should automatically migrate to resource. 2. New webhook form should create resource when type anything in action prop |
||
|
Ivan Starkov
|
ca4b0f89bc |
experimental: Animation publishing (#4976)
## Description 1. What is this PR about (link the issue and add a short description) ## Steps for reproduction 1. click button 2. expect xyz ## Code Review - [ ] hi @kof, I need you to do - conceptual review (architecture, feature-correctness) - detailed review (read every line) - test it on preview ## Before requesting a review - [ ] made a self-review - [ ] added inline comments where things may be not obvious (the "why", not "what") ## Before merging - [ ] tested locally and on preview environment (preview dev login: 0000) - [ ] updated [test cases](https://github.com/webstudio-is/webstudio/blob/main/apps/builder/docs/test-cases.md) document - [ ] added tests - [ ] if any new env variables are added, added them to `.env` file |
||
|
Bogdan Chadkin
|
a344b34cf7 |
refactor: generate only asset name for published sites (#4947)
We don't really need anything from asset object on published sites except the name to resolve the file. Here got rid from Asset objects for favicon, fonts and backgrounds. This should reduce client bundle where a lot of assets are used. |