mirror of
https://github.com/webstudio-is/webstudio.git
synced 2026-10-30 00:27:43 +09:00
codex/headers-settings-docs
41
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Oleg Isonen
|
3d0f19ffe0 |
fix: complete and speed up publish validation (#6478)
## Summary Complete Validate for large projects with linked MDX content, using the same validation API in the dialog and MCP. Publishing continues independently of diagnostics. - Load MDX sources omitted from the embedded artifact and report storage failures with typed diagnostic reasons. - Reuse parsed MDX and template scope indexes; cache source reads in the shared asset loader. - Exclude conservative non-MDX query candidates while retaining explicit invalid references for diagnostics. - Give validation requests a 60-second server execution budget. - Support both server bundles in E2E tests and make OAuth logging initialization repeatable. - Raise the project bundle import limit from 20 to 32 MiB to reproduce the customer project. ## TODO / acceptance checklist - [x] Import and inspect the customer project locally and on staging. - [x] Reproduce missing embedded MDX sources and slow template materialization. - [x] Load omitted MDX sources and discover nested template dependencies. - [x] Report source-read failures using typed reasons rather than matching error messages. - [x] Reuse template scope indexes and remove duplicate source caching. - [x] Route dialog Validate through the same API as MCP, after flushing editor content. - [x] Preserve publishing in parallel with diagnostics, including failed or unfinished checks. - [x] Fix JSON collection files being treated as MDX query candidates. - [x] Preserve explicit invalid file references so they still produce diagnostics. - [x] Fix E2E routing across both generated server bundles and repeated OAuth logging initialization. - [x] Demonstrate the new review regressions fail before the fixes and pass after them. - [x] Run targeted tests, relevant typechecks, lint, formatting, package boundaries, and generated API checks. - [x] Run the full fixture pipeline and production builder build; no generated fixture changes. - [x] Build the exact staging project locally after the final review changes. - [x] Validate the exact staging project through MCP/CLI after the final deployment. - [x] Complete an actual staging publish after the final deployment. - [x] Address PR comments: simplify the stream mock, explain every diagnostic reason, and retain the existing diagnostic code contract. - [x] Review documentation impact; these fixes restore expected behavior and require no user instruction changes. - [x] Finish CI on the final commit, including all six Builder E2E shards. ## Verification Local checks passed: 179 project-build tests, 78 CLI prebuild tests, 8 asset publication tests, 42 server tests, and 9 publish UI/helper tests. The imported staging project also builds successfully. The new explicit-reference regression and repeated-initialization regression both failed before their fixes and passed afterward. A production build serves the login route and dispatches tRPC requests to its separate server bundle. Validation on staging project `b9d23e54-5dbf-45c4-b6d8-bc4cf842a468` completed in 25.3 seconds after deployment and 23.5 seconds after publishing: 92 documents included, 0 omitted, and 0 MDX errors. The [original staging publish job rerun](https://github.com/webstudio-is/webstudio-saas/actions/runs/36647770120/job/109819143695) succeeded end to end with the final artifact, including CLI/Remix builds, worker deployment, and database updates. [CI on commit `01a9d772dc`](https://github.com/webstudio-is/webstudio/actions/runs/36696154831) is green, including all six Builder E2E shards and Windows/Ubuntu fixture checks. The comment follow-ups simplify the publication test mock and document all diagnostic reasons. Production behavior is unchanged. The 8 publication tests, 6 schema tests, package typecheck, formatter, and lint pass. Disabling the source cache makes the simplified regression fail as expected. Final staging validation passed in 27.1 seconds with 92 documents and no omissions or MDX errors. One additional publish attempt hit a database statement timeout during project sync, before the build. A [retry with no code changes succeeded end to end](https://github.com/webstudio-is/webstudio-saas/actions/runs/36647770120/job/109826767589). CI is green on `01a9d772dc`. Ref #3550 |
||
|
Oleg Isonen
|
2c9fd16689 |
feat: configure custom response headers by route (#6429)
## Description Project Settings starts with an empty response-header rule list. Users can add valid HTTP response headers for all paths (`/*`) or an Authentication-style route such as `/private/*`, and remove any configured rule. The Headers and Authentication views share the accessible rule-list form. The header form suggests standard response-header names and common values for the selected header, while accepting custom names and values. `Set-Cookie` is not suggested because a rule can store only one value per header and would replace existing cookies. `X-Powered-By`, `X-Content-Type-Options`, and `Strict-Transport-Security` are dispatcher-owned and cannot be configured. Users may configure `Content-Security-Policy`, `X-Frame-Options`, and `Referrer-Policy`; when absent, the dispatcher supplies their defaults. The SDK validates header names, values, routes, duplicates, and size limits. The CLI emits only configured rules. Customization requires Pro for custom-domain publishing, while unchanged defaults and staging remain available on Free plans. The server checks the saved production snapshot against the project owner's plan. Companion Cloud worker PR: https://github.com/webstudio-is/webstudio-saas/pull/664. Deploy its dispatcher support before exposing these settings in the Builder. Other hosts need an adapter that applies the generated rules; Webstudio Cloud dispatcher defaults do not apply automatically elsewhere. Static exports do not apply headers automatically. ## Validation - SDK, CLI, Builder browser, publish-warning, and production-build tests pass. - Builder, SDK, and CLI typechecks and lint pass. - The companion Cloud worker tests, typecheck, and lint pass. Replaces https://github.com/webstudio-is/webstudio/pull/6428 to use `headers.staging`. Closes https://github.com/webstudio-is/webstudio/issues/4748 |
||
|
Oleg Isonen
|
13952a7d34 |
fix: Preserve Global root tokens when importing templates (#6348)
Inserting the CMS Sitemap marketplace page could detach existing Global root tokens and break styles across existing pages. Preserve the root's existing style-source selections in their original order, then append mapped incoming sources without duplicates. Incoming local declarations continue to merge into the existing root local source. The MDX save fix is now included in main through `commitAssetContentUpdate`. This PR retains an E2E assertion that saved MDX revisions are not persisted again through `build.patch`. Closes #6347 Verification and delivery: - [x] Add four regression cases covering empty, local-only, token-only, and mixed incoming root selections, including repeated imports, token remapping, and local declaration merging. All four failed before the fix and pass after it. - [x] Rebase onto main and rerun style-copy and page-copy tests: 55 passed. Formatting and diff checks passed. - [x] Verify the cloned project locally: reproduce the broken Home canvas before the fix and restore it with Undo. After the fix, insert Sitemap and verify Home retains its logo sizing, typography, spacing, rounded corners, and both Global root tokens after reload. - [x] Verify the MDX E2E assertion fails with duplicate persistence and passes with the fix. Before rebase, all 13 shard 4 tests passed locally with four workers, along with relevant typecheck and lint checks. - [x] Resolve the rebase conflict using main's shared MDX implementation and remove the duplicate helper from this branch. - [x] Review documentation impact against marketplace, CSS-variable, and Content Block documentation: no documentation changes required. - [x] Review fixture impact: no fixture inputs, serialization formats, or generated contracts changed. No migrations are required. This prevents future imports from dropping root selections; it does not reconstruct selections already lost in previously saved projects. |
||
|
Oleg Isonen
|
e69a3d2f8e |
feat: Create structured MDX entries from collection folders (#6317)
## Paragraph editing follow-up — |
||
|
Oleg Isonen
|
05706b5425 |
feat: Make Content Block templates the MDX authoring interface (#6307)
Closes #6306 ## Outcome Content Block templates are now the public component model for MDX authoring. - Standard Markdown resolves through matching Content Block templates. - PascalCase JSX resolves to a named template first, then to a registered component. - Adding or changing a template immediately affects matching MDX, including content authored before that template existed. - Every supported standard MDX element is present in new Content Blocks by default. - Template `name` is a stable PascalCase JSX identifier. `label` remains optional display text only. - Lowercase JSX represents native HTML and SVG without exposing `ws.element` in authored syntax. - Unknown, ambiguous, and unsafe JSX stays in source, produces a source-ranged Builder diagnostic, and is omitted from published output. - Frontmatter images preserve exact `$ref` values and use the Asset description for `alt` text. - Keyboard and mouse insertion from the slash menu remove the trigger slash and preserve the selected template. ## Public MDX contract - Markdown is canonical for standard prose. - Lowercase JSX creates native elements: `<section><p>Text</p></section>`. - PascalCase JSX uses templates and registered components: `<PromotionCard />`, `<Heading tag="h1">Title</Heading>`, and `<YouTube src="https://youtu.be/KI5JHpzBK1s" />`. - Component-name collisions use stable qualified identifiers such as `Checkbox` and `RadixCheckbox`. - A self-closing template keeps its designed default subtree. An explicitly paired empty template clears it. - Static quoted values and bare booleans are supported. - Imports, exports, expressions, spreads, functions, executable JavaScript, unsafe attributes, `<$.*>`, and authored `<ws.element>` are rejected. - `ws` remains only for compiler primitives with no React component. ## Implementation - [x] Add stable, unique template names and keep them separate from labels. - [x] Parse and validate Markdown and JSX through their ASTs with precise source ranges. - [x] Share component identity between template JSX, Content MDX, Builder materialization, reload, copy, and publication. - [x] Resolve templates live after add, edit, rename, reorder, removal, undo/redo, collaboration updates, and reload. - [x] Preserve authored props, children, Asset references, template defaults, and the exact slash-menu template choice. - [x] Make one typed MDX semantic inventory drive default templates and slash-menu eligibility. - [x] Move component-dependent template assembly into the component registry. - [x] Replace production `$`, `animation`, and `radix` JSX namespaces with direct component imports. - [x] Keep only explicit compiler primitives in `ws` template JSX. - [x] Migrate repository templates, fixtures, stories, examples, MCP guidance, CLI guidance, and generated outputs. - [x] Add Builder unit and Playwright coverage for authoring, diagnostics, insertion, synchronization, reload, and publication. - [x] Review and update the authoritative Content Block and generated MCP documentation. ## Compatibility - Persisted project data continues to use namespaced component IDs and `ws:element` for generic HTML elements. No project-data migration is required. - Legacy `<ws.element ws:tag="…">`, `<ws.element ws:name="…">`, `<$.*>`, `<animation.*>`, and `<radix.*>` input remains readable during the compatibility window. - Current serializers and repository-owned sources emit only canonical Markdown, lowercase intrinsic JSX, or PascalCase identifiers. - Untouched user MDX is not rewritten automatically. Legacy syntax is canonicalized when its owning source is edited. - Deferred follow-up: remove legacy namespace readers and proxy support only after the canonical writer is deployed and existing projects are backfilled. This is intentionally outside this PR. ## Verification - [x] CI package tests, typechecks, lint, package boundaries, generated assets, migrations, and size checks pass. - [x] All six Builder E2E shards pass. - [x] Fixture checks pass on Ubuntu and Windows. - [x] Targeted local Content Engine, Template, SDK, registry, Project Build, migration, Builder, component, CLI, and Playwright tests pass. - [x] Runtime contracts, CLI docs, and MCP docs were regenerated from their owning sources. - [x] Local fixture generation completed without an unexpected fixture diff. - [x] Documentation impact reviewed and included in this PR. - [x] Review and approve the intentional List visual snapshot change. - [x] Run the complete agent evaluation suite after explicit approval. - [x] Run the network fixture link/sync pipeline against the matching PR deployment after explicit permission. ## Known visual change The generated List story now targets its actual default `ul` element instead of `ol`, so the default list styling appears in the visual snapshot. The visual change was reviewed and the approved visual check passes. |
||
|
Oleg Isonen
|
4273b7001f |
refactor: Organize Builder development and test tooling (#6285)
## Outcome Builder development, backend, E2E, and CI tooling now share one ownership model: Builder defines named commands and backend modes, while CI only schedules those commands. ## Technical choices - Group shared Compose files, PostgreSQL initialization, and backend helpers under `apps/builder/backend`. - Resolve `compose.<mode>.yaml` by convention instead of maintaining Compose filename lists in scripts and workflows. - Keep persistent development and disposable test Compose projects separate by default, while allowing the standard `COMPOSE_PROJECT_NAME` override. - Make schema snapshots invalidate from discovered migration sources and backend files. - Move E2E behavior recipes into named Builder commands; retain concise root proxies for developer entry points. - Let Vitest allocate available browser ports instead of maintaining a cross-package port registry. - Keep test-suite scheduling explicit in `main.yml`, but move executable test commands to root package scripts. - Move fixture-size enforcement from inline GitHub JavaScript into a locally runnable repository script. ## Todo - [x] Consolidate Builder backend and Compose ownership. - [x] Add named development, E2E, schema-cache, Supabase, and CI-shard commands. - [x] Remove Compose topology and behavior flags from CI. - [x] Remove the static Vitest browser-port registry. - [x] Extract fixture-size checking from the workflow. - [x] Verify migration bootstrap, snapshot-backed E2E, tests, builds, types, formatting, lint, generated contracts, and workflow syntax. - [x] Review documentation impact: updated the E2E developer guide; no user-facing `docs/` content is affected. ## Verification - Full package, CLI, Builder core, and Builder app test suites - Full workspace typecheck - Builder production build - Schema-cache generation through the complete migration history - Supabase migration compatibility - Snapshot-backed Playwright animation workflow - Backend regression tests, including demonstrated fail/pass coverage - All four Compose mode configurations - `oxfmt`, `oxlint`, package boundaries, generated API, and generated docs checks - `actionlint` and Bash syntax checks The fixture-size checker preserves the existing Linux `du` thresholds. Its three inputs build locally; macOS allocation sizes are not directly comparable to the Linux CI thresholds, so CI remains the authoritative size result. |
||
|
Oleg Isonen
|
bc8630f413 |
fix: Gate Builder releases on the published CLI contract (#6282)
Closes #6280 ## Summary - Expose the deployed Builder's CLI-facing bundle contract through a non-cached, read-only endpoint. - Resolve and execute the exact `webstudio@latest` registry artifact, observe the bundle version sent by its real `sync` request, and require an exact match before assigning the release staging alias. - Include expected and received bundle versions in runtime and release-gate errors. ## Technical choices - The existing generated `bundleVersion` remains the single compatibility boundary. Builder-only changes outside that project-bundle contract do not require a CLI release. - The CLI sync probe runs against a local fixture API and temporary directory. It reads the deployed runtime contract but never reads or modifies a Cloud project. - Verification runs after Vercel creates the exact production-built deployment and before the release staging alias is assigned. A mismatch therefore leaves production domains untouched. - Registry resolution requires version and integrity metadata, then executes the resolved immutable version rather than source from the checkout. - The gate requires the deployed Builder origin and cannot fall back to the checkout's contract. ## Todo 1. [x] Add the deployed Builder contract endpoint. 2. [x] Probe the exact published CLI through its real sync request. 3. [x] Gate the release staging alias on an exact contract match. 4. [x] Report expected and received versions for compatibility failures. 5. [x] Add regression tests and verify the production Builder build. 6. [x] Review documentation and fixture impact. ## Verification - `pnpm --filter @webstudio-is/builder exec vitest run app/services/build-router.server.test.ts app/services/cli-compatibility-route.node.test.ts` — 12 passed. - `pnpm --filter webstudio exec vitest run scripts/builder-contract-gate.test.ts scripts/release-smoke-registry.test.ts` — 7 passed. - Rebased onto `main` at `9ec4c820e1`. - Builder and CLI typechecks passed. - `pnpm lint` and changed-file formatting passed. - `pnpm --filter @webstudio-is/builder build` passed and emitted `/rest/cli-compatibility`. - The changed GitHub Action parses as valid YAML. - Real registry probe passed for `webstudio@0.293.0` against `bundle-kzb45v`. - Real registry probe correctly blocked current Builder contract `bundle-1mym82p` and reported expected `bundle-1mym82p`, received `bundle-kzb45v`. - Missing staged Builder origin fails closed before registry execution. - GitHub CI passed all four workflows, including all six Builder E2E shards. - The formerly failing empty-MDX insertion workflow passed 8 repeated runs with 4 workers locally, then passed in Builder E2E shard 4 in CI. - Full browser-based release smoke was not run, per the request not to use Chrome. Agent evaluations were not run because no evaluation approval was requested or received. - Documentation: no user-facing documentation changes are required. - Fixtures/generated outputs: no fixture inputs or generated outputs are affected. ## Operational boundary The check needs the exact deployed runtime, so Vercel creates an unaliased `--skip-domain` deployment before verification. A failed check cannot affect production domains or the normal release alias. Repository automation cannot prevent a Vercel administrator from manually promoting an arbitrary deployment outside this release path. |
||
|
Oleg Isonen
|
1b48716ff4 |
docs: Document MDX-backed Content Blocks (#6273)
## Summary - Document how people create, connect, edit, validate, and publish MDX-backed Content Blocks. - Document the complete MCP workflow, including scoped expression sources, frontmatter updates, diagnostics, conflicts, and disconnect behavior. - Add two focused screenshots for the human workflow. - Fix issues found by a low-reasoning MCP agent while executing every documented workflow. Ref #6182 ## Implementation notes - Connection confirmation uses the public `confirmReplacement` field because `confirm` is reserved as a hidden transport field. - MCP edits frontmatter with `update-content-block-frontmatter`; materialized MDX elements are not persistent targets for generic instance mutation tools. - Reload now intentionally discards a conflicted local source and accepts the Asset's latest storage revision before an edit is reapplied. - Generated MCP documentation and runtime contracts are updated from their source files. ## Checklist - [x] Document the human MDX Content Block workflow. - [x] Document MCP discovery, connection, editing, diagnostics, frontmatter, dynamic sources, conflicts, and disconnection. - [x] Add only screenshots that clarify otherwise hard-to-describe UI steps. - [x] Test every documented MDX capability through the local MCP CLI with a low-reasoning agent. - [x] Reproduce and fix the replacement-confirmation schema mismatch. - [x] Reproduce and fix persistent-session conflict reload and reapply. - [x] Add regression tests that failed before each product fix and pass afterward. - [x] Regenerate runtime contracts and generated MCP documentation. - [x] Review authoritative `docs/` content and update affected pages. - [x] Run affected tests, typechecks, formatting, and diff checks. ## Verification - Project Build: 208 affected tests passed. - Content Engine Asset session: 8 tests passed. - CLI documentation tests: 14 tests passed. - MCP docs generator: 3 tests passed. - Content Engine, Project Build, CLI, and MCP docs typechecks passed. - `pnpm oxfmt --check` passed. - `git diff --check` passed. - Low-reasoning MCP evaluation passed all documented workflows, including a real two-client revision conflict, reload, reapply, and final saved inspection. |
||
|
Oleg Isonen
|
8718feb465 |
test: simplify browser testing infrastructure (#6271)
## Summary Unifies frontend testing around three intentional boundaries: - Vitest Node projects for pure and server tests - Vitest Browser Mode with Playwright/Chromium for DOM and component tests - Playwright Test for Builder end-to-end workflows The Builder no longer uses jsdom. Ordinary E2E workflows use an automatically managed isolated `page` fixture; explicit contexts remain only for simultaneous identities and suite-scoped setup. ## Technical choices - Split Builder tests into one Node project and independent browser projects so each Chromium/Vite process has a bounded workload. - Added shared Browser Mode configuration for the affected SDK React packages. - Preserve the Builder's production top-window bridge ownership; Browser Mode uses explicit test-only installation seams. - Centralized Playwright context options and kept action/navigation timeouts bounded. - Validate Playwright filters before Docker, migrations, browser installation, or Builder builds start. - Preserve the first failed E2E attempt's report, traces, screenshots, and videos when CI retries a shard. - Split static checks, typechecks, CLI tests, core package tests, browser package tests, and Builder projects into parallel CI jobs with five-minute hard ceilings. - Distribute Builder E2E tests across ten isolated shards with two workers; parallel suites create unique per-worker setup data. - Reuse color contrast DOM/canvas samples so the exhaustive theme sweep takes about four seconds instead of timing out at 30 seconds. - The removed cost is jsdom behavior drift, oversized serial check jobs, and unnecessary E2E startup for empty selections. Real browser startup and E2E environment setup remain. ## Todo - [x] Split Builder pure/server tests from browser/component tests. - [x] Migrate Builder DOM tests from jsdom to Vitest Browser Mode. - [x] Migrate affected SDK React DOM tests to the shared Browser Mode setup. - [x] Remove jsdom annotations, dependencies, and compatibility setup. - [x] Replace ordinary manual E2E contexts with the isolated `page` fixture. - [x] Keep explicit contexts only for concurrent identities and `beforeAll` setup. - [x] Preserve setup authentication isolation from workflow pages. - [x] Add bounded Playwright action and navigation timeouts. - [x] Preserve first-attempt E2E failure evidence across CI retry. - [x] Reject empty E2E selections before expensive environment setup. - [x] Parallelize repository checks into bounded CI shards. - [x] Isolate parallel E2E setup data across workers. - [x] Remove the color contrast sweep bottleneck. - [x] Update E2E contributor documentation. - [x] Review product documentation impact; no user-facing product docs are affected. - [x] Run the full repository tests and typechecks. - [x] Run formatting, lint, package-boundary, generated-file, and fixture checks. - [x] Complete final correctness and simplification review. ## Verification - `pnpm --filter @webstudio-is/builder test` - `pnpm -r typecheck` - `pnpm lint` - `pnpm check:package-boundaries` - `pnpm check:generated-api` - `pnpm check:generated-docs` - `pnpm fixtures` - Design-system: 222 tests pass locally. - Builder: all seven Vitest projects pass locally. - The formerly slow E2E shard passes locally in 1m55s. - Full database-backed Builder E2E execution passes in CI. - All CI jobs pass; the slowest completed in 4m58s. - No fixture inputs/outputs or generated files changed. Ref #6270 |
||
|
Oleg Isonen
|
f6c524572d |
feat: add Builder dark theme and rebuild colors
## Summary
Rebuilds Webstudio’s color system, migrates the entire Builder to it,
and introduces a complete dark theme.
A small set of theme colors and contrast controls now drives the
interface through relative CSS colors:
```text
Theme colors and contrast
↓
Light/dark derivation bounds
↓
Derived color families
↓
Reusable semantic colors
↓
Component composition
↓
Builder interface
```
CSS is the runtime source of truth. Light and dark modes share the same
theme inputs, so changing a theme color automatically updates its
surfaces, content, borders, states, and component presentation. The
refactor turns dark mode into a first-class Builder theme rather than a
separately maintained palette.
This is an atomic migration. Legacy color names and compatibility
aliases will not remain after all consumers have moved.
## Scope
- The color source, derivation model, semantic vocabulary, validation,
typing, and Storybook overview.
- Every design-system component.
- Builder, dashboard, authentication, shared UI, Query Builder, stories,
and tests that consume the legacy color scale.
- Removal of the legacy Stitches color scale after its final consumer is
migrated.
- A complete dark theme across application UI while keeping authored
canvas content independent.
- Complete light and dark visual verification.
- Unified browser-test infrastructure: Vitest Browser Mode for Design
System browser contracts and standard Playwright Test for all Builder
E2E workflows.
Literal colors remain only where color is the actual content or belongs
to a deliberately separate system, such as user-authored values,
external previews, brand artwork, data visualization, functional canvas
overlays, and source assets. Every remaining exception must have an
explicit owner.
## Architecture decisions
- Theme authors control six color families and three bounded contrast
controls.
- Relative CSS colors derive the rest of the system.
- Dark mode changes derivation bounds instead of maintaining a second
independent palette.
- Components consume a small reusable semantic vocabulary.
- Component states and exceptional presentation are composed inside
components.
- Builder-specific categorical meaning uses focused composite recipes,
not intent semantics or a larger global vocabulary.
- TypeScript validates public semantic CSS-variable names but does not
generate runtime colors.
- No token-manager dependency, compatibility aliases, or
component-specific public color tokens.
- A new semantic or theme variable is added only when existing semantics
and local composition cannot represent a reusable cross-feature
decision.
## Todo
### Color-system foundation
- [x] Establish direct CSS as the runtime source of truth.
- [x] Define bounded theme colors and contrast controls.
- [x] Derive light and dark modes from the same theme inputs.
- [x] Reduce the semantic inventory to reusable decisions and explicit
accessibility pairs.
- [x] Generate only TypeScript names for public semantic CSS variables.
- [x] Validate layers, references, cycles, consumption, naming,
generated types, and browser-computed contrast.
- [x] Add the Storybook color-system overview and theme controls.
- [x] Document the architecture and its relationship to the public Craft
specification.
### Design-system migration
- [x] Replace every active legacy color consumer in design-system
components and Storybook chrome.
- [x] Compose component states locally without component-specific public
tokens.
- [x] Preserve intentional preview and sample-content colors.
- [x] Add complete positive, negative, warning, informative, and accent
accessibility pairings.
- [x] Validate light and dark scheme-bound references independently.
- [x] Verify the design-system package with typecheck, lint, formatting,
Storybook, and 222 tests.
### Inventory and contract
- [x] Inventory all 319 legacy references across 107 Builder, Query
Builder, story, test, and design-system files.
- [x] Classify every reference as a direct semantic replacement, local
composite recipe, documented literal exception, or removable code.
- [x] Record the intentional authored-content, external-preview,
functional-preview, data-visualization, brand, and source-asset
exceptions.
- [x] Treat the public semantic vocabulary as closed by default and
justify any proposed addition by cross-feature reuse and accessibility
need.
- [x] Keep Builder code limited to typed public variables accepted by
`cssVar()`.
### Theme infrastructure
- [x] Add `system` (default), `light`, and `dark` application
preferences.
- [x] Persist the preference in existing client settings and resolve
`system` through `matchMedia`.
- [x] Apply the resolved scheme before paint and respond to
operating-system changes.
- [x] Expose the persisted theme control under **View → Color scheme**.
- [x] Keep authored canvas documents independent from the application
theme.
- [x] Ensure portals, overlays, embedded surfaces, and third-party
controls receive the correct interface scheme.
### Generic interface migration
- [x] Migrate shared dialogs, project settings, notifications, controls,
and Query Builder.
- [x] Migrate dashboard, authentication, publishing, and shared
application UI.
- [x] Migrate the Builder loading state, topbar, sidebars, navigator,
workspace, footer, settings, pages, alerts, and menus.
- [x] Replace fixed interface literals alongside their owning surfaces.
- [x] Remove the old scrollbar variable and hardcoded loading color.
### Style panel and style-source recipes
- [x] Migrate ordinary style-panel inputs, labels, layouts, spacing,
alignment, menus, property states, and value editors.
- [x] Model style-source provenance, kind, interaction, and error as
separate axes.
- [x] Compose categorical style-source colors from neutral and
accent-relative colors.
- [x] Use negative semantics only for actual errors, not overwritten
provenance.
- [x] Reuse one feature-owned recipe across style-source controls,
badges, menus, and values without adding global tokens.
### Canvas and editors
- [x] Migrate canvas selection, reusable outlines, grid guides, resize
handles, labels, badges, and overlays.
- [x] Keep collaborator colors as data visualization while guaranteeing
readable labels.
- [x] Prevent Builder theme values from leaking into authored content.
- [x] Migrate editor surfaces, selections, autocomplete, lint states,
and tooltips.
- [x] Derive syntax visualization from neutral and accent-relative
colors; reserve intent colors for diagnostics.
### Explicit color domains
- [x] Isolate external search and social previews from the application
scheme.
- [x] Preserve user-authored CSS values, functional color previews, and
checkerboards.
- [x] Review brand surfaces independently in both schemes.
- [x] Document every retained literal in the exception ledger.
### Stories, tests, and legacy removal
- [x] Migrate all affected Builder stories and tests.
- [x] Replace the design-system test that intentionally references the
legacy scale.
- [x] Add theme-preference and scheme-resolution behavior tests.
- [x] Remove the legacy `design-tokens.color` registry.
- [x] Remove the Stitches `colors` scale and `rawTheme.colors`.
- [x] Remove color-name conversion code that exists only for the legacy
scale.
- [x] Add a focused repository check that rejects legacy references.
- [x] Confirm no compatibility mappings or deprecated aliases remain.
### Browser-test infrastructure
- [x] Run pure Design System tests in Node and browser-dependent tests
in Vitest Browser Mode with Playwright Chromium.
- [x] Migrate all 79 Builder E2E workflows from the custom harness to
standard Playwright Test.
- [x] Replace filename-based shard discovery with standard Playwright
sharding.
- [x] Preserve clean-backend shard retries and upload Playwright failure
artifacts in CI.
- [x] Remove the custom runner, registry, and timeout infrastructure.
### Verification and documentation
- [x] Compare light-mode visual regressions directly with `main`; review
every difference rather than accepting snapshots wholesale.
- [x] Add representative dark coverage for dashboard, Builder shell,
style panel, dialogs, editors, canvas tools, and status states.
- [x] Test default and extreme custom-theme presets.
- [x] Verify semantic and categorical-recipe contrast.
- [x] Test first paint, persistence, system changes, portals, and canvas
isolation.
- [x] Run affected tests, typechecks, lint, formatting, and Storybook
checks.
- [x] Review fixture impact; no fixture inputs or generated outputs are
affected.
- [x] Review authoritative `docs/` content and update it where needed.
- [x] Review the final diff and make this checklist reflect delivered
scope.
## Current verification
- Design-system typecheck and generated-type freshness: passed.
- Design-system tests: 222 passed, including the full theme-parameter
contrast sweep and browser component coverage.
- Builder typecheck and production build: passed.
- Theme preference and scheme-resolution tests: 10 passed.
- Scoped Oxlint, formatting, and diff checks: passed.
- Design-system Storybook build and representative light/dark visual
review: passed.
- Builder E2E: 80/80 Playwright entries passed (setup plus 79
workflows).
Ref #3818
Closes #6267
|
||
|
Oleg Isonen
|
f9681dc30f |
feat: MDX-backed Content Blocks (#6234)
## Summary Supersede #6182 with a clean implementation of MDX-backed Content Blocks on top of the unchanged `main` Content Block editing model. A connected `.mdx` Asset changes where authored children are loaded and saved; it must not change how Content mode inserts, edits, selects, moves, copies, or deletes instances. The previous implementation remains available in `agent/mdx-content-block-6124` and `agent/mdx-content-block-6124-review`. Reuse its independently tested parsers, converters, schemas, Asset repository, template resolution, publication, MCP, and UI work where those pieces fit the corrected architecture. Do not port its per-operation storage routing, projected editing stores, MDX undo/history, or atomic multi-root machinery. Ref #6124 Ref #6182 Closes #6261 ## Locked behavior - `.md` keeps its existing Markdown behavior; `.mdx` enables MDX. - Standard Markdown structures are stored as Markdown. JSX is used only for structures Markdown cannot represent losslessly. - Generic elements use `<ws.element ws:tag="…">`; template instances use `<ws.element ws:name="…">`. - Invalid, unsafe, or unsupported MDX never prevents editing the file or publishing the project. Builder and publication use the same recovery result: recoverable nodes are skipped with source-located warnings, while an unrecoverable document renders no authored file content in either environment. - Connecting a source never serializes existing Content Block children into the file. **Connect** removes ordinary persisted children, preserves Templates, sets `src`, and uses file content. - Disconnecting resets `src`, discards derived instances, leaves the Content Block empty, and never changes the Asset. - While connected, normal Content Block mutations update in-memory instances synchronously. Asset persistence happens afterward and never blocks rendering. - Genuine external revision conflicts use the standard reload-required flow. Rapid local edits are ordered/coalesced and must never conflict with each other. - Repeated scopes use Content Block ID plus Collection item identity. - Multiple Content Blocks may intentionally share one MDX Asset. They share one revisioned document session and edits propagate to every connected occurrence. - Missing templates show warnings/placeholders only in Builder and are skipped on published sites. ## Reuse strategy - [x] Audit all 62 original implementation commits and 5 later review/fix commits - [x] Reuse the shared Content Engine Markdown/MDX AST, parser, serializer, diagnostics, reference discovery, and tests - [x] Reuse SDK source schemas, instance-name utilities, unique template naming, and source-integrity validation - [x] Reuse template resolution/materialization and authored-tree conversion after removing storage-operation coupling - [x] Reuse revisioned Asset content repository, authorization, limits, and browser/server transport - [x] Reuse publication, document-graph, CLI, MCP, source-control UI, frontmatter UI, and E2E work where contracts remain valid - [x] Reuse tests as behavioral specifications; rewrite tests that assert the rejected projection/routing architecture - [x] Do not port per-operation MDX routing, alternate projected mutation stores, semantic reconciliation after each command, MDX Builder history/undo, atomic multi-file rollback, or canvas loading/saving UI ## Old-branch porting map — do not rewrite Before implementing any checklist batch, inspect the named implementation and tests on `agent/mdx-content-block-6124-review`. Move the implementation and its tests first; change only the boundary that conflicts with the rebuilt architecture. Do not create a second implementation under a new name. When a test encodes the rejected projected-store architecture, retain its behavioral assertion and replace only its setup. ### Already ported - `parseMarkdownAst`, `createMdxDocumentFromMarkdown`, `parseMdxDocument`, `serializeMdxDocument`, `preferMarkdownSyntax`, `discoverMdxBodyAssetReferences`, `discoverMdxAssetReferences`, and `previewMarkdownToMdxConversion` - Asset-reference, byte-stream, document-graph, Markdown, and MDX helpers included in commit `76bc388ba9` - Tests: the corresponding `packages/content-engine/src/**/*.test.ts` files already on this branch ### Port with their existing tests | Area | Functions to move or adapt | Tests to move with them | | --- | --- | --- | | Source contract and integrity | `parseContentBlockSourceProp`, `getContentBlockSource`, `isEqualContentBlockSource`, `formatContentBlockSourceIntegrityIssue`, `getContentBlockSourceIntegrityIssues`, `allocateUniqueContentBlockTemplateName` | `packages/sdk/src/content-block.test.ts`, `packages/sdk/src/schema/content-block.test.ts` | | Existing instance/name utilities | `findChildReferenceIndex`, `findParentInstanceReference`, `getInstanceName` | `packages/sdk/src/instances-utils.test.tsx` | | Template naming in mutations | `assignUniqueBlockTemplateNamesMutable`, `findBlockTemplateNameCollision`, `getSourceBackedBlockTemplateContext`, `getBlockTemplateNameConfirmation`, `getBlockTemplateInsertionIndex` | `packages/project-build/src/runtime/block.test.ts` | | Content-mode property permissions | `getContentModePropEligibility`; preserve its distinction between editable, design-only, incompatible, and unknown props | `packages/project-build/src/runtime/content-mode-permissions.test.ts` | | Template resolution | `resolveMdxTemplates` | `packages/project-build/src/runtime/mdx-template-resolution.test.ts` | | Template materialization | `createMdxScopeIdGenerator`, `materializeMdxTemplates` | `packages/project-build/src/runtime/mdx-materialization.test.ts` | | MDX → instances and instances → MDX | `isMaterializedMdxAuthoredContentRoot`, `preserveMdxAuthoredContentIds`, `materializeMdxAuthoredContent`, `reconcileMdxAuthoredContent`, `omitTransientEmptyParagraphs`, `serializeMdxTemplateInsertion` | `packages/project-build/src/runtime/mdx-authored-content.test.ts`; retain the pure serialization cases from `mdx-storage-adapter.test.ts` | | Revisioned Asset I/O | `createAssetContentRevision`, `readAssetContentBytes`, the package-owned `createHttpAssetContentRepository`, the thin Builder `createBuilderHttpAssetContentRepository`, `createAssetContentBridge`, `initAssetContentBridge`, `getAssetContentBridge`; keep one repository implementation rather than copying the old Builder duplicate | `packages/asset-uploader/src/content-repository.test.ts`, `apps/builder/app/builder/shared/assets/mdx-content-repository.test.ts`, `apps/builder/app/shared/asset-content-bridge.client.test.ts` | | Source lifecycle planning | `createContentBlockBodyRemoval`, `prepareContentBlockConnect`, `prepareContentBlockDisconnect`, `prepareContentBlockSwitch`, `ContentBlockSourceRevisionConflictError` | `packages/project-build/src/runtime/content-block-source-lifecycle.test.ts` | | Per-Asset revision/session behavior | `getContentBlockSessionSource`, `isContentBlockSessionSourceCommitted`; reuse the loading, authorization, revision, queue, conflict, retry, cancellation, and source-replacement internals of `createMdxAssetEditingSession`, but key the rebuilt session by Asset rather than Content Block/render root | Port the applicable cases from `packages/project-build/src/runtime/mdx-asset-session.test.ts`; replace root-projection setup with in-memory snapshot setup | | Template migrations | `validateMdxTemplateMigrationConfirmation`, `planMdxTemplateMigration` | `packages/project-build/src/runtime/mdx-template-migration.test.ts` | | Publication | `createPublishedMdxMaterializationCache`, `materializePublishedMdx`, `getPublishedMdxContentDatabaseMaxBytes`, `getDynamicPublishedMdxSourceBlockIds`, `hasDynamicPublishedMdxSources`, `createPublishedBuildContentCompilationPlan`, `resolvePublishedMdxDependencyClosure`, `resolvePublishedMdxAssetCandidates` | `packages/project-build/src/runtime/published-mdx.test.ts`, `packages/project-build/src/content-database.test.ts` | | Build/import integrity | `assertContentBlockSourceIntegrity` and the Content Block cases in `getBuildIntegrityIssues`/`formatBuildIntegrityIssue` | `packages/project-build/src/build-integrity.test.ts` | | MDX paste/MCP insertion | `insertMdxText` and its input/result schemas; retain parsing, confirmation, template-resolution, and insertion-index logic, then invoke the normal mutation path | `packages/project-build/src/runtime/mdx-paste.test.ts` | | Builder source control | `createContentBlockSourceController`, `createBuilderContentBlockSourceController` and the existing source-control components | `content-block-source-controller.test.ts`, `content-block-source.test.tsx` | | Diagnostics | `formatContentBlockDiagnostic`, `deduplicateContentBlockDiagnostics`, `takeNewContentBlockDiagnostics` | Port only their focused cases from `apps/builder/app/shared/content-block-content.test.ts` | ### Explicitly do not port - `projectContentStorageChanges`, `createContentStorageProjection`, `createContentStorageSelectorProjection`, `executeContentStorageMutation`, `executeContentStorageTextReplacement`, `executeContentStoragePropMutation`, and `executeContentStorageStructuralMutation` - `applyMdxContentStorageChanges` and `prepareMdxContentStorageWrites`; replace them with one post-mutation snapshot serializer built from the reusable pure `reconcileMdxAuthoredContent` converter - `createContentBlockApplicationOperations` and the persistence-plan wrappers that route individual editing commands - Builder `$runtime*` projected stores, materialized mutation functions, alternate selector/tree helpers, pending projected-change state, and MDX-specific mutation sequences from `content-block-content.ts` - Canvas loading/saving/open-file presentation state, MDX history/undo, and atomic rollback machinery ## Implementation checklist ### 1. Preserve the `main` editing architecture - [x] Keep normal synchronous insert, text, Enter, slash menu, property, delete, move, wrap, copy/paste, selection, and focus mutations unchanged - [x] Materialize connected content through the same runtime instance interfaces used by ordinary Content Blocks - [x] Keep connected authored children in memory and exclude them from persisted build data - [x] Route persistence only after a synchronous mutation has updated the canvas - [x] Never wait for Asset loading, conversion, or saving before rendering an interaction - [x] Add no MDX-specific implementation to individual editing commands ### 2. Shared Markdown and MDX conversion - [x] Parse Markdown and the Markdown portion of MDX through one shared AST and plugin configuration - [x] Implement one shared MDX-to-instances converter and one shared instances-to-MDX converter - [x] Serialize headings, paragraphs, links, images, lists, tables, task lists, blockquotes, thematic breaks, code, emphasis, strong, strikethrough, and other supported GFM structures as Markdown - [x] Serialize only non-lossless/custom structures as Webstudio JSX - [x] Preserve nested Markdown in MDX elements, frontmatter, comments, tight/loose list semantics, code whitespace, reference links, and Asset references - [x] Omit transient empty draft paragraphs instead of emitting `<ws.element ws:tag="p" />` - [x] Preserve unsupported source text and authored metadata whenever it cannot be converted safely - [x] Add deterministic round-trip and canonical-output coverage for every supported Markdown structure ### 3. Invalid and unsafe MDX - [x] Preserve file source exactly; never silently repair or rewrite invalid source - [x] Ignore imports, exports, expressions, executable JavaScript, invalid properties, and other unsupported nodes when they can be isolated - [x] Continue rendering valid siblings and report every skipped subtree once with filename, line, and column - [x] Use the same recovery pipeline in Builder and publication so both render the same authored result for the current file revision - [x] When structural syntax prevents recovery, render no authored file content in Builder or publication instead of retaining a stale materialization - [x] Skip an unrecoverable Content Block on published sites and continue the build with a warning - [x] Keep invalid and unsafe files editable and reconnectable - [x] Enforce UTF-8, byte-size, depth, node-count, property-count, and generated-output limits with source-located diagnostics where possible ### 4. File formats and Asset creation - [x] Keep `.md` parsing and raw-HTML behavior unchanged - [x] Support creating, opening, editing, deleting, and recreating `.mdx` files through the normal text-file UI - [x] Open `.mdx` in the same editor experience as `.md` - [x] Always create an independent empty MDX Asset even when identical empty files exist - [x] Allow renaming `.md` to `.mdx` without converting or rewriting its contents; after rename, handle invalid contents through the normal MDX diagnostics and recovery pipeline - [x] Update an already-open MDX editor immediately after canvas persistence - [x] Bound read/write buffering and verify the returned Asset identity and revision ### 5. Source contracts and lifecycle - [x] Support direct Asset bindings and dynamic expression bindings - [x] Keep source mutation controls available only in Design mode; show the connected filename and **Open** action read-only in Content mode - [x] Reject duplicate source props, unsupported prop types, missing Assets, non-MDX Assets, and unauthorized Assets - [x] Analyze selected file diagnostics before confirmation without modifying the block or file; MDX syntax diagnostics do not block connection - [x] For a non-empty block, show **Abort** and **Connect** and explain that ordinary children will be replaced - [x] On connect, remove every non-template child, preserve Templates, set `src`, and materialize file content - [x] Replace disconnect handoff with normal `src` reset; discard derived content and leave the block empty - [x] Remove disconnect confirmation and conversion; resetting a repeated binding leaves every occurrence empty - [x] Switching sources validates/loads the target before replacing the usable binding and leaves both files unchanged - [x] Revalidate Asset revision and project version when confirmation is submitted - [x] Make repeated connect/disconnect operations idempotent - [x] Prevent disposed, stale, or out-of-order loads from replacing the newest source - [x] Keep loading, saving, conflict, empty-file, and open-file actions off canvas ### 6. Asynchronous persistence and conflicts - [x] Snapshot the current in-memory authored root after normal mutations - [x] Serialize/coalesce rapid edits per resolved Asset and always prepare against the latest local snapshot - [x] Allow independent Assets to save concurrently without scope leakage - [x] Use one revisioned document session and one ordered save queue per Asset, even when multiple Content Blocks or render scopes reference it - [x] After a shared Asset changes, rematerialize every connected occurrence without persisting their generated instances to project data - [x] Pin the resolved Asset and exact loaded revision for each write - [x] Recheck authorization immediately before storage - [x] Ensure local queued edits never produce stale-revision conflicts with each other - [x] On a genuine external revision conflict, preserve local content, stop writes, and enter the standard reload-required state - [x] Keep failed non-conflicting writes retryable without discarding local content - [x] Finish already accepted persistence writes after a Content Block unmounts or switches source - [x] Cancel obsolete loads, materializations, and UI updates after unmount, reload, or source switch - [x] Dispose project-scoped roots, queues, parse caches, failure state, generations, and session subscriptions when the Asset session is replaced - [x] Invalidate Asset resources and synchronize open file editors after successful writes - [x] Keep multi-root persistence ordered, serial, and non-atomic; report partial completion without rollback - [x] Do not add Asset-history undo/redo; native text-editor undo remains local to the open editor ### 7. Content mode parity - [x] Empty connected blocks remain selectable and expose the normal Content-mode `+` affordance - [x] First insertion renders immediately - [x] Enter creates and focuses the next paragraph immediately - [x] Slash commands work in newly created empty paragraphs - [x] Deleting one selected instance deletes exactly that instance - [x] Mouse and keyboard moves use the same operation and permissions - [x] Preserve sibling reorder, nested reparenting, and moves between editable Content Blocks where `main` permits them - [x] Preserve content-editable props such as Link `href` - [x] Preserve copy, paste, cut, multi-cut, wrapping, unwrapping, grid insertion, selection, Navigator order, and canvas order - [x] Cover mutation routing centrally, retain the full unconnected Builder suite, and exercise representative connected interactions in real Builder E2E ### 8. Template names and permissions - [x] Resolve only exact user-facing names from the direct flat Templates list - [x] Enforce unique names within each Templates list - [x] Reject duplicate names during rename with inline validation - [x] Allocate unique names during insert, paste, duplicate, move, wrap, and unwrap - [x] Apply valid template properties using the same permissions as Content mode - [x] Preserve and warn about ignored design-only, invalid, or stale properties - [x] Preserve template shell provenance without serializing expanded template internals - [x] Generate deterministic materialized IDs from source identity, render scope, and authored usage path so focus/selection survives saves - [x] Warn before renaming or deleting referenced templates; files remain unchanged - [x] Provide shared AST-based rename/remove migration operations for MCP-assisted multi-file cleanup ### 9. Scope, frontmatter, and diagnostics - [x] Scope materialized content by Content Block ID plus exact Collection-item/render context - [x] Prevent repeated instances and dynamic candidates from leaking generated IDs, template resolution, or diagnostics across scopes while sharing the underlying Asset source and revision where applicable - [x] Remove frontmatter editing from Settings; edit the exact YAML through the MDX Asset file editor - [x] Preserve the exact frontmatter source byte-for-byte during canvas body saves - [x] Route canvas body changes and complete-file editor saves through one revisioned per-Asset coordinator - [x] Show a source warning indicator in Settings and precise source-located diagnostics in the Asset editor - [x] Deduplicate repeated diagnostics without losing distinct locations - [x] Keep unresolved-template placeholders selectable and accessible only in Builder ### 10. Asset references, transfer, and integrity - [x] Discover and rewrite Asset references in frontmatter, Markdown links/images, and MDX properties - [x] Preserve Asset references while editing unrelated content - [x] Include referenced Asset and template revisions in dependency/cache identity - [x] Route referenced Assets through published URL rewriting - [x] Validate Content Block source integrity during project import, fragment/page transfer, copy/paste, build integrity, and pre-publish audit - [x] Allow multiple Content Blocks and dynamic render scopes to reference the same MDX Asset without treating the connection as an integrity error - [x] Preserve Templates when copying a connected Content Block - [x] Preserve direct and dynamic Content Block source bindings when copying or pasting within the same project - [x] When copying only one rendered Collection item, resolve that item’s current MDX Asset and replace the copied Content Block’s dynamic source with a direct binding to that same Asset - [x] When copying an entire Collection, preserve the Collection, its instances, and its dynamic source bindings - [x] Preserve the complete MDX source and frontmatter by reference; copying a connected block must not rewrite the shared file - [x] During cross-project transfer, copy every discoverable distinct bound MDX Asset and rewrite transferred bindings to the destination Assets - [x] If some cross-project dynamic candidates cannot be discovered or copied, continue with successful copies, preserve what can be represented safely, and warn about skipped candidates - [x] Revalidate source revisions and the destination project before applying a cross-project transfer - [x] If cross-project Asset creation or project insertion fails, leave the source project unchanged and clean up any newly reserved destination Asset - [x] Use the same transfer operation through Builder and MCP ### 11. Dynamic sources and publication - [x] Resolve dynamic bindings once per render/save scope - [x] Require finite, discoverable candidate sets for publication and load no unrelated MDX - [x] Warn and skip unbounded, missing, unreadable, malformed, or unsafe sources without failing the complete build - [x] Materialize only rendered Content Blocks; ignore Templates definitions and `src` on non-Content-Block instances - [x] Skip unresolved template subtrees while keeping valid siblings - [x] Include selected template styles, fonts, resources, unresolved names, and referenced Asset revisions in generated dependency output - [x] Keep draft-only MDX Assets out of publication retention ### 12. Builder, MCP, CLI, and shared operations - [x] Use the same converters, source lifecycle, revision checks, persistence service, diagnostics, and template migrations in Builder and MCP - [x] Keep frontmatter editing file-based; retain whole-frontmatter runtime operations only where compatibility requires them - [x] Avoid Builder-only or MCP-only mutation/conversion implementations - [x] Regenerate runtime/API contracts from their source definitions - [x] Update CLI Content Engine reference and generated MCP documentation ### 13. Verification and delivery - [x] Add regression tests that reproduce failures from the previous implementation, including double deletion and blocked rendering - [x] Cover two Content Blocks sharing one Asset: edits from either block serialize once, update both canvases, preserve scope-specific Templates, and cannot overwrite a newer local snapshot - [x] Verify new tests fail against the broken implementation and pass with the corrected behavior - [x] Run SDK, Content Engine, Asset Uploader, Project Build, Builder, React SDK, CLI, Protocol, and HTTP Client affected suites and typechecks - [x] Run real Builder E2E for source lifecycle, live file refresh, empty insertion, Enter/slash focus, Link props, delete, keyboard reorder, switching, disconnect, conflicts, unresolved templates, dynamic sources, and repeated scopes - [x] Regenerate shared project fixture bundles and verify a second generation leaves the tree clean - [x] Run lint, package boundaries, `pnpm oxfmt --check`, and `git diff --check` - [x] Review and update authoritative `docs/` content and navigation - [ ] Run the complete agent evaluation suite — not run because it requires separate explicit approval ### 14. Content-mode parity and disconnect handoff - [x] Allow keyboard reordering in Content mode through the normal move command while keeping the editable Content Block boundary - [x] Let already accepted Asset writes drain independently before resetting the source; disconnect does not load, flush, or convert the file - [x] Remove MDX-to-instance copying from disconnect - [x] Reset `src`, remove the derived tree, and leave the ordinary Content Block empty - [x] Preserve automatic text-editor focus after inserting editable content from the template menu - [x] Remove Builder frontmatter-save UI and report complete-file save failures through the Asset editor - [x] Add regression coverage for keyboard movement, disconnect teardown, source switching, insertion focus, and paragraph deletion focus restoration - [x] Run the complete six-scenario MDX Content Block Builder E2E suite ### 15. Final architecture review fixes - [x] Keep one canonical ordered pending document per Asset so shared Content Block occurrences cannot conflict with one another - [x] Merge ordered local edits from different occurrences while preserving text changes, insertions, deletions, and sibling reordering; keep external revisions on the strict reload-required path - [x] Centralize typed render-scope parsing, repeated-occurrence detection, scoped variable lookup, and dynamic Asset resolution across Canvas, Settings, source lifecycle, and copy/paste - [x] Capture the owning Asset session when a save is accepted so normal unmount/source-switch teardown cannot redirect or lose it - [x] Dispose project-owned roots, queues, caches, generations, failure state, and subscriptions when replacing the project Asset session - [x] Add regression coverage for same-Asset edits from two occurrences, same-node last-write ordering, structural reorder, insertion preservation, scoped source resolution, and project cleanup - [x] Re-run affected full unit suites, typechecks, production build, static checks, and all six MDX Content Block Builder E2E scenarios ### 16. Final persistence and transfer hardening - [x] Drain every accepted per-Asset edit before project-session teardown while retaining the owning authorization context - [x] Drain canvas serialization queues and revisioned Asset writes before pre-publish audit and publication - [x] Block publication on a queued serialization failure or Asset revision conflict - [x] Copy actual Asset bytes during cross-project paste, preserve MDX reference names, rewrite direct and Collection-backed Asset IDs, and clean up newly created Assets when insertion fails - [x] Include every resolved dynamic MDX source when copying an entire Collection and warn while preserving successful copies when individual Assets cannot be transferred - [x] Keep MDX editor save failures visible with retry and route conflicts through the standard reload-required flow - [x] Give keyboard nested reparenting the same editable Content Block boundary used by pointer movement - [x] Add regression coverage and rerun the complete Content Engine and Builder unit suites plus type, lint, formatting, package-boundary, and diff checks ## Current rebuild status - The connected Content Block now owns one complete MDX document: frontmatter feeds designed-shell bindings and the explicit Body outlet owns the authored MDX body. - Builder and published output resolve the same frontmatter document graph, including nested document references and Asset references. - Direct frontmatter text/property edits use the existing ordered per-Asset session; referenced-document values remain file-owned. - Repeated Collection occurrences keep isolated runtime shell/Body identities and occurrence-specific document values. - The local blog article layout has been upgraded to the document/Body model and its bindings validate cleanly. - Full affected package suites and typechecks pass; every new regression was demonstrated fail/pass. - `pnpm lint`, package boundaries, `pnpm oxfmt --check`, generated API/docs checks, fixture builds, and `git diff --check` pass. - Full fixture synchronization is blocked by the development API rejecting this local CLI version before fixture generation; existing fixture bundles build successfully. - Complete agent evaluations remain pending explicit approval. ### 17. Treat Content Block source as a property - [x] Declare Content Block `src` in component metadata as a Design-mode initial property labeled **Source** - [x] Add the short property description: “Create an .mdx file in Assets, then select or bind it here.” - [x] Accept only a direct Asset or an expression resolving to an Asset ID; reject URL/string sources, missing Assets, and non-MDX Assets - [x] Extract and reuse the Image-style Asset picker and binding presentation without duplicating source-control UI - [x] Render Source through the normal properties section and remove the separate source-section placement and special `src` filtering - [x] Route Source changes through Content Block connect, switch, and disconnect lifecycle operations instead of generic prop updates - [x] Keep connect confirmation, Open, diagnostics, and persistence state attached to Source; remove frontmatter and disconnect UI - [x] Add regression coverage for direct Assets, dynamic Asset bindings, invalid resolved values, lifecycle operations, and Design/Content mode visibility - [x] Re-review authoritative `docs/` content for the Source-property UI change and update it if required ### 18. Source property UX - [x] Show one full-width bindable Connect .mdx file button while disconnected - [x] Show equal-width filename and Open buttons while connected - [x] Switch files from the filename button through the source lifecycle - [x] Override PropertyLabel reset and Alt-click with the disconnect lifecycle - [x] Preserve confirmation, pending, failure, repeated-scope, and dynamic-source behavior - [x] Add regression coverage for connect, switch, binding, reset, and disconnect - [x] Update Content Block documentation - [x] Run affected tests, typecheck, lint, formatting, package boundaries, and diff checks ### 19. Reduced file-owned frontmatter follow-up This checklist supersedes the earlier structured-frontmatter and disconnect-copy behavior. Frontmatter remains file-owned and is edited only in the MDX Asset editor. #### A. Remove structured frontmatter UI - [x] Remove the JSON frontmatter editor and unused Builder controller/state - [x] Keep source mutation Design-mode-only; show connected Source read-only in Content mode and allow authorized editors to open the complete file - [x] Retain runtime frontmatter APIs only where compatibility requires them #### B. Preserve source and separate validation - [x] Preserve the exact frontmatter envelope during every canvas body save without parsing/stringifying YAML - [x] Preserve comments, ordering, quoting, whitespace, BOM, newline style, delimiters, and invalid but closed YAML - [x] Keep the existing unsupported-node/property MDX recovery; do not add an ad hoc partial parser - [x] Parse body and frontmatter independently so either valid half remains usable - [x] Keep bounded Asset-query frontmatter extraction unchanged - [x] Treat structurally malformed MDX as having no safe body tree #### C. Share Asset state with the raw editor - [x] Route canvas and complete-file changes through one ordered, non-blocking per-Asset coordinator - [x] Update canvas and clean raw editors immediately from shared in-memory source - [x] Reject stale dirty-editor saves, preserve unsaved text, and require reload; never force-overwrite - [x] Drain accepted writes after source replacement/disconnect and report background failures globally - [x] Dispose sessions only after they have no consumers and no queued work #### D. Connect, replace, and disconnect - [x] Load and diagnose before connect; block only unreadable/unauthorized/non-MDX Assets - [x] Show **Abort / Connect** for non-empty blocks, including diagnostics and the recoverable-content warning - [x] Set `src` and delete persisted children through one runtime mutation - [x] Switch to a loaded source without waiting for the previous Asset's network save - [x] Remove disconnect confirmation and conversion - [x] Reset `src` immediately, discard the derived tree, leave the block empty, and preserve the Asset - [x] Resetting a bound source leaves every repeated occurrence empty #### E. Dynamic and shared sources - [x] Keep one source/revision state per Asset and a separate derived instances tree per occurrence - [x] Resolve Templates independently per Content Block and preserve Builder-only missing-template placeholders - [x] Resolve bound `src` from the selected Collection occurrence - [x] Make **Open** use the occurrence's resolved Asset; update the filename and disable Open when unresolved - [x] Add direct/bound tests including Collection item A opening `a.mdx` and item B opening `b.mdx` #### F. Diagnostics, MCP, and publication - [x] Show an accessible warning icon beside Source with the first diagnostic, line/column, and remaining count - [x] Show precise YAML/MDX diagnostics in the Asset editor without repetitive canvas/toast UI - [x] Reuse parser/diagnostic/revision contracts in Builder, build, publication, and MCP - [x] Keep MCP frontmatter editing file-based and return structured diagnostics for complete-source writes - [x] Emit build warnings; published sites skip invalid/unsupported content without diagnostic UI #### G. Verification and documentation - [x] Add fail/pass regressions for source preservation, independent validation, stale raw saves, lifecycle, dynamic Open, shared Assets, and cleanup - [x] Run affected tests, typechecks, lint, formatting, package boundaries, fixtures, and diff checks - [x] Open Markdown and MDX editors at twice the default dialog size with the right-side preview visible initially - [x] Document canvas body editing, file-owned frontmatter, Content-mode Asset access, and cloning an MDX Asset for a new post - [x] Perform a final correctness, data-loss, DRY, simplification, accessibility, and performance review - [x] Keep this checklist synchronized with implementation status ### 20. Content-mode component properties - [x] Show a connected Content Block Source read-only in Content mode with an active **Open** action - [x] Keep source switching, binding, reset, and disconnect available only in Design mode - [x] Expose HTML Embed code and preserve Code Text's code-editor metadata on authored Text Content - [x] Audit every registered component and expose authored payload while keeping design, structure, and behavior controls in Design mode - [x] Keep media dimensions and Code Text theme Design-only while preserving adapter-owned intrinsic Image metadata during MDX round trips - [x] Add exact component-permission and regression coverage - [x] Update authoritative component documentation - [x] Run affected full suites, typechecks, formatting, lint, package boundaries, and generated contract/documentation checks ### 21. Designed document shell and Body outlet - [x] Add one structural **Content Block Body** outlet and keep legacy Content Blocks without it compatible - [x] Materialize and persist only `document.body` children through the Body outlet while preserving the surrounding designed shell and Templates - [x] Expose the connected document frontmatter through a Content Block `document` parameter for shell text and property bindings - [x] Write direct frontmatter-bound text and properties back through the same ordered per-Asset session without blocking canvas mutations - [x] Resolve document `$ref` dependencies and Asset references consistently in Builder and published output - [x] Keep referenced-document values read-only on canvas and direct editors to the referenced file - [x] Isolate repeated Collection occurrences while retaining occurrence-specific source, frontmatter, shell, and Body content - [x] Render the same designed shell, frontmatter values, and Body substitution in generated/published output - [x] Upgrade the local blog article layout to bind title, excerpt, dates, author data, and images from frontmatter and render MDX through Body - [x] Add fail/pass regressions for nested Body discovery, lifecycle, permissions, repeated values, document references, and generated shell rendering - [x] Update authoritative Content Block documentation - [x] Run affected tests and typechecks, formatting, lint, package boundaries, generated API/docs checks, and fixture builds ### 22. Final nested references and frontmatter editing - [x] Keep direct frontmatter-bound text and property controls editable in Content mode without exposing or replacing their bindings - [x] Keep referenced-document values read-only and preserve legacy instance-property name normalization - [x] Render direct frontmatter text bindings through the plain-text editing path - [x] Resolve nested legacy and structured Asset references from referenced Markdown/MDX frontmatter - [x] Preserve missing Asset references, emit structured `ASSET_NOT_FOUND` diagnostics, and keep them out of the document graph - [x] Add fail/pass regressions for the real Settings control and root/nested missing Asset-reference cases - [x] Fix exact Content Block selection after the Body node appears in Navigator - [x] Run full affected suites/typechecks, all six MDX Content Block E2E scenarios, deterministic fixture builds, generated checks, lint, formatting, package boundaries, and diff checks ### Before merge - [x] Resolve #6261: Nested Asset references in referenced Markdown frontmatter work consistently in Assets queries, previews, publication, and static export ### 23. Explicit writable expression bindings - [x] Add an explicit `readwrite` mode to expression bindings; existing expressions remain read-only by default - [x] Accept `readwrite` only for direct static member paths and centralize their parsing and validation - [x] Require the expression root to provide a writable data source; initially support the connected Content Block document frontmatter - [x] Update Content Block permissions, canvas text editing, and property controls to require a valid writable binding - [x] Persist direct frontmatter edits through the existing ordered per-Asset session - [x] Create new direct frontmatter bindings with `readwrite` mode and migrate existing compatible bindings - [x] Preserve writable mode through schemas, serialization, loading, copy/paste, transfer, and export - [x] Add fail/pass regressions for valid paths, derived expressions, unsupported sources, permissions, persistence, and compatibility - [x] Review authoritative `docs/` content and update it if the persisted binding contract affects user-facing documentation - [x] Reject unsafe frontmatter paths and reject `readwrite` bindings outside a connected Content Block document - [x] Preserve expression binding mode through the public HTTP client and document the MCP authoring contract - [x] Limit frontmatter value write-back to Content mode so Design mode retains normal binding replacement/reset behavior - [x] Keep bindings inside Templates and across nested Content Block ownership boundaries read-only - [x] Upgrade compatible legacy bindings even when the Content Block is disconnected during migration - [x] Run affected tests, typechecks, formatting, lint, package boundaries, fixture checks, and `git diff --check` Verification: full SDK (567), Project Build (2,277), and Builder (2,695) suites pass, along with affected Expression, Migration, Template, HTTP Client, and MCP suites; typechecks, lint, formatting, package boundaries, and diff checks pass. Generated contracts are refreshed. Fixture synchronization remains blocked before generation because the development API rejects this local CLI version as incompatible. The checked-in bundles pass the current Protocol suite and every fixture build with the current bundle version. |
||
|
Oleg Isonen
|
5a57234c75 |
revert: Store Content Block content in MDX assets
Revert the accidental merge of #6125 so the implementation can continue in draft PR #6181. |
||
|
Oleg Isonen
|
86b6310fff | feat: Store Content Block content in MDX assets | ||
|
Oleg Isonen
|
c44ef705cf |
perf: Speed up Builder resource loading (#6011)
## Problem Assets-backed Builder pages can take 6–7 seconds to become usable even when their Assets resources only request metadata. On the production-derived Blog project, a single Assets query measured 8,703 ms and an immediate repeat measured 8,110 ms. Four sibling Assets resources repeated the same authorization, development-build load/parse, compilation, canonical-asset preparation, runtime metadata load, and document work. ## Architecture The loader now treats remote resources holistically while preserving provider boundaries: - The client scheduler is resource-provider neutral. It versions logical requests, frees capacity when obsolete requests are replaced, ignores stale results, and aborts physical batches only when every member is obsolete. - Ordinary Assets requests in one server resource-loader call use a deferred Assets provider and execute as one repository batch. Independent HTTP, GraphQL, and other resources start immediately instead of waiting for Assets preparation. - Arbitrary remote responses are **not** coalesced or persistently cached. Their auth headers, cookies, upstream freshness, and `Vary` semantics remain isolated. - The Assets provider authorizes and creates its API context once per batch, then loads only `props`, `dataSources`, and `resources` from the development Build. - Queries covered by the saved Build compilation plan reuse that exact plan/artifact, preserving published byte limits, truncation, and pinned-revision behavior. Uncovered unpinned queries use a deterministic literal union and safely fall back to their existing individual path if shared preparation cannot preserve behavior. - Request-scoped document loading uses npm's byte-bounded `lru-cache` so sibling queries do not reread the same referenced author/content files. - Sibling dynamic queries with the same filter, sort, output, and content shape stitch into one document-graph resolution while keeping filter values, pagination windows, results, and failures independent. Aggregate document/byte-limit failures fall back to isolated execution. - Provider-neutral batching lives in the SDK runtime, and project asset-query preview orchestration lives in the shared project package so Builder and MCP paths can reuse the same behavior. - Cancellation flows from the client batch through the SDK/server/build/query path, with checkpoints between expensive phases. - Runtime asset metadata remains conservatively artifact-wide because URL/size filters, structured references, markdown hydration, materialized rows, and saved plans have different reachability rules. To make 1,000-asset artifacts safe, PostgREST `.in(...)` lookups are deduplicated and split into 100-ID chunks with bounded concurrency. ## Implementation checklist - [x] Trace the production Blog resource path and identify repeated work - [x] Copy the production Blog project into an ignored disposable local project - [x] Record production and local baselines - [x] Add provider-neutral client scheduling, stale-result protection, and cancellation - [x] Batch ordinary Assets queries without coupling or caching HTTP/GraphQL responses - [x] Load only Build fields required to create the saved content compilation plan - [x] Preserve saved-plan, truncation, fallback, diagnostics, and revision semantics - [x] Share Assets artifacts, runtime metadata, and revision-keyed document reads per request - [x] Stitch compatible query shapes into one graph resolution with isolated fallback - [x] Extract reusable SDK batching and project preview orchestration for MCP/runtime consumers - [x] Bound large runtime asset metadata lookups for the 1,000-asset project limit - [x] Add regression coverage and demonstrate new tests failing before their fixes - [x] Run focused and broad tests, type checks, lint, format, package-boundary, and diff checks - [x] Review documentation impact; no authoritative `docs/` content changes are required - [x] Regenerate and verify the affected asset-query runtime fixture; no production data is changed - [x] Address all actionable PR review comments - [x] Rerun the final parity-preserving implementation against the copied Blog project ## Measurements Production trace before the change: - Updates Assets query: 8,703 ms - Immediate identical repeat: 8,110 ms - Query artifact: 9,344 bytes / 7 documents - Saved source database: 42,086 bytes / 28 documents Copied-project local baseline: - Builder navigation until resource loading: 5,111 ms - Exact four-resource batches: 1,448 / 797 / 1,377 / 774 / 3,018 ms During implementation, before the final saved-plan parity tightening: - Exact four-resource batch: 174 ms first run, then 51–80 ms - Actual Builder resource-loader requests: 234–246 ms Final parity-preserving copied-project rerun: - Preflight: 10 ms - Exact four-resource batch: 435 ms cold, then 119 / 97 / 90 / 79 ms warm - Every response returned HTTP 200 with the same 16,884-byte payload The final cold batch is 70% faster than the best copied-project cold baseline, and the fastest warm batch is 90% faster than the best warm baseline. The production trace remains the authoritative production baseline; these before/after comparisons use the same disposable local project and captured request payload. ## Verification Passed: - Full package suites: Builder (212 files / 2,496 tests), asset-uploader (22 / 193), content-engine (40 / 274), project (7 / 84), SDK (29 / 508), and project-build (82 / 2,070) - Final focused regressions: Builder preview/API/resource integration (3 files / 42 tests), SDK resource loader (23 tests), and document-graph resolution (6 tests) - Builder, asset-uploader, content-engine, project, SDK, and project-build type checks - Fixture regeneration is deterministic; the generated vendor exactly matches the source runtime bundle - Fixture typecheck and production build - Changed-file Oxfmt, repository Oxlint, and package-boundary checks - `git diff --check` - `git diff --name-only HEAD --exit-code` New behavioral tests were demonstrated to fail before implementation and pass afterward, including Assets batch coordination, saved-plan wiring, truncation and pinned revisions, shared document reads, cancellation checkpoints, stale/replaced client requests, and large PostgREST ID filters. Agent evaluations were not run because they require separate explicit approval. ## Compatibility and remaining work The public resource result/error shape and on-demand diagnostics contract stay compatible. Generic HTTP/GraphQL caching semantics are unchanged. Compilation already in progress is not force-cancelled because safely interrupting shared compilation requires ownership/reference tracking; cancellation prevents later phases from starting. Closes #6003 |
||
|
Oleg Isonen
|
9fe63c7501 |
fix: Submit webhook forms only once (#5979)
## Summary Webhook Form action resources were being persisted both as prop-bound actions and render-time data sources. Generated sites could therefore execute the POST while rendering and again when the visitor submitted the form. This change keeps prop-bound resources action-only and hardens generation for existing builds that still contain a stale data-source binding. Ref #5806 ## Implementation - Keep `resources.upsertProp` resources out of render-time data and remove stale exposure when an action is edited. - Make the mutation's `dataSourceId` result optional and regenerate the runtime contract. - Exclude action-bound resources from the generated data-resource map, preserving them only in the action map. - Extend the Webhook Form E2E with a local request-counting endpoint. ## Todo - [x] Prevent new Form actions from creating render-time data sources. - [x] Protect existing persisted builds from loading action resources during rendering. - [x] Regenerate the runtime operation contract. - [x] Add mutation, generation, Builder-store, and generated-site regression coverage. - [x] Verify zero calls on page load and one populated POST on submit. ## Verification - [x] Focused Builder E2E: `Webhook Form action submits once and persists after reload` - [x] Project Build test suite: 2,062 tests - [x] SDK test suite: 505 tests - [x] Builder resource mutation test file: 25 tests - [x] SDK, Project Build, and Builder typechecks - [x] Changed-file lint, formatting, and diff checks - [x] Generated API cleanliness check - [ ] Agent evaluation suite (not run; requires explicit approval) ## Notes No fixture inputs or generated fixture bundles are affected. |
||
|
Oleg Isonen
|
efbc3098c4 |
feat: add queryable Assets resources (#5946)
Closes #5919 Ref #5920 ## Outcome Extend the existing Assets system resource with optional queries for Markdown, JSON, and standard file metadata. Unconfigured Assets resources retain the existing fetch-all behavior and activate none of the content-query infrastructure. Files are the authoring model. A standalone content engine converts an immutable snapshot of only the files required by configured queries into one bounded, deterministic runtime database. Builder preview, SSR, and SSG use the same query semantics without requiring a database service for a published site. ## Final architecture ```text Markdown / JSON files │ ▼ ContentSource adapter (FS or Webstudio Assets) │ immutable snapshot ▼ ContentCompilationPlan (union of configured query requirements) │ ▼ ContentCompiler (parse, normalize, discover fields, enforce budget) │ ▼ ContentArtifact (bounded, deterministic, immutable JSON) │ ▼ ContentDatabase (filter, OR groups, sort, paginate, project, read content) │ ├── Builder preview ├── SSR runtime └── SSG build only ``` The authoring/storage layer owns mutable files. The content engine consumes immutable snapshots and produces immutable, disposable artifacts. Query authoring is a separate OpenAPI-driven path: ```text Assets REST OpenAPI operation + standard request JSON Schema + dynamic field paths │ ▼ Generic OpenAPI adapter │ resolved schemas, controls, paths, and transport mapping ▼ Generic query-builder core and React renderer │ arbitrary provider-defined value ▼ Assets REST request adapter │ ▼ { query: value } ``` The selected standard OpenAPI operation is the only source of query UI semantics. Request parameters and JSON Schema describe the editable data; the generic renderer derives controls and owns layout. ### Activation and compilation - The compiler does not index every asset. It combines all reachable configured Assets-resource queries into one `ContentCompilationPlan`. - Static constraints such as folder, path, and file type select candidate files. Dynamic values such as a route-bound slug cannot reduce the build-time candidate set. - The plan contains the union of fields used by filtering, sorting, routing, projection, excerpts, and requested full content. - Documents shared by multiple resources are stored once in one deployment-wide runtime database. - Markdown/frontmatter, JSON properties, excerpts, and full bodies are prepared only when required. - No configured content query means no parsing, artifact, or shipped runtime dependency. Fresh generated apps need no content-runtime cleanup; incremental generation removes stale integration if a previously configured query is deleted. ### Runtime database limit - `CONTENT_DATABASE_MAX_BYTES` limits the complete serialized runtime database, not source files, images, or unrelated assets. - The default is `500 * 1024` bytes and is presented as 500 KiB. - Counted data includes indexed metadata, frontmatter/JSON properties, excerpts, the field catalog, serialization overhead, and full Markdown bodies only when requested. - Prepublish validation builds and serializes the exact database that will be attached to the immutable build, so it knows the unbounded size, bounded size, included documents, omitted documents, and affected resources. - Overflow never stores partial documents. Candidates are ordered by `createdAt` descending, canonical path ascending, and stable document ID ascending. Missing dates follow dated documents. A document that does not fit is skipped while smaller later documents are still considered. - Exceeding the limit warns but does not block publishing. - Builder preview can report exact bounded versus unbounded results when dynamic expressions have concrete preview values. Prepublish warnings remain conservative for future route values. ### Snapshot consistency The source contract guarantees a consistent compilation, while adapters provide the mechanism: - Webstudio Assets captures asset metadata and immutable content references. - Filesystem sources capture file identity and verify files did not change before accepting the artifact. - A changed source retries once, then fails clearly instead of producing a mixed-revision database. - This requires no universal snapshot service, source-tree copy, or new versioned-asset infrastructure. ### Package ownership - `@webstudio-is/content-engine`: runtime artifact schema, query execution, field catalog, compact runtime statistics. - `@webstudio-is/content-engine/compiler`: source snapshots, compilation plans, Markdown/JSON parsing, deterministic compilation, byte accounting, and compiler diagnostics. This entry exists to keep Remark/YAML/future MDX dependencies out of runtime bundles. - `@webstudio-is/query-builder`: resolves standard OpenAPI operation schemas into an internal provider-neutral editor model, validation, mutations, and source serialization. It owns no Assets, provider-authored UI contract, or fixed REST-query semantics. - `@webstudio-is/query-builder/react`: container-neutral schema renderer whose layout is renderer policy. It remains separate from the content engine. - SDK: Webstudio Assets/Resource domain models and mapping Webstudio expressions into generic content requirements. - Protocol: external Assets REST/OpenAPI transport contracts. - Asset uploader/repository: asset and folder mutations, generic metadata, immutable byte references, and storage access. Its hosted adapter supplies Webstudio Assets snapshots and parsed-metadata cache persistence to the content engine; parsing, artifact creation, and query semantics remain engine-owned. - Builder and generated-app integrations decide when to compile, cache, embed, load, or discard a database. ### Runtime lifecycle - Builder caches immutable `ContentDatabase` instances by source revision plus compilation-plan hash and coalesces concurrent compilation. Matching operations reuse the same instance; field discovery uses a dedicated minimal all-properties plan that excludes excerpts and bodies. - SSR attaches the validated bounded database to the build and includes the small runtime only when required. - SSG uses the database only while generating pages and omits it and the runtime from final output unless a future runtime-query feature requires them. - Local development compiles from the filesystem and uses identical query semantics. - Published pages query their local artifact; they do not call Builder REST endpoints. - Published artifacts need no external index store, invalidation, garbage collection, SaaS content proxy, new bucket, or new environment variable. ## Implemented foundation These existing parts remain useful and should be moved behind the final ownership boundaries rather than reimplemented. - [x] Upgrade the existing Assets system resource in place and preserve unconfigured fetch-all behavior. - [x] Implement typed filtering, nested boolean groups, deterministic sorting, pagination, projections, dynamic values, and content hydration. - [x] Add Markdown frontmatter, excerpt, JSON-property, and field-catalog extraction. - [x] Add reusable query-control implementations and a separately imported React UI as a foundation for the final OpenAPI-driven contract. - [x] Add Assets query authoring and dynamic field discovery as existing behavior to migrate into the Assets OpenAPI operation. - [x] Add synchronized visual and editable expression authoring. - [x] Unify Assets REST operations under `/rest/assets/*`. - [x] Define authenticated collection, item, content, upload, folder, query, field-catalog, and OpenAPI operations. - [x] Support Builder cookie authentication and public API token authentication. - [x] Add Builder preview, MCP/CLI operations, SSR execution, SSG path generation, and conditional activation. - [x] Add revision-keyed parsed metadata persistence for the hosted Builder. - [x] Add focused query, API, authorization, Builder, CLI, SSR, SSG, migration, and fixture coverage. ## Remaining implementation plan ### 1. Establish the content engine - [x] Rename the generic `@webstudio-is/asset-resource` implementation to `@webstudio-is/content-engine`. - [x] Move existing behavior instead of creating a parallel implementation. - [x] Keep the primary entry runtime-only and add only the compiler entry needed for bundle separation. - [x] Remove the old package after all consumers migrate. - [x] Ensure the engine has no Builder, SDK, asset-uploader, PostgreSQL, R2, or deployment-target dependency. ### 2. Define source snapshots and compilation plans - [x] Introduce `ContentSource.openSnapshot()` and a revision-bearing `ContentSourceSnapshot`. - [x] Include canonical path, stable document ID, content type, immutable content reference, size, and optional `createdAt`. - [x] Implement Webstudio Assets and filesystem snapshot guarantees using their adapter-specific mechanisms. - [x] Introduce `ContentCompilationPlan` for the union of reachable configured resource requirements. - [x] Separate statically selectable candidates from dynamic runtime predicates. - [x] Add mutation-during-compilation tests for replacement, deletion, folder movement, and filesystem changes. ### 3. Separate storage from content processing - [x] Keep upload, replacement, folders, byte storage, revisions, and generic metadata in the Assets repository/uploader. - [x] Move parsing, excerpts, field discovery, query execution, and artifact creation into the content engine. - [x] Remove content-index coordination from asset mutation business logic. - [x] Preserve original source `createdAt` across content replacement and expose it as optional queryable metadata. - [x] Keep hosted parsed-document persistence behind the content compiler's optional cache interface. ### 4. Build deterministic compilation - [x] Move the existing Markdown, frontmatter, JSON, canonicalization, hashing, document-building, and tests into the compiler. - [x] Keep frontmatter behind a dedicated module and continue using the existing Remark/Micromark pipeline. - [x] Parse only candidate files and only the tiers required by the compilation plan. - [x] Deduplicate documents and fields required by multiple resources. - [x] Define a versioned, deterministic `ContentArtifact` with integrity information. - [x] Keep verbose parse and omission diagnostics outside the runtime artifact. - [x] Reject incompatible artifact versions cleanly. ### 5. Enforce the deployment-wide database budget - [x] Replace the old asset-index limit with `CONTENT_DATABASE_MAX_BYTES`, defaulting to 500 KiB. - [x] Have applications read configuration and pass the byte limit into the compiler. - [x] Measure the exact serialized database, including selected bodies and overhead. - [x] Implement deterministic skip-and-continue packing of complete documents. - [x] Return compiler diagnostics with unbounded/bounded bytes, included/omitted counts, omitted identities, largest contributors, and affected resources. - [x] Expose only compact artifact statistics at runtime through `ContentDatabase.getStats()`. - [x] Add exact boundary, deterministic ordering, deduplication, and oversized-document tests. ### 6. Finalize the runtime database - [x] Define `ContentDatabase` over an immutable artifact. - [x] Preserve filtering, nested AND/OR groups, sorting, offset/limit, projection, field references, and content retrieval. - [x] Expose `getFieldCatalog()` and compact `getStats()`. - [x] Keep Markdown/YAML/compiler dependencies out of the runtime entry. - [x] Add execution guards and maximum-size runtime benchmarks. ### 7. Make query authoring OpenAPI-driven The selected standard OpenAPI operation is the complete query-authoring contract. The query builder resolves parameters and request-body JSON Schema, derives its internal editor model, and applies renderer-owned layout policy. It must not embed or fetch a second capabilities/UI language through `x-webstudio-query` or another side channel. - [x] Remove `x-webstudio-query` and all vendor query-definition metadata from the Assets OpenAPI operation. - [x] Resolve local OpenAPI references and derive the editable request value, defaults, validation, labels, choices, arrays, objects, and unions solely from standard operation parameters and request-body JSON Schema. - [x] Represent dynamic Assets field paths with standard JSON Schema vocabulary in the `queryAssets` request schema. - [x] Keep the normalized editor representation internal to query-builder; do not expose another provider-authored capabilities contract. - [x] Remove provider-controlled layout metadata such as `group`; make compact scalar packing and all other layout renderer policy. - [x] Keep Webstudio expression binding as a consumer/editor concern layered over schema values, not part of the OpenAPI contract. - [x] Preserve the existing Assets filter, sort, output selection, full-content/range selection, limit/offset, editable source, and configured-field behavior. - [x] Keep the Assets resource adapter responsible for serializing the edited query into its request body. - [x] Remove the obsolete Assets query-capabilities endpoint, URL, and UI-side capability parsing. - [x] Add a non-Assets OpenAPI proof with differently named parameters and no vendor metadata. - [x] Verify the generic packages contain no Assets imports, fixed Assets property contract, or provider-authored layout model. - [x] Update focused core, React, SDK, protocol/OpenAPI, Builder, and source round-trip tests. ### 8. Manage ContentDatabase lifecycle in Builder - [x] Cache databases by source revision plus normalized compilation-plan hash. - [x] Coalesce concurrent compilation requests. - [x] Reuse cached database instances across field discovery, autocomplete, query preview, and response preview when their normalized compilation plans match. - [x] Replace it when source revision or query requirements change. - [x] Preserve the optional revision-keyed hosted parsed-document cache without making it a source of truth. - [x] Show exact preview warnings when concrete values reveal missing bounded results. ### 9. Materialize and verify immutable builds - [x] Compile and serialize the exact deployment-wide database during prepublish validation. - [x] Attach that validated bounded database to the immutable build without rebuilding it from later asset state. - [x] Show non-blocking prepublish warnings with database bytes, omissions, and potentially affected resources. - [x] Avoid claiming exact future route matches when dynamic values are unknown. - [x] Include the database/runtime only when a reachable configured query requires them. - [x] Ensure detail resources requesting Markdown content include all statically eligible bodies needed by dynamic routes. ### 10. Integrate all targets - [x] SSR: load the build's bounded database and conditionally include the small runtime. - [x] SSG: query the same database during generation, then omit database/runtime output when no runtime query is needed. - [x] Local/CLI: compile from the filesystem source with the same semantics. - [x] Preserve overview pagination, alternative route fields, draft-as-user-data behavior, and full article rendering. - [x] Verify ordinary Assets and fresh query-free sites perform no parser, artifact, content-runtime dependency install, or cleanup work. ### 11. Correct API and package ownership - [x] Move external Assets REST/OpenAPI contracts to `@webstudio-is/protocol`. - [x] Keep Webstudio domain models and Assets system-resource configuration in the SDK. - [x] Keep both cookie and token authentication behavior on the unified Assets REST surface. - [x] Remove pass-through route wrappers/reexports that add no transport behavior. - [x] Keep published runtime queries local to the artifact rather than routing them through Builder REST. ### 12. Remove superseded deployment infrastructure - [x] Remove external deployment-index upload, fetch, invalidation, cleanup, and garbage-collection code. - [x] Remove the SaaS asset-origin adapter requirement; close webstudio-is/webstudio-saas#652 as superseded because it has no independent purpose. - [x] Remove index-bucket compatibility configuration and avoid adding Infisical variables. - [x] Remove obsolete external-index paths, package dependencies, routes, scripts, and tests. - [x] Consolidate unreleased intermediate migrations where safe and retain only schema required by the final hosted cache/API design. ### 13. Verification and release readiness - [x] Move and preserve existing tests while changing ownership; add new tests only for changed contracts and behavior. - [x] Run content-engine compiler/runtime tests, query-builder tests, SDK/protocol/API tests, Builder tests, CLI tests, and affected typechecks. - [x] Run formatting, generated-file idempotence, fixtures, clean Supabase-compatible migrations, E2E shards, and complete agent evaluations. - [x] Verify SSR and SSG conditional inclusion and query-free generated package manifests. - [x] Run the five-article MCP blog workflow: authoring, field discovery, overview, dynamic detail routes, full Markdown rendering, publishing, and published-site verification. - [x] Run a maximum-budget database test and a mutation-during-compilation test. - [x] Perform a final architecture, security, API compatibility, migration, bundle, correctness, and DRY review. - [x] Reconcile every checkbox and replace the verification section with results from the final architecture before release. ## Future MDX work MDX is intentionally separate from this release. - The compiler contract should allow a future build-only MDX parser. - MDX compilation must not enter the published runtime. - Embedded JSX must use an explicit compiled representation rather than runtime source evaluation. - Webstudio components, tokens, styles, and component allowlists remain Webstudio integration concerns. - The future MDX work should reuse `ContentArtifact` and `ContentDatabase` where practical. ## Superseded design The following earlier design choices are no longer the target architecture: - GROQ or GraphQL as the Assets query language. - One external or long-lived index object per resource/query. - A deployment index fetched by the Worker from SaaS/CDN infrastructure. - Lazy production reads of source Markdown bodies. - A separate index bucket, asset origin, invalidation service, or garbage collector. - A 4 MiB published index limit. - Rebuilding an old build's database from the current mutable asset set. - Bundling query infrastructure into projects without a configured content query. ## Verification baseline The embedded runtime-database architecture previously passed the checks below. The OpenAPI-driven query-authoring change additionally passed: - Query-builder core: 4 files / 19 tests. - Query-builder React: 1 file / 7 tests. - SDK Assets source/config: 1 file / 7 tests. - Protocol Assets OpenAPI: 1 file / 9 tests, including independent OpenAPI validation. - Builder Assets OpenAPI description: 1 file / 5 tests. - Query-builder, React, SDK, protocol, and Builder typechecks. - Package-boundary validation and formatting of every affected file. Local verification baseline: - Content engine: 15 files / 104 tests. - Asset uploader: 24 files / 175 tests. - Builder: 204 files / 2,423 tests. - CLI: 54 files / 866 tests. - Complete workspace typecheck, formatting, lint, package boundaries, generated fixtures, and fixture builds. - The complete migration history applied successfully to a fresh Supabase-compatible database, including every migration after the consolidated Assets migration. - Builder production build and full local E2E suite passed. - Complete agent evaluations passed: authenticated page (170,859 ms), design input (207,101 ms), and font assets (67,050 ms). - Final review additionally verified correct REST edit/view/build permits, cookie and public-token read paths, immutable snapshot retries, exact database budgeting, query-free dependency omission, and SSR/SSG conditional runtime inclusion. Final staging verification passed against the embedded runtime-database architecture: MCP uploaded and updated five Markdown articles, discovered dynamic frontmatter fields, previewed a five-item sorted overview and one full-body slug query, updated both Assets resources, found zero binding issues, rendered `/blog` and `/blog/dynamic-routes` at HTTP 200, published build version 266, and verified both published routes at HTTP 200 with the expected article metadata and Markdown body. The workflow also found and fixed offset-form asset timestamps failing canonical datetime validation and made MCP `preview.start({ port: 0 })` correctly select an available local port. ## Release constraints - No new SaaS content service, bucket, or environment variable. - No published-site database dependency. - No parser/compiler dependency in query-free sites or runtime-only bundles. - The validated content database is immutable build output. - Publishing warnings do not block deployment. - The selected standard OpenAPI operation and its request JSON Schema are the only source of query UI semantics. |
||
|
Oleg Isonen
|
620f8b54ed |
feat: Create blank text assets (#5931)
## Summary - add a **Create text file** action to the Assets toolbar and shared context/dropdown menu - validate filenames against the central editable-asset registry, create the empty file in the current asset folder, and open it in the text editor - keep simultaneous empty uploads distinct by including the filename in the upload fingerprint - keep internal Nano ID storage suffixes out of visible asset names and subsequent content revisions - show the immutable file extension beside the editable basename in Asset settings Asset querying, content hydration, structured data, bindings, generated runtime support, and MCP integration remain owned by #5921. ## Verification - Builder asset/text tests: 34 passed - SDK asset tests: 110 passed - project-build asset tests: 71 passed - asset-uploader revision tests: 5 passed - SDK, project-build, asset-uploader, and Builder typechecks - lint and package-boundary checks Ref #5824 |
||
|
Oleg Isonen
|
0070fa66a1 |
ci: deduplicate builder E2E setup (#5917)
## What changed - reuse the shared CI setup action in both Builder E2E jobs - use one build command for CI and local E2E runs - centralize shard filename parsing and suite-name formatting - remove redundant suite-level shard filtering - parse each E2E filename's shard metadata once ## Why The merged image-asset PR introduced multiple owners for Builder E2E setup, build dependencies, and shard filename semantics. Consolidating them prevents CI and local behavior from drifting as the shard layout evolves. ## Validation - `pnpm --filter=@webstudio-is/builder test -- e2e/test-modules.test.ts` - `pnpm --filter=@webstudio-is/builder typecheck` - `pnpm lint` - `pnpm e2e:builder:build` - shell syntax and YAML parsing - shard discovery returned shards 1–6 |
||
|
Oleg Isonen
|
ef94d85ccf |
feat: drag image assets onto canvas (#5912)
## Summary - allow dragging a single image asset from the Assets panel onto the canvas - insert an Image component with its source bound to the dragged asset in one undoable mutation - preserve asset-folder drops and canceled drags - build the Builder once in CI and reuse the compact build artifact across all E2E shards ## E2E timing Recent CI showed shard test runtimes of roughly 101–145 seconds, while every shard also repeated a roughly two-minute Builder build. The four shards are already reasonably balanced, so a fifth shard would not address the primary bottleneck. The shared artifact is about 30 MB uncompressed and roughly 5 MB compressed because checked-out public assets are excluded. This should return individual shard jobs to the 2–3 minute range. ## Validation - 45 focused unit tests - image asset to canvas E2E using the prebuilt-artifact path - image asset to folder regression E2E - Builder typecheck - lint - package-boundary validation - workflow YAML validation - shell syntax validation |
||
|
Oleg Isonen
|
7d7c23d30a |
fix: Center empty asset folder drop state (#5911)
## Summary - center the reusable “Drop files here” state against the full empty-folder area - keep the Back thumbnail independently positioned and interactive - add regression coverage for empty folders with additional content - add a fourth builder E2E shard and split the serial asset-folder workflows using measured CI durations ## Root cause The Back thumbnail participated in the same column flow as the drop state, reducing the drop state’s available height and shifting its center downward. The asset-folder E2E module also accumulated ten serial workflows and became shard 1’s 214-second bottleneck. ## User impact Empty folders now align the drop message consistently with the root empty state and full-panel drag overlay. The E2E split reduces shard 1’s measured workload from about 230 seconds to a projected 113 seconds, with shard 4 at about 100 seconds. ## Validation - focused asset tests: 62 passed - E2E shard discovery test - shard 1 and shard 4 module loading - builder typecheck - workspace typechecks - lint - package boundaries - generated API and docs checks - fixtures - full workspace tests; three CLI subprocess tests initially timed out under concurrent load and passed when rerun in isolation |
||
|
Oleg Isonen
|
cca9f93e3d |
feat: Improve asset folder move workflows (#5908)
## Summary - use the design-system native scroll area and auto-scroll long asset lists during internal drag operations - add a shared **Move** action to thumbnail and context menus for individual and multiselected assets and folders - add a folder-only move dialog that excludes invalid descendant destinations - consolidate bulk move, duplicate, delete, and clipboard behavior into shared asset-manager operations with undo/redo coverage - simplify dialog state, folder exclusion traversal, and long-list E2E setup ## User impact Users can drag assets or folders to destinations outside the currently visible part of a long panel, and can move one or many selected items through an explicit folder picker. ## Validation - `pnpm --filter @webstudio-is/builder test app/builder/shared/asset-manager` — 126 tests passed - `pnpm --filter @webstudio-is/builder typecheck` - `pnpm lint` - `pnpm check:package-boundaries` - focused builder E2E: dragging auto-scrolls a long folder list - focused builder E2E: multiselected assets move from the context menu ## Review note The current validation prevents missing destinations and folder cycles. A small follow-up is still needed to disable moves when every selected item is already in the chosen destination, avoiding a no-op mutation and announcement. |
||
|
Oleg Isonen
|
878d6f1de1 |
feat: Add asset folders to asset manager and improve asset manager UX (#5892)
Closes https://github.com/webstudio-is/webstudio/issues/5877 Closes https://github.com/webstudio-is/webstudio/issues/4775 - Organize assets into nested folders. - Navigate folders with breadcrumbs and Back controls. - Select and manage multiple assets and folders at once. - Drag, copy, cut, paste, duplicate, move, and delete items. - Use context menus and familiar keyboard shortcuts for common actions. - Search and sort assets and folders, with folders grouped first. - Upload files directly into the active folder. - Manage folder placement from asset and folder settings. - Automate asset and folder management through MCP. |
||
|
Oleg Isonen
|
d579607ed8 |
feat: Add draft pages (#5885)
Closes https://github.com/webstudio-is/webstudio/issues/3036 ## Summary - add an optional page-level draft state with a presentation-only `[Draft]` prefix in Builder - add **Mark as draft** and **Stage for publish** actions to the Page details header menu - keep drafts editable, previewable, linkable, duplicable, and path-reserving in Builder - exclude drafts from generated routes, public page collections, sitemap output, staging, and production builds - expose draft state and usage guidance through the runtime API, HTTP client, CLI docs, and MCP contracts ## Why Pages need a way to remain unfinished inside Builder without being included in the next site deployment. Draft is an exclusion flag, not a separate revision or publishing workflow state. ## Behavior and fixes - home and `/*` catch-all pages cannot be drafts - draft pages must be staged before becoming the home page - restricted features used only on draft pages no longer block publishing the publishable site - **Stage for publish** only clears draft state; it does not deploy the site - missing or explicit `false` draft state remains backward-compatible and publishable ## Validation - 5,179 package tests passed across SDK, migrations, project-build, CLI, and Builder - targeted Builder browser E2E passed for draft/stage actions, persistence, home-page guidance, copy, duplicate, delete, and generated-app build behavior - all affected package typechecks passed - lint and package-boundary checks passed - generated runtime contracts and CLI docs are reproducible - `git diff --check` passed |
||
|
Oleg Isonen
|
dde30ffb54 |
feat: Improve MCP authoring, compact output, and agent onboarding (#5883)
## Summary This improves how agents discover, author, inspect, and verify Webstudio projects while reducing unnecessary MCP context usage. ## What changed - Adds an **Agents** section to Project Settings with ready-to-run setup commands and client configuration for Claude Code, Codex, Cursor, and VS Code. - Makes large list and schema responses compact and paginated by default, with **verbose** available when complete records or diagnostics are needed. - Applies consistent compact output to assets, instances, styles, design tokens, CLI schemas, and project-session metadata. - Returns actionable expression warnings for props, resources, and Collection bindings, including unknown variables and risky render-time mutation resources. - Improves Collection expression guidance so agents can bind array items with the correct scoped variables. - Reports rendered audit outcomes explicitly as complete, partial, confirmation-required, or failed. - Expands runtime output schemas so MCP clients can validate responses reliably. - Strengthens the packaged CLI smoke workflow with a realistic seven-page stdio MCP scenario. ## Architecture - Consolidates `project-build` from 65 file-oriented entrypoints to 11 architecture-owned surfaces. - Separates the core model, contracts, runtime authoring, transfer, state, project sessions, MCP, visual tooling, persistence codecs, and authenticated server services. - Isolates Node-only JSX evaluation from browser-safe transfer utilities and authenticated server modules. - Keeps persistence codecs independent from authenticated database services, preventing unrelated module initialization. - Defines shared instance-path types once and reuses them across runtime modules. - Preserves Immer initialization through explicit package side-effect metadata. - Adds package tests that prevent a return to source-file entrypoints or generic exposure of Node-only surfaces. ## Validation - Full repository checks, tests, typechecks, generated-file checks, and fixtures - All 163 Builder test files and 2,141 tests - All 71 project-build test files and 1,794 tests - Lint and package-boundary validation - Full packed CLI release smoke test, including install, preview, agent setup, seven-page stdio MCP workflow, rendered audit, and structured commands |
||
|
Oleg Isonen
|
e57bd297f5 | test: cover drag gestures in builder e2e (#5876) | ||
|
Oleg Isonen
|
d186d24501 |
feat: Improve MCP project audits, visual verification, and dynamic content authoring (#5874)
This update helps AI agents understand, audit, and visually verify Webstudio projects more reliably. It also adds clearer guidance for expressions, Collections, screenshots, and responsive rendered audits. ### Audit complete projects - Run one `audit` command across accessibility, SEO, security, assets, styles, and performance. - Audit the whole project or focus on a specific page, scope, or severity. - Receive stable findings with locations, explanations, and recommended fixes. - Continue large result sets through pagination without repeating general guidance. ### Audit the rendered website - Build and inspect the generated website, rather than relying only on stored project data. - Check every static page at representative mobile, tablet, and desktop widths. - Use one suitable viewport inside each configured breakpoint range. - Reuse the same browser and resize pages between captures for substantially faster audits. - Process multiple pages concurrently while keeping resource use bounded. - Supply concrete examples for dynamic routes such as blog posts. - Review and confirm unusually large screenshot plans before they run. ### Find visual and performance problems Rendered audits can detect: - Horizontal page overflow. - Broken images. - Images loaded eagerly below the fold. - Images that are much larger than their rendered size. - Render-blocking resources. - Legacy font formats. - Pages that fail to load or redirect somewhere unexpected. - Pages captured before their layout becomes stable. Detailed screenshots, measurements, failures, and timing evidence are saved locally for further inspection. ### Capture generated pages directly - Capture a project route with `webstudio screenshot --path /route`. - Automatically generate and start a temporary production-style preview. - Stop the temporary preview after the screenshot is captured. - Return navigation, responsive layout, and page-size evidence with JSON output. - Allow external image domains needed by the generated preview. ### Reuse generated previews - Cache generated preview builds when the project has not changed. - Reuse preview dependencies across regenerations. - Keep preview dependencies isolated from the user’s project dependencies. - Surface preview build and startup failures with useful diagnostic output. - Include local project assets in generated previews. ### Author Collections correctly - Explain when Collection is required for arrays and objects. - Guide agents to bind the complete iterable instead of one item or the response wrapper. - Explain how Collection repeats its children for every entry. - Explain current-item and current-key scope. - Preserve Webstudio’s generated Collection parameters. - Guide agents to use stable values for repeated interactive components. ### Write expressions with fewer mistakes - Add a dedicated `webstudio://project/expressions` reference. - Document expression scope, supported syntax, and supported methods. - Explain how resources, variables, Collections, actions, and `system` values are accessed. - Distinguish fixed values from runtime expressions. - Provide valid examples for text, props, resource URLs, headers, query parameters, and bodies. - Warn agents that valid syntax does not guarantee that runtime data has the expected shape. ### Improve agent discovery - Make Collection and expression workflows discoverable through `meta.guide`. - Return Collection-specific instructions through `components.get`. - Point relevant command schemas to the expression reference. - Keep detailed guidance available without placing it in every tool response. ### Improve audit reliability - Report which rendered captures passed, found issues, failed, or were skipped. - Distinguish planning, preview, capture, and cleanup failures. - Include actionable retry instructions with rendered failures. - Preserve completed evidence when part of an audit fails. - Detect stale or conflicting preview servers more clearly. - Support cancellation and bounded capture timeouts. ### Improve packaged CLI verification - Add a release-shaped CLI smoke test. - Install the packed CLI in an isolated temporary project. - Fail on peer-dependency warning floods. - Verify preview, screenshots, MCP calls, rendered audits, and generated assets. - Confirm that generic React Router previews do not include Docker-only image dependencies. - Keep CLI and MCP stdout machine-readable while progress remains on stderr. ## User Impact Agents can now audit a real rendered Webstudio site, inspect responsive behavior, find concrete visual and performance problems, and provide evidence for their conclusions. They also receive much clearer instructions when turning API data into repeated content or writing dynamic expressions. |
||
|
Oleg Isonen
|
7d781bdc86 |
fix: MCP fixes (#5870)
Improves CLI and MCP reliability across project synchronization, asset editing, imports, audits, and clipboard workflows. ## Changes - Keeps asset filename and description updates after session refresh. - Prevents intermittent Immer patch errors during consecutive MCP mutations. - Prevents false edit conflicts caused by stale local session state. - Preserves asset uploads without treating them as redundant metadata updates. - Exposes editable asset filenames consistently through MCP. - Imports project assets correctly when copying projects between environments. - Supports refreshing partially available project snapshots. - Keeps MCP operations working when optional project data is unavailable. - Produces clearer compatibility errors when the API and CLI versions differ. - Makes rendered audits more resilient when individual screenshots fail. - Prevents rendered audits from hanging while starting browser captures. - Handles denied clipboard permissions without uncaught browser errors. - Improves copy and paste reliability for pages, components, styles, and animations. ## Verification - Asset uploader tests and typechecking pass. - Focused asset workflow E2E passes. - Complete E2E shard 1 passes. - Realistic MCP subagent test confirms: - Asset filename and description persist after refresh. - Consecutive mutations work without manual refresh. - No Immer patch error occurs. - No false conflict occurs. - Original asset values can be restored successfully. |
||
|
Oleg IsonenandClaude Fable 5
|
071a4d9a85 |
feat: MCP update 3 (#5864)
## Summary This update adds new MCP and CLI capabilities for project auditing, structured data, project-specific agent instructions, focused project search, safer bulk editing, image descriptions, font discovery, visual verification, and external registry inspection. ## New Capabilities ### Audit projects - Run a single `audit` command across accessibility, SEO, security, assets, styles, and performance. - Limit an audit to selected scopes, pages, or severity levels. - Receive stable findings with locations and suggested fixes. - Run rendered audits across desktop, mobile, and Builder breakpoint widths. - Detect horizontal overflow, broken images, oversized images, eager below-fold images, render-blocking resources, and legacy font formats. - Identify unused assets, tokens, CSS variables, local styles, and breakpoints. - Identify duplicate token declarations. - Detect accessibility issues involving names, labels, headings, landmarks, IDs, keyboard focus, ARIA, iframes, and autoplaying media. - Detect SEO issues involving titles, descriptions, language, social images, and structured data. - Detect unsafe new-tab links and write resources exposed during page rendering. ### Search an entire project - Search project text, labels, props, resources, assets, and styles through `search-project`. - Find relevant records without downloading and reparsing the full project snapshot. - Scope searches to the relevant project-data areas. ### Add validated JSON-LD - Add structured data with a dedicated JSON-LD component. - Place JSON-LD inside the page Head Slot. - Edit JSON in a purpose-built code editor. - Prevent structurally invalid JSON-LD from being saved. - Validate Schema.org types, properties, and values. - Find invalid, missing, unknown, deprecated, or incompatible structured data through the SEO audit. - Let agents discover the correct JSON-LD workflow without using page custom metadata or a generic HTML embed. ### Configure project-specific agent instructions - Add AI-agent instructions from the new **Agents** section in Project Settings. - Synchronize those instructions into a managed root `AGENTS.md`. - Update or remove the managed file automatically during project sync. - Protect existing user-owned `AGENTS.md` files from being overwritten. - Share the instructions with authenticated CLI and MCP agents. - Keep agent instructions out of publicly readable production bundles. ### Generate and manage image descriptions - Audit image assets for missing descriptions. - Let a vision-capable agent inspect each image in its rendered context. - Save generated descriptions to multiple image assets in one operation. - Mark images as intentionally decorative. - Reuse the saved asset description wherever the image is used. - Re-run the audit to verify that missing-description findings are resolved. ### Perform bounded bulk replacements - Replace matching literal text across selected pages with `replace-text`. - Replace matching fixed prop text with `replace-prop-text`. - Replace matching resource names or URLs with `replace-resource-text`. - Restrict replacements by page, instance, prop, resource, field, match mode, and maximum change count. - Leave dynamic expressions unchanged. ### Inspect element ancestry - Return an element’s direct parent in instance listings. - Request the complete ancestor chain when inspecting an element. - Understand tree placement without rebuilding the instance hierarchy from raw project data. ### Discover available fonts - List uploaded font families through `list-fonts`. - Discover available system font stacks. - Inspect font family, style, weight, format, and associated asset information before applying typography. ### Inspect external component registries - Inspect a local or remote shadcn-style registry item. - Read its description, files, dependencies, targets, and documentation. - Receive an explicit Webstudio compatibility report. - Inspect registry metadata without installing or executing external code. ### Capture generated project routes directly - Run `webstudio screenshot --path /route` against the configured project. - Generate a temporary production-style preview automatically. - Capture the requested route. - Stop the temporary preview after capture. - Avoid manually building and starting a preview for a single screenshot. ### Assert visual changes - Add expected text to screenshot comparisons. - Set maximum pixel mismatch limits. - Set minimum or maximum changed-region counts. - Assert whether overall brightness or a color channel moved in the expected direction. - Use these assertions as pass/fail evidence for visual changes. ### Require compatible CLI versions - Send the project-bundle contract version with CLI synchronization requests. - Stop incompatible CLI versions before they receive incomplete project data. - Tell users exactly how to update the CLI or run the latest version once. - Prevent older clients from silently dropping project settings. ## Upgrade Note This branch introduces a new synchronized project-bundle contract. Older CLI versions must update before synchronizing against the new API. ```sh npm install -g webstudio@latest ``` Or run the latest version once: ```sh npx webstudio@latest <command> ``` --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oleg IsonenandClaude Fable 5
|
de23e2a888 |
chore: migrate toolchain to oxc (oxfmt, oxlint, rolldown-vite) and typeScript 7 (#5868)
## prettier -> oxfmt (0.58.0)
- .oxfmtrc.json generated with `oxfmt --migrate=prettier`: trailingComma
es5, printWidth 80 (oxfmt defaults to 100), sortPackageJson disabled,
.prettierignore patterns folded in (file deleted).
- ignorePatterns additionally exclude yml/yaml/toml/jsonc and
packages/cli/templates/** so oxfmt covers exactly the file set prettier
used to format (ts,tsx,js,json,css,md) and never touches user-facing
scaffolding artifacts.
- root `format` script, nano-staged pre-commit hook and the CI check
(`pnpm oxfmt --check`, replacing `prettier --cache --check` and its
actions/cache step — oxfmt checks the whole repo in ~0.7s) now run
oxfmt.
- every generator script that ran `prettier --write` on __generated__
output now runs `oxfmt` (css-data, icons, html-data, postgrest, sdk,
sdk-components-*, apps/builder tailwind preflight).
- the four scripts that used prettier's JS `format()` API write raw
source instead and their package.json scripts pipe the generated file
through the oxfmt CLI: project-build generate-docs +
generate-runtime-operation-contracts, cli generate-docs, apps/builder
generate-server-only-api-metadata. oxfmt rejects paths containing "..",
so cross-package generated files are formatted via a subshell cd into
the owning package.
- one-time reformat: oxfmt's canonical output differs from prettier on 8
repo source files (binary-operator continuation indent, import() call
wrapping, sub-pixel spacing of empty for-clauses); oxfmt honors existing
`// prettier-ignore` comments so intentional layouts are kept.
- prettier remains ONLY as a cli devDependency for the
prettier/standalone HTML normalizer inside html-to-jsx.test.ts, and
inside fixtures/ + packages/cli/templates/ (ships to scaffolded user
projects; out of scope here).
- editor config switched from esbenp.prettier-vscode to oxc.oxc-vscode
(devcontainer extension list, .vscode editor.defaultFormatter).
## oxlint 1.56 -> 1.73
- oxlint 1.73 rules-of-hooks special-cases hooks NAMED useEffectEvent:
their results may only be called from effects. Our
app/shared/hook-utils/effect-event.ts is a stable-callback helper
(useEvent pattern), intentionally called from event handlers and passed
as props. The rule is disabled via .oxlintrc.json overrides for the five
files doing this (product code untouched); note those files lose
hook-order linting entirely until the rule becomes configurable.
## TypeScript 5.8.2 -> 7.0.2 (native compiler)
- `tsc` is now the Go-based native binary; all `tsgo` script usages
(workspace packages and fixtures) renamed back to `tsc` and
@typescript/native-preview removed (the package is being phased out).
- fixes for TS7 changed defaults and stricter checks:
- `types` now defaults to [] -> base tsconfig sets "types": ["node"]
(restores the implicit-@types/node behavior of TS5).
- `rootDir` now defaults to ./ -> every tsconfig.dts.json sets
"rootDir": "./src" (TS5011 otherwise).
- import assertions removed -> cli/src/cli.ts uses `with { type: "json"
}` (TS2880).
- stricter isolatedDeclarations inference -> explicit annotations in
css-data bin/property-filter.ts (TS9010) and image image-dev.stories.tsx
(TS9013).
- declaration serialization limit -> project-build builder-runtime.ts
runtimeOperationContracts gets an explicit type (TS7056); the annotation
preserves the narrow RuntimeOperationId union.
- TS7 ships no stable programmatic API:
- react-sdk (component-generator tests drive the compiler API) aliases
its typescript devDependency to @typescript/typescript6@^6.0.2; its bin
is `tsc6`, so the package's typecheck still runs native TS7 tsc from the
workspace root.
- generate-arg-types keeps typescript 5.8.2 for react-docgen-typescript.
- fixtures/ and packages/cli/templates/ keep typescript 5.8.2
(user-facing); fixture scripts still work because plain `tsc` resolves
to their local dependency.
## vite -> rolldown-vite 7.3.1 (oxc bundler)
- global pnpm override `"vite": "npm:rolldown-vite@^7.3.1"`: the builder
app (remix v2 vite plugin works unmodified), every package vite build
and the storybook build now bundle on rolldown/oxc.
- vitest is scoped BACK to vanilla vite via `vitest>vite`,
`vite-node>vite` and `@vitest/mocker>vite` overrides because vitest
3.1's runner does not understand rolldown's SSR transform
(__vite_ssr_exportName__ ReferenceError in every suite).
- those overrides pin vite EXACTLY to 6.3.4: letting ^6.3.4 float to
6.4.3 breaks cli prebuild tests (dynamic import of a generated
app/routes/$.tsx fails to resolve under 6.4.3's loader).
## unrelated cleanup requested during review
- apps/builder commands.ts imports duplicate-tokens-group statically
instead of via dynamic import. The circular dependency the dynamic
import guarded against only exists through the command-panel barrel
(command-panel.tsx > groups/index.ts > commands-group.tsx >
commands.ts); importing the group module directly is cycle-free
(verified with madge), matching the neighboring wrap-group and
convert-group imports.
## verification
- pnpm lint, oxfmt --check: clean.
- pnpm -r typecheck (TS7 native) and pnpm dts: clean.
- pnpm -r test: all suites pass (builder 2093 tests, cli 630, react-sdk
45, project-build 11, ...).
- builder app build, full workspace build, cli build and storybook build
all succeed on rolldown-vite.
## Description
1. What is this PR about (link the issue and add a short description)
## Steps for reproduction
1. click button
2. expect xyz
## Code Review
- [ ] hi @kof, I need you to do
- conceptual review (architecture, feature-correctness)
- detailed review (read every line)
- test it on preview
## Before requesting a review
- [ ] made a self-review
- [ ] added inline comments where things may be not obvious (the "why",
not "what")
## Before merging
- [ ] tested locally and on preview environment (preview dev login:
0000)
- [ ] updated [test
cases](https://github.com/webstudio-is/webstudio/blob/main/apps/builder/docs/test-cases.md)
document
- [ ] added tests
- [ ] if any new env variables are added, added them to `.env` file
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Oleg Isonen
|
6983ecfdb0 | fix: Document MCP consumer capabilities & more fixes (#5862) | ||
|
Oleg Isonen
|
6e10ae3809 |
fix: Fix Builder paste transfer data handling and publish sync edge cases (#5861)
- Fix publish sync for empty JSON data variables - Retry transient asset downloads once with shared sync-client backoff - Fix Builder paste handling for Webstudio transfer data - Add transfer/data-envelope terminology and colocated runtime data-format modules - Rename page clipboard runtime operation to page transfer semantics - Report malformed Webstudio/Webflow paste data through paste results instead of inserting it as text - Add browser-level E2E coverage for instance, page, folder, template, generic plugin, malformed data, native input, and content-mode paste flows ## Details - Allows empty JSON data variables to sync/publish correctly. - Adds asset download retry coverage and uses shared backoff logic. - Introduces runtime `data-formats` modules for data envelopes, instance transfer data, and page transfer data. - Removes stale clipboard aliases and `validateTemplateStructure`. - Keeps clipboard as a Builder UI transport concern while runtime/protocol use transfer-data terminology. - Changes paste plugins to return expressive success/error results so callers can show toast errors and stop fallback insertion. - Prevents malformed Webstudio transfer payloads from falling through to text/HTML/markdown insertion. - Handles malformed Webflow-owned paste data without throwing. - Adds shared copy-paste fragment insertion helpers to reduce duplicated breakpoint-warning/fragment-data logic. |
||
|
Oleg Isonen
|
d291e4b671 |
feat: Builder api runtime contracts guardrails for MCP (#5857)
This PR expands Webstudio MCP and CLI into a more complete agent authoring interface. Agents can discover project capabilities, edit real Webstudio content, use data/resources, verify visual output, and report useful issues when workflows fail. ## MCP User Capabilities - Create, update, duplicate, reorder, and delete pages. - Create pages from templates and manage reusable page templates. - Inspect project pages, instances, props, styles, assets, variables, resources, and metadata. - Insert Webstudio components and authored JSX fragments into pages. - Edit text, props, styles, breakpoints, design tokens, and CSS variables. - Update page metadata, SEO fields, redirects, marketplace metadata, and project settings. - Create and use static data variables. - Create and use dynamic resources fetched over HTTP. - Use REST, GraphQL, and built-in system resources in authored pages. - Bind data variables and resources to visible page content. - Create resource-backed pages such as sitemap XML, structured SEO data, and API-driven content sections. - Preview current project state without requiring a publish step. - Capture screenshots for visual verification. - Verify responsive designs across desktop, tablet, and mobile breakpoints. - Use screenshot diff and OCR-style checks to inspect rendered output. - Reset and refresh local MCP project state when needed. - Discover available CLI/MCP commands from generated manuals. - Use clearer examples for resources, data variables, JSX authoring, previewing, and visual checks. - Read marketplace product metadata without requiring a full project snapshot. - Get safer guidance when using low-level patch tools. - Get a ready-to-copy Discord report template when MCP/CLI behavior is confusing or broken. |
||
|
Oleg Isonen
|
a515a4fcbf |
chore: Change the local setup to use docker only for supabase (#5851)
It was disappearing after played on builder |
||
|
Oleg Isonen
|
1ddf6b02c9 |
feat: API and CLI for everything (#5838)
Closes #5287 |
||
|
Oleg Isonen
|
2a77c2a8b5 |
feat: Multiselect (#5834)
Closes https://github.com/webstudio-is/webstudio/issues/5310 |
||
|
Oleg Isonen
|
b194d3e0b3 | feat: Import project via CLI (#5828) | ||
|
Oleg Isonen
|
ac29cf7d1d |
feat: Keyboard shorthands for instance positioning and slot bug fixes (#5821)
Closes https://github.com/webstudio-is/webstudio/issues/4586 Closes https://github.com/webstudio-is/webstudio/issues/4619 ## Summary Fixes shared Slot editing so linked Slot occurrences stay consistent across structural edits and labels. ## Changes - Fixed moving a direct Slot child out of a Slot so the child is removed from all linked Slot occurrences. - Fixed unwrapping a direct Slot child so the child is removed from all linked Slot occurrences. - Added linked Slot label sync when renaming from Navigator or Settings panel. - Added/updated tests for canonical Slots, legacy Slots, nested Slot children, keyboard moves, unwrap, reparent, and label sync. - Simplified stale Slot detach logic that was incorrectly cloning only the selected Slot occurrence. ## Testing - `pnpm --filter @webstudio-is/builder test -- app/shared/instance-utils/...` - `pnpm --filter @webstudio-is/builder test -- app/builder/shared/commands.test.ts` - `pnpm --filter @webstudio-is/builder typecheck` - `pnpm lint` - `git diff --check` |
||
|
Oleg Isonen
|
c3d2fa7fa3 |
feat: Render current page state on links in preview mode (#5810)
Closes https://github.com/webstudio-is/webstudio/issues/5207 |
||
|
Oleg Isonen
|
7a6ffcb229 | fix: Fix share link options saving (#5805) | ||
|
Oleg Isonen
|
d49582bcf5 |
feat: Copy-paste pages, folders and page templates (#5798)
Closes https://github.com/webstudio-is/webstudio/issues/5797 ## Summary - Add page, page template, and folder copy/paste support, including cross-project paste. - Reuse a shared page item actions menu for page headers and context menus with Copy, Duplicate, Delete, and shortcut hints. - Add Paste to the pages context menu. - Preserve template semantics when copying/pasting templates. - Support folder copy with nested folders/pages. - Add Backspace/Delete page deletion shortcuts and focus destructive dialog actions for Enter confirmation. - Rename `init-copy-paste.ts` to `copy-paste.ts`. ## Testing - `pnpm --filter @webstudio-is/builder typecheck` - `pnpm --filter @webstudio-is/builder test app/shared/copy-paste/copy-paste.test.ts app/shared/copy-paste/plugin-page.test.ts app/shared/page-utils.test.tsx app/builder/features/pages/page-utils.test.ts` - `git diff --check` - E2E copy/delete flows were added in `apps/builder/e2e/tests/pages-actions.e2e.ts` |
||
|
Oleg Isonen
|
392c9089be |
fix: content mode permissions (#5795)
**Summary** Fixes content-mode edit permissions so editor-only users can perform allowed content workflows without backend “build permission” failures. **Changes** - Added shared content-mode permission validation used by backend authorization. - Aligned content-mode capabilities around editable props, editable instances, content roots, pages, assets, and template insertion. - Allowed content-safe instance/text/page/template/style changes while keeping design/data/resource changes build-only. - Updated template insertion to preserve content-safe local styles and reuse tokens by id. - Added accessible `SyncStatusDot` for e2e-visible sync state without replacing the existing sync status UI. - Added builder e2e setup with Postgres/PostgREST and a first content-mode browser test: - edit existing text as an editor-only user - wait for saved state via the sync status dot - reload and verify persistence **Verification** - `pnpm --filter=@webstudio-is/builder test -- sync-status.test.tsx` - `pnpm --filter=@webstudio-is/builder typecheck` - `git diff --check` - `pnpm e2e:builder` |