6 Commits

Include renames

Author	SHA1	Message	Date
Oleg Isonen	75d710e759	fix: Let Editor links patch Content Block text and props (#6575) ... Editor links with API access can edit text and editable props inside Content Blocks through raw CLI/MCP patches. Builder and Viewer permissions stay unchanged. UI and API requests use the same Content-mode policy. The API keeps its narrower text-and-props scope; page, asset, style, expression, and structure changes remain blocked for raw Editor patches. ### Implementation - Use the persistence patch engine for instance, prop, and style state changes. Remove the separate instance patch interpreter and record writes from permission validation. - Check prop changes against the resulting state. Image dimensions work even when their patches appear before the source patch. - Move Image source, dimensions, and alt updates into the shared runtime. Video source changes preserve its existing dimensions. The UI no longer sends separate follow-up writes. Explicit overrides and expression bindings are preserved. - Separate capability discovery from transaction validation and move the server permission adapter out of the router. - Reject missing replacement targets and protected or renamed prop-ID reuse. Keep template-copy protections. - This refactor removes 19 non-test lines. It removes eager whole-instance cloning, but the server still loads the project and validates its content tree. Empty transactions skip that validation. Closes #6573 ### Checklist - [x] Reproduce the Editor-token denial and add regression coverage - [x] Keep Editor raw patches limited to text and editable props - [x] Use the existing patch engine and test missing targets, immutable input, and permission boundaries - [x] Share media updates between UI and MCP; test atomic updates, patch ordering, overrides, and bindings - [x] Separate capability discovery, permission validation, and API orchestration - [x] Verify new regression tests fail with the relevant behavior disabled, then pass after restoration - [x] Run runtime tests, API/sync tests, typechecks, lint, formatter, package-boundary, and generated-API checks - [x] Run Content-mode E2E checks for templates, assets, page creation, collaboration, and save/reload - [x] Add a CLI MCP Editor-token save/reload E2E, including protected template rejection - [x] Return a structured forbidden error for rejected Editor MCP content changes - [x] Run the complete Luna/low evaluation suite; record failures below - [x] Review documentation and fixture impact: no new API inputs, schema changes, migrations, or fixture generation changes; existing guidance remains applicable - [x] Correct the project-package permission fixture to include its owning instance and run all package tests - [x] Cover repeated media source updates and bound namespace batches in Editor requests - [x] Preserve stale retry handling before Editor content validation, with API regression coverage - [x] Preserve expression-bound media metadata during source replacement and test saved Content-mode output - [x] Reuse batch merging so split prop payloads validate against their final state; test split and combined forms - [x] Push the refactor and verify required CI completes - [ ] Deferred: reproduce the separate Builder UI report on a session-free Editor link - [ ] Deferred: confirm image-description editing expectations separately ### Verification - Full project-build suite: 2,597 tests passed. Focused permission/prop run: 121 passed. - All package tests (`pnpm test:packages`) passed after correcting the project-package fixture. All 1,297 CLI tests passed. - Builder API and sync authorization: 54 passed; settings-panel tests: 11 passed. - Content-mode E2E: 9 passed, including setup, plus the earlier focused Image replacement run. - Builder/project-build typechecks, lint, formatting, package boundaries, and generated-API checks passed. The CLI content-runtime bundle test passed; no generated fixture output changed. - Earlier full Luna/low evaluations: **7/9 passed**. `managed-form-runtime-v1` failed `runtimeFormAction`; `markdown-blog-v1` failed `blogRouteEvidence`. These checks also failed before this refactor. MDX editing and Markdown-reference discovery passed. Baselines for this model/effort were incompatible or missing and were not updated. This suite is not fully green. - Latest full Luna/low rerun after review fixes: **6/9 passed**. Form runtime, blog authoring, and MDX editing failed; the MDX fixture passed in the earlier run. Baselines were not updated. The evaluation suite remains incomplete. - Retry follow-up: all 59 API/router tests and 28 persistence patch tests passed, along with Builder typecheck and lint. Older retries use existing persistence handling; current and future versions remain permission-checked. - Bound-media follow-up: Image and Video source replacement preserves expression-bound width/alt and still passes shared Content-mode validation. Full runtime tests, typecheck, and lint passed after this fix. The agent evaluation results above precede the final retry and binding guards; those guards have dedicated regression coverage. - Split-payload follow-up: reuse the existing batch merge helper before permission validation. Combined and split media patches pass the same final-state checks. Full runtime tests, 59 API tests, typechecks, and lint passed. Agent evaluations were not repeated after this follow-up. - CI passed on `1242ba8344`: all package, Builder, and CLI test jobs; all six Builder E2E shards; visual regression; typecheck/lint; generated-file checks; packaged SSG; size; migrations; and submodule checks. Development deployment checks passed. All review threads are resolved. ### Final review follow-up (2fbb30ffe6; CI passed) - [x] Reject future base versions before saving; test an otherwise allowed Editor patch - [x] Preserve sequential patch behavior and test authorization/persistence parity - [x] Run focused tests, typechecks, lint, and review documentation/fixture impact - [x] Commit and push these follow-up fixes, then verify CI Local follow-up verification: 2,600 project-build tests, 96 project tests, and 60 API tests passed. Builder/project-build typechecks, changed-file lint and formatting, package boundaries, and diff checks passed. Regression tests failed without the fixes and passed with them. The shared patch engine uses one Immer draft and supports namespace replacement without mutating input. Net production change: 6 lines. No documentation, fixture inputs/outputs, schemas, or generated contracts changed. Browser E2E and agent evaluations were not rerun for this follow-up; the race and patch-order cases are covered by deterministic tests. CI passed on `2fbb30ffe6`: all six Builder E2E shards, package/CLI/Builder tests, typecheck/lint/generated checks, visual regression, migrations, SSG build, size, and submodule checks. No new blocking findings in the follow-up review. Agent evaluations were not rerun; earlier limitations remain recorded above. ### Asset review follow-up - [x] Reproduce and block asset-valued design-only props, retaining matching Image metadata - [x] Stop generating image metadata for Video source updates - [x] Verify regression tests, documentation/fixture impact, and CI; reply to both review threads Asset follow-up `2ea28028c0`: shared metadata eligibility now checks asset values and their matching Image source, including MDX materialization. Video source edits do not generate Image-only metadata. New and updated regression tests failed without the fixes and pass with them. All 2,604 runtime tests, 60 API tests, 30 persistence tests, both typechecks, lint, formatting, and package boundaries passed. No documentation, schemas, or fixture data changes are needed. Agent evaluations were not rerun. Both review threads have replies and are resolved. CI passed on `2ea28028c0`, including all six E2E shards, visual regression, package/CLI/Builder tests, typecheck/lint/generated checks, migrations, SSG build, size, and submodules. - [x] Reproduce repeated source updates without a saved source, coalesce before ID assignment, and verify regression tests/CI Final source follow-up `16c1c5f68e`: repeated instance/property updates are coalesced before assigning IDs, replacing the later source-only check. Both new Image/Video regression cases failed before the fix and pass now. All 2,606 runtime tests, typecheck, and lint passed. CI passed on this commit, including all six E2E shards and visual regression. No documentation or fixture-data changes were needed; evaluations were not rerun. ### Coverage audit - [x] Audit changed behavior against assertions, including UI/API and persistence boundaries - [x] Add test assertions for identified coverage gaps; automatic CI passed - [x] Review documentation and fixture impact: no product documentation change; one new SVG used only by browser E2E Per request, no local tests or agent evaluations were run for this audit. Automatic CI passed on `4591940169`, including package/API tests, all six browser-test shards, visual regression, typechecks, lint, and generated-file checks. Coverage additions are pushed in `2371d9796a` and `4591940169`: check actual text sent to persistence; duplicate ordinary props across instances; rejected edits leaving capabilities reusable; merged patch ordering and input immutability; accepted/rejected linked Image dimensions in MDX templates; returned style/source/selection state; and UI Image dimension changes followed by reload. The previous image E2E used equal-sized files and could miss broken dimension synchronization. Its new SVG has different dimensions. Production code is unchanged. No tests or evaluations were run for this audit, as requested; no new pass claim is made. - [x] Inventory every changed/moved/removed production function and map it to assertions - [x] Add missing mixed-child and dependent-transaction coverage - [x] Push inventory follow-up and inspect CI results ### Function-by-function coverage inventory This inventories production functions changed, moved, or removed by this PR. “Covered” means the named test asserts the behavior; it does not claim every possible input or branch is covered. No local tests or evaluations were run for this inventory. | Function / handler | Assertions traced | | --- | --- | | `renderProperty` and its `onChange` callback | Content-mode props E2E checks text, href, Image/Video source, alt, protected controls, and reload. Asset E2E now changes dimensions from 120×80 to 240×160, then replaces/reloads to 120×80. | | `createContentModeCapabilitiesFromBuild` (moved unchanged) | API “commits content-mode mutations with editor api tokens” and “limits editor api patches to content instances and props” exercise conversion from saved build data. | | `assertContentOrBuildTransactions` | Editor vs Builder, design denial, transaction/payload limits, all-or-nothing rejection; new exact 10-transaction boundary and dependent append-then-edit transactions. | | `contentOrBuildMutation` | API semantic text edit asserts version, authorization, actual saved text, unchanged outside content, and unchanged input. | | `build.patch` handler | API tests assert allowed text forms, denied structure/styles/assets/expressions/outside roots, Viewer/no-API denial, stale retries, current-version validation, and future-version CONFLICT before persistence. | | `getContentModePropNamesByTag` (moved unchanged) | Derives names by tag and merges names for matching tags. | | `getDefaultContentRootIds` (moved unchanged) | Default vs explicit roots, Body selection, template exclusion. | | `getContentModeFrontmatterBoundTargets` (moved unchanged) | Explicit Body/frontmatter scope, read vs readwrite expressions, recursive instances. | | `getContentModeEditableInstanceIds` (moved unchanged) | Outside roots, nested templates, missing references/roots, non-id children, cycles. | | `getContentModeEditableInstanceRootIds` and its recursive `visit` (moved unchanged) | Same traversal cases; transaction tests reject moving content between roots and duplicate references. | | `getInstancePropMetas` (moved unchanged) | Tag-derived props, unknown/design-only/incompatible props, no generic HTML-attribute permission bypass. | | `getContentModeEditableStaticPropNames` (moved unchanged) | MDX template tests assert exact editable names, primitive conversions, and retained/ignored design-only props. | | `getContentModePropEligibility` | Matching/missing/different asset sources, protected asset props, exact primitive types, UI/transaction parity, Image vs Video. | | `isContentModeCopyableProp` (moved unchanged) | Editable/copyable classification and content-mode copy-policy test. | | `getContentModeCapabilities` | Editable IDs, roots, frontmatter bindings, asset metadata; shared UI and transaction eligibility test. | | `isContentModePropTransition` | Schema/id/owner/name checks, unknown instance, protected existing props, permitted source type changes, linked dimensions. | | `validatePropPatches` | Add/replace/remove, new-then-update, missing replacement target, protected ID reuse, copied template defaults, final-state media patches in split/reversed order, failed-edit immutability and retry. | | `hasSameInstanceMetadata` | Whole-instance text replacement allowed; tag/name changes and outside references rejected. | | `validateStyleSourceSelectionPatch` | New content selection allowed; existing/token misuse and dangling cleanup rejected; now asserts resulting selection map. | | `validateStyleSourcePatch` | New local source allowed; token, duplicate/unused source and incomplete cleanup rejected; now asserts resulting source map. | | `validateStylePatch` | New local declarations allowed; existing design styles, missing breakpoint and incomplete cleanup rejected; now asserts resulting declaration map. | | `validateMapChange` | Above three map paths, invalid patch targets, resulting state and input immutability. | | `validatePatchChange` | Instance/prop/style/asset/page paths and rejected design namespaces through transaction tests. | | `preservesNonTextChildren` and local `keepNonText` | Text-only changes allowed; insertion/deletion/reordering of instance references and expression changes denied; new mixed text/reference positional cases. | | `applyContentModeTransaction` | Authorization/persistence parity, split batches, empty transactions, scoped permissions, copy protections, roots, cleanup, and immutable input on success/failure. | | `materializeMdxAuthoredContent`, its `visit`, and `overlayChildren` | MDX-source “round-trips an authored Asset Image through a styled composite descendant” asserts src/width/height and styles after reload; “serializes an Asset selected on an inserted Image template” asserts all four linked props; nested and standalone component-permission tests reject protected props. | | `getMdxPropEligibility` | Template tests check allowed/ignored props, aliases, incompatible types; new matching vs mismatched Image dimension asset test exercises passed asset values. | | `materializeMdxTemplates` | New linked-dimension test plus existing copied-fragment, unchanged-input, authored-prop, diagnostic, and alias tests. | | `appendBuilderPatchChanges` | Mutation accumulator test asserts sequential state and combined payload; merge test asserts ordering and input preservation. | | `mergeBuilderPatchChanges` | New repeated/interleaved namespace, empty input, repeat-call and patch-array immutability assertions; permission tests exercise split payload integration. | | `getMediaAssetPropNames` | Image generates width/height/alt; Video and unrelated components do not; explicit overrides and expression-bound metadata retained. | | `getPropIdsToDelete` | Image src removes dimensions; non-Image removes only requested prop; resource cleanup and missing-instance cases. | | `updateProps` | Atomic Image/Video updates, repeated sources with/without saved props, overrides, bindings, original-state preservation; new ordinary prop deduplication across names and instances. | | `getPatchParent` (type-only change) | Mutable nested patch and missing-path tests. | | `assertReplacePatchTargetsExistingPath` (type-only change) | Missing target rejected; preceding add/root replacement makes later replacement valid; preceding remove makes it invalid. | | `applyBuilderNamespacePatches` | Sequential/root replacements, append then edit, remove then replace, immutable state, permission/persistence parity. | Removed functions are covered through their replacements: `shouldSyncMediaAssetProps` → shared `updateProps` plus UI E2E; `assertContentOrBuildPayload` → transaction adapter; `isContentModePropForInstance` → eligibility; `isContentModePropPatchValue`/`validatePropPatch` → transition/batch validation; `applyValidatedInstancePatch` → shared patch engine. No behavior test is needed for import-only changes, type declarations, public-entry exports, or regenerated metadata/docs; their callers and contracts retain tests. The original reporter-specific UI reproduction and image-description product question remain separate limitations. This inventory does not claim either is resolved. ### Latest review follow-up - [x] Reject fresh prop IDs that shadow existing instance/property records; cover protected bindings and valid edits - [x] Coalesce updates in linear time while preserving last-update order and input immutability - [x] Check documentation/fixture impact, push, inspect CI, and reply to both review threads Review fixes pushed in `5dd03135e7`: 10 net production lines. Added binding-shadow and batch-work regression assertions; no local tests or evaluations were run, per request. Formatting and diff checks passed. Documentation and generated fixture inputs/outputs are unchanged. Automatic CI is pending. - [x] Reject duplicate fresh props within the same transaction and test split/combined batches - [x] Preserve resource and parameter media bindings using the existing dynamic-type check - [x] Bound total Editor patches before validation; test the boundary and unchanged Builder access Review follow-up `d478446eb1`: all package, Builder, and CLI CI test jobs and typecheck/lint/generated checks passed. Five addressed review threads are resolved. Browser CI checks are still running. One newly posted review about non-image assets assigned to Image sources remains open pending the user's behavior choice. No local tests or agent evaluations were run; local typechecks passed. ### Shared asset constraints - [x] Reuse component file-control metadata and the UI MIME matcher in shared property writes; preserve manually entered URLs and pending uploads - [x] Generate linked Image metadata only when the selected asset supports it; cover legacy file-backed images - [x] Add shared-runtime coverage for accepted/rejected Image and Video assets, custom props, URLs, and atomic rejection - [x] Review documentation, generated contracts, fixture impact, and diff size; verify typechecks and CI (no local tests or evaluations, as requested) Shared asset follow-up `5f0054edd7`: property writes use the UI's existing file-control `accept` metadata and MIME matcher. Known incompatible assets are rejected with the approved message. Legacy image files remain selectable without generating unsupported metadata. URLs, custom asset props, and unresolved uploads keep their existing behavior. Asset records are now loaded before CLI/MCP property writes; this may require an asset-list request when not already loaded. This change does not add asset-file downloads. Net handwritten production change: 24 lines. Builder/project-build typechecks, changed-file lint, formatting, and diff checks passed. No local tests or evaluations were run. Regression and runtime-entry tests were added; CI is pending. Reviewed Image/Video docs: no guidance change needed. No fixture inputs or generated fixture bundles changed; regenerated runtime contract metadata only. - [x] Keep Editor permission checks when a shared edit operation loads assets; add an API regression test before final CI verification Follow-up `d33b70e1c7` preserves Editor authorization when property writes load assets. The API regression test asserts a valid Image source reaches persistence, incompatible files are rejected, and outside-content edits remain forbidden. This reuses the existing asset loader and permission adapters (3 net production lines). CI found and corrected an old session namespace expectation and a mistaken test assumption: Video's existing unsupported `.mpg`/`.mpeg` accept entries cause the shared matcher to allow all assets. This PR preserves that UI behavior; it does not tighten Video formats. Total handwritten production increase for this asset follow-up: 27 lines. Builder typecheck/lint and server metadata generation passed; metadata output was unchanged. CI is rerunning. Final asset follow-up verification (`8b2efd8508`): all package, CLI, Builder API/unit, and six browser E2E jobs passed, together with lint/typechecks, generated checks, visual regression, migrations, size, packaged SSG, and branch deployment. The direct API test now checks the typed authorization cause, not an HTTP status that this test caller does not map. All review threads are resolved. No local tests or agent evaluations were run for this follow-up; CI provided execution coverage. The separate Video filter limitation noted above remains unchanged. ### Shared write-boundary validation - [x] Reuse one asset-property validator in UI/MCP property updates and raw patch persistence, independent of permission level - [x] Cover whole-property and field patches, component changes, invalid metadata, final-state validation, unchanged legacy data, and stale retries - [x] Review documentation/fixture impact, typecheck and lint, then verify CI; no local tests or evaluations per the earlier request Expected change: about 70–90 net production lines. Patch saves load only referenced asset metadata; no asset-file downloads. Shared-save follow-up: 133 net production lines, above the initial estimate because one save can replace an asset file and its references together. The save validator reuses the asset writer’s canonical file preparation rather than trusting client-declared asset types. It also checks asset-only replacements, before folder or Build writes, while leaving unchanged legacy references alone. Ordinary property writes load only referenced asset metadata; asset-changing saves with references additionally prepare the full asset map, then the writer repeats that metadata preparation. No file contents are downloaded by the new check. This does not change schemas, migrations, generated contracts, documentation, or fixture inputs/outputs. Added regression coverage; no local tests/evaluations were run, per the earlier request. Independent Astra review found no remaining defect after the projection fix. CI pending. - [x] Fix selective UI sync loading to include asset-validation dependencies; add selection regression coverage and rerun CI The first browser run caught missing props/instances in selectively loaded UI saves. Package/API tests passed but their fixtures provided complete build rows. The development deployment succeeded; its status-reporting step failed with GitHub HTTP 500. Final verification on `a22d059717`: all package, CLI, Builder unit/API, six browser shards, visual regression, typecheck/lint/generated checks, migrations, size, packaged SSG, development deployment, and Ubuntu/Windows deployment fixtures passed. Browser shard 3 passed on one retry after a page-drag ordering assertion; the save/parsing errors were fixed by loading the required fields. No local tests or evaluations were run. The added persistence suite has 46 passing tests and runtime props has 74; selective-sync tests now omit unselected fields. No unresolved review threads. Total net production increase across this follow-up: 134 lines. ### Deep review follow-up - [x] Make MDX linked-image validation independent of authored attribute order; cover changed sources, mismatches, and the affected materialization paths - [x] Review DRY and diff size, documentation/fixture impact, and independent review findings - [x] Verify typechecks/lint and CI; no local tests or evaluations per the earlier request Review found that linked dimensions can be checked against the old template source before the new source is applied. Fix should reuse the shared eligibility rule with the final source, without another property parser. Initial estimate: under 60 added non-test lines; revisit the approach if it needs more. Deep-review fix `f7c3d805a8`: apply Image sources before linked metadata in all three MDX materialization paths, keeping original attribute indexes for asset references. Reuses the existing eligibility checks and parser; 23 net production lines. Regression cases cover source order, changed sources, mismatched dimensions, and nested overlays. Independent Astra review, project-build typecheck, focused lint, formatting, and diff checks passed. No schema, documentation, generated contract, or fixture-data changes. Local tests/evaluations have not run; focused-test permission was requested. CI pending. Final deep-review verification on `f7c3d805a8`: all package/CLI/Builder test jobs, all six browser shards, visual regression, typechecks/lint/generated checks, migrations, packaged SSG, size, submodules, and development deployment/fixtures passed. CI ran 60 authored-content and 24 template-materialization tests successfully. No local tests or evaluations ran; fail-before/pass-after was not executed locally. All review threads are resolved and the worktree is clean. No further actionable defect was found in the independent reviews. Known remaining cost: asset-changing saves can prepare metadata twice; removing that safely would expand this fix into persistence changes. ### Editor MCP browser E2E follow-up `0ee978e465` adds a real CLI/MCP Editor-token round trip: insert content into a Content Block, reload through a fresh CLI process, verify it in the Builder canvas, and reject a protected template insertion without changing the saved Build. A direct API denial now returns `FORBIDDEN` instead of a generic internal error. The focused E2E and API tests, typecheck, lint, formatting, and diff checks passed locally. Fail-before/pass-after was verified by temporarily disabling Editor fragment access, then restoring it. The full Luna/low evaluation run was attempted but did not pass: the form-runtime fixture received 503 responses. The suite is not claimed green. CI for this push passed on `0ee978e465`, including all six Builder E2E shards, package/CLI/Builder tests, visual regression, typecheck/lint/generated checks, migrations, size, SSG build, and submodules. Live MCP permission probe against the branch deployment: Builder, Editor, and Viewer role metadata matched their tokens. Editor content text, rich text, fragment, link href, and compatible Image source edits committed and survived a fresh MCP session. Outside-root, protected-template, design-only prop, mismatched Image width, and incompatible asset writes were denied without persistence. The updated deployment returned structured `FORBIDDEN` for protected content writes even when `contentMode` was false or omitted. Mixed valid/invalid writes were atomic; stale Editor writes refreshed and retried. Viewer MCP could not be exercised beyond the API gate because this Viewer token has API access disabled. MDX frontmatter, token revocation, and UI-only scenarios were not tested. A wrong-project token was rejected but surfaced `INTERNAL_SERVER_ERROR`, which is a diagnostic-quality gap, not an authorization bypass. The isolated probe page `/codex-permission-probe-6575` remains: cleanup was stopped after a concurrent-edit `CONFLICT`, to avoid deleting another session's work. ### Fixture-based parity follow-up - [ ] Verify the full Content-mode UI/MCP parity matrix with an isolated MDX/template/media/page fixture, excluding publishing and token/plan changes - [x] Fix the reproduced Editor MCP template/page-settings gap and cover fail-before/pass-after routing - [x] Verify a real CLI Editor-token create-from-template call, fresh readback, Builder reload, allowed settings, and protected-setting rejection - [x] Update MCP guidance, review fixture impact, and verify required CI The UI already allows an Editor to create a page from a template and edit allowed page settings. MCP incorrectly required Builder permission, and a separate MCP-only legacy Webhook Form copy guard blocked the seeded template. Those restrictions are corrected. The server now derives Content-mode page copying from Editor permission even when `contentMode` is omitted or false; Builder copies retain their existing behavior. Editor semantic commands use the server's shared Content-mode validator, while raw Editor patches stay limited to text and editable props. This adds a named server call and refetch for Editor semantic edits; the existing initial project sync still occurs. The new isolated E2E passed on `03e7ca6953`, including flag-free template creation, settings edit, fresh CLI readback, Builder reload, and a denied protected setting without a saved change. All required CI passed on that commit: all six Builder E2E shards, package/CLI/Builder tests, generated checks, typecheck/lint, visual regression, migrations, packaged SSG, size, and submodules. Focused local runtime/API tests, typechecks, lint, formatting, and generated-doc checks passed. Browser E2E and full agent evaluations were not run locally; the requested browser approval and a fresh evaluation approval were not received. The full UI/MCP parity matrix remains open; Editor asset operations are a separately identified gap, not claimed fixed here. ### Exploratory permission coverage follow-up - [x] Inventory observed Builder/Editor/Viewer UI and MCP permission scenarios against existing tests - [x] Add fail-before/pass-after regression tests for confirmed uncovered failures - [x] Run focused tests, typechecks, lint, and fixture checks; verify required CI - [x] Record unsupported UI/MCP parity as open product work, without calling the matrix complete The UI share-link E2E already checks Viewer read-only controls, Editor text editing, and Builder design controls. Existing MCP E2E checks Editor content/template writes, page creation, asset upload, and protected edits. This follow-up adds one real CLI/MCP flow with API-enabled Viewer and Builder tokens: Viewer can list pages but cannot create a page or asset folder or upload a file; Builder can change an HTTP status setting that Editor cannot. The test checks the denied page writes leave the build unchanged and the Builder setting survives readback. The run exposed two diagnostic bugs: local permission denial and REST asset-upload denial reached MCP as generic failures. Both now retain `FORBIDDEN`. Focused unit tests failed before these changes and pass after; the full local browser/MCP test passes. The only non-test change is three net lines. No fixture data, schemas, generated contracts, or product docs changed. Local checks passed: 2,654 project-build tests, 48 HTTP-client tests, the focused Builder E2E (2 tests including setup), affected typechecks, lint, formatting, and package boundaries. `check:fixture-sizes` could not run locally because the SSG fixture's built `dist/client` is absent; CI builds that fixture before checking its size. The full UI/MCP parity matrix remains open. In particular, Editor UI asset update/replacement/duplication/deletion and collection-entry creation still lack equivalent MCP support until the UI's collection protections are enforced at the shared server boundary. Publishing and token/plan changes were outside this test scope. CI passed on `6490efc838`: all six Builder E2E shards, package/CLI/Builder tests, visual regression, typecheck/lint/generated checks, migrations, packaged SSG, size, and submodules. Shard 3 first hit Playwright's 120-second limit under four-worker load; its logic passed locally. The longer role flow now uses the same 240-second per-test limit as other MCP E2E tests, and shard 3 passed in the final run. ### Final UI/MCP parity audit - [x] Trace Editor UI asset and folder actions through REST, runtime, and persistence - [x] Reproduce Editor MCP upload and folder-create denials; add fail-before/pass-after permission tests - [x] Add isolated Editor MCP upload/folder and UI folder-reload browser regressions - [x] Keep collection-sensitive asset/folder mutations Builder-only until UI collection protections are shared at the server boundary - [x] Review docs and fixture impact: MCP guidance is updated; no fixture input or bundle changes - [x] Restrict Editor folder saves to new IDs; test overwrite and delete denials at both UI and API boundaries - [x] Reject folder-add collisions at persistence and test the mixed-writer/stale-snapshot case - [x] Keep exact Editor folder retries authorized after a partial save; add fail-before/pass-after coverage - [x] Verify the new browser E2E in required CI - [ ] Run the full Luna/low agent evaluations after fresh approval (not granted for this follow-up) - [x] Verify required CI for `6fd3beefc8` The safe scope is small: the Assets REST upload path already authorizes Editor tokens and enforces collection rules, but the MCP catalog incorrectly required Builder access. Creating an ordinary asset folder is available in the Editor UI, but the shared Content-mode save check rejected it. Both now use Editor permission. The isolated browser tests check fresh MCP readback and Builder reload; they have not run yet. Further parity work remains open: Editor UI allows updates, replacement, duplication, deletion, and collection-entry creation in cases MCP cannot currently perform. These need the UI's collection protections enforced at the shared server boundary before their MCP permits can be widened. Raw Editor `apply-patch` stays text-and-props-only. This PR does not claim the full matrix is complete. Asset/folder follow-up `6fd3beefc8`: Editor semantic tools now share the UI's edit permission for uploads and ordinary folder creation. Folder saves reject same-ID changes, allow identical retries after a lost response, and use insert for add-only persistence. The retry regression failed before the fix and passes with JSON-normalized parent IDs/timestamps. Independent Astra review found no remaining concrete defect. Focused runtime (86), asset persistence (52), protocol (14), and API/sync (71) tests passed; Builder/project-build/asset-uploader typechecks, lint, package boundaries, formatting, generated API, and MCP docs checks passed. Full project-build, protocol, and CLI suites passed earlier in this follow-up; Builder core had resource-contention failures but its affected tests passed in isolation. The new browser E2E and agent evaluations were not run locally pending fresh approval; CI is running them automatically. CI passed on `6fd3beefc8`: all six Builder browser shards, package/CLI/Builder tests, generated checks, typechecks, lint, visual regression, migrations, packaged SSG, size, and submodules. The new Editor MCP asset round trip and UI folder-reload tests ran in the passing browser shards.	2026-10-23 14:40:26 +02:00
Oleg Isonen	13eb7527ef	fix: route deprecated Webhook Forms through Form backend (#6553) ... Deprecated Webhook Forms reuse the existing Form execution pipeline without teaching the modern backend about WebhookForm. ## User-facing behavior and how to test it - Existing saved Webhook Forms keep their markup, action settings, and success/error feedback. Submit a published Webhook Form and check both outcomes. - URL and Resource actions preserve configured methods, headers, dependencies, body overrides, and the last submitted value for repeated field names. Test a configured action and inspect the received request. - Default owner notifications use the Email Worker, including submitted files. Submit a Webhook Form without a custom action and inspect the owner email and attachment. - Configured Email actions validate recipients and retain threading metadata. Test valid recipients and a rejected recipient limit. ## Implementation - One deprecated Webhook Form adapter calls existing bot validation, Email preparation, recipient validation, resource loading, and response handling. - Both generated route templates call that adapter. Both paths reuse an internal resource-graph traversal helper; modern submission behavior is unchanged. - Root actions use the protected, uncached request override. Dependencies reuse existing generated Assets and Resource caching behavior. - Keep reachable-resource filtering because Email preparation runs before loading: unrelated Email settings must not break the selected action. - Remove dead `getDefaultActionResource` wiring from generated templates. ## Verification - Focused SDK tests: 116 passed (11 deprecated Webhook Form adapter and 105 modern submission, Email, and handler tests), including unrelated resources, missing dependencies, cycles, Email preflight/threading, and single-attempt Email failures. - Resource generator tests: 26 passed. - Full CLI prebuild suite: 139 passed after tracked fixture regeneration; focused deprecated Webhook Form adapter tests: 9 passed. - Repository fixture pipeline (`pnpm -r fixtures:build`) passed. All tracked generated `getDefaultActionResource` references were removed, and no Builder `N8N_FORM_EMAIL_HOOK` reference remains. - Final MCP-generated local Preview E2E rerun: 3/3 passed. - Focused saved Webhook Form custom-action E2E rerun: 2/2 passed (setup and test), including a single POST through the protected test receiver after Builder reload. - SDK, CLI, and Builder typechecks, focused lint and formatting, and `git diff --check` passed. - Deprecated component picker regression test passed (2/2); verified it fails when WebhookForm deprecation is disabled, then passes with deprecation restored. - Independent Astra review found no blocker, including the 62 generated fixture changes. No production deployment or live secret change was performed. - Required staging CI checks passed on previous head `15635fc` (Main workflow, visual regression, submodules, and lint). An unrelated existing MCP success-redirect E2E assertion failed once in shard 5, passed in a focused local rerun (2/2 including setup), then shard 5 passed on GitHub rerun without code changes. - Terminology pass (`4a69ab573`): renamed the dedicated SDK adapter and migration helper, updated both route templates and generated fixtures, corrected the active identity migration marker comments, and synchronized docs and test names. Focused tests passed: SDK 35, CLI prebuild 12 plus 2 after final title change, MCP 7, migrations 8, React browser 7, CLI docs 9. Seven affected package typechecks, fixture builds, formatting, and `git diff --check` passed. That historical head was superseded; latest-head CI results are recorded below. - ResourceForm identity migration (`29a21edbb`): focused migration (23), DB patch (25), project-build DB/session (93), Builder API/import/auth/patch (60), React SDK (162), runtime/MCP (239), registry (11), CLI evaluation (90), CLI session/sync (55), and protocol (15) tests passed; full CLI prebuild (139) passed. Affected typechecks, lint, formatting, and `git diff --check` passed. Independent Astra review found no blocker. No deployment was performed. - Prepublish Webhook Form warning (`25855e8f00`): audit checks every non-draft published page and links nested/shared instances to the correct page; modern ResourceForm, native forms, draft-only forms, and orphans are excluded. Focused audit tests (13), Builder navigation tests (6), project-build and Builder typechecks, lint, formatting, and `git diff --check` passed. Independent Astra review found no blocker; publishing remains nonblocking. No deployment was performed. - ResourceForm test and CI expectation repair (`e1a527fbcc`): focused React SDK (24), Builder preview/registry (38), project-build pages (1), and CLI fixture API (8) tests passed; Remix and React Router targeted browser suites, affected package typechecks, full formatting (3375 files), lint, and `git diff --check` passed. Astra review found no blocker. Persisted identity and runtime exports were unchanged. - Project bundle fixture repair (`70889ef27d`): eight bundles now carry the current contract version; the two unmarked historical Form instances in `webstudio-features` normalize to WebhookForm. Protocol fixture tests (17), protocol typecheck, CLI fixture build, formatting, lint, and diff check passed. Independent Astra review found no blocker. - Story/MCP E2E CI repair (`483614e6c5`): SDK CLI story tests (7), ResourceForm fragment tests (34), and focused local managed-form E2E (4 including setup) passed. SDK CLI and Builder typechecks, lint, formatting, and diff check passed. Astra review found no blocker. The initial local Preview build used an ignored stale React Router `lib` artifact; rebuilding it restored the passing E2E run. - Latest PR head `483614e6c5`: Main workflow [run 39317777032](https://github.com/webstudio-is/webstudio/actions/runs/39317777032) passed, including all package matrices and six Builder E2E shards. Visual regression [run 39317796922](https://github.com/webstudio-is/webstudio/actions/runs/39317796922), submodules, and PR lint checks passed. The overlapping unapproved visual run was cancelled by the approved rerun. - Shared bot-value generator (`ecc4cad1e9`): SDK helper tests (3), React Router WebhookForm/ResourceForm browser tests (5), SDK/Router/Remix typechecks and builds, SDK browser-boundary check, formatting, and diff check passed. Brave bypasses media queries, working media queries produce a fresh timestamp per submission, and failed queries produce `jsdom`; each managed/legacy submission has one bot field. Independent Astra review found no blocker. Latest-head Main workflow [run 39325677117](https://github.com/webstudio-is/webstudio/actions/runs/39325677117) passed, including all six Builder E2E shards; approved visual regression [run 39325702835](https://github.com/webstudio-is/webstudio/actions/runs/39325702835), submodules, and PR lint passed. The superseded visual run was cancelled by the approved rerun.\n\n- Historical Form clipboard guard (`e864283d1c`): instance/page transfer parsers reject ambiguous legacy `Form` before asset import; direct fragment/page/folder mutations preflight before changes. Canonical ResourceForm, WebhookForm, and NativeForm remain accepted. Project-build tests (89), Builder paste browser tests (69), both typechecks, focused lint/formatting, and diff check passed. Independent Astra review found no blocker. Latest-head Main workflow [run 39329184890](https://github.com/webstudio-is/webstudio/actions/runs/39329184890) passed, including all six Builder E2E shards; approved visual regression [run 39329562692](https://github.com/webstudio-is/webstudio/actions/runs/39329562692), submodules, and PR lint passed. The earlier unapproved visual run reported two expected story differences.\n\n## Compatibility notes - Root HTTP actions reject private and same-origin destinations. Dependencies retain their existing fetch/cache behavior. - Routes retain the 25 MiB request limit. Shared execution applies bot validation, a 10-second Resource timeout, and one retry for eligible HTTP failures; an ambiguous failure can duplicate an HTTP action. Email actions are not retried. - Default email uses the Email Worker's sender, plain text, recipient limits, and attachment limits instead of the legacy n8n HTML/basic-auth path. - Self-hosted sites need normal Email sender configuration. Direct Builder draft Preview handling is outside this published-route change. - Coordinate this CLI release with [SaaS PR #669](https://github.com/webstudio-is/webstudio-saas/pull/669), which removes the n8n helper and `N8N_FORM_EMAIL_HOOK` from new published-site builds. Publish with a CLI artifact containing this adapter. Existing deployed workers retain their bundled code and injected hook until republished, so keep the actual n8n endpoint available while those sites remain active. ## Implementation checklist - [x] Align saved Webhook Form custom-action E2E with the protected receiver and verify one submission. - [x] Regenerate tracked published-site fixtures with the updated CLI and confirm no `getDefaultActionResource` or `N8N_FORM_EMAIL_HOOK` references remain in Builder source/config/fixtures. - [x] Review authoritative `docs/` impact of removing the obsolete Builder hook wiring; related deprecation and integration docs were updated. - [x] Route deprecated Webhook Form submissions through the shared execution pipeline with one adapter. - [x] Preserve configured actions, dependencies, repeated fields, body overrides, and saved feedback. - [x] Prepare and validate Email actions using existing shared logic. - [x] Remove duplicated orchestration and dead generated context wiring. - [x] Verify generated routes and MCP-generated Preview submissions. - [x] Mark the Webhook Form component and its dedicated dependencies visibly deprecated, while leaving shared Form infrastructure active. - [x] Group Webhook Form-only SDK code for straightforward removal while preserving public APIs and generated output. - [x] Use deprecated terminology consistently for Webhook Form-only PR code, and clarify the active identity migration marker. - [x] Confirm required staging CI checks. - [x] Use ResourceForm as the one active modern component identity across Builder, SDK, runtime, templates, generated code, and docs. - [x] Migrate historical Form, NativeForm, ResourceForm, and WebhookForm identities in one idempotent pass using the historical pages marker. - [x] Remove the marker from ordinary project and snapshot APIs while preserving historical input readers. - [x] Persist pages and instances together when a historical marked project is saved, including a pages-only edit. - [x] Verify migration, export/reimport, old CLI and server snapshots, page-only save, generation, and affected typechecks. - [x] Complete an independent architecture review before commit or push. - [x] Warn at prepublish for deprecated Webhook Forms on published pages, with a link to each affected element. - [x] Verify modern ResourceForm is excluded and nested/shared Webhook Forms navigate to the correct page. - [x] Complete independent review, focused tests, and affected typechecks before push. - [x] Remove local `ResourceForm as Form` aliases from React SDK tests and fixtures; use the canonical component name throughout. - [x] Fix stale identity expectations found in CI and verify affected tests/typechecks. - [x] Complete independent review before push. - [x] Refresh tracked project bundle fixtures to the current bundle contract and canonical WebhookForm identity; verify protocol and affected CLI flows. - [x] Align SDK CLI story assertions and MCP E2E fragment fixtures with the persisted ResourceForm component name; rerun affected checks. - [x] Share one browser bot-value generator across ResourceForm submission and deprecated Webhook Form wrappers, preserving field ownership and behavior. - [x] Verify bot timestamps, Brave/media failure handling, repeated submissions, affected typechecks, and independent review before push. - [x] Reject ambiguous historical `Form` clipboard fragments before any insertion or transfer mutation, with clear recopy guidance; preserve canonical ResourceForm action remapping.\n- [x] Verify instance/page/template/folder paste regressions, independent review, and latest-head CI.\n\n - [x] Explain Email attachment size limits in the Email action UI and give a useful error when the aggregate encoded attachments exceed the local guard. - [x] Check Preview and published behavior, boundary tests, and authoritative Form docs; verify the focused change and get independent review. Ref #4093 - [x] Preserve safe email codes/messages through MCP, including local attachment and upload limit errors. - [x] Correct Form documentation for each enforced limit and verify Preview/published/MCP paths. - [x] Verify Builder SDK and MCP focused tests, typechecks, and independent review before push. - Email limits and failure copy (`6d215f3ec3`): the shared SDK checks 32 attachments and 7 MiB aggregate Base64 data before dispatch; the 25 MiB multipart response and Email action guidance use approved copy. Managed Email error codes pass through Form and MCP results with resource IDs, while MCP still omits resource bodies. Form docs distinguish the Worker 7 MiB JSON cap from Cloudflare's 5 MiB complete message and verified-destination exception, and correct the no-retry behavior. Focused SDK tests 89, Builder API tests 44, UI browser tests 42, SDK/Builder/CLI/docs typechecks, lint, formatting, and generated MCP docs check passed. Astra found no blocker. No real email or production deployment occurred. - Latest Email limits head `6d215f3ec3`: Main workflow [run 39393037290](https://github.com/webstudio-is/webstudio/actions/runs/39393037290) passed on rerun, including all six Builder E2E shards; approved visual regression [run 39393456618](https://github.com/webstudio-is/webstudio/actions/runs/39393456618), submodules, and PR lint passed. The initial unapproved visual run showed the same two previously accepted story differences. - [x] Address review: normalize previous-Builder `Form` instance patches before persistence, preserve historical WebhookForm identity, and cover marked/unmarked cases. Documentation impact reviewed: this restores intended save behavior, so no user guide change is needed. - Review finding follow-up (`ce05dd733a`): three Form identity patch regressions passed red/green; full project build-patch tests 28/28, project typecheck, lint, formatting, and diff check passed. Independent Astra review found no blocker. Incoming PATCH `Form` is treated as the immediately previous Builder's modern form alias; already stored unmarked historical `Form` continues to migrate to WebhookForm. The old patch protocol has no client-release identifier, so pre-October-14 live clients adding new ambiguous IDs cannot be identified separately. - Latest review-fix head `ce05dd733a`: Main workflow [run 39397337356](https://github.com/webstudio-is/webstudio/actions/runs/39397337356) passed, including six E2E shards after a targeted shard-4 rerun for an unrelated MDX slash-menu timeout. Approved visual regression [run 39397395206](https://github.com/webstudio-is/webstudio/actions/runs/39397395206), submodules, and PR lint passed. The unapproved synchronize visual run was cancelled by the approved rerun.	2026-10-21 00:25:55 +02:00
Oleg Isonen	efbc3098c4	feat: add queryable Assets resources (#5946) ... Closes #5919 Ref #5920 ## Outcome Extend the existing Assets system resource with optional queries for Markdown, JSON, and standard file metadata. Unconfigured Assets resources retain the existing fetch-all behavior and activate none of the content-query infrastructure. Files are the authoring model. A standalone content engine converts an immutable snapshot of only the files required by configured queries into one bounded, deterministic runtime database. Builder preview, SSR, and SSG use the same query semantics without requiring a database service for a published site. ## Final architecture ```text Markdown / JSON files │ ▼ ContentSource adapter (FS or Webstudio Assets) │ immutable snapshot ▼ ContentCompilationPlan (union of configured query requirements) │ ▼ ContentCompiler (parse, normalize, discover fields, enforce budget) │ ▼ ContentArtifact (bounded, deterministic, immutable JSON) │ ▼ ContentDatabase (filter, OR groups, sort, paginate, project, read content) │ ├── Builder preview ├── SSR runtime └── SSG build only ``` The authoring/storage layer owns mutable files. The content engine consumes immutable snapshots and produces immutable, disposable artifacts. Query authoring is a separate OpenAPI-driven path: ```text Assets REST OpenAPI operation + standard request JSON Schema + dynamic field paths │ ▼ Generic OpenAPI adapter │ resolved schemas, controls, paths, and transport mapping ▼ Generic query-builder core and React renderer │ arbitrary provider-defined value ▼ Assets REST request adapter │ ▼ { query: value } ``` The selected standard OpenAPI operation is the only source of query UI semantics. Request parameters and JSON Schema describe the editable data; the generic renderer derives controls and owns layout. ### Activation and compilation - The compiler does not index every asset. It combines all reachable configured Assets-resource queries into one `ContentCompilationPlan`. - Static constraints such as folder, path, and file type select candidate files. Dynamic values such as a route-bound slug cannot reduce the build-time candidate set. - The plan contains the union of fields used by filtering, sorting, routing, projection, excerpts, and requested full content. - Documents shared by multiple resources are stored once in one deployment-wide runtime database. - Markdown/frontmatter, JSON properties, excerpts, and full bodies are prepared only when required. - No configured content query means no parsing, artifact, or shipped runtime dependency. Fresh generated apps need no content-runtime cleanup; incremental generation removes stale integration if a previously configured query is deleted. ### Runtime database limit - `CONTENT_DATABASE_MAX_BYTES` limits the complete serialized runtime database, not source files, images, or unrelated assets. - The default is `500 * 1024` bytes and is presented as 500 KiB. - Counted data includes indexed metadata, frontmatter/JSON properties, excerpts, the field catalog, serialization overhead, and full Markdown bodies only when requested. - Prepublish validation builds and serializes the exact database that will be attached to the immutable build, so it knows the unbounded size, bounded size, included documents, omitted documents, and affected resources. - Overflow never stores partial documents. Candidates are ordered by `createdAt` descending, canonical path ascending, and stable document ID ascending. Missing dates follow dated documents. A document that does not fit is skipped while smaller later documents are still considered. - Exceeding the limit warns but does not block publishing. - Builder preview can report exact bounded versus unbounded results when dynamic expressions have concrete preview values. Prepublish warnings remain conservative for future route values. ### Snapshot consistency The source contract guarantees a consistent compilation, while adapters provide the mechanism: - Webstudio Assets captures asset metadata and immutable content references. - Filesystem sources capture file identity and verify files did not change before accepting the artifact. - A changed source retries once, then fails clearly instead of producing a mixed-revision database. - This requires no universal snapshot service, source-tree copy, or new versioned-asset infrastructure. ### Package ownership - `@webstudio-is/content-engine`: runtime artifact schema, query execution, field catalog, compact runtime statistics. - `@webstudio-is/content-engine/compiler`: source snapshots, compilation plans, Markdown/JSON parsing, deterministic compilation, byte accounting, and compiler diagnostics. This entry exists to keep Remark/YAML/future MDX dependencies out of runtime bundles. - `@webstudio-is/query-builder`: resolves standard OpenAPI operation schemas into an internal provider-neutral editor model, validation, mutations, and source serialization. It owns no Assets, provider-authored UI contract, or fixed REST-query semantics. - `@webstudio-is/query-builder/react`: container-neutral schema renderer whose layout is renderer policy. It remains separate from the content engine. - SDK: Webstudio Assets/Resource domain models and mapping Webstudio expressions into generic content requirements. - Protocol: external Assets REST/OpenAPI transport contracts. - Asset uploader/repository: asset and folder mutations, generic metadata, immutable byte references, and storage access. Its hosted adapter supplies Webstudio Assets snapshots and parsed-metadata cache persistence to the content engine; parsing, artifact creation, and query semantics remain engine-owned. - Builder and generated-app integrations decide when to compile, cache, embed, load, or discard a database. ### Runtime lifecycle - Builder caches immutable `ContentDatabase` instances by source revision plus compilation-plan hash and coalesces concurrent compilation. Matching operations reuse the same instance; field discovery uses a dedicated minimal all-properties plan that excludes excerpts and bodies. - SSR attaches the validated bounded database to the build and includes the small runtime only when required. - SSG uses the database only while generating pages and omits it and the runtime from final output unless a future runtime-query feature requires them. - Local development compiles from the filesystem and uses identical query semantics. - Published pages query their local artifact; they do not call Builder REST endpoints. - Published artifacts need no external index store, invalidation, garbage collection, SaaS content proxy, new bucket, or new environment variable. ## Implemented foundation These existing parts remain useful and should be moved behind the final ownership boundaries rather than reimplemented. - [x] Upgrade the existing Assets system resource in place and preserve unconfigured fetch-all behavior. - [x] Implement typed filtering, nested boolean groups, deterministic sorting, pagination, projections, dynamic values, and content hydration. - [x] Add Markdown frontmatter, excerpt, JSON-property, and field-catalog extraction. - [x] Add reusable query-control implementations and a separately imported React UI as a foundation for the final OpenAPI-driven contract. - [x] Add Assets query authoring and dynamic field discovery as existing behavior to migrate into the Assets OpenAPI operation. - [x] Add synchronized visual and editable expression authoring. - [x] Unify Assets REST operations under `/rest/assets/*`. - [x] Define authenticated collection, item, content, upload, folder, query, field-catalog, and OpenAPI operations. - [x] Support Builder cookie authentication and public API token authentication. - [x] Add Builder preview, MCP/CLI operations, SSR execution, SSG path generation, and conditional activation. - [x] Add revision-keyed parsed metadata persistence for the hosted Builder. - [x] Add focused query, API, authorization, Builder, CLI, SSR, SSG, migration, and fixture coverage. ## Remaining implementation plan ### 1. Establish the content engine - [x] Rename the generic `@webstudio-is/asset-resource` implementation to `@webstudio-is/content-engine`. - [x] Move existing behavior instead of creating a parallel implementation. - [x] Keep the primary entry runtime-only and add only the compiler entry needed for bundle separation. - [x] Remove the old package after all consumers migrate. - [x] Ensure the engine has no Builder, SDK, asset-uploader, PostgreSQL, R2, or deployment-target dependency. ### 2. Define source snapshots and compilation plans - [x] Introduce `ContentSource.openSnapshot()` and a revision-bearing `ContentSourceSnapshot`. - [x] Include canonical path, stable document ID, content type, immutable content reference, size, and optional `createdAt`. - [x] Implement Webstudio Assets and filesystem snapshot guarantees using their adapter-specific mechanisms. - [x] Introduce `ContentCompilationPlan` for the union of reachable configured resource requirements. - [x] Separate statically selectable candidates from dynamic runtime predicates. - [x] Add mutation-during-compilation tests for replacement, deletion, folder movement, and filesystem changes. ### 3. Separate storage from content processing - [x] Keep upload, replacement, folders, byte storage, revisions, and generic metadata in the Assets repository/uploader. - [x] Move parsing, excerpts, field discovery, query execution, and artifact creation into the content engine. - [x] Remove content-index coordination from asset mutation business logic. - [x] Preserve original source `createdAt` across content replacement and expose it as optional queryable metadata. - [x] Keep hosted parsed-document persistence behind the content compiler's optional cache interface. ### 4. Build deterministic compilation - [x] Move the existing Markdown, frontmatter, JSON, canonicalization, hashing, document-building, and tests into the compiler. - [x] Keep frontmatter behind a dedicated module and continue using the existing Remark/Micromark pipeline. - [x] Parse only candidate files and only the tiers required by the compilation plan. - [x] Deduplicate documents and fields required by multiple resources. - [x] Define a versioned, deterministic `ContentArtifact` with integrity information. - [x] Keep verbose parse and omission diagnostics outside the runtime artifact. - [x] Reject incompatible artifact versions cleanly. ### 5. Enforce the deployment-wide database budget - [x] Replace the old asset-index limit with `CONTENT_DATABASE_MAX_BYTES`, defaulting to 500 KiB. - [x] Have applications read configuration and pass the byte limit into the compiler. - [x] Measure the exact serialized database, including selected bodies and overhead. - [x] Implement deterministic skip-and-continue packing of complete documents. - [x] Return compiler diagnostics with unbounded/bounded bytes, included/omitted counts, omitted identities, largest contributors, and affected resources. - [x] Expose only compact artifact statistics at runtime through `ContentDatabase.getStats()`. - [x] Add exact boundary, deterministic ordering, deduplication, and oversized-document tests. ### 6. Finalize the runtime database - [x] Define `ContentDatabase` over an immutable artifact. - [x] Preserve filtering, nested AND/OR groups, sorting, offset/limit, projection, field references, and content retrieval. - [x] Expose `getFieldCatalog()` and compact `getStats()`. - [x] Keep Markdown/YAML/compiler dependencies out of the runtime entry. - [x] Add execution guards and maximum-size runtime benchmarks. ### 7. Make query authoring OpenAPI-driven The selected standard OpenAPI operation is the complete query-authoring contract. The query builder resolves parameters and request-body JSON Schema, derives its internal editor model, and applies renderer-owned layout policy. It must not embed or fetch a second capabilities/UI language through `x-webstudio-query` or another side channel. - [x] Remove `x-webstudio-query` and all vendor query-definition metadata from the Assets OpenAPI operation. - [x] Resolve local OpenAPI references and derive the editable request value, defaults, validation, labels, choices, arrays, objects, and unions solely from standard operation parameters and request-body JSON Schema. - [x] Represent dynamic Assets field paths with standard JSON Schema vocabulary in the `queryAssets` request schema. - [x] Keep the normalized editor representation internal to query-builder; do not expose another provider-authored capabilities contract. - [x] Remove provider-controlled layout metadata such as `group`; make compact scalar packing and all other layout renderer policy. - [x] Keep Webstudio expression binding as a consumer/editor concern layered over schema values, not part of the OpenAPI contract. - [x] Preserve the existing Assets filter, sort, output selection, full-content/range selection, limit/offset, editable source, and configured-field behavior. - [x] Keep the Assets resource adapter responsible for serializing the edited query into its request body. - [x] Remove the obsolete Assets query-capabilities endpoint, URL, and UI-side capability parsing. - [x] Add a non-Assets OpenAPI proof with differently named parameters and no vendor metadata. - [x] Verify the generic packages contain no Assets imports, fixed Assets property contract, or provider-authored layout model. - [x] Update focused core, React, SDK, protocol/OpenAPI, Builder, and source round-trip tests. ### 8. Manage ContentDatabase lifecycle in Builder - [x] Cache databases by source revision plus normalized compilation-plan hash. - [x] Coalesce concurrent compilation requests. - [x] Reuse cached database instances across field discovery, autocomplete, query preview, and response preview when their normalized compilation plans match. - [x] Replace it when source revision or query requirements change. - [x] Preserve the optional revision-keyed hosted parsed-document cache without making it a source of truth. - [x] Show exact preview warnings when concrete values reveal missing bounded results. ### 9. Materialize and verify immutable builds - [x] Compile and serialize the exact deployment-wide database during prepublish validation. - [x] Attach that validated bounded database to the immutable build without rebuilding it from later asset state. - [x] Show non-blocking prepublish warnings with database bytes, omissions, and potentially affected resources. - [x] Avoid claiming exact future route matches when dynamic values are unknown. - [x] Include the database/runtime only when a reachable configured query requires them. - [x] Ensure detail resources requesting Markdown content include all statically eligible bodies needed by dynamic routes. ### 10. Integrate all targets - [x] SSR: load the build's bounded database and conditionally include the small runtime. - [x] SSG: query the same database during generation, then omit database/runtime output when no runtime query is needed. - [x] Local/CLI: compile from the filesystem source with the same semantics. - [x] Preserve overview pagination, alternative route fields, draft-as-user-data behavior, and full article rendering. - [x] Verify ordinary Assets and fresh query-free sites perform no parser, artifact, content-runtime dependency install, or cleanup work. ### 11. Correct API and package ownership - [x] Move external Assets REST/OpenAPI contracts to `@webstudio-is/protocol`. - [x] Keep Webstudio domain models and Assets system-resource configuration in the SDK. - [x] Keep both cookie and token authentication behavior on the unified Assets REST surface. - [x] Remove pass-through route wrappers/reexports that add no transport behavior. - [x] Keep published runtime queries local to the artifact rather than routing them through Builder REST. ### 12. Remove superseded deployment infrastructure - [x] Remove external deployment-index upload, fetch, invalidation, cleanup, and garbage-collection code. - [x] Remove the SaaS asset-origin adapter requirement; close webstudio-is/webstudio-saas#652 as superseded because it has no independent purpose. - [x] Remove index-bucket compatibility configuration and avoid adding Infisical variables. - [x] Remove obsolete external-index paths, package dependencies, routes, scripts, and tests. - [x] Consolidate unreleased intermediate migrations where safe and retain only schema required by the final hosted cache/API design. ### 13. Verification and release readiness - [x] Move and preserve existing tests while changing ownership; add new tests only for changed contracts and behavior. - [x] Run content-engine compiler/runtime tests, query-builder tests, SDK/protocol/API tests, Builder tests, CLI tests, and affected typechecks. - [x] Run formatting, generated-file idempotence, fixtures, clean Supabase-compatible migrations, E2E shards, and complete agent evaluations. - [x] Verify SSR and SSG conditional inclusion and query-free generated package manifests. - [x] Run the five-article MCP blog workflow: authoring, field discovery, overview, dynamic detail routes, full Markdown rendering, publishing, and published-site verification. - [x] Run a maximum-budget database test and a mutation-during-compilation test. - [x] Perform a final architecture, security, API compatibility, migration, bundle, correctness, and DRY review. - [x] Reconcile every checkbox and replace the verification section with results from the final architecture before release. ## Future MDX work MDX is intentionally separate from this release. - The compiler contract should allow a future build-only MDX parser. - MDX compilation must not enter the published runtime. - Embedded JSX must use an explicit compiled representation rather than runtime source evaluation. - Webstudio components, tokens, styles, and component allowlists remain Webstudio integration concerns. - The future MDX work should reuse `ContentArtifact` and `ContentDatabase` where practical. ## Superseded design The following earlier design choices are no longer the target architecture: - GROQ or GraphQL as the Assets query language. - One external or long-lived index object per resource/query. - A deployment index fetched by the Worker from SaaS/CDN infrastructure. - Lazy production reads of source Markdown bodies. - A separate index bucket, asset origin, invalidation service, or garbage collector. - A 4 MiB published index limit. - Rebuilding an old build's database from the current mutable asset set. - Bundling query infrastructure into projects without a configured content query. ## Verification baseline The embedded runtime-database architecture previously passed the checks below. The OpenAPI-driven query-authoring change additionally passed: - Query-builder core: 4 files / 19 tests. - Query-builder React: 1 file / 7 tests. - SDK Assets source/config: 1 file / 7 tests. - Protocol Assets OpenAPI: 1 file / 9 tests, including independent OpenAPI validation. - Builder Assets OpenAPI description: 1 file / 5 tests. - Query-builder, React, SDK, protocol, and Builder typechecks. - Package-boundary validation and formatting of every affected file. Local verification baseline: - Content engine: 15 files / 104 tests. - Asset uploader: 24 files / 175 tests. - Builder: 204 files / 2,423 tests. - CLI: 54 files / 866 tests. - Complete workspace typecheck, formatting, lint, package boundaries, generated fixtures, and fixture builds. - The complete migration history applied successfully to a fresh Supabase-compatible database, including every migration after the consolidated Assets migration. - Builder production build and full local E2E suite passed. - Complete agent evaluations passed: authenticated page (170,859 ms), design input (207,101 ms), and font assets (67,050 ms). - Final review additionally verified correct REST edit/view/build permits, cookie and public-token read paths, immutable snapshot retries, exact database budgeting, query-free dependency omission, and SSR/SSG conditional runtime inclusion. Final staging verification passed against the embedded runtime-database architecture: MCP uploaded and updated five Markdown articles, discovered dynamic frontmatter fields, previewed a five-item sorted overview and one full-body slug query, updated both Assets resources, found zero binding issues, rendered `/blog` and `/blog/dynamic-routes` at HTTP 200, published build version 266, and verified both published routes at HTTP 200 with the expected article metadata and Markdown body. The workflow also found and fixed offset-form asset timestamps failing canonical datetime validation and made MCP `preview.start({ port: 0 })` correctly select an available local port. ## Release constraints - No new SaaS content service, bucket, or environment variable. - No published-site database dependency. - No parser/compiler dependency in query-free sites or runtime-only bundles. - The validated content database is immutable build output. - Publishing warnings do not block deployment. - The selected standard OpenAPI operation and its request JSON Schema are the only source of query UI semantics.	2026-07-31 00:17:46 +02:00
Oleg Isonen	e7b7faeca5	fix: performance optimizations (#5827)	2026-06-19 12:23:44 +02:00
Oleg Isonen	392c9089be	fix: content mode permissions (#5795) ... **Summary** Fixes content-mode edit permissions so editor-only users can perform allowed content workflows without backend “build permission” failures. **Changes** - Added shared content-mode permission validation used by backend authorization. - Aligned content-mode capabilities around editable props, editable instances, content roots, pages, assets, and template insertion. - Allowed content-safe instance/text/page/template/style changes while keeping design/data/resource changes build-only. - Updated template insertion to preserve content-safe local styles and reuse tokens by id. - Added accessible `SyncStatusDot` for e2e-visible sync state without replacing the existing sync status UI. - Added builder e2e setup with Postgres/PostgREST and a first content-mode browser test: - edit existing text as an editor-only user - wait for saved state via the sync status dot - reload and verify persistence **Verification** - `pnpm --filter=@webstudio-is/builder test -- sync-status.test.tsx` - `pnpm --filter=@webstudio-is/builder typecheck` - `git diff --check` - `pnpm e2e:builder`	2026-06-09 01:48:23 +02:00
Oleg Isonen	b006947a85	feat: WIP realtime collaboration to builder (#5726) ... https://github.com/webstudio-is/webstudio/issues/46	2026-04-27 23:19:21 +02:00