mirror of
https://github.com/webstudio-is/webstudio.git
synced 2026-10-23 03:27:27 +09:00
Closes #5919 Ref #5920 ## Outcome Extend the existing Assets system resource with optional queries for Markdown, JSON, and standard file metadata. Unconfigured Assets resources retain the existing fetch-all behavior and activate none of the content-query infrastructure. Files are the authoring model. A standalone content engine converts an immutable snapshot of only the files required by configured queries into one bounded, deterministic runtime database. Builder preview, SSR, and SSG use the same query semantics without requiring a database service for a published site. ## Final architecture ```text Markdown / JSON files │ ▼ ContentSource adapter (FS or Webstudio Assets) │ immutable snapshot ▼ ContentCompilationPlan (union of configured query requirements) │ ▼ ContentCompiler (parse, normalize, discover fields, enforce budget) │ ▼ ContentArtifact (bounded, deterministic, immutable JSON) │ ▼ ContentDatabase (filter, OR groups, sort, paginate, project, read content) │ ├── Builder preview ├── SSR runtime └── SSG build only ``` The authoring/storage layer owns mutable files. The content engine consumes immutable snapshots and produces immutable, disposable artifacts. Query authoring is a separate OpenAPI-driven path: ```text Assets REST OpenAPI operation + standard request JSON Schema + dynamic field paths │ ▼ Generic OpenAPI adapter │ resolved schemas, controls, paths, and transport mapping ▼ Generic query-builder core and React renderer │ arbitrary provider-defined value ▼ Assets REST request adapter │ ▼ { query: value } ``` The selected standard OpenAPI operation is the only source of query UI semantics. Request parameters and JSON Schema describe the editable data; the generic renderer derives controls and owns layout. ### Activation and compilation - The compiler does not index every asset. It combines all reachable configured Assets-resource queries into one `ContentCompilationPlan`. - Static constraints such as folder, path, and file type select candidate files. Dynamic values such as a route-bound slug cannot reduce the build-time candidate set. - The plan contains the union of fields used by filtering, sorting, routing, projection, excerpts, and requested full content. - Documents shared by multiple resources are stored once in one deployment-wide runtime database. - Markdown/frontmatter, JSON properties, excerpts, and full bodies are prepared only when required. - No configured content query means no parsing, artifact, or shipped runtime dependency. Fresh generated apps need no content-runtime cleanup; incremental generation removes stale integration if a previously configured query is deleted. ### Runtime database limit - `CONTENT_DATABASE_MAX_BYTES` limits the complete serialized runtime database, not source files, images, or unrelated assets. - The default is `500 * 1024` bytes and is presented as 500 KiB. - Counted data includes indexed metadata, frontmatter/JSON properties, excerpts, the field catalog, serialization overhead, and full Markdown bodies only when requested. - Prepublish validation builds and serializes the exact database that will be attached to the immutable build, so it knows the unbounded size, bounded size, included documents, omitted documents, and affected resources. - Overflow never stores partial documents. Candidates are ordered by `createdAt` descending, canonical path ascending, and stable document ID ascending. Missing dates follow dated documents. A document that does not fit is skipped while smaller later documents are still considered. - Exceeding the limit warns but does not block publishing. - Builder preview can report exact bounded versus unbounded results when dynamic expressions have concrete preview values. Prepublish warnings remain conservative for future route values. ### Snapshot consistency The source contract guarantees a consistent compilation, while adapters provide the mechanism: - Webstudio Assets captures asset metadata and immutable content references. - Filesystem sources capture file identity and verify files did not change before accepting the artifact. - A changed source retries once, then fails clearly instead of producing a mixed-revision database. - This requires no universal snapshot service, source-tree copy, or new versioned-asset infrastructure. ### Package ownership - `@webstudio-is/content-engine`: runtime artifact schema, query execution, field catalog, compact runtime statistics. - `@webstudio-is/content-engine/compiler`: source snapshots, compilation plans, Markdown/JSON parsing, deterministic compilation, byte accounting, and compiler diagnostics. This entry exists to keep Remark/YAML/future MDX dependencies out of runtime bundles. - `@webstudio-is/query-builder`: resolves standard OpenAPI operation schemas into an internal provider-neutral editor model, validation, mutations, and source serialization. It owns no Assets, provider-authored UI contract, or fixed REST-query semantics. - `@webstudio-is/query-builder/react`: container-neutral schema renderer whose layout is renderer policy. It remains separate from the content engine. - SDK: Webstudio Assets/Resource domain models and mapping Webstudio expressions into generic content requirements. - Protocol: external Assets REST/OpenAPI transport contracts. - Asset uploader/repository: asset and folder mutations, generic metadata, immutable byte references, and storage access. Its hosted adapter supplies Webstudio Assets snapshots and parsed-metadata cache persistence to the content engine; parsing, artifact creation, and query semantics remain engine-owned. - Builder and generated-app integrations decide when to compile, cache, embed, load, or discard a database. ### Runtime lifecycle - Builder caches immutable `ContentDatabase` instances by source revision plus compilation-plan hash and coalesces concurrent compilation. Matching operations reuse the same instance; field discovery uses a dedicated minimal all-properties plan that excludes excerpts and bodies. - SSR attaches the validated bounded database to the build and includes the small runtime only when required. - SSG uses the database only while generating pages and omits it and the runtime from final output unless a future runtime-query feature requires them. - Local development compiles from the filesystem and uses identical query semantics. - Published pages query their local artifact; they do not call Builder REST endpoints. - Published artifacts need no external index store, invalidation, garbage collection, SaaS content proxy, new bucket, or new environment variable. ## Implemented foundation These existing parts remain useful and should be moved behind the final ownership boundaries rather than reimplemented. - [x] Upgrade the existing Assets system resource in place and preserve unconfigured fetch-all behavior. - [x] Implement typed filtering, nested boolean groups, deterministic sorting, pagination, projections, dynamic values, and content hydration. - [x] Add Markdown frontmatter, excerpt, JSON-property, and field-catalog extraction. - [x] Add reusable query-control implementations and a separately imported React UI as a foundation for the final OpenAPI-driven contract. - [x] Add Assets query authoring and dynamic field discovery as existing behavior to migrate into the Assets OpenAPI operation. - [x] Add synchronized visual and editable expression authoring. - [x] Unify Assets REST operations under `/rest/assets/*`. - [x] Define authenticated collection, item, content, upload, folder, query, field-catalog, and OpenAPI operations. - [x] Support Builder cookie authentication and public API token authentication. - [x] Add Builder preview, MCP/CLI operations, SSR execution, SSG path generation, and conditional activation. - [x] Add revision-keyed parsed metadata persistence for the hosted Builder. - [x] Add focused query, API, authorization, Builder, CLI, SSR, SSG, migration, and fixture coverage. ## Remaining implementation plan ### 1. Establish the content engine - [x] Rename the generic `@webstudio-is/asset-resource` implementation to `@webstudio-is/content-engine`. - [x] Move existing behavior instead of creating a parallel implementation. - [x] Keep the primary entry runtime-only and add only the compiler entry needed for bundle separation. - [x] Remove the old package after all consumers migrate. - [x] Ensure the engine has no Builder, SDK, asset-uploader, PostgreSQL, R2, or deployment-target dependency. ### 2. Define source snapshots and compilation plans - [x] Introduce `ContentSource.openSnapshot()` and a revision-bearing `ContentSourceSnapshot`. - [x] Include canonical path, stable document ID, content type, immutable content reference, size, and optional `createdAt`. - [x] Implement Webstudio Assets and filesystem snapshot guarantees using their adapter-specific mechanisms. - [x] Introduce `ContentCompilationPlan` for the union of reachable configured resource requirements. - [x] Separate statically selectable candidates from dynamic runtime predicates. - [x] Add mutation-during-compilation tests for replacement, deletion, folder movement, and filesystem changes. ### 3. Separate storage from content processing - [x] Keep upload, replacement, folders, byte storage, revisions, and generic metadata in the Assets repository/uploader. - [x] Move parsing, excerpts, field discovery, query execution, and artifact creation into the content engine. - [x] Remove content-index coordination from asset mutation business logic. - [x] Preserve original source `createdAt` across content replacement and expose it as optional queryable metadata. - [x] Keep hosted parsed-document persistence behind the content compiler's optional cache interface. ### 4. Build deterministic compilation - [x] Move the existing Markdown, frontmatter, JSON, canonicalization, hashing, document-building, and tests into the compiler. - [x] Keep frontmatter behind a dedicated module and continue using the existing Remark/Micromark pipeline. - [x] Parse only candidate files and only the tiers required by the compilation plan. - [x] Deduplicate documents and fields required by multiple resources. - [x] Define a versioned, deterministic `ContentArtifact` with integrity information. - [x] Keep verbose parse and omission diagnostics outside the runtime artifact. - [x] Reject incompatible artifact versions cleanly. ### 5. Enforce the deployment-wide database budget - [x] Replace the old asset-index limit with `CONTENT_DATABASE_MAX_BYTES`, defaulting to 500 KiB. - [x] Have applications read configuration and pass the byte limit into the compiler. - [x] Measure the exact serialized database, including selected bodies and overhead. - [x] Implement deterministic skip-and-continue packing of complete documents. - [x] Return compiler diagnostics with unbounded/bounded bytes, included/omitted counts, omitted identities, largest contributors, and affected resources. - [x] Expose only compact artifact statistics at runtime through `ContentDatabase.getStats()`. - [x] Add exact boundary, deterministic ordering, deduplication, and oversized-document tests. ### 6. Finalize the runtime database - [x] Define `ContentDatabase` over an immutable artifact. - [x] Preserve filtering, nested AND/OR groups, sorting, offset/limit, projection, field references, and content retrieval. - [x] Expose `getFieldCatalog()` and compact `getStats()`. - [x] Keep Markdown/YAML/compiler dependencies out of the runtime entry. - [x] Add execution guards and maximum-size runtime benchmarks. ### 7. Make query authoring OpenAPI-driven The selected standard OpenAPI operation is the complete query-authoring contract. The query builder resolves parameters and request-body JSON Schema, derives its internal editor model, and applies renderer-owned layout policy. It must not embed or fetch a second capabilities/UI language through `x-webstudio-query` or another side channel. - [x] Remove `x-webstudio-query` and all vendor query-definition metadata from the Assets OpenAPI operation. - [x] Resolve local OpenAPI references and derive the editable request value, defaults, validation, labels, choices, arrays, objects, and unions solely from standard operation parameters and request-body JSON Schema. - [x] Represent dynamic Assets field paths with standard JSON Schema vocabulary in the `queryAssets` request schema. - [x] Keep the normalized editor representation internal to query-builder; do not expose another provider-authored capabilities contract. - [x] Remove provider-controlled layout metadata such as `group`; make compact scalar packing and all other layout renderer policy. - [x] Keep Webstudio expression binding as a consumer/editor concern layered over schema values, not part of the OpenAPI contract. - [x] Preserve the existing Assets filter, sort, output selection, full-content/range selection, limit/offset, editable source, and configured-field behavior. - [x] Keep the Assets resource adapter responsible for serializing the edited query into its request body. - [x] Remove the obsolete Assets query-capabilities endpoint, URL, and UI-side capability parsing. - [x] Add a non-Assets OpenAPI proof with differently named parameters and no vendor metadata. - [x] Verify the generic packages contain no Assets imports, fixed Assets property contract, or provider-authored layout model. - [x] Update focused core, React, SDK, protocol/OpenAPI, Builder, and source round-trip tests. ### 8. Manage ContentDatabase lifecycle in Builder - [x] Cache databases by source revision plus normalized compilation-plan hash. - [x] Coalesce concurrent compilation requests. - [x] Reuse cached database instances across field discovery, autocomplete, query preview, and response preview when their normalized compilation plans match. - [x] Replace it when source revision or query requirements change. - [x] Preserve the optional revision-keyed hosted parsed-document cache without making it a source of truth. - [x] Show exact preview warnings when concrete values reveal missing bounded results. ### 9. Materialize and verify immutable builds - [x] Compile and serialize the exact deployment-wide database during prepublish validation. - [x] Attach that validated bounded database to the immutable build without rebuilding it from later asset state. - [x] Show non-blocking prepublish warnings with database bytes, omissions, and potentially affected resources. - [x] Avoid claiming exact future route matches when dynamic values are unknown. - [x] Include the database/runtime only when a reachable configured query requires them. - [x] Ensure detail resources requesting Markdown content include all statically eligible bodies needed by dynamic routes. ### 10. Integrate all targets - [x] SSR: load the build's bounded database and conditionally include the small runtime. - [x] SSG: query the same database during generation, then omit database/runtime output when no runtime query is needed. - [x] Local/CLI: compile from the filesystem source with the same semantics. - [x] Preserve overview pagination, alternative route fields, draft-as-user-data behavior, and full article rendering. - [x] Verify ordinary Assets and fresh query-free sites perform no parser, artifact, content-runtime dependency install, or cleanup work. ### 11. Correct API and package ownership - [x] Move external Assets REST/OpenAPI contracts to `@webstudio-is/protocol`. - [x] Keep Webstudio domain models and Assets system-resource configuration in the SDK. - [x] Keep both cookie and token authentication behavior on the unified Assets REST surface. - [x] Remove pass-through route wrappers/reexports that add no transport behavior. - [x] Keep published runtime queries local to the artifact rather than routing them through Builder REST. ### 12. Remove superseded deployment infrastructure - [x] Remove external deployment-index upload, fetch, invalidation, cleanup, and garbage-collection code. - [x] Remove the SaaS asset-origin adapter requirement; close webstudio-is/webstudio-saas#652 as superseded because it has no independent purpose. - [x] Remove index-bucket compatibility configuration and avoid adding Infisical variables. - [x] Remove obsolete external-index paths, package dependencies, routes, scripts, and tests. - [x] Consolidate unreleased intermediate migrations where safe and retain only schema required by the final hosted cache/API design. ### 13. Verification and release readiness - [x] Move and preserve existing tests while changing ownership; add new tests only for changed contracts and behavior. - [x] Run content-engine compiler/runtime tests, query-builder tests, SDK/protocol/API tests, Builder tests, CLI tests, and affected typechecks. - [x] Run formatting, generated-file idempotence, fixtures, clean Supabase-compatible migrations, E2E shards, and complete agent evaluations. - [x] Verify SSR and SSG conditional inclusion and query-free generated package manifests. - [x] Run the five-article MCP blog workflow: authoring, field discovery, overview, dynamic detail routes, full Markdown rendering, publishing, and published-site verification. - [x] Run a maximum-budget database test and a mutation-during-compilation test. - [x] Perform a final architecture, security, API compatibility, migration, bundle, correctness, and DRY review. - [x] Reconcile every checkbox and replace the verification section with results from the final architecture before release. ## Future MDX work MDX is intentionally separate from this release. - The compiler contract should allow a future build-only MDX parser. - MDX compilation must not enter the published runtime. - Embedded JSX must use an explicit compiled representation rather than runtime source evaluation. - Webstudio components, tokens, styles, and component allowlists remain Webstudio integration concerns. - The future MDX work should reuse `ContentArtifact` and `ContentDatabase` where practical. ## Superseded design The following earlier design choices are no longer the target architecture: - GROQ or GraphQL as the Assets query language. - One external or long-lived index object per resource/query. - A deployment index fetched by the Worker from SaaS/CDN infrastructure. - Lazy production reads of source Markdown bodies. - A separate index bucket, asset origin, invalidation service, or garbage collector. - A 4 MiB published index limit. - Rebuilding an old build's database from the current mutable asset set. - Bundling query infrastructure into projects without a configured content query. ## Verification baseline The embedded runtime-database architecture previously passed the checks below. The OpenAPI-driven query-authoring change additionally passed: - Query-builder core: 4 files / 19 tests. - Query-builder React: 1 file / 7 tests. - SDK Assets source/config: 1 file / 7 tests. - Protocol Assets OpenAPI: 1 file / 9 tests, including independent OpenAPI validation. - Builder Assets OpenAPI description: 1 file / 5 tests. - Query-builder, React, SDK, protocol, and Builder typechecks. - Package-boundary validation and formatting of every affected file. Local verification baseline: - Content engine: 15 files / 104 tests. - Asset uploader: 24 files / 175 tests. - Builder: 204 files / 2,423 tests. - CLI: 54 files / 866 tests. - Complete workspace typecheck, formatting, lint, package boundaries, generated fixtures, and fixture builds. - The complete migration history applied successfully to a fresh Supabase-compatible database, including every migration after the consolidated Assets migration. - Builder production build and full local E2E suite passed. - Complete agent evaluations passed: authenticated page (170,859 ms), design input (207,101 ms), and font assets (67,050 ms). - Final review additionally verified correct REST edit/view/build permits, cookie and public-token read paths, immutable snapshot retries, exact database budgeting, query-free dependency omission, and SSR/SSG conditional runtime inclusion. Final staging verification passed against the embedded runtime-database architecture: MCP uploaded and updated five Markdown articles, discovered dynamic frontmatter fields, previewed a five-item sorted overview and one full-body slug query, updated both Assets resources, found zero binding issues, rendered `/blog` and `/blog/dynamic-routes` at HTTP 200, published build version 266, and verified both published routes at HTTP 200 with the expected article metadata and Markdown body. The workflow also found and fixed offset-form asset timestamps failing canonical datetime validation and made MCP `preview.start({ port: 0 })` correctly select an available local port. ## Release constraints - No new SaaS content service, bucket, or environment variable. - No published-site database dependency. - No parser/compiler dependency in query-free sites or runtime-only bundles. - The validated content database is immutable build output. - Publishing warnings do not block deployment. - The selected standard OpenAPI operation and its request JSON Schema are the only source of query UI semantics.
89 lines
2.6 KiB
YAML
89 lines
2.6 KiB
YAML
name: Fixtures tests
|
|
|
|
on:
|
|
workflow_call:
|
|
inputs:
|
|
builder-host:
|
|
required: true
|
|
type: string
|
|
environment:
|
|
required: true
|
|
type: string
|
|
secrets:
|
|
PRIVATE_GITHUB_DEPLOY_TOKEN:
|
|
required: true
|
|
|
|
permissions:
|
|
contents: read # to fetch code (actions/checkout)
|
|
|
|
jobs:
|
|
checks:
|
|
timeout-minutes: 20
|
|
|
|
strategy:
|
|
matrix:
|
|
os: [ubuntu-latest, windows-latest]
|
|
|
|
runs-on: ${{ matrix.os }}
|
|
|
|
environment:
|
|
name: ${{ inputs.environment }}
|
|
|
|
env:
|
|
DATABASE_URL: postgres://
|
|
AUTH_SECRET: test
|
|
BUILDER_HOST: ${{ inputs.builder-host }}
|
|
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
ref: ${{ github.event.pull_request.head.sha || github.sha }}
|
|
# Test that everything is working with submodules
|
|
- uses: ./.github/actions/submodules-checkout
|
|
with:
|
|
submodules-ssh-key: ${{ secrets.PRIVATE_GITHUB_DEPLOY_TOKEN }}
|
|
|
|
- uses: ./.github/actions/ci-setup
|
|
|
|
# Testing fixtures for vercel template
|
|
- name: Test cli --help flag
|
|
working-directory: ./fixtures/webstudio-features
|
|
run: pnpm cli:local --help
|
|
|
|
- name: Testing cli link command
|
|
run: pnpm --filter='./fixtures/*' --sequential run fixtures:link
|
|
|
|
- name: Testing cli sync command
|
|
shell: bash
|
|
run: |
|
|
set +e
|
|
pnpm --filter='./fixtures/*' --sequential run fixtures:sync 2>&1 | tee fixtures-sync.log
|
|
sync_status=${PIPESTATUS[0]}
|
|
if [ "$sync_status" -ne 0 ]; then
|
|
message=$(tail -n 40 fixtures-sync.log | sed -e 's/%/%25/g' -e 's/\r/%0D/g' -e 's/$/%0A/' | tr -d '\n')
|
|
echo "::error title=Fixture sync failed::$message"
|
|
exit "$sync_status"
|
|
fi
|
|
|
|
- name: Build CLI content runtime
|
|
run: mkdir -p packages/cli/lib && pnpm --filter webstudio build:content-runtime
|
|
|
|
- name: Testing cli build command
|
|
run: pnpm --filter='./fixtures/*' run --parallel fixtures:build
|
|
|
|
- name: Prepare for diffing
|
|
shell: bash
|
|
run: |
|
|
find . -type f -path "./fixtures/*/.webstudio/data.json" -exec sed -i 's|"origin": ".*"|"origin": "https://main.development.webstudio.is"|g' {} +
|
|
|
|
- name: Test git diff
|
|
# This command will fail if there are uncommitted changes, i.e something has broken
|
|
run: git diff --name-only HEAD --exit-code
|
|
|
|
- name: Show changed files and diff
|
|
if: ${{ failure() }}
|
|
run: |
|
|
echo "Changed files are:"
|
|
git diff --name-only HEAD
|
|
git diff HEAD | head -n 1000
|