mirror of
https://github.com/webstudio-is/webstudio.git
synced 2026-11-05 19:49:57 +09:00
Oleg Isonen
56a22fde8a
## Summary Introduce a document graph architecture for Assets resources so structured documents can reference one another while keeping full document payloads on CDN-backed storage. The content database will retain the searchable index and dependency graph. At runtime, the server will select a root document, determine its complete dependency set before performing I/O, fetch each unique document concurrently, and assemble the final result server-side. ## Implementation todos ### Document graph foundation - [x] Add normalized internal reference contracts for whole documents, JSON paths, Markdown body, and Markdown frontmatter. - [x] Add an immutable graph abstraction with deterministic ordering, dependency traversal, revision validation, and cycle detection. - [x] Add a storage-independent resolver with dependency closure, bounded parallel loading, shared-content deduplication, representation selection, assembly hooks, cancellation, and structured errors. - [x] Add focused unit coverage for the isolated document-graph abstraction. - [x] Define and validate the external remote-document reference syntax. - [x] Add a pure graph compiler that resolves canonical document URLs into IDs and revisions and validates missing, duplicate, and cyclic dependencies. ### Build-time graph - [x] Add a storage-independent source-catalog compiler for bounded parallel reference discovery and deterministic graph compilation. - [x] Add a versioned deterministic document graph artifact with canonical serialization and integrity verification. - [x] Discover and normalize statically declared references while Assets resources are processed. - [x] Resolve document identities and revisions and report missing, stale, or cyclic dependencies with document context. - [x] Persist document nodes and dependency edges alongside searchable resource metadata. - [x] Include persisted graph data in artifact integrity and size validation. ### Document representations - [x] Add a standalone JSON document adapter for exact reference markers, compiler occurrences, JSON path selection, and immutable assembly. - [x] Add a standalone Markdown document adapter for bounded decoding, frontmatter references, immutable assembly, and source, body, and frontmatter selection. - [x] Parse fetched JSON and Markdown documents through format-specific adapters. - [x] Select complete documents, JSON fragments, Markdown body, and Markdown frontmatter through the shared reference model. - [x] Assemble nested and shared references into the final server-side value. ### Runtime resolution - [x] Enforce reachable-document, per-document byte, and aggregate-byte limits before parsing and assembly. - [x] Add a revision-keyed bounded source-cache wrapper with replayable bytes, stale-entry fallback, and in-flight deduplication. - [x] Add a storage-neutral loaded-source contract that rejects revision mismatches before parsing. - [x] Resolve graph-backed property paths before filtering, sorting, pagination, and projection; skip graph loading for unrelated queries. - [x] Add an injected HTTP/CDN document loader with revision metadata validation. - [x] Apply runtime payload limits, bounded concurrency, caching, and structured failure mapping. - [x] Ensure shared documents are fetched and parsed once per resolution. ### Product integration - [x] Integrate equivalent behavior in local preview, SSR, SSG, and published runtime paths. - [x] Preserve existing embedded Assets resource behavior without requiring migration. - [x] Add observability for graph loading, document fetching, caching, and resolution failures. - [x] Document the public reference and resolution behavior. ### Review hardening - [x] Enforce graph, artifact, cache, and published-storage identity consistency. - [x] Compile only graph sources reachable from selected build entries. - [x] Resolve graph-backed fields before query evaluation and avoid unnecessary graph loading. - [x] Consolidate duplicated graph schemas, utilities, and document content-type detection. - [x] Correct runtime documentation and use structural evaluation validation. ### Verification - [x] Keep a deterministic high-impact agent evaluation for Markdown-to-JSON references across the dynamic blog overview and article routes. - [x] Add a separate discoverability evaluation that must learn the reference workflow from MCP guidance without prompt-supplied syntax, queries, bindings, or call order. - [ ] Re-run the complete local agent evaluation suite after the discoverability fixture change and compare every fixture with its accepted baseline. - [x] Add integration coverage for CDN-backed hydration, parallel fetching, caching, and shared dependencies. - [x] Add integration coverage for missing references, cycles, revision changes, limits, and cancellation. - [x] Verify equivalent behavior across preview, SSR, SSG, and published deployments. - [x] Verify that existing embedded Assets resources retain their current behavior. ## Intended outcome - Storage-backed documents do not require their complete payloads to be embedded in the content database. - The server knows the complete statically discoverable dependency set before fetching document content. - Shared dependencies are fetched once per resolution and independent resources can be fetched in parallel. - JSON fragments and Markdown body/frontmatter representations use the same document-resolution system. - Reference and payload revisions cannot be mixed silently. - Existing Assets resources continue to work without migration. ## Verification results - `@webstudio-is/content-engine`: 37 files / 244 tests passed; typecheck passed. - `@webstudio-is/asset-uploader`: 22 files / 182 tests passed; typecheck passed. - CLI: 57 files / 925 tests passed, including generated SSR and SSG build/prerender coverage; typecheck passed. - Repository-wide `pnpm oxfmt --check` passed across 2,757 files. - Package-boundary validation passed. - Previous `pnpm evaluations` run: all 4 original high-impact fixtures passed. The complete 5-fixture rerun is pending explicit approval after adding the discoverability fixture.
Webstudio Fixtures
Fixtures are test projects that ensure the CLI and build process work correctly across different deployment targets.
Updating Fixtures After Changes
# From repository root - updates all fixtures
pnpm fixtures
This command will:
- Link all fixtures to their respective projects
- Sync the latest data from the development environment
- Build all fixtures with the latest code
Working with Individual Fixtures
# Go to fixture folder
cd fixtures/webstudio-features
# Link to project
pnpm fixtures:link
# Sync latest data
pnpm fixtures:sync
# Build fixture
pnpm fixtures:build
# Start it
pnpm run dev
Adding Assets to Fixtures
To add a new asset (image, video, audio, etc.) to a fixture:
-
Publish to staging
- Make your changes in the builder
- Publish to the staging environment
-
Get the build ID from CI
- After publishing, check the CI logs
- Find the build ID that was created
-
Update the fixture's build ID
- Edit
fixtures/[fixture-name]/package.json - Update the
--buildIdparameter in thefixtures:syncscript
"fixtures:sync": "pnpm cli sync --buildId NEW_BUILD_ID_HERE && pnpm oxfmt ./.webstudio/" - Edit
-
Sync with the correct environment
# Go to fixture folder cd fixtures/webstudio-features # Sync from staging environment (replace with your branch) BUILDER_HOST=fix-assets.staging.webstudio.is pnpm run fixtures -
Fix the origin URL
- After syncing, edit
.webstudio/data.json - Change the origin back to:
"origin": "https://main.development.webstudio.is"- This is necessary because sync changes it to the current branch
- After syncing, edit
-
Commit the changes
git add fixtures/ git commit -m "chore: update fixtures with new assets" git push