mirror of
https://github.com/webstudio-is/webstudio.git
synced 2026-11-06 03:58:34 +09:00
Oleg Isonen
56a22fde8a
## Summary Introduce a document graph architecture for Assets resources so structured documents can reference one another while keeping full document payloads on CDN-backed storage. The content database will retain the searchable index and dependency graph. At runtime, the server will select a root document, determine its complete dependency set before performing I/O, fetch each unique document concurrently, and assemble the final result server-side. ## Implementation todos ### Document graph foundation - [x] Add normalized internal reference contracts for whole documents, JSON paths, Markdown body, and Markdown frontmatter. - [x] Add an immutable graph abstraction with deterministic ordering, dependency traversal, revision validation, and cycle detection. - [x] Add a storage-independent resolver with dependency closure, bounded parallel loading, shared-content deduplication, representation selection, assembly hooks, cancellation, and structured errors. - [x] Add focused unit coverage for the isolated document-graph abstraction. - [x] Define and validate the external remote-document reference syntax. - [x] Add a pure graph compiler that resolves canonical document URLs into IDs and revisions and validates missing, duplicate, and cyclic dependencies. ### Build-time graph - [x] Add a storage-independent source-catalog compiler for bounded parallel reference discovery and deterministic graph compilation. - [x] Add a versioned deterministic document graph artifact with canonical serialization and integrity verification. - [x] Discover and normalize statically declared references while Assets resources are processed. - [x] Resolve document identities and revisions and report missing, stale, or cyclic dependencies with document context. - [x] Persist document nodes and dependency edges alongside searchable resource metadata. - [x] Include persisted graph data in artifact integrity and size validation. ### Document representations - [x] Add a standalone JSON document adapter for exact reference markers, compiler occurrences, JSON path selection, and immutable assembly. - [x] Add a standalone Markdown document adapter for bounded decoding, frontmatter references, immutable assembly, and source, body, and frontmatter selection. - [x] Parse fetched JSON and Markdown documents through format-specific adapters. - [x] Select complete documents, JSON fragments, Markdown body, and Markdown frontmatter through the shared reference model. - [x] Assemble nested and shared references into the final server-side value. ### Runtime resolution - [x] Enforce reachable-document, per-document byte, and aggregate-byte limits before parsing and assembly. - [x] Add a revision-keyed bounded source-cache wrapper with replayable bytes, stale-entry fallback, and in-flight deduplication. - [x] Add a storage-neutral loaded-source contract that rejects revision mismatches before parsing. - [x] Resolve graph-backed property paths before filtering, sorting, pagination, and projection; skip graph loading for unrelated queries. - [x] Add an injected HTTP/CDN document loader with revision metadata validation. - [x] Apply runtime payload limits, bounded concurrency, caching, and structured failure mapping. - [x] Ensure shared documents are fetched and parsed once per resolution. ### Product integration - [x] Integrate equivalent behavior in local preview, SSR, SSG, and published runtime paths. - [x] Preserve existing embedded Assets resource behavior without requiring migration. - [x] Add observability for graph loading, document fetching, caching, and resolution failures. - [x] Document the public reference and resolution behavior. ### Review hardening - [x] Enforce graph, artifact, cache, and published-storage identity consistency. - [x] Compile only graph sources reachable from selected build entries. - [x] Resolve graph-backed fields before query evaluation and avoid unnecessary graph loading. - [x] Consolidate duplicated graph schemas, utilities, and document content-type detection. - [x] Correct runtime documentation and use structural evaluation validation. ### Verification - [x] Keep a deterministic high-impact agent evaluation for Markdown-to-JSON references across the dynamic blog overview and article routes. - [x] Add a separate discoverability evaluation that must learn the reference workflow from MCP guidance without prompt-supplied syntax, queries, bindings, or call order. - [ ] Re-run the complete local agent evaluation suite after the discoverability fixture change and compare every fixture with its accepted baseline. - [x] Add integration coverage for CDN-backed hydration, parallel fetching, caching, and shared dependencies. - [x] Add integration coverage for missing references, cycles, revision changes, limits, and cancellation. - [x] Verify equivalent behavior across preview, SSR, SSG, and published deployments. - [x] Verify that existing embedded Assets resources retain their current behavior. ## Intended outcome - Storage-backed documents do not require their complete payloads to be embedded in the content database. - The server knows the complete statically discoverable dependency set before fetching document content. - Shared dependencies are fetched once per resolution and independent resources can be fetched in parallel. - JSON fragments and Markdown body/frontmatter representations use the same document-resolution system. - Reference and payload revisions cannot be mixed silently. - Existing Assets resources continue to work without migration. ## Verification results - `@webstudio-is/content-engine`: 37 files / 244 tests passed; typecheck passed. - `@webstudio-is/asset-uploader`: 22 files / 182 tests passed; typecheck passed. - CLI: 57 files / 925 tests passed, including generated SSR and SSG build/prerender coverage; typecheck passed. - Repository-wide `pnpm oxfmt --check` passed across 2,757 files. - Package-boundary validation passed. - Previous `pnpm evaluations` run: all 4 original high-impact fixtures passed. The complete 5-fixture rerun is pending explicit approval after adding the discoverability fixture.
78 lines
2.5 KiB
JSON
78 lines
2.5 KiB
JSON
{
|
|
"name": "@webstudio-is/content-engine",
|
|
"version": "0.0.0-webstudio-version",
|
|
"description": "Compile file-based content into a portable query runtime",
|
|
"author": "Webstudio <github@webstudio.is>",
|
|
"homepage": "https://webstudio.is",
|
|
"license": "AGPL-3.0-or-later",
|
|
"type": "module",
|
|
"sideEffects": false,
|
|
"scripts": {
|
|
"benchmark:index": "NODE_OPTIONS=--conditions=webstudio pnpm tsx scripts/benchmark-index.mjs",
|
|
"benchmark:scale": "NODE_OPTIONS=--conditions=webstudio pnpm tsx scripts/benchmark-scale.mjs",
|
|
"build": "rm -rf lib && esbuild src/index.ts src/compiler.ts src/limits.ts src/markdown-assets.ts src/markdown-references.ts src/published-runtime.ts --outdir=lib --bundle --format=esm --packages=external",
|
|
"dts": "tsc --project tsconfig.dts.json",
|
|
"test": "vitest run",
|
|
"typecheck": "tsc --noEmit"
|
|
},
|
|
"dependencies": {
|
|
"@hyperjump/json-pointer": "1.1.2",
|
|
"@webstudio-is/expression": "workspace:*",
|
|
"@webstudio-is/query-builder": "workspace:*",
|
|
"canonicalize": "3.0.0",
|
|
"micromark": "4.0.2",
|
|
"micromark-util-decode-string": "2.0.1",
|
|
"remark-parse": "10.0.2",
|
|
"unified": "10.1.2",
|
|
"yaml": "2.9.0",
|
|
"zod": "^4.4.3"
|
|
},
|
|
"devDependencies": {
|
|
"@webstudio-is/tsconfig": "workspace:*",
|
|
"esbuild": "^0.25.3",
|
|
"vitest": "^3.1.2"
|
|
},
|
|
"exports": {
|
|
".": {
|
|
"webstudio": "./src/index.ts",
|
|
"types": "./lib/types/index.d.ts",
|
|
"import": "./lib/index.js",
|
|
"default": "./src/index.ts"
|
|
},
|
|
"./compiler": {
|
|
"webstudio": "./src/compiler.ts",
|
|
"types": "./lib/types/compiler.d.ts",
|
|
"import": "./lib/compiler.js",
|
|
"default": "./src/compiler.ts"
|
|
},
|
|
"./runtime": {
|
|
"webstudio": "./src/published-runtime.ts",
|
|
"types": "./lib/types/published-runtime.d.ts",
|
|
"import": "./lib/published-runtime.js",
|
|
"default": "./src/published-runtime.ts"
|
|
},
|
|
"./limits": {
|
|
"webstudio": "./src/limits.ts",
|
|
"types": "./lib/types/limits.d.ts",
|
|
"import": "./lib/limits.js",
|
|
"default": "./src/limits.ts"
|
|
},
|
|
"./markdown-assets": {
|
|
"webstudio": "./src/markdown-assets.ts",
|
|
"types": "./lib/types/markdown-assets.d.ts",
|
|
"import": "./lib/markdown-assets.js",
|
|
"default": "./src/markdown-assets.ts"
|
|
},
|
|
"./markdown-references": {
|
|
"webstudio": "./src/markdown-references.ts",
|
|
"types": "./lib/types/markdown-references.d.ts",
|
|
"import": "./lib/markdown-references.js",
|
|
"default": "./src/markdown-references.ts"
|
|
}
|
|
},
|
|
"files": [
|
|
"lib/**/*",
|
|
"!*.test.*"
|
|
]
|
|
}
|