mirror of
https://github.com/webstudio-is/webstudio.git
synced 2026-11-03 02:27:57 +09:00
Oleg Isonen
db2d092268
## Summary Replace the defunct Lost Pixel service and the interim Chromatic workflow with a fast, self-hosted visual comparison pipeline. The workflow indexes the existing CSF stories without building Storybook, builds and serves a lightweight browser app with Vite, renders the merge base and current revision through Webstudio's shared browser tooling, and compares screenshots with Webstudio's pixel-diff and OCR engine. It uses GitHub Actions cache and artifacts only—no R2 bucket or external visual-testing service. Closes #6064 ## Implementation - [x] Remove the Lost Pixel workflow, configuration, acknowledgement, and global two-second wait. - [x] Remove the interim Chromatic workflow and settings. - [x] Extract source-agnostic capture, diff, OCR, audit, cache, and report tooling into `@webstudio-is/vision`; rename the Webstudio-specific preview boundary to `@webstudio-is/project-build/preview`. - [x] Expose the reusable browser capture session through the shared vision boundary. - [x] Make OCR and diff-artifact generation conditional on detected pixel changes. - [x] Render existing CSF stories through a typed, lightweight browser harness built with Vite. - [x] Replace the embedded HTML and handwritten HTTP/MIME server with a real HTML entry, Vite production build, and Vite preview server; isolate declarative build configuration in `story-build.ts`. - [x] Apply Storybook preview annotations, decorators, global CSS, backgrounds, and story metadata. - [x] Store story scopes as shared data and load each revision's own configuration. - [x] Discover changed, added, and removed stories from both revisions. - [x] Populate private story sources symmetrically in the baseline worktree. - [x] Isolate every story with a fresh React root and deterministic time, randomness, intervals, focus, animation, and layout readiness. - [x] Preserve story-owned visual options for masking, settling, and interval control. - [x] Reuse Vite preview servers, one Chromium process, grouped browser pages, production bundles, and a bounded CPU worker pool for image comparison. - [x] Recheck initial pixel differences with serial captures, then recapture persistent baseline/current pairs on the same browser page so renderer variance cannot create false failures. - [x] Report individual story render failures without discarding successful captures. - [x] Enforce a 0.1 per-pixel threshold and 0.001% total mismatch tolerance. - [x] Run OCR only for changed comparisons and include text-change evidence. - [x] Move the generic report model, writer, static viewer, and opener into the shared Vision package for reuse by CI and future CLI comparisons. - [x] Bundle report HTML, styling, and browser behavior as static Vision assets without runtime source-file lookups. - [x] Copy external screenshot inputs into self-contained report artifacts for CLI reuse. - [x] Handle cached removed-only comparisons without rendering the removed story on the current server. - [x] Support focused local reproduction and opening the report with one command. - [x] Pin CI to the lockfile's Playwright Chromium, cache complete revision screenshots with matching restore/save paths and automatically derived runtime invalidation, and render both revisions on a miss. - [x] Keep cache archives bounded to one revision instead of accumulating the cache history. - [x] Run the complete visual suite in one CI job and upload one report as a 14-day artifact. - [x] Fail PR checks on visual differences, empty runs, missing screenshots, missing or empty report artifacts, and infrastructure failures. - [x] Rerun fork checks when `safe-to-deploy` is added. - [x] Allow reviewed visual-only differences through `visual-change-approved` and invalidate approval after new commits. - [x] Keep capture, pixel diff, OCR, and report generation source-agnostic for future draft-versus-published project comparisons. - [x] Use Vite for the complete story HTML, module, asset, production-build, and preview-server lifecycle; inline the selected stories and skip ineffective tree-shaking to keep the cached run under one minute. - [x] Review Builder product documentation impact: no user-facing Builder documentation is affected; contributor usage is documented in `scripts/visual-regression/README.md`. - [x] Review fixture impact: no fixture inputs or generated fixture outputs are affected. ## Verification - [x] Project-build Vision tests, including the reusable report test, and entrypoint contract tests pass. - [x] CLI screenshot, preview, and browser-session tests pass. - [x] Project-build, CLI, Builder, design-system, and visual-runner typechecks pass. - [x] Visual runner tests pass (10), and the complete Vision package passes (79), including demonstrated fail-before/pass-after coverage. - [x] Full Storybook production build passes. - [x] Full local Vite run completes all 238 stories in 69.9s cold and 50.3s with a cached baseline plus two same-page serial verifications, with zero differences; identical-image comparison remains sub-second. - [x] The report writer passes a bundled-output smoke test and materializes external screenshot assets. - [x] The generated report validates all 238 entries and referenced images. - [x] Hosted cached-baseline comparison completes all 238 stories in 52.9s with zero differences; the cache-miss fallback completes in 101.3s. - [x] Cache-miss fallback renders both Git revisions on demand. - [x] Visual differences produce baseline, current, diff, OCR, JSON, and HTML evidence and fail without approval. - [x] Hosted single-job visual workflow restores both the pinned browser and baseline cache, validates all 238 public and private stories with zero differences, and uploads the report. - [x] The final Vite production build renders all 238 stories without dependency invalidation or duplicate React runtimes, and restores `NODE_ENV` so cached serial fallback installs the baseline's dev dependencies. - [x] Root lint, package-boundary validation, and relevant formatting checks pass. - [x] Workflow YAML parses successfully. - [x] `git diff --check`, staged-diff review, and final worktree review pass. ## Usage Run the complete local comparison and open the visual report: ```sh pnpm visual-regression:ui ``` Focus one component while iterating: ```sh pnpm visual-regression:ui -- --grep "Design system/Button" ``` Reopen the latest report without rerunning: ```sh pnpm visual-regression:report ``` The CI report is available as the `visual-regression-report` artifact. ## Tradeoffs - Baseline screenshots are a disposable acceleration cache, not source of truth. A missing or invalid cache renders the merge base directly from Git. - GitHub artifact storage provides downloadable one-click reports; permanently hosted browser links would require another hosting surface. - OCR runs after pixel triage because invoking Tesseract for every unchanged story would make the common path unnecessarily slow. - Vite production builds add Rollup's module graph and HTML/asset pipeline. The runner emits one bundle because every selected story is rendered, and skips tree-shaking because it removed little while increasing build time. - The verified local path is 50.3s cached and 69.9s cold. The hosted path is 52.9s cached and 101.3s cold. A new browser/runtime cache must render its baseline once; the pinned browser keeps later runners on the reusable path. - The complete suite stays in one Actions job; browser-page concurrency is internal and bounded.
104 lines
6.1 KiB
JSON
104 lines
6.1 KiB
JSON
{
|
|
"packageManager": "pnpm@9.14.4",
|
|
"name": "webstudio-root",
|
|
"version": "0.0.0",
|
|
"private": true,
|
|
"type": "module",
|
|
"scripts": {
|
|
"build": "pnpm -r --filter='!./fixtures/*' build",
|
|
"webstudio": "node packages/cli/local.js",
|
|
"dts": "pnpm -r dts",
|
|
"dev": "bash ./apps/builder/dev/run-local.sh",
|
|
"evaluations": "pnpm --filter webstudio evaluations",
|
|
"lint": "oxlint --deny-warnings",
|
|
"check:package-boundaries": "node scripts/check-package-boundaries.mjs",
|
|
"check:generated-api": "pnpm --filter @webstudio-is/project-build generate-runtime-operation-contracts && pnpm --filter @webstudio-is/builder generate-server-only-api-metadata && node scripts/check-clean-files.mjs packages/project-build/src/contracts/__generated__/runtime-operation-contracts.ts packages/protocol/src/builder-api/__generated__/server-only-router-operation-metadata.ts",
|
|
"check:generated-docs": "pnpm --filter @webstudio-is/project-build generate-docs && pnpm --filter webstudio generate-docs && pnpm docs:mcp:test && pnpm docs:mcp:typecheck && pnpm docs:content-engine:test && pnpm docs:mcp:check && pnpm docs:content-engine:check && node scripts/check-clean-files.mjs packages/project-build/src/docs.generated.ts packages/cli/src/docs.generated.ts",
|
|
"docs:mcp:sync": "tsx scripts/docs/generate-mcp-docs.ts",
|
|
"docs:mcp:check": "pnpm docs:mcp:sync && node scripts/check-clean-files.mjs docs/university/mcp.md",
|
|
"docs:mcp:test": "tsx --test scripts/docs/generate-mcp-docs.test.ts",
|
|
"docs:mcp:typecheck": "tsc --noEmit -p scripts/docs/tsconfig.json",
|
|
"docs:content-engine:sync": "tsx scripts/docs/generate-content-engine-docs.ts",
|
|
"docs:content-engine:check": "pnpm docs:content-engine:sync && node scripts/check-clean-files.mjs docs/university/foundations/content-engine-reference.md",
|
|
"docs:content-engine:test": "tsx --test scripts/docs/generate-content-engine-docs.test.ts && pnpm --filter webstudio exec vitest run src/content-engine-reference.test.ts",
|
|
"fixtures": "pnpm -r fixtures:link && pnpm -r fixtures:sync && pnpm -r fixtures:build",
|
|
"checks": "pnpm -r test && pnpm -r typecheck && pnpm lint && pnpm check:package-boundaries && pnpm check:generated-api && pnpm check:generated-docs && pnpm fixtures",
|
|
"e2e:builder": "bash ./apps/builder/e2e/run-local.sh",
|
|
"e2e:builder:build": "pnpm --filter=@webstudio-is/builder build",
|
|
"e2e:builder:dev:backend": "E2E_SKIP_CLEANUP=true E2E_RUN_TESTS=false E2E_START_POSTGREST=true E2E_DB_BOOTSTRAP=if-empty E2E_INSTALL_PLAYWRIGHT=false bash ./apps/builder/e2e/run-local.sh",
|
|
"e2e:builder:dev": "E2E_SKIP_CLEANUP=true E2E_DB_BOOTSTRAP=if-empty E2E_INSTALL_PLAYWRIGHT=false E2E_BUILDER_URL=${E2E_BUILDER_URL:-https://127.0.0.1:3000} bash ./apps/builder/e2e/run-local.sh",
|
|
"e2e:builder:refresh-schema-cache": "E2E_DB_BOOTSTRAP=migrations E2E_WRITE_SCHEMA_SNAPSHOT=true E2E_RUN_TESTS=false bash ./apps/builder/e2e/run-local.sh",
|
|
"playwright": "pnpm -r playwright-init",
|
|
"format": "oxfmt",
|
|
"storybook:dev": "storybook dev -p 6006",
|
|
"storybook:build": "storybook build",
|
|
"visual-regression": "tsx --conditions=webstudio scripts/visual-regression/run.ts",
|
|
"visual-regression:ui": "tsx --conditions=webstudio scripts/visual-regression/run.ts --open-report",
|
|
"visual-regression:report": "tsx scripts/visual-regression/open-report.ts",
|
|
"visual-regression:test": "tsx --test scripts/visual-regression/*.test.ts",
|
|
"visual-regression:typecheck": "tsc -p scripts/visual-regression/tsconfig.json",
|
|
"migrations": "pnpm --filter=@webstudio-is/prisma-client generate && pnpm --filter=./packages/prisma-client migrations --dev --cwd ../../apps/builder",
|
|
"prepare": "which git && git config core.hooksPath .git/hooks/ && simple-git-hooks || echo git not installed",
|
|
"local:version-snapshot": "pnpm -r exec pnpm version prepatch --preid $(cat /dev/urandom | LC_ALL=C tr -dc 'a-z' | fold -w 8 | head -n 1)",
|
|
"local:publish-snapshot": "pnpm -r publish --access public --no-git-checks --registry http://localhost:4873",
|
|
"local:dangerously-undo-version-snapshot": "git restore --source=$(git branch --show-current) '**/*/package.json'",
|
|
"local:release": "pnpm build && pnpm dts && pnpm local:version-snapshot && pnpm local:publish-snapshot && pnpm local:dangerously-undo-version-snapshot && echo \"now execute\npnpm up -r -L '@webstudio-is/*' --registry http://localhost:4873\"",
|
|
"clean:deep": "bash ./cleanup.sh"
|
|
},
|
|
"simple-git-hooks": {
|
|
"pre-commit": "./node_modules/.bin/nano-staged"
|
|
},
|
|
"devDependencies": {
|
|
"@fontsource-variable/inter": "^5.0.20",
|
|
"@fontsource-variable/manrope": "^5.0.20",
|
|
"@fontsource/roboto-mono": "^5.0.18",
|
|
"@modelcontextprotocol/sdk": "^1.10.0",
|
|
"@radix-ui/react-tooltip": "^1.2.4",
|
|
"@storybook/addon-actions": "^8.6.4",
|
|
"@storybook/addon-backgrounds": "^8.6.4",
|
|
"@storybook/addon-controls": "^8.6.4",
|
|
"@storybook/react": "^8.6.4",
|
|
"@storybook/react-vite": "^8.6.4",
|
|
"@types/node": "^22.13.10",
|
|
"@types/react": "^18.2.70",
|
|
"@types/react-dom": "^18.2.25",
|
|
"@webstudio-is/project-build": "workspace:^",
|
|
"@webstudio-is/vision": "workspace:^",
|
|
"esbuild": "^0.25.3",
|
|
"nano-staged": "^0.8.0",
|
|
"oxfmt": "0.58.0",
|
|
"oxlint": "^1.73.0",
|
|
"parse5": "7.3.0",
|
|
"react": "18.3.0-canary-14898b6a9-20240318",
|
|
"react-dom": "18.3.0-canary-14898b6a9-20240318",
|
|
"simple-git-hooks": "^2.11.1",
|
|
"storybook": "^8.6.4",
|
|
"tsx": "^4.19.3",
|
|
"typescript": "7.0.2",
|
|
"webstudio": "workspace:^",
|
|
"vite": "^6.3.4"
|
|
},
|
|
"engines": {
|
|
"node": "22",
|
|
"pnpm": "^9.14.0",
|
|
"yarn": "This project is configured to use pnpm"
|
|
},
|
|
"nano-staged": {
|
|
"*.{ts,tsx,js,json,css,md}": "oxfmt"
|
|
},
|
|
"pnpm": {
|
|
"overrides": {
|
|
"@types/react": "^18.2.70",
|
|
"@types/react-dom": "^18.2.25",
|
|
"react": "18.3.0-canary-14898b6a9-20240318",
|
|
"react-dom": "18.3.0-canary-14898b6a9-20240318",
|
|
"vite@<7": "6.3.4"
|
|
},
|
|
"patchedDependencies": {
|
|
"@stitches/react@1.3.1-1": "patches/@stitches__react@1.3.1-1.patch",
|
|
"@radix-ui/react-scroll-area@1.0.5": "patches/@radix-ui__react-scroll-area@1.0.5.patch",
|
|
"@remix-run/dev": "patches/@remix-run__dev.patch"
|
|
}
|
|
}
|
|
}
|