admin/webstudio
1
0 Fork 0
mirror of https://github.com/webstudio-is/webstudio.git synced 2026-11-04 02:58:23 +09:00
Code Issues Packages Projects Releases Wiki Activity
Files
agent/validate-css-before-publish
webstudio/packages/cli/package.json
T
Oleg Isonen db2d092268 ci: replace Lost Pixel with self-hosted visual regression (#6086) 
## Summary

Replace the defunct Lost Pixel service and the interim Chromatic
workflow with a fast, self-hosted visual comparison pipeline.

The workflow indexes the existing CSF stories without building
Storybook, builds and serves a lightweight browser app with Vite,
renders the merge base and current revision through Webstudio's shared
browser tooling, and compares screenshots with Webstudio's pixel-diff
and OCR engine. It uses GitHub Actions cache and artifacts only—no R2
bucket or external visual-testing service.

Closes #6064

## Implementation

- [x] Remove the Lost Pixel workflow, configuration, acknowledgement,
and global two-second wait.
- [x] Remove the interim Chromatic workflow and settings.
- [x] Extract source-agnostic capture, diff, OCR, audit, cache, and
report tooling into `@webstudio-is/vision`; rename the
Webstudio-specific preview boundary to
`@webstudio-is/project-build/preview`.
- [x] Expose the reusable browser capture session through the shared
vision boundary.
- [x] Make OCR and diff-artifact generation conditional on detected
pixel changes.
- [x] Render existing CSF stories through a typed, lightweight browser
harness built with Vite.
- [x] Replace the embedded HTML and handwritten HTTP/MIME server with a
real HTML entry, Vite production build, and Vite preview server; isolate
declarative build configuration in `story-build.ts`.
- [x] Apply Storybook preview annotations, decorators, global CSS,
backgrounds, and story metadata.
- [x] Store story scopes as shared data and load each revision's own
configuration.
- [x] Discover changed, added, and removed stories from both revisions.
- [x] Populate private story sources symmetrically in the baseline
worktree.
- [x] Isolate every story with a fresh React root and deterministic
time, randomness, intervals, focus, animation, and layout readiness.
- [x] Preserve story-owned visual options for masking, settling, and
interval control.
- [x] Reuse Vite preview servers, one Chromium process, grouped browser
pages, production bundles, and a bounded CPU worker pool for image
comparison.
- [x] Recheck initial pixel differences with serial captures, then
recapture persistent baseline/current pairs on the same browser page so
renderer variance cannot create false failures.
- [x] Report individual story render failures without discarding
successful captures.
- [x] Enforce a 0.1 per-pixel threshold and 0.001% total mismatch
tolerance.
- [x] Run OCR only for changed comparisons and include text-change
evidence.
- [x] Move the generic report model, writer, static viewer, and opener
into the shared Vision package for reuse by CI and future CLI
comparisons.
- [x] Bundle report HTML, styling, and browser behavior as static Vision
assets without runtime source-file lookups.
- [x] Copy external screenshot inputs into self-contained report
artifacts for CLI reuse.
- [x] Handle cached removed-only comparisons without rendering the
removed story on the current server.
- [x] Support focused local reproduction and opening the report with one
command.
- [x] Pin CI to the lockfile's Playwright Chromium, cache complete
revision screenshots with matching restore/save paths and automatically
derived runtime invalidation, and render both revisions on a miss.
- [x] Keep cache archives bounded to one revision instead of
accumulating the cache history.
- [x] Run the complete visual suite in one CI job and upload one report
as a 14-day artifact.
- [x] Fail PR checks on visual differences, empty runs, missing
screenshots, missing or empty report artifacts, and infrastructure
failures.
- [x] Rerun fork checks when `safe-to-deploy` is added.
- [x] Allow reviewed visual-only differences through
`visual-change-approved` and invalidate approval after new commits.
- [x] Keep capture, pixel diff, OCR, and report generation
source-agnostic for future draft-versus-published project comparisons.
- [x] Use Vite for the complete story HTML, module, asset,
production-build, and preview-server lifecycle; inline the selected
stories and skip ineffective tree-shaking to keep the cached run under
one minute.
- [x] Review Builder product documentation impact: no user-facing
Builder documentation is affected; contributor usage is documented in
`scripts/visual-regression/README.md`.
- [x] Review fixture impact: no fixture inputs or generated fixture
outputs are affected.

## Verification

- [x] Project-build Vision tests, including the reusable report test,
and entrypoint contract tests pass.
- [x] CLI screenshot, preview, and browser-session tests pass.
- [x] Project-build, CLI, Builder, design-system, and visual-runner
typechecks pass.
- [x] Visual runner tests pass (10), and the complete Vision package
passes (79), including demonstrated fail-before/pass-after coverage.
- [x] Full Storybook production build passes.
- [x] Full local Vite run completes all 238 stories in 69.9s cold and
50.3s with a cached baseline plus two same-page serial verifications,
with zero differences; identical-image comparison remains sub-second.
- [x] The report writer passes a bundled-output smoke test and
materializes external screenshot assets.
- [x] The generated report validates all 238 entries and referenced
images.
- [x] Hosted cached-baseline comparison completes all 238 stories in
52.9s with zero differences; the cache-miss fallback completes in
101.3s.
- [x] Cache-miss fallback renders both Git revisions on demand.
- [x] Visual differences produce baseline, current, diff, OCR, JSON, and
HTML evidence and fail without approval.
- [x] Hosted single-job visual workflow restores both the pinned browser
and baseline cache, validates all 238 public and private stories with
zero differences, and uploads the report.
- [x] The final Vite production build renders all 238 stories without
dependency invalidation or duplicate React runtimes, and restores
`NODE_ENV` so cached serial fallback installs the baseline's dev
dependencies.
- [x] Root lint, package-boundary validation, and relevant formatting
checks pass.
- [x] Workflow YAML parses successfully.
- [x] `git diff --check`, staged-diff review, and final worktree review
pass.

## Usage

Run the complete local comparison and open the visual report:

```sh
pnpm visual-regression:ui
```

Focus one component while iterating:

```sh
pnpm visual-regression:ui -- --grep "Design system/Button"
```

Reopen the latest report without rerunning:

```sh
pnpm visual-regression:report
```

The CI report is available as the `visual-regression-report` artifact.

## Tradeoffs

- Baseline screenshots are a disposable acceleration cache, not source
of truth. A missing or invalid cache renders the merge base directly
from Git.
- GitHub artifact storage provides downloadable one-click reports;
permanently hosted browser links would require another hosting surface.
- OCR runs after pixel triage because invoking Tesseract for every
unchanged story would make the common path unnecessarily slow.
- Vite production builds add Rollup's module graph and HTML/asset
pipeline. The runner emits one bundle because every selected story is
rendered, and skips tree-shaking because it removed little while
increasing build time.
- The verified local path is 50.3s cached and 69.9s cold. The hosted
path is 52.9s cached and 101.3s cold. A new browser/runtime cache must
render its baseline once; the pinned browser keeps later runners on the
reusable path.
- The complete suite stays in one Actions job; browser-page concurrency
is internal and bounded.
2026-08-12 10:55:52 +02:00

135 lines
4.8 KiB
JSON
Raw Permalink Blame History

{
"name": "webstudio",
"version": "0.0.0-webstudio-version",
"description": "Webstudio CLI",
"author": "Webstudio <github@webstudio.is>",
"homepage": "https://webstudio.is",
"type": "module",
"bin": {
"webstudio-cli": "./bin.js",
"webstudio": "./bin.js"
},
"exports": {
"./vision": {
"webstudio": "./src/vision.ts"
}
},
"files": [
"lib/*",
"templates/*",
"bin.js",
"!*.{test,stories}.*"
],
"scripts": {
"typecheck": "tsc --noEmit",
"generate-docs": "tsx scripts/generate-docs.ts && oxfmt src/docs.generated.ts",
"build:content-runtime": "esbuild @webstudio-is/content-engine/runtime --bundle --format=esm --platform=browser --target=es2022 --conditions=webstudio --minify --outfile=lib/content-runtime.js",
"fixtures:build": "pnpm build:content-runtime",
"build": "pnpm generate-docs && rm -rf lib && vite build && pnpm build:content-runtime",
"pretest": "mkdir -p lib && pnpm build:content-runtime",
"evaluations": "tsx --import ../../scripts/register-react-global.ts --conditions=webstudio evaluations/high-impact/run-local-agent.ts",
"test": "vitest run",
"test:ssg-package": "tsx --conditions=webstudio scripts/test-ssg-package.ts",
"test:release-smoke": "pnpm build && tsx --conditions=webstudio scripts/release-smoke.ts",
"test:release-smoke:model": "WEBSTUDIO_RELEASE_SMOKE_MODEL_EVAL_REQUIRED=1 pnpm test:release-smoke",
"test:release-smoke:registry": "tsx --conditions=webstudio scripts/release-smoke.ts"
},
"license": "AGPL-3.0-or-later",
"engines": {
"node": ">=22.12.0"
},
"dependencies": {
"@clack/prompts": "^0.10.0",
"@emotion/hash": "^0.9.2",
"@trpc/client": "^10.45.2",
"@webstudio-is/http-client": "workspace:*",
"@webstudio-is/project-migrations": "workspace:*",
"@webstudio-is/protocol": "workspace:*",
"@webstudio-is/sync-client": "workspace:*",
"acorn": "^8.14.1",
"acorn-jsx": "^5.3.2",
"acorn-walk": "^8.3.4",
"change-case": "^5.4.4",
"chrome-launcher": "^1.2.1",
"deepmerge": "^4.3.1",
"decode-named-character-reference": "^1.0.2",
"detect-port": "^2.1.0",
"env-paths": "^3.0.0",
"esbuild": "^0.25.3",
"fast-deep-equal": "^3.1.3",
"get-port": "^7.2.0",
"immer": "^10.1.1",
"nanoid": "^5.1.5",
"p-limit": "^6.2.0",
"parse5": "7.3.0",
"path-key": "^4.0.0",
"picocolors": "^1.1.1",
"reserved-identifiers": "^1.0.0",
"tinyexec": "^0.3.2",
"tus-js-client": "^4.3.1",
"warn-once": "^0.1.1",
"which": "^5.0.0",
"yargs": "^17.7.2",
"zod": "^4.4.3"
},
"devDependencies": {
"@cloudflare/vite-plugin": "^1.1.0",
"@netlify/vite-plugin-react-router": "^1.0.1",
"@react-router/dev": "^7.5.3",
"@react-router/fs-routes": "^7.5.3",
"@react-router/node": "^7.5.3",
"@react-router/serve": "^7.5.3",
"@remix-run/cloudflare": "^2.16.5",
"@remix-run/cloudflare-pages": "^2.16.5",
"@remix-run/dev": "^2.16.5",
"@remix-run/node": "^2.16.5",
"@remix-run/react": "^2.16.5",
"@remix-run/server-runtime": "^2.16.5",
"@shikijs/langs": "4.4.1",
"@shikijs/themes": "4.4.1",
"@types/mdast": "^4.0.4",
"@types/react": "^18.2.70",
"@types/react-dom": "^18.2.25",
"@types/which": "^3.0.4",
"@types/yargs": "^17.0.33",
"@vercel/react-router": "^1.1.0",
"@vitejs/plugin-react": "^4.4.1",
"@webstudio-is/css-engine": "workspace:*",
"@webstudio-is/content-engine": "workspace:*",
"@webstudio-is/expression": "workspace:*",
"@webstudio-is/fonts": "workspace:*",
"@webstudio-is/image": "workspace:*",
"@webstudio-is/project-build": "workspace:*",
"@webstudio-is/query-builder": "workspace:*",
"@webstudio-is/react-sdk": "workspace:*",
"@webstudio-is/sdk": "workspace:*",
"@webstudio-is/sdk-components-animation": "workspace:*",
"@webstudio-is/sdk-components-react": "workspace:*",
"@webstudio-is/sdk-components-react-radix": "workspace:*",
"@webstudio-is/sdk-components-react-remix": "workspace:*",
"@webstudio-is/sdk-components-react-router": "workspace:*",
"@webstudio-is/sdk-components-registry": "workspace:*",
"@webstudio-is/tsconfig": "workspace:*",
"@webstudio-is/vision": "workspace:*",
"@webstudio-is/wsauth": "workspace:*",
"h3": "^1.15.1",
"ipx": "^3.0.3",
"isbot": "^5.1.25",
"mdast-util-directive": "^3.1.0",
"mdast-util-from-markdown": "^2.0.3",
"mdast-util-to-string": "^4.0.0",
"micromark-extension-directive": "^4.0.0",
"oxfmt": "0.58.0",
"react": "18.3.0-canary-14898b6a9-20240318",
"react-dom": "18.3.0-canary-14898b6a9-20240318",
"react-router": "^7.5.3",
"shiki": "4.4.1",
"ts-expect": "^1.3.0",
"typescript": "7.0.2",
"vike": "^0.4.229",
"vite": "^6.3.4",
"vitest": "^3.1.2",
"wrangler": "^3.63.2"
}
}
Reference in new issue View Git Blame Copy Permalink