Oleg Isonen
db2d092268
## Summary Replace the defunct Lost Pixel service and the interim Chromatic workflow with a fast, self-hosted visual comparison pipeline. The workflow indexes the existing CSF stories without building Storybook, builds and serves a lightweight browser app with Vite, renders the merge base and current revision through Webstudio's shared browser tooling, and compares screenshots with Webstudio's pixel-diff and OCR engine. It uses GitHub Actions cache and artifacts only—no R2 bucket or external visual-testing service. Closes #6064 ## Implementation - [x] Remove the Lost Pixel workflow, configuration, acknowledgement, and global two-second wait. - [x] Remove the interim Chromatic workflow and settings. - [x] Extract source-agnostic capture, diff, OCR, audit, cache, and report tooling into `@webstudio-is/vision`; rename the Webstudio-specific preview boundary to `@webstudio-is/project-build/preview`. - [x] Expose the reusable browser capture session through the shared vision boundary. - [x] Make OCR and diff-artifact generation conditional on detected pixel changes. - [x] Render existing CSF stories through a typed, lightweight browser harness built with Vite. - [x] Replace the embedded HTML and handwritten HTTP/MIME server with a real HTML entry, Vite production build, and Vite preview server; isolate declarative build configuration in `story-build.ts`. - [x] Apply Storybook preview annotations, decorators, global CSS, backgrounds, and story metadata. - [x] Store story scopes as shared data and load each revision's own configuration. - [x] Discover changed, added, and removed stories from both revisions. - [x] Populate private story sources symmetrically in the baseline worktree. - [x] Isolate every story with a fresh React root and deterministic time, randomness, intervals, focus, animation, and layout readiness. - [x] Preserve story-owned visual options for masking, settling, and interval control. - [x] Reuse Vite preview servers, one Chromium process, grouped browser pages, production bundles, and a bounded CPU worker pool for image comparison. - [x] Recheck initial pixel differences with serial captures, then recapture persistent baseline/current pairs on the same browser page so renderer variance cannot create false failures. - [x] Report individual story render failures without discarding successful captures. - [x] Enforce a 0.1 per-pixel threshold and 0.001% total mismatch tolerance. - [x] Run OCR only for changed comparisons and include text-change evidence. - [x] Move the generic report model, writer, static viewer, and opener into the shared Vision package for reuse by CI and future CLI comparisons. - [x] Bundle report HTML, styling, and browser behavior as static Vision assets without runtime source-file lookups. - [x] Copy external screenshot inputs into self-contained report artifacts for CLI reuse. - [x] Handle cached removed-only comparisons without rendering the removed story on the current server. - [x] Support focused local reproduction and opening the report with one command. - [x] Pin CI to the lockfile's Playwright Chromium, cache complete revision screenshots with matching restore/save paths and automatically derived runtime invalidation, and render both revisions on a miss. - [x] Keep cache archives bounded to one revision instead of accumulating the cache history. - [x] Run the complete visual suite in one CI job and upload one report as a 14-day artifact. - [x] Fail PR checks on visual differences, empty runs, missing screenshots, missing or empty report artifacts, and infrastructure failures. - [x] Rerun fork checks when `safe-to-deploy` is added. - [x] Allow reviewed visual-only differences through `visual-change-approved` and invalidate approval after new commits. - [x] Keep capture, pixel diff, OCR, and report generation source-agnostic for future draft-versus-published project comparisons. - [x] Use Vite for the complete story HTML, module, asset, production-build, and preview-server lifecycle; inline the selected stories and skip ineffective tree-shaking to keep the cached run under one minute. - [x] Review Builder product documentation impact: no user-facing Builder documentation is affected; contributor usage is documented in `scripts/visual-regression/README.md`. - [x] Review fixture impact: no fixture inputs or generated fixture outputs are affected. ## Verification - [x] Project-build Vision tests, including the reusable report test, and entrypoint contract tests pass. - [x] CLI screenshot, preview, and browser-session tests pass. - [x] Project-build, CLI, Builder, design-system, and visual-runner typechecks pass. - [x] Visual runner tests pass (10), and the complete Vision package passes (79), including demonstrated fail-before/pass-after coverage. - [x] Full Storybook production build passes. - [x] Full local Vite run completes all 238 stories in 69.9s cold and 50.3s with a cached baseline plus two same-page serial verifications, with zero differences; identical-image comparison remains sub-second. - [x] The report writer passes a bundled-output smoke test and materializes external screenshot assets. - [x] The generated report validates all 238 entries and referenced images. - [x] Hosted cached-baseline comparison completes all 238 stories in 52.9s with zero differences; the cache-miss fallback completes in 101.3s. - [x] Cache-miss fallback renders both Git revisions on demand. - [x] Visual differences produce baseline, current, diff, OCR, JSON, and HTML evidence and fail without approval. - [x] Hosted single-job visual workflow restores both the pinned browser and baseline cache, validates all 238 public and private stories with zero differences, and uploads the report. - [x] The final Vite production build renders all 238 stories without dependency invalidation or duplicate React runtimes, and restores `NODE_ENV` so cached serial fallback installs the baseline's dev dependencies. - [x] Root lint, package-boundary validation, and relevant formatting checks pass. - [x] Workflow YAML parses successfully. - [x] `git diff --check`, staged-diff review, and final worktree review pass. ## Usage Run the complete local comparison and open the visual report: ```sh pnpm visual-regression:ui ``` Focus one component while iterating: ```sh pnpm visual-regression:ui -- --grep "Design system/Button" ``` Reopen the latest report without rerunning: ```sh pnpm visual-regression:report ``` The CI report is available as the `visual-regression-report` artifact. ## Tradeoffs - Baseline screenshots are a disposable acceleration cache, not source of truth. A missing or invalid cache renders the merge base directly from Git. - GitHub artifact storage provides downloadable one-click reports; permanently hosted browser links would require another hosting surface. - OCR runs after pixel triage because invoking Tesseract for every unchanged story would make the common path unnecessarily slow. - Vite production builds add Rollup's module graph and HTML/asset pipeline. The runner emits one bundle because every selected story is rendered, and skips tree-shaking because it removed little while increasing build time. - The verified local path is 50.3s cached and 69.9s cold. The hosted path is 52.9s cached and 101.3s cold. A new browser/runtime cache must render its baseline once; the pinned browser keeps later runners on the reusable path. - The complete suite stays in one Actions job; browser-page concurrency is internal and bounded.
Webstudio CLI
The Webstudio CLI helps you to link, sync and build projects from Webstudio Cloud. This README will guide you through the process of setting up a Webstudio project on your local machine and continually sync it with the cloud.
Prerequisites: Installing Node.js
You need Node.js to use the Webstudio CLI. If Node.js is already installed in your system, you can skip ahead to the section on installing the Webstudio CLI.
To install Node.js using NVM, first install NVM by running:
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.5/install.sh | bash
Once NVM is installed, you can install Node.js version 22 by running:
nvm install 22
Verify your Node.js installation by checking its version:
node --version
Installing the Webstudio CLI
To get started with the Webstudio CLI:
-
Download and install the CLI using the following command:
npm install -g webstudio@latest -
Confirm the installation by checking the CLI version:
webstudio --version -
To keep your CLI updated, use the same command used for installation whenever a new release is available.
Initiating a Webstudio Project
Now, you can run a Webstudio project on your local machine using this command:
webstudio
This will initiate the flow to connect your Webstudio Cloud project and build it in your local computer. The default flow will guide you through the steps. You can also perform all the operations individually using independent commands.
Commands
Here is the list of independent commands:
- version
- help
- link
- sync
- build
link
The link command stores access to a Webstudio Cloud project. MCP clients use that connection to read and edit the latest Builder build directly; the project does not need to be published or synchronized first.
Shell-capable agents can call MCP tools directly as CLI subcommands immediately after linking. This does not require MCP client configuration or a client restart:
webstudio meta.index
webstudio components.search '{"brief":"button"}'
webstudio insert-fragment --input-file .temp/insert-fragment.json --dry-run
.temp/insert-fragment.json:
{
"parentInstanceId": "<parent-id>",
"fragment": "<ws.element ws:tag='section' />"
}
The explicit equivalent is webstudio mcp single-op-call <tool> '<json>'.
Use webstudio connect <client> only when you want persistent native MCP tools
inside Claude Code, Codex, Cursor, or VS Code. A persistent client may need to
reconnect after its configuration or cached tool catalog changes.
See the Webstudio MCP documentation for setup, discovery, editing, verification, and safety guidance.
You can link a project from Webstudio Cloud with the following command:
webstudio link
This command will prompt you to paste a link which you can create using the Share option in your project.
Make sure to provide Build access when generating the link in Webstudio Cloud.
sync
Once the project is linked, use the sync command to sync it with the cloud:
webstudio sync
Make sure to publish the project in Webstudio Cloud before running the sync command in your local Webstudio project.
sync is only for downloading a published project export. It is not part of the MCP editing workflow.
build
Now, you can build your project with the build command:
webstudio build
During this phase, the CLI establishes the necessary routes and pages, scaffolding the entire application using the default Remix template. Additionally, all assets, such as images and fonts are downloaded to the assets folder inside the public directory.
Once the project is scaffolded, you can run npm install and then npm run dev to run your app in development mode.
If you want to build a production version of the app, you can run npm run build.
Deployment
Vercel
Once you've built the project locally, you can use the Vercel CLI to deploy your project directly to Vercel:
vercel deploy
Follow the instructions here to install the vercel CLI. We plan to add more deployment targets in future.
Netlify
If you want to deploy to netlify, you can use Netlify CLI to deploy your project directly to Netlify:
netlify deploy
You can configure the project to support netlify serverless/edge-functions respectively, as deployment target at the time of initially setting up your project. Please check the initiating-a-webstudio-project section.
You can manually change it using the build command. For serverless functions:
webstudio build --template netlify
Important Notes
If you use vercel build before vercel deploy, make sure to clean your app folder in the project afterward.
Vercel injects a few files to support and deploy Remix using their CLI, but these files are not necessary for your project when you use it locally.