Files
webstudio/scripts/visual-regression/harness.tsx
Oleg Isonen db2d092268 ci: replace Lost Pixel with self-hosted visual regression (#6086)
## Summary

Replace the defunct Lost Pixel service and the interim Chromatic
workflow with a fast, self-hosted visual comparison pipeline.

The workflow indexes the existing CSF stories without building
Storybook, builds and serves a lightweight browser app with Vite,
renders the merge base and current revision through Webstudio's shared
browser tooling, and compares screenshots with Webstudio's pixel-diff
and OCR engine. It uses GitHub Actions cache and artifacts only—no R2
bucket or external visual-testing service.

Closes #6064

## Implementation

- [x] Remove the Lost Pixel workflow, configuration, acknowledgement,
and global two-second wait.
- [x] Remove the interim Chromatic workflow and settings.
- [x] Extract source-agnostic capture, diff, OCR, audit, cache, and
report tooling into `@webstudio-is/vision`; rename the
Webstudio-specific preview boundary to
`@webstudio-is/project-build/preview`.
- [x] Expose the reusable browser capture session through the shared
vision boundary.
- [x] Make OCR and diff-artifact generation conditional on detected
pixel changes.
- [x] Render existing CSF stories through a typed, lightweight browser
harness built with Vite.
- [x] Replace the embedded HTML and handwritten HTTP/MIME server with a
real HTML entry, Vite production build, and Vite preview server; isolate
declarative build configuration in `story-build.ts`.
- [x] Apply Storybook preview annotations, decorators, global CSS,
backgrounds, and story metadata.
- [x] Store story scopes as shared data and load each revision's own
configuration.
- [x] Discover changed, added, and removed stories from both revisions.
- [x] Populate private story sources symmetrically in the baseline
worktree.
- [x] Isolate every story with a fresh React root and deterministic
time, randomness, intervals, focus, animation, and layout readiness.
- [x] Preserve story-owned visual options for masking, settling, and
interval control.
- [x] Reuse Vite preview servers, one Chromium process, grouped browser
pages, production bundles, and a bounded CPU worker pool for image
comparison.
- [x] Recheck initial pixel differences with serial captures, then
recapture persistent baseline/current pairs on the same browser page so
renderer variance cannot create false failures.
- [x] Report individual story render failures without discarding
successful captures.
- [x] Enforce a 0.1 per-pixel threshold and 0.001% total mismatch
tolerance.
- [x] Run OCR only for changed comparisons and include text-change
evidence.
- [x] Move the generic report model, writer, static viewer, and opener
into the shared Vision package for reuse by CI and future CLI
comparisons.
- [x] Bundle report HTML, styling, and browser behavior as static Vision
assets without runtime source-file lookups.
- [x] Copy external screenshot inputs into self-contained report
artifacts for CLI reuse.
- [x] Handle cached removed-only comparisons without rendering the
removed story on the current server.
- [x] Support focused local reproduction and opening the report with one
command.
- [x] Pin CI to the lockfile's Playwright Chromium, cache complete
revision screenshots with matching restore/save paths and automatically
derived runtime invalidation, and render both revisions on a miss.
- [x] Keep cache archives bounded to one revision instead of
accumulating the cache history.
- [x] Run the complete visual suite in one CI job and upload one report
as a 14-day artifact.
- [x] Fail PR checks on visual differences, empty runs, missing
screenshots, missing or empty report artifacts, and infrastructure
failures.
- [x] Rerun fork checks when `safe-to-deploy` is added.
- [x] Allow reviewed visual-only differences through
`visual-change-approved` and invalidate approval after new commits.
- [x] Keep capture, pixel diff, OCR, and report generation
source-agnostic for future draft-versus-published project comparisons.
- [x] Use Vite for the complete story HTML, module, asset,
production-build, and preview-server lifecycle; inline the selected
stories and skip ineffective tree-shaking to keep the cached run under
one minute.
- [x] Review Builder product documentation impact: no user-facing
Builder documentation is affected; contributor usage is documented in
`scripts/visual-regression/README.md`.
- [x] Review fixture impact: no fixture inputs or generated fixture
outputs are affected.

## Verification

- [x] Project-build Vision tests, including the reusable report test,
and entrypoint contract tests pass.
- [x] CLI screenshot, preview, and browser-session tests pass.
- [x] Project-build, CLI, Builder, design-system, and visual-runner
typechecks pass.
- [x] Visual runner tests pass (10), and the complete Vision package
passes (79), including demonstrated fail-before/pass-after coverage.
- [x] Full Storybook production build passes.
- [x] Full local Vite run completes all 238 stories in 69.9s cold and
50.3s with a cached baseline plus two same-page serial verifications,
with zero differences; identical-image comparison remains sub-second.
- [x] The report writer passes a bundled-output smoke test and
materializes external screenshot assets.
- [x] The generated report validates all 238 entries and referenced
images.
- [x] Hosted cached-baseline comparison completes all 238 stories in
52.9s with zero differences; the cache-miss fallback completes in
101.3s.
- [x] Cache-miss fallback renders both Git revisions on demand.
- [x] Visual differences produce baseline, current, diff, OCR, JSON, and
HTML evidence and fail without approval.
- [x] Hosted single-job visual workflow restores both the pinned browser
and baseline cache, validates all 238 public and private stories with
zero differences, and uploads the report.
- [x] The final Vite production build renders all 238 stories without
dependency invalidation or duplicate React runtimes, and restores
`NODE_ENV` so cached serial fallback installs the baseline's dev
dependencies.
- [x] Root lint, package-boundary validation, and relevant formatting
checks pass.
- [x] Workflow YAML parses successfully.
- [x] `git diff --check`, staged-diff review, and final worktree review
pass.

## Usage

Run the complete local comparison and open the visual report:

```sh
pnpm visual-regression:ui
```

Focus one component while iterating:

```sh
pnpm visual-regression:ui -- --grep "Design system/Button"
```

Reopen the latest report without rerunning:

```sh
pnpm visual-regression:report
```

The CI report is available as the `visual-regression-report` artifact.

## Tradeoffs

- Baseline screenshots are a disposable acceleration cache, not source
of truth. A missing or invalid cache renders the merge base directly
from Git.
- GitHub artifact storage provides downloadable one-click reports;
permanently hosted browser links would require another hosting surface.
- OCR runs after pixel triage because invoking Tesseract for every
unchanged story would make the common path unnecessarily slow.
- Vite production builds add Rollup's module graph and HTML/asset
pipeline. The runner emits one bundle because every selected story is
rendered, and skips tree-shaking because it removed little while
increasing build time.
- The verified local path is 50.3s cached and 69.9s cold. The hosted
path is 52.9s cached and 101.3s cold. A new browser/runtime cache must
render its baseline once; the pinned browser keeps later runners on the
reusable path.
- The complete suite stays in one Actions job; browser-page concurrency
is internal and bounded.
2026-08-12 10:55:52 +02:00

235 lines
6.9 KiB
TypeScript

import React from "react";
import { flushSync } from "react-dom";
import { createRoot } from "react-dom/client";
import { composeStories, setProjectAnnotations } from "@storybook/react";
import preview from "virtual:webstudio-visual-preview";
import { modules } from "virtual:webstudio-visual-story-modules";
type VisualRegressionParameters = {
disableAnimations?: boolean;
disableIntervals?: boolean;
hideSelectors?: readonly string[];
settleTime?: number;
};
type VisualStoryInput = {
file: string;
exportName: string;
title: string;
};
type ComposedStory = React.ComponentType & {
parameters?: {
backgrounds?: {
default?: string;
values?: Array<{ name: string; value: string }>;
};
visualRegression?: VisualRegressionParameters;
};
};
declare global {
interface Window {
finishVisualStory: () => Promise<void>;
renderVisualStory: (input: VisualStoryInput) => Promise<void>;
showVisualError: (error: unknown) => void;
}
}
setProjectAnnotations(preview);
const waitForFrames = () =>
new Promise((resolve) =>
requestAnimationFrame(() => requestAnimationFrame(resolve))
);
const markReady = async (disableAnimations: boolean) => {
await waitForFrames();
if (disableAnimations) {
for (const animation of document.getAnimations()) {
animation.cancel();
}
// Animation components can react to cancellation on the next frame. Wait
// for those updates and cancel anything they created in response.
await waitForFrames();
for (const animation of document.getAnimations()) {
animation.cancel();
}
await waitForFrames();
}
if (document.activeElement instanceof HTMLElement) {
document.activeElement.blur();
}
const ready = document.createElement("div");
ready.id = "visual-ready";
ready.hidden = true;
document.body.append(ready);
};
const showError = (error: unknown) => {
document.querySelector("#visual-ready")?.remove();
document.querySelector("#visual-error")?.remove();
const output = document.createElement("pre");
output.id = "visual-error";
output.textContent =
error instanceof Error ? (error.stack ?? error.message) : String(error);
document.body.append(output);
};
window.showVisualError = showError;
window.addEventListener("error", (event) =>
showError(event.error ?? event.message)
);
window.addEventListener("unhandledrejection", (event) =>
showError(event.reason)
);
const rootElement = document.querySelector("#root");
if (rootElement === null) {
throw new Error("Visual story root is missing");
}
let root = createRoot(rootElement);
const originalSetInterval = window.setInterval;
const originalClearInterval = window.clearInterval;
const storyIntervals = new Set<number>();
let disableCurrentAnimations = true;
const OriginalDate = Date;
const fixedTime = Date.UTC(2020, 0, 1);
window.Date = new Proxy(OriginalDate, {
construct(target, argumentsList, newTarget) {
return Reflect.construct(
target,
argumentsList.length === 0 ? [fixedTime] : argumentsList,
newTarget
);
},
apply() {
return new OriginalDate(fixedTime).toString();
},
});
window.Date.now = () => fixedTime;
window.renderVisualStory = async ({ file, exportName, title }) => {
flushSync(() => root.unmount());
rootElement.replaceChildren();
root = createRoot(rootElement);
for (const interval of storyIntervals) {
originalClearInterval(interval);
}
storyIntervals.clear();
for (const element of document.querySelectorAll(
"body > :not(#root), [data-visual-regression-style]"
)) {
element.remove();
}
document.body.style.background = "";
let randomState = 0x12345678;
Math.random = () => {
randomState = (randomState * 1664525 + 1013904223) >>> 0;
return randomState / 0x100000000;
};
const load = modules[`/${file}`];
if (load === undefined) {
throw new Error(`Visual story module not found: ${file}`);
}
const module = await load();
const meta =
typeof module.default === "object" && module.default !== null
? module.default
: {};
const csfModule = {
...module,
default: { ...meta, title },
} as Parameters<typeof composeStories>[0];
const stories = composeStories(csfModule) as Record<string, ComposedStory>;
const Story = stories[exportName];
if (Story === undefined) {
throw new Error(`Visual story export not found: ${exportName}`);
}
const options = (Story.parameters?.visualRegression ??
{}) as VisualRegressionParameters;
disableCurrentAnimations = options.disableAnimations !== false;
Object.defineProperty(window, "setInterval", {
configurable: true,
value:
options.disableIntervals !== false
? () => 0
: (
handler: TimerHandler,
timeout?: number,
...arguments_: unknown[]
) => {
const interval = originalSetInterval(
handler,
timeout,
...arguments_
);
storyIntervals.add(interval);
return interval;
},
});
if (
options.disableAnimations !== false ||
options.hideSelectors?.length !== undefined
) {
const style = document.createElement("style");
style.dataset.visualRegressionStyle = "";
const rules =
options.hideSelectors?.map(
(selector) => `${selector} { visibility: hidden !important; }`
) ?? [];
if (options.disableAnimations !== false) {
rules.push(`*, *::before, *::after {
animation-delay: 0s !important;
animation-duration: 0s !important;
transition: none !important;
}`);
}
style.textContent = rules.join("\n");
document.head.append(style);
}
const backgrounds = Story.parameters?.backgrounds;
const background = backgrounds?.values?.find(
({ name }: { name: string }) => name === backgrounds.default
);
if (background?.value !== undefined) {
document.body.style.background = background.value;
}
flushSync(() => root.render(React.createElement(Story)));
if (options.settleTime !== undefined && options.settleTime > 0) {
await new Promise((resolve) =>
window.setTimeout(resolve, options.settleTime)
);
}
await markReady(options.disableAnimations !== false);
};
window.finishVisualStory = async () => {
if (disableCurrentAnimations === false) {
return;
}
// The browser calls this after fonts and component frames settle. Give
// animation consumers one frame to react, then freeze the final static state
// immediately before screenshot capture.
for (const animation of document.getAnimations()) {
animation.cancel();
}
await waitForFrames();
for (const animation of document.getAnimations()) {
animation.cancel();
}
};
const params = new URLSearchParams(window.location.search);
if (params.has("file")) {
const file = params.get("file");
const exportName = params.get("exportName");
const title = params.get("title");
if (file !== null && exportName !== null && title !== null) {
window.renderVisualStory({ file, exportName, title }).catch(showError);
}
}