Files
webstudio/scripts/visual-regression/manifest.ts
Oleg Isonen db2d092268 ci: replace Lost Pixel with self-hosted visual regression (#6086)
## Summary

Replace the defunct Lost Pixel service and the interim Chromatic
workflow with a fast, self-hosted visual comparison pipeline.

The workflow indexes the existing CSF stories without building
Storybook, builds and serves a lightweight browser app with Vite,
renders the merge base and current revision through Webstudio's shared
browser tooling, and compares screenshots with Webstudio's pixel-diff
and OCR engine. It uses GitHub Actions cache and artifacts only—no R2
bucket or external visual-testing service.

Closes #6064

## Implementation

- [x] Remove the Lost Pixel workflow, configuration, acknowledgement,
and global two-second wait.
- [x] Remove the interim Chromatic workflow and settings.
- [x] Extract source-agnostic capture, diff, OCR, audit, cache, and
report tooling into `@webstudio-is/vision`; rename the
Webstudio-specific preview boundary to
`@webstudio-is/project-build/preview`.
- [x] Expose the reusable browser capture session through the shared
vision boundary.
- [x] Make OCR and diff-artifact generation conditional on detected
pixel changes.
- [x] Render existing CSF stories through a typed, lightweight browser
harness built with Vite.
- [x] Replace the embedded HTML and handwritten HTTP/MIME server with a
real HTML entry, Vite production build, and Vite preview server; isolate
declarative build configuration in `story-build.ts`.
- [x] Apply Storybook preview annotations, decorators, global CSS,
backgrounds, and story metadata.
- [x] Store story scopes as shared data and load each revision's own
configuration.
- [x] Discover changed, added, and removed stories from both revisions.
- [x] Populate private story sources symmetrically in the baseline
worktree.
- [x] Isolate every story with a fresh React root and deterministic
time, randomness, intervals, focus, animation, and layout readiness.
- [x] Preserve story-owned visual options for masking, settling, and
interval control.
- [x] Reuse Vite preview servers, one Chromium process, grouped browser
pages, production bundles, and a bounded CPU worker pool for image
comparison.
- [x] Recheck initial pixel differences with serial captures, then
recapture persistent baseline/current pairs on the same browser page so
renderer variance cannot create false failures.
- [x] Report individual story render failures without discarding
successful captures.
- [x] Enforce a 0.1 per-pixel threshold and 0.001% total mismatch
tolerance.
- [x] Run OCR only for changed comparisons and include text-change
evidence.
- [x] Move the generic report model, writer, static viewer, and opener
into the shared Vision package for reuse by CI and future CLI
comparisons.
- [x] Bundle report HTML, styling, and browser behavior as static Vision
assets without runtime source-file lookups.
- [x] Copy external screenshot inputs into self-contained report
artifacts for CLI reuse.
- [x] Handle cached removed-only comparisons without rendering the
removed story on the current server.
- [x] Support focused local reproduction and opening the report with one
command.
- [x] Pin CI to the lockfile's Playwright Chromium, cache complete
revision screenshots with matching restore/save paths and automatically
derived runtime invalidation, and render both revisions on a miss.
- [x] Keep cache archives bounded to one revision instead of
accumulating the cache history.
- [x] Run the complete visual suite in one CI job and upload one report
as a 14-day artifact.
- [x] Fail PR checks on visual differences, empty runs, missing
screenshots, missing or empty report artifacts, and infrastructure
failures.
- [x] Rerun fork checks when `safe-to-deploy` is added.
- [x] Allow reviewed visual-only differences through
`visual-change-approved` and invalidate approval after new commits.
- [x] Keep capture, pixel diff, OCR, and report generation
source-agnostic for future draft-versus-published project comparisons.
- [x] Use Vite for the complete story HTML, module, asset,
production-build, and preview-server lifecycle; inline the selected
stories and skip ineffective tree-shaking to keep the cached run under
one minute.
- [x] Review Builder product documentation impact: no user-facing
Builder documentation is affected; contributor usage is documented in
`scripts/visual-regression/README.md`.
- [x] Review fixture impact: no fixture inputs or generated fixture
outputs are affected.

## Verification

- [x] Project-build Vision tests, including the reusable report test,
and entrypoint contract tests pass.
- [x] CLI screenshot, preview, and browser-session tests pass.
- [x] Project-build, CLI, Builder, design-system, and visual-runner
typechecks pass.
- [x] Visual runner tests pass (10), and the complete Vision package
passes (79), including demonstrated fail-before/pass-after coverage.
- [x] Full Storybook production build passes.
- [x] Full local Vite run completes all 238 stories in 69.9s cold and
50.3s with a cached baseline plus two same-page serial verifications,
with zero differences; identical-image comparison remains sub-second.
- [x] The report writer passes a bundled-output smoke test and
materializes external screenshot assets.
- [x] The generated report validates all 238 entries and referenced
images.
- [x] Hosted cached-baseline comparison completes all 238 stories in
52.9s with zero differences; the cache-miss fallback completes in
101.3s.
- [x] Cache-miss fallback renders both Git revisions on demand.
- [x] Visual differences produce baseline, current, diff, OCR, JSON, and
HTML evidence and fail without approval.
- [x] Hosted single-job visual workflow restores both the pinned browser
and baseline cache, validates all 238 public and private stories with
zero differences, and uploads the report.
- [x] The final Vite production build renders all 238 stories without
dependency invalidation or duplicate React runtimes, and restores
`NODE_ENV` so cached serial fallback installs the baseline's dev
dependencies.
- [x] Root lint, package-boundary validation, and relevant formatting
checks pass.
- [x] Workflow YAML parses successfully.
- [x] `git diff --check`, staged-diff review, and final worktree review
pass.

## Usage

Run the complete local comparison and open the visual report:

```sh
pnpm visual-regression:ui
```

Focus one component while iterating:

```sh
pnpm visual-regression:ui -- --grep "Design system/Button"
```

Reopen the latest report without rerunning:

```sh
pnpm visual-regression:report
```

The CI report is available as the `visual-regression-report` artifact.

## Tradeoffs

- Baseline screenshots are a disposable acceleration cache, not source
of truth. A missing or invalid cache renders the merge base directly
from Git.
- GitHub artifact storage provides downloadable one-click reports;
permanently hosted browser links would require another hosting surface.
- OCR runs after pixel triage because invoking Tesseract for every
unchanged story would make the common path unnecessarily slow.
- Vite production builds add Rollup's module graph and HTML/asset
pipeline. The runner emits one bundle because every selected story is
rendered, and skips tree-shaking because it removed little while
increasing build time.
- The verified local path is 50.3s cached and 69.9s cold. The hosted
path is 52.9s cached and 101.3s cold. A new browser/runtime cache must
render its baseline once; the pinned browser keeps later runners on the
reusable path.
- The complete suite stays in one Actions job; browser-page concurrency
is internal and bounded.
2026-08-12 10:55:52 +02:00

126 lines
3.2 KiB
TypeScript

import { readFile } from "node:fs/promises";
import path from "node:path";
import { glob } from "node:fs/promises";
import { loadCsf } from "storybook/internal/csf-tools";
import { toId } from "storybook/internal/csf";
import type { VisualEntry } from "@webstudio-is/vision/comparison";
export type StorySource = {
directory: string;
files: string;
titlePrefix: string;
};
const isStorySource = (value: unknown): value is StorySource =>
typeof value === "object" &&
value !== null &&
"directory" in value &&
typeof value.directory === "string" &&
"files" in value &&
typeof value.files === "string" &&
"titlePrefix" in value &&
typeof value.titlePrefix === "string";
export const readStorySources = async (
root: string
): Promise<readonly StorySource[] | undefined> => {
let source: string;
try {
source = await readFile(
path.join(root, ".storybook/story-sources.json"),
"utf8"
);
} catch (error) {
if (
typeof error === "object" &&
error !== null &&
"code" in error &&
error.code === "ENOENT"
) {
return;
}
throw error;
}
const value: unknown = JSON.parse(source);
if (Array.isArray(value) === false || value.every(isStorySource) === false) {
throw new Error(".storybook/story-sources.json is invalid.");
}
return value;
};
export type VisualStoryEntry = VisualEntry & {
title: string;
name: string;
exportName: string;
file: string;
titlePrefix: string;
};
const readScopeEntries = async ({
root,
pattern,
titlePrefix,
}: {
root: string;
pattern: string;
titlePrefix: string;
}) => {
const entries: VisualStoryEntry[] = [];
for await (const file of glob(pattern, { cwd: root })) {
const source = await readFile(path.join(root, file), "utf8");
const csf = loadCsf(source, {
fileName: file,
makeTitle: (title) => `${titlePrefix}/${title}`,
});
csf.parse();
for (const story of csf.indexInputs) {
if (story.type !== "story") {
continue;
}
if (
story.title === undefined ||
story.name === undefined ||
story.exportName === undefined
) {
throw new Error(`Story index metadata is incomplete in ${file}`);
}
entries.push({
id: story.__id ?? toId(story.title, story.name),
title: story.title,
name: story.name,
exportName: story.exportName,
file,
titlePrefix,
});
}
}
return entries;
};
export const readStoryManifest = async ({
root,
storySources,
}: {
root: string;
storySources: readonly StorySource[];
}) => {
const entries = (
await Promise.all(
storySources.map(({ directory, files, titlePrefix }) => {
const pattern = path.posix.join(".storybook", directory, files);
return readScopeEntries({ root, pattern, titlePrefix });
})
)
)
.flat()
.sort((left, right) => left.id.localeCompare(right.id));
const duplicate = entries.find(
(entry, index) => entry.id === entries[index - 1]?.id
);
if (duplicate !== undefined) {
throw new Error(`Duplicate visual story id: ${duplicate.id}`);
}
return Object.fromEntries(entries.map((entry) => [entry.id, entry]));
};