mirror of
https://github.com/webstudio-is/webstudio.git
synced 2026-10-28 23:57:17 +09:00
## Summary Replace the defunct Lost Pixel service and the interim Chromatic workflow with a fast, self-hosted visual comparison pipeline. The workflow indexes the existing CSF stories without building Storybook, builds and serves a lightweight browser app with Vite, renders the merge base and current revision through Webstudio's shared browser tooling, and compares screenshots with Webstudio's pixel-diff and OCR engine. It uses GitHub Actions cache and artifacts only—no R2 bucket or external visual-testing service. Closes #6064 ## Implementation - [x] Remove the Lost Pixel workflow, configuration, acknowledgement, and global two-second wait. - [x] Remove the interim Chromatic workflow and settings. - [x] Extract source-agnostic capture, diff, OCR, audit, cache, and report tooling into `@webstudio-is/vision`; rename the Webstudio-specific preview boundary to `@webstudio-is/project-build/preview`. - [x] Expose the reusable browser capture session through the shared vision boundary. - [x] Make OCR and diff-artifact generation conditional on detected pixel changes. - [x] Render existing CSF stories through a typed, lightweight browser harness built with Vite. - [x] Replace the embedded HTML and handwritten HTTP/MIME server with a real HTML entry, Vite production build, and Vite preview server; isolate declarative build configuration in `story-build.ts`. - [x] Apply Storybook preview annotations, decorators, global CSS, backgrounds, and story metadata. - [x] Store story scopes as shared data and load each revision's own configuration. - [x] Discover changed, added, and removed stories from both revisions. - [x] Populate private story sources symmetrically in the baseline worktree. - [x] Isolate every story with a fresh React root and deterministic time, randomness, intervals, focus, animation, and layout readiness. - [x] Preserve story-owned visual options for masking, settling, and interval control. - [x] Reuse Vite preview servers, one Chromium process, grouped browser pages, production bundles, and a bounded CPU worker pool for image comparison. - [x] Recheck initial pixel differences with serial captures, then recapture persistent baseline/current pairs on the same browser page so renderer variance cannot create false failures. - [x] Report individual story render failures without discarding successful captures. - [x] Enforce a 0.1 per-pixel threshold and 0.001% total mismatch tolerance. - [x] Run OCR only for changed comparisons and include text-change evidence. - [x] Move the generic report model, writer, static viewer, and opener into the shared Vision package for reuse by CI and future CLI comparisons. - [x] Bundle report HTML, styling, and browser behavior as static Vision assets without runtime source-file lookups. - [x] Copy external screenshot inputs into self-contained report artifacts for CLI reuse. - [x] Handle cached removed-only comparisons without rendering the removed story on the current server. - [x] Support focused local reproduction and opening the report with one command. - [x] Pin CI to the lockfile's Playwright Chromium, cache complete revision screenshots with matching restore/save paths and automatically derived runtime invalidation, and render both revisions on a miss. - [x] Keep cache archives bounded to one revision instead of accumulating the cache history. - [x] Run the complete visual suite in one CI job and upload one report as a 14-day artifact. - [x] Fail PR checks on visual differences, empty runs, missing screenshots, missing or empty report artifacts, and infrastructure failures. - [x] Rerun fork checks when `safe-to-deploy` is added. - [x] Allow reviewed visual-only differences through `visual-change-approved` and invalidate approval after new commits. - [x] Keep capture, pixel diff, OCR, and report generation source-agnostic for future draft-versus-published project comparisons. - [x] Use Vite for the complete story HTML, module, asset, production-build, and preview-server lifecycle; inline the selected stories and skip ineffective tree-shaking to keep the cached run under one minute. - [x] Review Builder product documentation impact: no user-facing Builder documentation is affected; contributor usage is documented in `scripts/visual-regression/README.md`. - [x] Review fixture impact: no fixture inputs or generated fixture outputs are affected. ## Verification - [x] Project-build Vision tests, including the reusable report test, and entrypoint contract tests pass. - [x] CLI screenshot, preview, and browser-session tests pass. - [x] Project-build, CLI, Builder, design-system, and visual-runner typechecks pass. - [x] Visual runner tests pass (10), and the complete Vision package passes (79), including demonstrated fail-before/pass-after coverage. - [x] Full Storybook production build passes. - [x] Full local Vite run completes all 238 stories in 69.9s cold and 50.3s with a cached baseline plus two same-page serial verifications, with zero differences; identical-image comparison remains sub-second. - [x] The report writer passes a bundled-output smoke test and materializes external screenshot assets. - [x] The generated report validates all 238 entries and referenced images. - [x] Hosted cached-baseline comparison completes all 238 stories in 52.9s with zero differences; the cache-miss fallback completes in 101.3s. - [x] Cache-miss fallback renders both Git revisions on demand. - [x] Visual differences produce baseline, current, diff, OCR, JSON, and HTML evidence and fail without approval. - [x] Hosted single-job visual workflow restores both the pinned browser and baseline cache, validates all 238 public and private stories with zero differences, and uploads the report. - [x] The final Vite production build renders all 238 stories without dependency invalidation or duplicate React runtimes, and restores `NODE_ENV` so cached serial fallback installs the baseline's dev dependencies. - [x] Root lint, package-boundary validation, and relevant formatting checks pass. - [x] Workflow YAML parses successfully. - [x] `git diff --check`, staged-diff review, and final worktree review pass. ## Usage Run the complete local comparison and open the visual report: ```sh pnpm visual-regression:ui ``` Focus one component while iterating: ```sh pnpm visual-regression:ui -- --grep "Design system/Button" ``` Reopen the latest report without rerunning: ```sh pnpm visual-regression:report ``` The CI report is available as the `visual-regression-report` artifact. ## Tradeoffs - Baseline screenshots are a disposable acceleration cache, not source of truth. A missing or invalid cache renders the merge base directly from Git. - GitHub artifact storage provides downloadable one-click reports; permanently hosted browser links would require another hosting surface. - OCR runs after pixel triage because invoking Tesseract for every unchanged story would make the common path unnecessarily slow. - Vite production builds add Rollup's module graph and HTML/asset pipeline. The runner emits one bundle because every selected story is rendered, and skips tree-shaking because it removed little while increasing build time. - The verified local path is 50.3s cached and 69.9s cold. The hosted path is 52.9s cached and 101.3s cold. A new browser/runtime cache must render its baseline once; the pinned browser keeps later runners on the reusable path. - The complete suite stays in one Actions job; browser-page concurrency is internal and bounded.
115 lines
3.5 KiB
TypeScript
115 lines
3.5 KiB
TypeScript
import assert from "node:assert/strict";
|
|
import { mkdtemp, readFile, rm } from "node:fs/promises";
|
|
import { tmpdir } from "node:os";
|
|
import path from "node:path";
|
|
import test from "node:test";
|
|
import { parse, type DefaultTreeAdapterTypes as Html } from "parse5";
|
|
import { buildVisualStoryApp } from "./story-build";
|
|
import {
|
|
startVisualStoryServer,
|
|
startVisualStoryServers,
|
|
} from "./story-server";
|
|
|
|
const findElement = (
|
|
node: Html.Node,
|
|
predicate: (element: Html.Element) => boolean
|
|
): Html.Element | undefined => {
|
|
if ("tagName" in node && predicate(node)) {
|
|
return node;
|
|
}
|
|
if ("childNodes" in node) {
|
|
for (const child of node.childNodes) {
|
|
const match = findElement(child, predicate);
|
|
if (match !== undefined) {
|
|
return match;
|
|
}
|
|
}
|
|
}
|
|
};
|
|
|
|
const readVisualStoryDocument = async () =>
|
|
parse(await readFile(new URL("./index.html", import.meta.url), "utf8"));
|
|
|
|
test("loads the visual harness from a proper HTML entry", async () => {
|
|
const document = await readVisualStoryDocument();
|
|
const root = findElement(document, (element) =>
|
|
element.attrs.some(({ name, value }) => name === "id" && value === "root")
|
|
);
|
|
const script = findElement(
|
|
document,
|
|
(element) => element.tagName === "script"
|
|
);
|
|
assert.equal(root?.parentNode?.nodeName, "body");
|
|
assert.ok(
|
|
script?.attrs.some(
|
|
({ name, value }) => name === "src" && value === "/harness.tsx"
|
|
)
|
|
);
|
|
});
|
|
|
|
test("does not disable animations before reading the story parameters", async () => {
|
|
const document = await readVisualStoryDocument();
|
|
const style = findElement(document, (element) => element.tagName === "style");
|
|
const css = style?.childNodes
|
|
.filter((node): node is Html.TextNode => node.nodeName === "#text")
|
|
.map((node) => node.value)
|
|
.join("");
|
|
|
|
assert.doesNotMatch(css ?? "", /animation-(?:delay|duration)|transition/);
|
|
});
|
|
|
|
test("does not leak Vite's production environment", async () => {
|
|
const outputDirectory = await mkdtemp(path.join(tmpdir(), "visual-build-"));
|
|
const nodeEnvironment = process.env.NODE_ENV;
|
|
delete process.env.NODE_ENV;
|
|
try {
|
|
await buildVisualStoryApp({
|
|
root: process.cwd(),
|
|
outputDirectory,
|
|
storyFiles: [
|
|
"packages/design-system/src/components/color-picker.stories.tsx",
|
|
],
|
|
});
|
|
assert.equal(process.env.NODE_ENV, undefined);
|
|
} finally {
|
|
if (nodeEnvironment === undefined) {
|
|
delete process.env.NODE_ENV;
|
|
} else {
|
|
process.env.NODE_ENV = nodeEnvironment;
|
|
}
|
|
await rm(outputDirectory, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
test("closes servers that started before a concurrent startup failure", async () => {
|
|
let releaseSuccessfulServer: (() => void) | undefined;
|
|
const successfulServerReady = new Promise<void>((resolve) => {
|
|
releaseSuccessfulServer = resolve;
|
|
});
|
|
let closed = false;
|
|
const startServer = async (
|
|
options: Parameters<typeof startVisualStoryServer>[0]
|
|
) => {
|
|
if (options.port === 6102) {
|
|
throw new Error("Current story server failed");
|
|
}
|
|
await successfulServerReady;
|
|
return {
|
|
async close() {
|
|
closed = true;
|
|
},
|
|
};
|
|
};
|
|
const startup = startVisualStoryServers(
|
|
[
|
|
{ root: "/repo", port: 6101, outputDirectory: "/one", storyFiles: [] },
|
|
{ root: "/repo", port: 6102, outputDirectory: "/two", storyFiles: [] },
|
|
],
|
|
startServer
|
|
);
|
|
releaseSuccessfulServer?.();
|
|
|
|
await assert.rejects(startup, /Current story server failed/);
|
|
assert.equal(closed, true);
|
|
});
|