Files
webstudio/apps/builder/app/services/content-database.server.ts
Oleg Isonen fc15ebf81a perf: avoid repeated MDX parsing in publish diagnostics (#6407)
## Description

The cold `build.contentDatabasePublishDiagnostics` path was parsing the
same reachable MDX documents while the publication plan converged, then
parsing them a third time to produce omission diagnostics. Content-heavy
projects can approach or exceed the 15-second serverless limit on that
first request.

This change:

- creates a request-scoped MDX dependency resolver that reuses parse
results by asset revision while the publication plan converges;
- carries the final template omissions out of the existing publication
pass, so the diagnostics endpoint does not walk the artifact again;
- keeps the existing sequential asset validation/preparation/stability
checks unchanged;
- does not use module-level caching or depend on a warm serverless
instance;
- leaves the timeout error handling from #6406 unchanged.

## Performance evidence

An isolated cold-path benchmark used the same MDX payload (2,000
sections) and compared the previous three independent
dependency-resolution walks with the new request-scoped convergence
path:

| path | time |
| --- | ---: |
| before | 1,178 ms |
| after | 404 ms |

That is a 66% reduction in the measured MDX resolution phase. The tests
also assert the underlying work reduction directly: an unchanged
revision is parsed once across convergence passes, while a changed
revision is parsed again. For a stable 214-document candidate set, this
removes up to 428 of the previous 642 parse operations.

## Verification

- `pnpm --filter @webstudio-is/project-build test` — 2,420 tests passed
- `pnpm --filter @webstudio-is/builder exec vitest run --project
core-server` — 336 tests passed, including server-side e2e runner tests
- `pnpm --filter @webstudio-is/project-build typecheck`
- `pnpm --filter @webstudio-is/builder typecheck`
- `pnpm lint`
- `pnpm check:package-boundaries`
- `pnpm check:generated-api`

## Code Review

- [x] made a self-review
- [x] added comments for the request-scoped cache and revision safety
- [x] added tests for parse reuse, revision invalidation, and the
optimized endpoint integration path

## Before merging

- [ ] tested on a deployment preview with a production-sized content
project

## Additional content-read optimization

Successive compilation plans now share already-fetched, revision-keyed
content bytes within one callback-scoped preparation session. The byte
store:

- is created inside one asset-stability attempt and discarded when that
attempt finishes;
- is not stored on the repository, in a module, or outside the request;
- remains bounded by the existing 2 MiB hydration limit;
- refetches changed revisions and starts clean on stability retries.

Tests verify that two overlapping plans read unchanged content once, a
revision change forces another read, a new session reads again, and a
stability retry creates a new session. A fully overlapping two-pass
workload therefore performs up to 50% fewer object-storage content
reads.

Additional verification:

- `pnpm --filter @webstudio-is/asset-uploader test` — 292 tests passed
- `pnpm --filter @webstudio-is/asset-uploader typecheck`

## Remaining optimization assessment

The percentages below estimate whole-request improvement and are
workload-dependent; they are not additive.

| optimization | estimated improvement | complexity | decision |
| --- | ---: | ---: | --- |
| Request-local content-byte reuse | 10–35% | 3/10 | Implemented |
| Stop when the initial dependency plan is already a fixed point | 5–15%
for static/known MDX; ~0% for dynamic discovery | 2/10 | Implemented |
| Reuse a canonical source snapshot/materialized entries across
different plans | 15–30% | 8/10 | Deferred: large semantic and
invalidation surface |
| Reuse the bundle's initial asset-data snapshot | 1–5% | 5/10 | Not
worthwhile |
| Add a diagnostics-specific bundle loader | 2–8% | 6/10 | Not
worthwhile for this PR |
| Parallelize or skip unrelated user/project metadata | 0–3% | 2/10 |
Not meaningful |
| Combine collection validation and reserved-asset inspection | 0–5%
normally | 4/10 | Only meaningful for projects with many collections |

The fixed-point fast path is covered by a test asserting one artifact
preparation and one dependency-resolution call when the original plan is
already stable.
2026-09-17 10:47:48 +02:00

80 lines
2.6 KiB
TypeScript

import { parseContentDatabaseMaxBytes } from "@webstudio-is/content-engine";
import { getContentDatabasePublishDiagnostics as analyzeContentDatabasePublish } from "@webstudio-is/project-build";
import type { PublishedProjectBundle } from "@webstudio-is/protocol";
import { resolvePublishedMdxDependencyClosure } from "@webstudio-is/project-build";
import type { PublishedMdxTemplateOmission } from "@webstudio-is/project-build";
import { formatAssetName } from "@webstudio-is/sdk";
import { migratePages } from "@webstudio-is/project-migrations/pages";
export const formatMdxTemplatePublishDiagnostics = (
bundle: PublishedProjectBundle,
issues: readonly PublishedMdxTemplateOmission[]
) => {
const omissions = new Map<
string,
{
assetId: string;
filename: string;
templateName: string;
blockInstanceId: string;
}
>();
const assets = new Map(bundle.assets.map((asset) => [asset.id, asset]));
for (const issue of issues) {
const asset = assets.get(issue.assetId);
omissions.set(JSON.stringify(issue), {
...issue,
filename: asset === undefined ? issue.assetId : formatAssetName(asset),
});
}
return [...omissions.values()];
};
export const getMdxTemplatePublishDiagnostics = async (
bundle: PublishedProjectBundle
) => {
if (bundle.assetIndex === undefined) {
return [];
}
const issues: PublishedMdxTemplateOmission[] = [];
await resolvePublishedMdxDependencyClosure({
build: { ...bundle.build, pages: migratePages(bundle.build.pages) },
artifact: bundle.assetIndex,
onTemplateOmission: (issue) => issues.push(issue),
});
return formatMdxTemplatePublishDiagnostics(bundle, issues);
};
export const getContentDatabaseMaxBytes = () =>
parseContentDatabaseMaxBytes(process.env.CONTENT_DATABASE_MAX_BYTES);
export type ContentDatabasePublishDiagnostics = {
stats: NonNullable<ReturnType<typeof analyzeContentDatabasePublish>>["stats"];
affectedResources: Array<{
name: string;
kind: "dynamic" | "static";
}>;
};
export const getContentDatabasePublishDiagnostics = (
bundle: PublishedProjectBundle
): ContentDatabasePublishDiagnostics | undefined => {
const diagnostics = analyzeContentDatabasePublish({
build: bundle.build,
artifact: bundle.assetIndex,
});
if (diagnostics === undefined) {
return;
}
const resourceNameById = new Map(
bundle.build.resources.map(([, resource]) => [resource.id, resource.name])
);
return {
stats: diagnostics.stats,
affectedResources: diagnostics.queries.map(({ id, kind }) => ({
name: resourceNameById.get(id) ?? id,
kind,
})),
};
};