Files
Oleg Isonen 55920c57c4 fix: Show all Markdown and Assets query diagnostics (#6308)
Ref #6305

## Outcome

Show actionable Markdown, MDX, and Assets query problems in the editor,
query Diagnostics, API, and MCP.

The current implementation substantially improves source diagnostics,
query error aggregation, query scoping, exact paths, MCP feedback, and
loading states. The exhaustive audit below also records what is not
complete yet.

## Completion rule

An error state is complete only when all applicable checks pass:

- [x] The editor uses the production validator and shows every issue for
the open file.
- [x] Query execution uses the same validation and returns every issue
for every file that matches the query, not only the returned page.
- [ ] MCP can reproduce the state through a real tool call and returns
the same code, severity, message, file, and location.
- [x] Diagnostics shows fatal errors and nonfatal warnings in the
correct place.
- [x] Exact file is always present; line and column are present whenever
the parser can determine them.
- [x] Multiple issues in the same file, multiple files, and multiple
validation phases are returned together.
- [ ] A regression test proves failure without the behavior and success
with it.

## Cross-surface contract

### Required validator parity

- [x] Editor, query execution, and MCP file operations invoke the same
exported production file-validation pipeline for byte decoding,
source/frontmatter, Markdown/MDX, Content Block semantics, and
references. Copied schemas, adapter-only checks, or reimplementations do
not count as parity.
- [x] Query builder, query execution, `validate-asset-query`, and
`preview-asset-query` invoke the same exported structural and
catalog-aware semantic query validator with the same project snapshot
and context.
- [x] Every surface preserves the same complete, deterministically
ordered diagnostic array: severity, code, message, phase/scope, asset
ID, file path, line/column/range, query path, and
fatal-versus-warning/result behavior. UI, API, and MCP serializers must
not drop or remap fields.
- [ ] A table-driven contract suite calls the real editor, query, API,
and MCP entry points—including MCP upload, update, download, validate,
and preview—and compares full normalized arrays for multiple errors,
multiple warnings, multiple files, and pre-pagination matches.

- [x] Markdown and MDX source syntax/frontmatter use
validateTextAssetSource in editor, query compilation, and MCP file
tools.
- [x] Query request execution and MCP local parsing use
assetQueryRequest.
- [x] Fatal query setup errors retain all detected semantic errors and
accompanying warnings.
- [x] Nonfatal query warnings retain a stable code and exact query path.
- [x] Source diagnostics retain severity, phase, code, asset ID, path,
and available source position.
- [x] Query Diagnostics excludes diagnostics produced only by unrelated
published queries.
- [x] Query, Preview, and Diagnostics tabs show loading states.
- [x] Authenticated diagnostics can show a sanitized internal message;
public/pre-auth errors remain redacted.
- [x] Share byte-level UTF-8 validation instead of decoding separately
in query and MCP.
- [x] Make the query editor and MCP call one literal shared
query-validation function before execution.
- [x] Share cross-file reference validation across editor, query, and
MCP file operations.
- [x] Share contextual Content Block/template validation across all
applicable surfaces.
- [x] Preserve an invalid stored query in the editor and explain it
instead of replacing it with defaults.
- [x] Define and enforce whether “matches the query” includes every
pre-pagination match; current source validation follows
returned/paginated IDs.
- [x] Continue through source, reference, and runtime phases where safe
so one phase does not hide later issues.
- [x] Map every user-triggerable graph/storage/runtime failure to a
structured public error instead of INTERNAL_ERROR.
- [ ] Label candidate-file warnings separately from returned-file
diagnostics.

## File transport and encoding

- [x] Empty Markdown and MDX are accepted.
- [x] UTF-8 BOM and CRLF are accepted.
- [x] Whole Markdown/MDX file over 1 MiB is detected.
- [x] Missing or unavailable selected content is reported.
- [x] Invalid UTF-8 frontmatter returns the same structured diagnostic
in query and MCP.
- [x] Invalid UTF-8 body returns the same structured diagnostic in query
and MCP.
- [x] MCP download must use fatal UTF-8 decoding rather than replacement
characters.
- [x] Validate bytes before committing an upload, or return the
committed Asset ID and explicit cleanup state when post-commit feedback
fails.
- [x] Reject filename/format mismatches instead of allowing either hint
to bypass Markdown/MDX validation.
- [x] Storage status, missing body, invalid metadata, revision mismatch,
format mismatch, short read, and excess read each return structured file
context.
- [x] Content changing during validation has a clear retryable conflict
diagnostic.

## Frontmatter

- [x] No frontmatter.
- [x] Empty frontmatter.
- [x] Opening delimiter without a closing delimiter.
- [x] Frontmatter over 64 KiB.
- [x] Invalid YAML grammar.
- [x] Multiple YAML parser errors.
- [x] Duplicate keys.
- [x] YAML aliases.
- [x] Unsupported YAML tags.
- [x] Scalar or array root instead of an object.
- [x] Non-JSON values, NaN, Infinity, and unsafe integers.
- [x] Nesting deeper than 8.
- [x] More than 256 fields/array entries.
- [x] One string over 16 KiB.
- [x] Normalized indexed properties over 64 KiB.
- [x] Frontmatter warnings remain nonfatal and visible.
- [x] Collect all post-parse normalization/limit errors; this phase
currently stops at the first.
- [ ] Prove every state above through real MCP upload, update, download,
query, and Diagnostics paths.

## Markdown body

- [x] Treat ordinary CommonMark/GFM as permissive; malformed-looking
Markdown and raw HTML are not automatically syntax errors.
- [x] Preserve and validate the body when frontmatter is invalid.
- [x] Detect the 1 MiB body/file limit.
- [x] Handle body extraction and bounded excerpts.
- [x] Return a shared structured invalid-UTF-8 body error from every
byte-reading surface.
- [ ] Distinguish author errors from internal
excerpt-limit/configuration errors.
- [ ] Prove body-size and decoding states through real MCP and
Diagnostics.

## MDX grammar and safety

- [x] Malformed MDX/JSX grammar.
- [x] Imports.
- [x] Exports.
- [x] Executable flow expressions.
- [x] Executable text expressions.
- [x] JSX expression attributes.
- [x] JSX spread attributes.
- [x] Arbitrary components.
- [x] HTML-looking JSX outside ws.element.
- [x] Unsupported/raw MDX node types.
- [x] Duplicate attributes and normalized alias collisions.
- [x] Event-handler attributes.
- [x] srcdoc, srcset, and dangerouslySetInnerHTML.
- [x] Unsafe URL protocols in link/source/action-like props.
- [x] Forbidden tags: base, link, meta, noscript, script, style,
template, title.
- [x] Conflicting, empty, or non-static ws:name and ws:tag.
- [x] More than 20,000 nodes.
- [x] Nesting deeper than 100.
- [x] More than 4,000 props.
- [x] Multiple recoverable unsafe-mdx warnings are collected.
- [ ] Collect multiple fatal grammar/structural errors when parser
recovery can do so; unrecoverable parsing currently stops at the first
fatal error.
- [x] Preserve sibling expression warnings when unsupported JSX is also
present; {first()}, <Card />, {second()} currently reports only the Card
warning.
- [ ] Prove every state above through real MCP and Diagnostics, not only
validator unit tests.

## Content Block semantics

- [x] Unresolved template warning exists.
- [x] Unknown template prop warning exists.
- [x] Incompatible template prop warning exists.
- [x] Design-only template prop warning exists.
- [x] Stale template prop warning exists.
- [x] Invalid materialized HTML content model error exists.
- [x] Run these checks for an MDX file even when no active editor
Content Block root has materialized it.
- [x] Run applicable semantic checks in query Diagnostics.
- [x] Run applicable semantic checks for MCP upload as well as
contextual update/download.
- [x] Return all Content Block frontmatter diagnostics; one path still
throws only diagnostics[0].
- [ ] Add real MCP and UI coverage for every semantic state.

## Cross-file references

- [x] Non-string $ref marker.
- [x] Invalid reference URL.
- [x] Invalid fragment encoding.
- [x] Invalid fragment.
- [x] Invalid JSON Pointer.
- [x] Missing referenced document.
- [x] Missing referenced JSON path.
- [x] Missing referenced local Asset warning.
- [x] Unsupported target representation.
- [x] Reference cycle.
- [x] Duplicate document ID or URL.
- [x] Missing graph source or target.
- [x] Root not found.
- [x] Duplicate/unused/unresolved reference.
- [x] Invalid resolved value.
- [x] Content identity conflict.
- [x] Target revision mismatch.
- [x] Representation selector required/invalid.
- [ ] Diagnose {$ref: value, extraKey: value} as an invalid marker or
explicitly document it as ordinary data.
- [x] Collect all bad references and targets instead of stopping at the
first graph failure.
- [x] Validate references from the editor and MCP file tools, not only
query compilation.
- [x] Map graph compilation/resolution, Markdown-document, and
JSON-document errors to exact structured Diagnostics rows.
- [x] Prevent an unrelated file's graph error or duplicate canonical URL
from blocking an exact-file query.
- [ ] Prove missing targets, cycles, bad representations, and bad JSON
paths through real MCP.

## Query request and structure

- [x] Invalid JSON, invalid request UTF-8, empty/non-object request, and
request over 512 KiB.
- [x] Missing query and unknown request/query keys.
- [x] Invalid result mode.
- [x] Malformed where/all/any/filter shape.
- [x] More than 32 filters or children.
- [x] Filter nesting deeper than 8.
- [x] Empty field path or segment.
- [x] Field path deeper than 9.
- [x] Bare properties path.
- [x] Unknown or nested standard field.
- [x] Missing or unknown operator.
- [x] Non-JSON operator value.
- [x] Invalid in value or more than 1,000 in values.
- [x] Non-boolean exists/isEmpty value.
- [x] More than 8 sort entries.
- [x] Invalid sort field or direction.
- [x] first/last without a sort.
- [x] Invalid output mode or includeMetadata.
- [x] Duplicate output fields or more than 256 fields.
- [x] No selected output.
- [x] Negative, fractional, or over-1,000 limit.
- [x] Negative, fractional, or over-1,000 offset.
- [x] Invalid content mode.
- [x] Invalid full/markdown-body-ref maxBytes outside 1..1 MiB.
- [x] Invalid range offset/length or range over 256 KiB.
- [x] Empty or over-255-character index revision.
- [x] Multiple simultaneous schema issues retain exact query paths.
- [x] Make content option objects strict; unknown keys are currently
stripped.
- [x] Show an invalid persisted resource body rather than silently
loading the default query.
- [x] Aggregate unknown-key errors with every other structural issue;
MCP currently stops at the first unsupported key.
- [x] Add field-value type validation, for example reject size gt
"large" and id contains {}.
- [x] Warn about unobserved dynamic fields selected only as output.
- [ ] Prove the full structural matrix through the MCP server/API end to
end; the current broad matrix tests the local adapter.

## Query semantic states

- [x] INCOMPATIBLE_OPERATOR is fatal for standard fields.
- [x] UNOBSERVED_FIELD is a nonfatal warning.
- [x] INCOMPATIBLE_OBSERVED_TYPES is a nonfatal warning.
- [x] STRUCTURED_SORT_ORDER is a nonfatal warning.
- [x] Every incompatible standard-field operator is collected.
- [x] Fatal semantic errors retain nonfatal warnings.
- [x] Successful warnings appear with exact query paths.
- [ ] Reproduce every catalog-dependent semantic state through real MCP.
- [x] Test repeated warnings at distinct query paths without accidental
deduplication.

## Query execution and hydration

- [x] Stale index revision.
- [x] result one matching multiple files, including matchedCount.
- [x] Candidate document limit of 1,000.
- [x] Result payload limit of 16 MiB.
- [x] Hydrated file count limit of 20.
- [x] Per-file hydration limit of 1 MiB.
- [x] Total hydration limit of 2 MiB.
- [x] Range limit of 256 KiB.
- [x] Binary content requested as text.
- [x] Invalid UTF-8 during hydration.
- [x] Missing, stale, or conflicting content identity.
- [x] Canonical metadata/byte-count mismatch.
- [x] Storage returning more bytes than requested.
- [x] Missing content reader.
- [x] Selected content not returned.
- [x] Runtime Asset URL unavailable.
- [x] Document graph document-count/per-file/total limits.
- [x] Classify candidate-document and result-byte limits as
CONTENT_LIMIT_EXCEEDED instead of INVALID_REQUEST.
- [x] Give unavailable URL, missing reader/content, storage, selection,
and assembly failures stable codes and exact asset/file details instead
of INTERNAL_ERROR.
- [x] Never silently turn an exact-file unavailable-content failure into
an empty successful result.
- [x] Add direct regression tests for duplicate identity, excess bytes,
and canonical byte mismatch.
- [ ] Reproduce every runtime/hydration state through real MCP.

## Scoping and aggregation

- [x] Errors and warnings from multiple loaded result files are returned
together.
- [x] Nonfatal errors do not suppress successful query results.
- [x] Unrelated published-query diagnostics do not replace current-query
diagnostics.
- [x] File path and available line/column are shown.
- [x] Exact nested query paths are shown.
- [x] Validate and report every pre-pagination matching file if that is
the product definition.
- [ ] Explain candidate metadata warnings when invalid metadata prevents
deciding whether a file matches.
- [ ] Stop including METADATA_PREPARATION_FAILED globally unless the UI
labels its candidate scope.
- [x] Aggregate reference errors instead of stopping at the first.
- [x] Aggregate across validation phases where safe.
- [x] Add a matrix for same-file multiple errors, multiple files,
outside-page errors, filter-only candidate files, and unrelated queries.

## Transport, authorization, cancellation, and internal failures

- [x] UNAUTHORIZED and FORBIDDEN.
- [x] REQUEST_CANCELLED in query HTTP paths.
- [x] Sanitized authenticated internal message.
- [x] Generic unauthenticated/public internal message.
- [x] Preserve structured causes for document-load HTTP
status/body/metadata failures.
- [x] Preserve structured causes for object-storage read failures.
- [x] Preserve structured causes for graph selection/assembly failures.
- [ ] Test MCP/API cancellation directly.
- [ ] Reproduce authorization, cancellation, storage, and
sanitized-internal states through real MCP where safe.

## MCP reproduction

- [x] Real MCP reproduction: multiple Markdown YAML errors.
- [x] Real MCP reproduction: multiple simultaneous structural query
errors.
- [x] MCP update returns Markdown and MDX source diagnostics.
- [x] MCP upload returns Markdown and MDX source diagnostics.
- [x] MCP download returns Markdown and MDX source diagnostics.
- [x] Temporary remote fixtures are removed after reproduction.
- [ ] Real MCP reproduction for every frontmatter code.
- [ ] Real MCP reproduction for every MDX fatal/warning class.
- [ ] Real MCP reproduction for every Content Block semantic diagnostic.
- [ ] Real MCP reproduction for every reference/graph error.
- [ ] Real MCP reproduction for every semantic query warning/error.
- [ ] Real MCP reproduction for every hydration/runtime/limit error.
- [ ] Real MCP reproduction for auth, cancellation, storage, and
internal errors where safe.
- [ ] A durable table-driven test proves code/severity/message/path
parity between editor, query, Diagnostics, and MCP.
- [x] Preserve stable top-level error identity and structured details
through MCP instead of flattening known errors to MCP_TOOL_FAILED.
- [x] Prove failed post-commit MCP feedback reports whether a mutation
committed and returns enough identity for cleanup.

## Real MCP recheck and follow-up fixes (2026-09-03)

Ran the branch CLI against the PR preview backend. The first run exposed
that the saved CLI origin pointed to a different backend, so those
results were discarded. Every MCP result below came from the PR preview.
All temporary remote and local fixtures were removed afterward. The
failures found in that run were then fixed and covered by regression
tests.

- [x] Exact broken Markdown query returns FRONTMATTER_INVALID with
severity, filename, message, line, and column.
- [x] Exact MDX query returns unsafe-mdx with severity, filename,
message, line, and column.
- [x] A no-extension query returns warnings from both matching Markdown
and MDX files.
- [x] A Markdown-only query excludes the unrelated MDX warning.
- [x] Two fatal MDX files inside the returned page are both reported;
matching warnings remain attached.
- [x] Structural query validation returns three independent errors with
exact paths in one call.
- [x] Semantic query validation returns three fatal operator errors and
two warnings with exact paths in one call.
- [x] MCP upload, update, download, and exact-file query return multiple
source diagnostics for the same MDX source.
- [x] Query preview reports a missing Asset reference as ASSET_NOT_FOUND
with the source file and reference.
- [x] All temporary MCP fixtures were deleted and the saved CLI
connection was restored.
- [x] Fixed after recheck: validate every matching file before
pagination, including broken files outside the returned page.
- [x] Fixed after recheck: unrelated graph failures no longer block an
exact-file query.
- [x] Fixed after recheck: update, download, editor, and query paths use
shared reference validation.
- [x] Fixed after recheck: graph failures map to structured diagnostics
with the exact file.
- [x] Fixed after recheck: shared fatal UTF-8 validation runs before
upload writes and preserves structured error identity.
- [x] Fixed after recheck: MCP rejects filename/format mismatches.
- [x] Fixed after recheck: MDX validation keeps sibling and nested
warnings around unsupported components.
- [x] Fixed after recheck: MCP preserves stable top-level error codes
and structured details.
- [x] Fixed after recheck: query and MCP serializers preserve full
normalized diagnostic fields.
- [x] Fixed after recheck: unknown keys are aggregated with all other
structural query errors.
- [x] Fixed after recheck: shared query validation covers literal types,
output-only fields, and strict content options.
- [x] Fixed after recheck: unavailable selected content returns an
exact-file diagnostic instead of a silent empty result.

## Tests and delivery

- [x] Full Builder suite: 2,707 tests.
- [x] Content Engine suite: 445 tests.
- [x] Asset uploader suite: 211 tests.
- [x] Project runtime/MCP targeted suite: 128 tests.
- [x] CLI/MCP suite: 1,051 tests.
- [x] Relevant package typechecks, lint, generated docs, and diff
checks.
- [x] Documentation impact reviewed and generated Content Engine/MCP
reference updated.
- [x] Add the missing cross-surface error matrix described above.
- [ ] Demonstrate red/green behavior for every new matrix case.
- [ ] Run the complete agent evaluation suite after explicit approval.
- [ ] Re-run all checks after the remaining checklist is implemented.

## Compatibility and risk

Normal query result and expression-binding shapes remain unchanged.
Diagnostics add optional fields. Existing public/pre-auth error
redaction remains intact. The remaining unchecked items are broader
exhaustive real-MCP coverage, parser-recovery limits, candidate-warning
labeling, cancellation coverage, and agent evaluations.
2026-09-04 16:00:28 +02:00

148 lines
4.9 KiB
TypeScript

import { assetQueryPreviewDiagnostics } from "@webstudio-is/content-engine";
import type { ResourceRequest } from "@webstudio-is/sdk";
import { isAssetsResourceRequest } from "@webstudio-is/sdk/runtime";
import { z } from "zod";
export const resourcePerformance = z.object({
serverDurationMs: z.number().nonnegative().optional(),
loaderDurationMs: z.number().nonnegative().optional(),
responseBytes: z.number().int().nonnegative().optional(),
assetQuery: z
.object({
phases: z
.object({
authorization: z.number().nonnegative().optional(),
buildPlan: z.number().nonnegative().optional(),
repositoryAuthorization: z.number().nonnegative().optional(),
sourceSnapshot: z.number().nonnegative().optional(),
canonicalMetadata: z.number().nonnegative().optional(),
compilerEntries: z.number().nonnegative().optional(),
compilerContentRead: z.number().nonnegative().optional(),
documentGraph: z.number().nonnegative().optional(),
documentGraphContentRead: z.number().nonnegative().optional(),
assetReferences: z.number().nonnegative().optional(),
sourceValidation: z.number().nonnegative().optional(),
artifactCompilation: z.number().nonnegative().optional(),
indexPreparation: z.number().nonnegative().optional(),
diagnosticsPreparation: z.number().nonnegative().optional(),
runtimeAssets: z.number().nonnegative().optional(),
documentResolution: z.number().nonnegative().optional(),
})
.optional(),
compilationCache: z
.enum(["hit", "coalesced", "miss", "disabled"])
.optional(),
resolvedDocumentCount: z.number().int().nonnegative().optional(),
documentFetchCount: z.number().int().nonnegative().optional(),
compilerContentFetchCount: z.number().int().nonnegative().optional(),
compilerContentBytes: z.number().int().nonnegative().optional(),
documentGraphContentFetchCount: z.number().int().nonnegative().optional(),
documentGraphContentBytes: z.number().int().nonnegative().optional(),
})
.optional(),
});
export type ResourcePerformance = z.infer<typeof resourcePerformance>;
type ResourceDiagnosticsSchemaIssue = {
code: string;
path: string[];
message: string;
};
export const createResourceDiagnosticsResponseError = ({
message,
issues,
}: {
message: string;
issues: readonly ResourceDiagnosticsSchemaIssue[];
}) => ({
ok: false as const,
status: 500,
data: {
error: {
code: "INVALID_DIAGNOSTICS_RESPONSE",
message,
retryable: true,
details: {
issues: issues.map((issue) => ({
...issue,
severity: "error" as const,
scope: "diagnostics" as const,
})),
},
},
},
});
type SeparatedResourceDiagnostics = {
result: unknown;
diagnostics?: z.infer<typeof assetQueryPreviewDiagnostics>;
diagnosticsError?: ReturnType<typeof createResourceDiagnosticsResponseError>;
performance?: ResourcePerformance;
};
// Assets diagnostics may be shown by Builder, but must never enter the value
// exposed to expressions. Ordinary APIs may legitimately use the same field.
export const separateResourceDiagnostics = ({
request,
result,
}: {
request?: ResourceRequest;
result: unknown;
}): SeparatedResourceDiagnostics => {
if (typeof result !== "object" || result === null) {
return { result, diagnostics: undefined, performance: undefined };
}
const { __performance__, ...resultWithoutPerformance } = result as Record<
string,
unknown
>;
const performance = resourcePerformance.safeParse(__performance__).data;
if (
request === undefined ||
isAssetsResourceRequest(request) === false ||
"__diagnostics__" in resultWithoutPerformance === false
) {
return {
result: resultWithoutPerformance,
diagnostics: undefined,
performance,
};
}
const { __diagnostics__, ...resourceResult } = resultWithoutPerformance;
const parsedDiagnostics =
assetQueryPreviewDiagnostics.safeParse(__diagnostics__);
if (parsedDiagnostics.success === false) {
const issues: ResourceDiagnosticsSchemaIssue[] = [];
for (const issue of parsedDiagnostics.error.issues) {
const path = ["__diagnostics__", ...issue.path.map(String)];
if (issue.code === "unrecognized_keys") {
for (const key of issue.keys) {
issues.push({
code: issue.code,
path: [...path, key],
message: issue.message,
});
}
continue;
}
issues.push({ code: issue.code, path, message: issue.message });
}
return {
result: resourceResult,
diagnostics: undefined,
diagnosticsError: createResourceDiagnosticsResponseError({
message: "Resource diagnostics response is invalid",
issues,
}),
performance,
};
}
return {
result: resourceResult,
diagnostics: parsedDiagnostics.data,
performance,
};
};