DataTable v2: sorts become an ordered array (datatables.net model) — plain
header click cycles asc/desc/none, shift+click stacks sort levels with a
priority badge. New useTableSort/useColumnVisibility hooks (optional
localStorage persistence), SortMenu/ColumnsMenu toolbar popovers, and a
shared DataTableFooter (Shown X of Y, page-size segment, load-more, pager).
All backward compatible; existing call sites keep working unchanged.
With the remote index memoized, re-parsing every bundled YAML on each call
was the entire remaining cost of a Templates page load. Entries are now
memoized per file and keyed on (st_mtime_ns, st_size), so a sync or an edit
invalidates exactly the file that changed: measured on the real catalog,
list_local_templates() goes from 304ms to 0.89ms (118 templates), and the
full catalog listing from 1707ms to 1.26ms.
The writers need no invalidation call - mtime keying covers create, overwrite
and delete, the last because the listing only returns what the directory scan
finds. Cache entries are pruned to what is still on disk so a long-running
panel that churns templates cannot grow it without bound.
Failed parses are cached too, keyed on the same stat tuple, so one malformed
YAML no longer costs a parse on every render; it is retried the moment the
file changes. A file that fails to parse still does not claim its id, so a
valid file of the same name in the fallback directory can supply it - pinned
by a test, since that is easy to lose in a refactor.
Entries are deep-copied out: a shallow copy would share the nested
categories/repo/engine containers, so a caller appending to categories would
corrupt every later listing.
Also swaps os.listdir for os.scandir (DirEntry.stat() reuses data the scan
already returned) and sorts directory entries, which makes a same-id
collision (foo.yaml vs foo.yml) resolve identically on every platform instead
of depending on filesystem order.
Both bugs were latent only while DEFAULT_REPOS pointed at a 404. Now that
serverkit.ai proxies a live registry, they bite on every Templates page load.
list_all_templates() concatenated local + remote with no id merge, so every
bundled template was listed twice the moment a real registry answered.
Measured against the live registry: 224 entries for 116 unique ids. It now
merges by id with local winning, matching get_template(), which already
resolves local directories before any repo; among repos, the first enabled
one to claim an id keeps it.
fetch_remote_templates() ran inline on every listing with a 30s timeout and
no memoization, so one unreachable repo stalled the whole catalog on every
render. It is now memoized with a TTL (5m success, 60s failure), a 10s
timeout, and a last-good fallback so a transient blip cannot blank a catalog
the panel has already shown. The last-good payload is carried forward across
repeated failures rather than being overwritten by the first one. The cache
is dropped in save_config(), the single choke point for repo add/remove/
enable (sync_templates routes through it too), and force=True bypasses it.
11 proving tests, including the repeated-failure case and a regression test
that a cached entry cannot be corrupted by a caller mutating it.
pytest-split falls back to splitting by test COUNT when no durations file
exists, and these tests range from 0.01s to 10s+. That produced visibly lopsided
shards on the first sharded CI run — 6m17s for the fastest against 10m46s for
the slowest, and the slowest shard is the whole job's wall clock:
pytest (4/4) 6m 17s
pytest (2/4) 8m 42s
pytest (1/4) 10m 01s
pytest (3/4) 10m 46s <- Backend CI was 11m06s because of this one
With real durations, pytest-split packs the shards to equal TIME instead, which
should pull the slowest shard down toward the ~9m average.
Generated from a full local run: 3175 entries, 20 failed / 3108 passed /
47 skipped in 11m53s. All 20 failures are pre-existing and unrelated (a
serverkit-wordpress checkout absent locally, plus Linux-only fail2ban tests) —
verified by re-running those same files against the pre-plan-64 conftest.
Regenerating this file was only affordable because plan 64 Phase 1 landed
first; before it, a full run was slow enough that Phase 0 abandoned the attempt.
Regenerate it whenever suite timings shift materially:
cd backend && python -m pytest tests --store-durations
Plan 64 Phase 3.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing catches a revert of `_flask_app` to scope='function'. The suite would
still pass — just ~3x slower — and CI would drift back to a 20-minute wait,
which is how it got that way in the first place. Same idea as the
collected-test-count ratchet: turn a silent regression into a red test.
Also asserts the inverse for `app`, which must stay function-scoped: it is what
resets the database between tests, and session-scoping it would leak each
test's rows into the next and fail while pointing at the wrong test.
Verified both directions — flipping _flask_app back to scope='function' fails
the guard with the explanation, and restoring it passes.
Ratchet 3173 -> 3175 for the two new tests.
Plan 64 Phase 4.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The `app` fixture called create_app('testing') for every one of 3173 tests,
registering 90+ blueprints each time. Measured on a fixed 199-test sample
(pytest --splits 16 --group 5 --durations=0):
setup 144.9s 89.5%
call 8.8s 5.4%
teardown 8.2s 5.0%
95% of the suite's time was preparing to test; 5% was testing. The app is now
built once per process and shared.
Same 794-test slice, before and after — identical results, 2.66x faster:
before 4 failed, 773 passed, 17 skipped 593.30s
after 4 failed, 773 passed, 17 skipped 223.11s
(the 4 are pre-existing locally: serverkit-wordpress isn't checked out here.)
Sharing one app surfaces three things a per-test app hid, all handled in the
fixture rather than patched test by test:
1. create_app() is not a pure constructor — it SEEDS the bundled flagship
extension rows as a boot side effect (test_cloudflare_extraction asserts
they are "seeded on boot"). The per-test drop_all wipes them, and a shared
app never re-boots, so every test after the first saw no flagship and the
extension routes answered 503. seed_flagship_extensions() is now re-run
after each create_all. The session fixture also boots twice on purpose:
once to create the schema, once to seed into a schema that exists.
2. Config mutation now leaks where it used to die with the app (e.g.
test_demo_deploys sets DEMO_DEPLOYS_ENABLED=False and never restores it),
so config is snapshotted and restored around every test.
3. Structure mutation cannot be undone at all — Flask has no
unregister_blueprint, and refuses setup methods once an app has served a
request. Those tests opt out via `pytestmark = pytest.mark.fresh_app` and
get a private app. Three groups needed it: 13 modules calling
register_blueprint/add_url_rule directly, 11 that install plugins (
plugin_service hot-loads their blueprints onto the live app), and
test_audit_log which adds routes with @app.route.
Tests using the `wp_extension` fixture are detected from the fixture graph
instead of being marked, since that fixture mounts blueprints on whatever app
it is handed — miss one marker and the failure lands in an unrelated test much
later in the run.
Plan 64 Phase 1. Phase 2 (savepoint rollback instead of create_all/drop_all)
is NOT included: it currently takes the slice from 4 failures to 100 failures
and 201 errors, and needs its own round.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Backend CI was the entire pipeline wait. Measured on the last dev push:
Extensions CI 12s
Version Bump 18s
Test System Utilities 35s
Security Scan 41s
Scripts CI 66s
Release Build Smoke 68s
Backend CI 21m 50s <- everything else finished in a minute
Step timings put all of it in the test run itself (install deps 22s, ratchet
9s), so there is no caching win to take — it is 3173 tests against a
function-scoped `app` fixture that rebuilds the Flask app and all 80+ tables
per test. Fixing that fixture is the real repair and is planned separately;
this change buys the wall-clock back now without touching a single test.
Split over 4 runners via pytest-split (794/794/794/791). Sharding rather than
pytest-xdist is deliberate: each shard is its own VM, so the process-shared
state that makes in-process parallelism unsafe here (templates.json, APPS_DIR
— see the per-PID DB dance in tests/conftest.py) simply isn't shared, and no
test code has to change.
The ratchet moves to its own job because it must see the whole suite; a shard
only collects its quarter. It runs beside the shards and costs no wall-clock.
Two details:
- The shard command is scoped to `tests` rather than a bare `pytest`.
Identical in CI (3173 either way, backend/dev-data/ being gitignored), but a
bare pytest on a dev box tries to collect the locally deployed apps under
backend/dev-data/ and dies during collection. The line is now copy-pasteable
for debugging a red shard locally.
- With no .test_durations file, pytest-split balances by test count, not time,
and these tests range from 0.1s to 10s+. If one shard becomes the new
critical path, run once with --store-durations and commit the file.
main also drops out of `push` here: release.yml gates itself on this workflow
via ci-gate, so listing main ran the whole suite a second time on every merge.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`npm run lint` was defined in frontend/package.json and documented in
CLAUDE.md, but no workflow ever invoked it. Grepping the whole workflow dir for
`npm run lint`/`eslint`/`npm run build` returned nothing, so three project-
specific checkers chained behind eslint were silently unenforced:
check-settings-index every Settings tab has a search-index entry
check-theme-tokens the theme-token whitelist stays in 3-way sync
check-html-sinks every raw-HTML sink is sanitized or annotated (XSS)
Lint currently passes: 926 warnings, 0 errors, and all three checkers green —
so this is safe to add as a blocking check today. eslint exits 0 on warnings,
so the gate is on errors only; add --max-warnings to freeze the count if the
warning drift ever needs a ratchet of its own.
Lint only, no build step: the frontend is already compiled in CI by Release
Build Smoke Test, whose scripts/build-release.sh runs `npm ci && npm run
build`. backend/app/** is in the paths because check-html-sinks scans it too,
for `|safe`, `Markup(` and `render_template_string`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These five workflows triggered on both `push: [dev, main]` and
`pull_request: [dev, main]`, so one change was validated up to three times: on
the dev push, again on the dev->main PR, and a third time on the post-merge
push to main.
Narrow them to `push: [dev]` + `pull_request: [main]`:
- `pull_request: [dev]` was dead config. Every PR this repo has had targets
main (checked back through #77, dependabot's included), so it never produced
a run.
- `push: [main]` was redundant for release-smoke: release.yml's build-release
job runs that same scripts/build-release.sh for real, on the same commit,
moments later.
Also drops test-system-utils' `unit-tests` job. It ran
`pytest tests/test_utils_system.py` (44 tests) that Backend CI's bare `pytest`
already collects — there is no pytest.ini, addopts or collect_ignore narrowing
collection. The distro matrix and the raw-subprocess audit stay; those are the
parts Backend CI genuinely cannot do.
Coverage of main is unchanged: a pull_request run tests the merge result, and
release.yml still gates itself on the full backend suite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
10 from the template repo/sync suite added alongside the DEFAULT_REPOS fix;
the rest accumulated since the last ratchet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reasoning about a redirect validator is how they ship broken, so this was
driven by a probe that asks the URL parser directly: for ~50 candidate
strings, does `new URL(candidate, base)` leave the origin, and does
sanitizeRedirect agree? Zero leaks before or after -- the origin property was
never broken. What the probe did surface was three ways to be wrong that are
not origin escapes:
1. Dot segments were accepted. `/..//evil.com` normalizes to `//evil.com`,
which stays same-origin only because it is resolved against a base;
assigned to window.location.href it leaves the site. Rather than audit
every present and future consumer of the stored value, the gadget is now
refused outright -- plain and percent-encoded, in any position. A dot
inside a segment name (/files/.env) is untouched.
2. Auth routes were matched as exact strings, so /login/ and, worse,
/login/callback/<provider> got through -- the latter re-runs the SSO
callback with no code and lands the user on an error page right after a
successful login. Now a prefix match on a "/" boundary, case-insensitive,
which still leaves /logins and /login-help as valid destinations.
3. No expiry. A destination parked early in a tab session fired on any later
login in that tab: park an install link, abandon it, log in normally half
an hour later, get taken somewhere you had forgotten about. Now stamped and
good for 30 minutes -- long enough for a password manager, a TOTP prompt,
or an SSO round trip that includes signing up at the provider.
Also a 2048-char cap, and consume now refuses a missing, non-numeric or
future timestamp rather than trusting it: rememberRedirect is the only writer,
so anything else did not come from this module.
Tests go 9 -> 19, and now cover the remember/consume round trip (single use,
expiry, tampered storage, storage unavailable) against a fake sessionStorage
so the whole module still runs under plain node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DEFAULT_REPOS pointed at raw.githubusercontent.com/serverkit/templates --
the org `serverkit` does not exist (the registry is jhd3197/serverkit-templates,
and its default branch is master, not main). So it has 404'd for its entire
existence: no panel has ever fetched a template remotely, and every /templates
page has been showing only the 118 bundled backend/templates/*.yaml.
Now points at https://serverkit.ai/templates, which proxies the registry and
was built for this consumer -- it serves <repo_url>/index.json and the
<repo_url>/templates/<id>.yaml path this class derives, behind a TTL cache
with last-good fallback. Using the product domain also means a branch rename
upstream cannot silently empty every panel's catalog, which is the exact
failure this is fixing.
Fixing the default alone would only help fresh installs, because get_config
returns whatever a saved templates.json holds. Known-dead URLs are therefore
healed on read. Nothing is lost by rewriting a URL that never once resolved.
The checksum half is here rather than in a follow-up because this commit
ACTIVATES a download path that has never run: sync_templates wrote whatever
came back straight to disk, ignoring the sha256 the index pins for every one
of the 106 official entries. Turning on an unverified fetch days after the
panel learned to verify ed25519 signatures on extensions would be shipping a
known downgrade. Mismatch is now a hard refusal that never reaches disk;
absent hash is allowed and counted, mirroring unsigned-vs-invalid for
extensions. Content is written as bytes so the file on disk is exactly what
was hashed.
Note get_template's in-memory remote read (it parses without saving) is not
covered -- it has no index entry on hand and would need an extra fetch.
DEPLOY ORDER: serverkit.ai must ship the master-branch fix before this helps.
Until then the proxy 502s and sync finds nothing, same as today.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The counterpart to Templates.jsx's ?install=<id>, which has worked for a
while; the extensions page never grew one, so serverkit.ai install links and
README badges had nowhere to land.
It opens the extension's detail modal rather than installing. A URL arriving
from another site must not be able to install anything on its own -- the
operator still presses Install, and the trust gates behind it (unreviewed,
unverified checksum, untrusted publisher key) all still fire exactly as they
do from a card click. An unknown slug says so instead of failing silently.
Deliberately reads the param on /extensions and not /marketplace: the old
path resolves through a <Navigate>, which drops the query string, so a link
to /marketplace?install=x would arrive with nothing to act on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PrivateRoute redirected to /login with no record of where the visitor was
going, and every post-auth path then navigated to '/'. Any deep link opened
without a session was silently discarded -- you landed on the dashboard with
no explanation of why the thing you clicked did not happen.
That was survivable when deep links were rare. It is not now: serverkit.ai
install links (/extensions?install=<slug>, /templates?install=<id>) arrive
from README badges, and by construction they are clicked by people who may
have no open session. Every first-time click hit this path.
The destination goes in sessionStorage rather than react-router location
state because the SSO flow leaves the origin entirely for the identity
provider and returns through /login/callback/<provider>; router state cannot
survive that. One mechanism covering all four post-auth exits (login link
redemption, password, 2FA, SSO) beats two each covering half.
sanitizeRedirect is the pure half and is validated on write AND on read.
Login is exactly where an open redirect hurts, so `//evil.com` and
`/\evil.com` -- both of which browsers read as protocol-relative -- are
rejected along with control characters and the auth routes themselves,
which would loop. Proving test: node --test
src/utils/__tests__/redirectAfterLogin.test.mjs (9 tests).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Piped execution leaves BASH_SOURCE unset, so the sourceability guard's bare
${BASH_SOURCE[0]} was an unbound variable under `set -u` and aborted the
install before main() ever ran:
bash: line 2107: BASH_SOURCE[0]: unbound variable
This broke the one-liner README/DEPLOYMENT document, on every distro — not
just the Debian 13 it was reported on (reproduced on debian:13 bash 5.2.37,
ubuntu:22.04 bash 5.1.16, almalinux:9 bash 5.1.8). CI never saw it because
every job runs `bash install.sh` from a checkout, where BASH_SOURCE IS set.
Use the same non-empty-first guard scripts/install.sh already carries, and
add the proving test the panel installer was missing (T41) — the agent
installer has had its twin since the same fix landed there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The full WordPress suite now lives at github.com/jhd3197/serverkit-wordpress
as a standalone runtime-ESM extension (backend blueprints/services, game
dashboard UI, tests, CI). Installed from the registry or offered during
onboarding from the bundled snapshot. WP data models stay core per D1
(documented core-data-seam).
Runtime-ESM extensions (starting with serverkit-wordpress) need slot
mounting and error containment from the host; additive, versioned in
lockstep frontend/backend.
Create/update validation fails loudly naming the missing extension
instead of orphaning monitors; health sweep no-ops with zero WP sites
(plan 52 phase 3 task 17). StatusComponent.wordpress_site_id stays core
per D1.
The wordpress template kind + wordpress-external-db preflight move behind
a template-provider hook the extension supplies; the backend filters the
catalog so the WP card simply does not appear when WP is absent — no
frontend change needed (plan 52 phase 3 task 16).
Extensions register event types at load; wordpress.* types now come from
the WP extension instead of a hardcoded core catalog (plan 52 phase 3
task 15). emit_wp stays core as a dumb emitter helper.
VALID_TARGET_TYPES is now provider-driven; the wordpress_site branch
moves behind a provider the WP extension registers. ProtectionPanel /
RestoreDrawer read available types from the API; policies whose provider
is absent surface a clear missing-provider state instead of crashing
(plan 52 phase 3 task 13).
Extensions can register providers into core engines at load (backup
targets, event types, template kinds) via a core_hooks module; idempotent,
process-lifetime like blueprint registration. WP extension supplies its
entries through builtin-extensions/serverkit-wordpress/backend/core_hooks.py
(plan 52 phase 3, D4).
App.jsx /wordpress* title fallbacks removed (manifest page_titles cover
the routes); sidebar preset hiddenItems no longer hardcode the wordpress
slug — nav sources from extension contributions.
5 WordPress* pages + components/wordpress (40 files) + 3 _wordpress*.scss
partials move from core into builtin-extensions/serverkit-wordpress/
frontend/ (synced to frontend/src/plugins via the baked-glob pattern).
200 imports rewritten to serverkit-sdk; remaining @/ imports only for
genuinely host-internal modules. main.scss drops the WP partials — the
extension bundle imports its own styles. npm build + lint green.
TabGroupLayout, Modal, ConfirmDialog, Spinner/LoadingState, EmptyState,
Skeleton(SkeletonBoundary), and the ui/* form kit (Button, Input, Select,
Switch, ...) as generic re-exports so an extension page can mirror core
page chrome without deep imports. Proven out by the WordPress frontend
relocation (plan 52 phase 4).
Unknown categories already derived/filtered gracefully; this adds
games to the canonical order and gives it a lucide Gamepad2 icon
ahead of serverkit-minecraft landing in the registry.
The four minecraft test files now live in serverkit-minecraft/tests/
(panel-hosted via SERVERKIT_MINECRAFT_DIR loader). Ratchet floor
3280 -> 3197 (-83 minecraft tests; collected 3222, floor intact).
The Minecraft game-server extension now lives at
github.com/jhd3197/serverkit-minecraft as a standalone runtime-ESM
extension (analytics/gpu repo pattern). It is installed from the
registry like any first-party extension, no longer bundled in-tree.
scripts/sign-extension.mjs signs a bundle with an ed25519 author key;
EXTENSIONS.md documents the format, key model, and verdict behaviors.
Generated private keys live in gitignored scripts/keys/.
Consent card shows permissions, version gate, and signature state
(Signed/Unsigned/Unknown signer/Invalid) with honest copy; 'Install
anyway' destructive path for acknowledged risk; invalid = disabled.
Juan Denis