A name used before it exists only fails when that line runs, so a
rarely-taken branch ships broken. Two routes 500'd on every call and
Stop crashed on compose-slot apps, both found by pyflakes in seconds.
The check now runs in the ratchet job and in preflight.
Opening a folder while the previous listing was still loading let the
slow answer land last and set the current path back, pulling the user
into the folder they had just left. Only the newest browse may now
write.
The API now answers 503 when docker ps fails instead of an empty list,
so the Containers page no longer says "No containers" on a host
running many. The Containers and Images tabs show that error, and a
late answer from the previously selected server can no longer replace
the current server's list. Also fixes the local branch of the remote
Docker listing, which passed all= to a parameter named all_containers
and always failed.
Changing the line count, clearing a search or picking a journal unit
re-fetched through a closure from the previous render, so the request
used the old count, the old search pattern or the previous unit. The
loaders now take the new values explicitly. Also drops a stray brace
that kept the selected journal unit from ever highlighting.
CONTRIBUTING asks contributors to target dev, but every workflow only
triggered on PRs into main, so contributor PRs merged with no checks
and failures surfaced only after the push to dev.
scrub() walked element children only, so comments, processing
instructions and CDATA passed through. They survive XML serialization
but close early when React re-parses the string as HTML (<!-->, <?x >),
so a registry or plugin icon could smuggle out an <img onerror> and run
script with the tokens in localStorage. Keep only allowed elements and
text; a browser regression runs eight bypass payloads in CI.
Full VM installs and Test Sandbox full mode now record a short
post-install runtime observation (60s settle + 120s sampling) with an
explicit measured/unavailable/failed/interrupted status that never
rewrites install success. Adds the standalone scripts/measure-runtime.py
sampler, the measure-installed-runtime.py harness wrapper, E2E report
rendering, results-API summaries, CI coverage and METRICS.md accounting
boundaries.
The smoke job already builds the release tarball on every push to dev and
threw it away. It now names the build v<VERSION>-dev.<sha7>, writes that
into VERSION so the installed panel reports it, and publishes the tarball
and checksums as a GitHub pre-release. The three newest dev builds are
kept; older ones and their tags are removed. Pull-request runs are
unchanged. The ServerKit Cloud release lab lists these rows as dev builds
and can install and upgrade-test a commit before it is promoted to main.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VndLQNwYw4vYNxQhAFosHY
Generate all four README tables from reviewed source and build snapshots, distinguish collection and asset sizes from runtime results, and remove unsupported footprint claims. Add a consistency check in CI and document how to reproduce profiling and measurements.
Centralize backup schedule calculations and shared request/form flows, encode query values, and move repository finalization into its service. Add browser CI coverage, repair stale test fixtures, and provide opt-in request and build measurement tools.
Schedule dependency and extension scans, audit production frontend dependencies, and replace category suppressions with reviewed function-specific exceptions. Preserve full reports and test the exception gate.
Expose MFA and passkey enrollment, use shared confirmations and locale formatting, and guard pending user and invitation actions. Add Chromium regressions for both themes and failure states to frontend CI.
Copilot review on #135:
- `_repo_steps()` now falls back to port 3000 when `params.port` is not
numeric instead of raising ValueError mid-stream.
- `params.pace` must be a positive number; zero, negative and non-numeric
values are ignored so `line_delay` can never go negative.
CI: the ubuntu integration job sat in "Install Python & pytest" for ~2 h on
a wedged apt mirror, and Version Bump waits on every sibling. Cap the job at
15 minutes and give apt retries, a fetch timeout and non-interactive dpkg
conffile options.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVRnRXqNMZagqQ3RTqG5pz
test_buildpacks.py asserts on Dockerfile TEXT, which cannot see a bug that
only fails at build or boot time -- the `--omit=dev` dead fallback shipped
`vite: not found` to a user's deploy. New @docker_builds leg (gated on
SERVERKIT_DOCKER_BUILDS=1, sibling of @real_binaries) runs the full path
per stack: detect() -> generate_dockerfile() -> `docker build` ->
`docker run` -> HTTP 200. Fixtures: vite (the regression), express with no
build step (prod-only install must still boot), flask, static. Wired as a
backend-ci job beside real-binaries.
Writing the flask fixture surfaced the same dead-at-boot class one layer
up: detection invents `gunicorn app:app` / `uvicorn main:app` as the start
command but never installed the server, so a repo that doesn't declare
gunicorn built green and died at boot. The build command now installs the
invented server (a no-op when the repo already pins it); Procfile-authored
start commands remain the author's contract.
All four stacks verified locally: build + boot + HTTP 200 in 53s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The panel's crontab/nginx code only ever met the fake_subprocess stub —
the 8-distro matrix runs stubs and the container legs run 12 detection
tests, so a syntax the real parser rejects failed on a user's box, not
in CI (the #117 class).
- new `real_binaries` marker (registered in conftest); the tests also
gate on SERVERKIT_REAL_BINARIES=1 so ordinary suite runs never mutate
a machine's crontab by surprise. Local loop (WSL works):
SERVERKIT_REAL_BINARIES=1 pytest tests -m real_binaries
- test_real_crontab_roundtrip: add -> list -> toggle -> remove through
the real `crontab -`/`crontab -l`, operator lines preserved, name
injection blocked; user crontab saved/restored around each test
(verified green on WSL against the real binary)
- test_real_nginx_vhost_syntax: every render_site_config flavor through
a real `nginx -t` in a tmp wrapper conf, plus a garbage probe so the
harness can't go vacuous
- backend-ci gains a real-binaries job (ubuntu runner, apt nginx+cron)
- scripts-ci gains the probe-matrix job: scripts/test/probe-matrix.sh
(real ufw/nginx/firewalld in 4 container shapes) existed since plan
75 but was wired into NO workflow; the ubuntu-root-sbinless leg was
re-verified green locally before wiring
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The frontend's 141 node --test assertions were invocable but invoked by
no workflow — written insurance never cashed in. frontend-ci now runs
npm test after lint (verified green locally: 141/141 in ~2s).
release.yml's ci-gate covered backend-ci only, so a release could ship
with a red frontend or drifted builtin extensions. The gate now also
requires frontend-ci and extensions-ci (both fast, both already
workflow_call-able). scripts-ci stays ungated on purpose — its
update-e2e leg takes ~45 min — with a comment showing how to add it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The amd64 tarball job on run 32270325886 hung inside apt-get update
(unreachable azure apt mirror on the runner) and sat for GitHub's
6-hour default job timeout. Normal runs finish in ~13 minutes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 75 §G1 — the door and the ratchet. The 33-file migration is NOT
done; see below.
run_checked(cmd, *, privileged=False, timeout=60, input=None). Around
backend/app's raw subprocess calls sit 220 copies of capture_output/text,
56 hand-rolled TimeoutExpired handlers, 19 FileNotFoundError handlers and
1,164 literal {'success': False, 'error': ...} dicts. Each is a place an
exec failure becomes a false fact.
It keeps the distinction those handlers lose: returncode is None when the
command never ran. Exit 1 means the command answered "no"; no exit code
at all means nobody answered. A caller rendering "not installed" can
finally tell them apart — plan 74's outage in one field. Unprivileged
calls get the same argv[0] resolution as privileged ones, because the
sbin outage is not something that only happens when you ask for root.
The ratchet is tests/raw_subprocess_census.py + RAW_SUBPROCESS_CEILING,
wired into the existing "Audit Raw Subprocess Patterns" CI job and
enforced by test_raw_subprocess_ratchet.py. The grep steps already there
check eight hand-listed files for three known-bad patterns, so a new file
was invisible to the audit entirely; the census counts every call site
under backend/app. It is AST-based because a grep for subprocess.run also
matches comments, docstrings and `except subprocess.SubprocessError`, and
a miscounting ratchet reads green either way. test_the_ceiling_is_not_stale
fails if the ceiling drifts more than 10 above reality — a stale ceiling
silently re-authorises everything a migration removed.
First migration: cron_service, 13 raw calls -> 3. Its `crontab -l` read
and Popen-with-stdin `crontab -` install were written out five times each
and are now two helpers over run_checked. Census 267 -> 258.
The other 32 files are untouched: docker_service (49), database_service
(24), git_service (18), security_service (14), python_service (13) are
the bulk. The door plus the ratchet are what make that incremental
instead of one unreviewable diff.
test_utils_system.py loads system.py under its own importlib spec, so
@patch('app.utils.system.<name>') binds to whichever module object is in
sys.modules at patch time — not necessarily the one run_checked closes
over. shutil/subprocess are singletons so the file's existing patches are
safe; these are the first tests to patch a module-level function of
system.py, and they use patch.object(_module, ...) instead.
BASELINE_COUNT 3383 -> 4081. Most of that is pre-existing drift (the
ratchet only fails on a drop, so an un-bumped floor never complains), not
tests added here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scripts/install.sh and scripts/install.ps1 are now VENDORED copies. The agent
repo owns them; this repo carries a copy only because the panel serves them off
its own disk at GET /api/v1/servers/install.sh|.ps1 and cannot link to another
repository.
Two copies of one file is what caused the second half of #101. When the agent
moved out of this monorepo it took its installer with it, and from then on there
were two -- actually four: the agent repo also carried a third, older pair under
its own scripts/ that nothing referenced. Each looked fine in isolation, nothing
ever compared them, and the panel went on serving one that downloaded from
`agent-v*` tags which exist in no repository. Unit tests are structurally blind
to this: they exercise whichever copy lives in their own repo and have no idea
another exists.
So:
scripts/sync-agent-installers.sh re-vendors from the agent repo (a sibling
checkout if present, else the published
branch), normalising to LF so this repo's
.gitattributes can apply its own EOL rule
scripts/test/check-installer-drift.sh fails, with the diff and the fix, if a
vendored copy differs from canonical
(EOL-insensitive: *.ps1 is checked out
CRLF here and LF there, which is not drift)
The drift check runs nightly alongside the release-chain verification, since
both need to reach GitHub. Both scripts run locally against a sibling checkout,
so this works offline and before an agent change is published.
Both installers carry a provenance header naming the canonical location, so
anyone who opens the vendored copy to patch it is told where to patch instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Issue #101 shipped because release.yml went straight from `build` to `push`
with nothing in between. The image was missing scripts/, so every containerised
panel 404'd its own enrollment URL, while the entire test suite stayed green --
pytest runs against the source tree, where the file always exists. Only booting
the artifact catches that class of bug.
scripts/test/smoke-docker-image.sh builds (or takes) an image, asserts the files
the panel reads at request time are present, boots it, and requests every route
that serves a file off disk. release.yml now builds amd64, runs it, and only
then pushes multi-arch (layers are cached, so the push reuses the build).
release-smoke.yml gets the same job on dev and PRs, and its path filters now
include Dockerfile, .dockerignore and scripts/ -- none of which used to trigger
it.
scripts/test/verify-agent-release-chain.sh covers the other half, which unit
tests structurally cannot: they stub curl, so they only ever prove the installer
is self-consistent, never that the URL it builds resolves to a real file. This
one reads the coordinates back out of the script, asks GitHub, downloads the
archive, verifies the checksum and confirms the tarball entry name the installer
moves. It runs nightly, and is also the tripwire for the reverse case -- the
agent repo changing its release naming and breaking the panel from the other
side of the repo split.
Both are plain scripts so the same check runs locally:
bash scripts/test/smoke-docker-image.sh
bash scripts/test/verify-agent-release-chain.sh
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven distro images were green throughout issue #99 because every one of them
runs the unit suites against STUBS. That is the right trade for logic, but it
left the one thing that talks to real package repos completely untested — so a
new Debian release dropping the Python we ask for could only ever be found by a
user.
scripts/test/test_provision_python.sh runs identify_system, choose_pkg_manager
and provision_python for real, then proves the result is usable: in range, can
create a virtualenv, has working pip, and imports ssl/sqlite3/ctypes. It needs
no systemd, nginx or ports, so it fits the plain containers the matrix already
pulls. Reaching the source build counts as a FAILURE, not a pass: a distro that
gets there is one we have not really added support for.
Wired up as a new `python-provisioning` job over the same images, and debian:13
added to both matrices. Adding trixie to the existing matrix alone would have
been decorative — those legs would have gone green on a distro that cannot
install.
Proven in both directions before being relied on: against the pre-fix tree it
reproduced #99 exactly on debian:13 (the reporter's apt errors, the fall
through to a source build, and the abort message), while debian:12 passed 7/7.
Also fixed the suite pinning the bug. T2 asserted `ver_in_range 3.13` must be
FALSE and T1's stub used 3.13 as its "too new" case, so the gate could not be
widened without the tests failing. Both moved to 3.14, plus a direct guard on
PYTHON_MAX — otherwise a regression to 3.12 could be "fixed" by editing the
assertion to match. 122 passed, 0 failed.
full-stack-test.ps1 documents why debian13 is NOT in the VM harness: no trixie
box publishes a Hyper-V provider (generic/debian13 does not exist,
debian/trixie64 is libvirt-only, bento/debian-13 has no hyperv) and Multipass
is Ubuntu-only. Named there with what remains uncovered on trixie — the full
systemd + nginx + API end-to-end — rather than left as a silent gap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Backend CI was the entire pipeline wait. Measured on the last dev push:
Extensions CI 12s
Version Bump 18s
Test System Utilities 35s
Security Scan 41s
Scripts CI 66s
Release Build Smoke 68s
Backend CI 21m 50s <- everything else finished in a minute
Step timings put all of it in the test run itself (install deps 22s, ratchet
9s), so there is no caching win to take — it is 3173 tests against a
function-scoped `app` fixture that rebuilds the Flask app and all 80+ tables
per test. Fixing that fixture is the real repair and is planned separately;
this change buys the wall-clock back now without touching a single test.
Split over 4 runners via pytest-split (794/794/794/791). Sharding rather than
pytest-xdist is deliberate: each shard is its own VM, so the process-shared
state that makes in-process parallelism unsafe here (templates.json, APPS_DIR
— see the per-PID DB dance in tests/conftest.py) simply isn't shared, and no
test code has to change.
The ratchet moves to its own job because it must see the whole suite; a shard
only collects its quarter. It runs beside the shards and costs no wall-clock.
Two details:
- The shard command is scoped to `tests` rather than a bare `pytest`.
Identical in CI (3173 either way, backend/dev-data/ being gitignored), but a
bare pytest on a dev box tries to collect the locally deployed apps under
backend/dev-data/ and dies during collection. The line is now copy-pasteable
for debugging a red shard locally.
- With no .test_durations file, pytest-split balances by test count, not time,
and these tests range from 0.1s to 10s+. If one shard becomes the new
critical path, run once with --store-durations and commit the file.
main also drops out of `push` here: release.yml gates itself on this workflow
via ci-gate, so listing main ran the whole suite a second time on every merge.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`npm run lint` was defined in frontend/package.json and documented in
CLAUDE.md, but no workflow ever invoked it. Grepping the whole workflow dir for
`npm run lint`/`eslint`/`npm run build` returned nothing, so three project-
specific checkers chained behind eslint were silently unenforced:
check-settings-index every Settings tab has a search-index entry
check-theme-tokens the theme-token whitelist stays in 3-way sync
check-html-sinks every raw-HTML sink is sanitized or annotated (XSS)
Lint currently passes: 926 warnings, 0 errors, and all three checkers green —
so this is safe to add as a blocking check today. eslint exits 0 on warnings,
so the gate is on errors only; add --max-warnings to freeze the count if the
warning drift ever needs a ratchet of its own.
Lint only, no build step: the frontend is already compiled in CI by Release
Build Smoke Test, whose scripts/build-release.sh runs `npm ci && npm run
build`. backend/app/** is in the paths because check-html-sinks scans it too,
for `|safe`, `Markup(` and `render_template_string`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These five workflows triggered on both `push: [dev, main]` and
`pull_request: [dev, main]`, so one change was validated up to three times: on
the dev push, again on the dev->main PR, and a third time on the post-merge
push to main.
Narrow them to `push: [dev]` + `pull_request: [main]`:
- `pull_request: [dev]` was dead config. Every PR this repo has had targets
main (checked back through #77, dependabot's included), so it never produced
a run.
- `push: [main]` was redundant for release-smoke: release.yml's build-release
job runs that same scripts/build-release.sh for real, on the same commit,
moments later.
Also drops test-system-utils' `unit-tests` job. It ran
`pytest tests/test_utils_system.py` (44 tests) that Backend CI's bare `pytest`
already collects — there is no pytest.ini, addopts or collect_ignore narrowing
collection. The distro matrix and the raw-subprocess audit stay; those are the
parts Backend CI genuinely cannot do.
Coverage of main is unchanged: a pull_request run tests the merge result, and
release.yml still gates itself on the full backend suite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The retry added in 1f2381c behaved exactly as designed and still failed the
job: Hub was unreachable for the full two minutes ("Client.Timeout exceeded
while awaiting headers" on every attempt), not flaky. Retrying a single
registry harder cannot survive that registry being down -- it only delays the
failure. Two consecutive runs were blocked this way.
Fall back to Google's public Hub mirror, a separate failure domain, and re-tag
to the original name so the run step needs no changes (`docker run` resolves
the local image; default pull policy is "missing"). Hub stays primary.
Ref construction differs by image kind -- `mirror.gcr.io/library/<img>` for
official images, `mirror.gcr.io/<ns>/<img>` for namespaced ones -- and all
seven matrix images were confirmed to resolve on the mirror via
`docker manifest inspect`, opensuse/leap included.
ONLY the pull falls back. Wrapping `docker run` would re-run the suites on
failure too, quietly turning a flaky TEST green.
Verified locally end-to-end against the step text extracted from this YAML,
under `bash -e`, with a docker shim reproducing the runner's exact Hub error:
* Hub down, mirror up -> 3 warnings, REAL pull from mirror.gcr.io, re-tag,
and `docker run <image>` then starts a container with the repo mounted
while Hub is still blocked
* both down -> 6 warnings, one error annotation, exit 1 (fails honestly)
* Hub healthy -> exits 0 on the first pull, no mirror attempt, no noise
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`docker run` pulls implicitly, so a registry timeout ("context deadline
exceeded" reaching registry-1.docker.io) killed the whole matrix leg with exit
125 before a single test ran, and cost a manual re-run. Seven images per matrix
run makes that a recurring tax rather than a one-off.
Pull up front with backoff (10/20/30s) and only then run the suites.
ONLY the pull is retried. Wrapping `docker run` would re-run the suites on
failure too, which would quietly turn a flaky TEST green -- the opposite of
what this job is for.
Left test-system-utils.yml alone: it uses a job-level `container:`, which the
runner pulls itself, so there is nothing to wrap.
Verified against the step text extracted from the YAML, under `bash -e` (the
shell GitHub actually uses for `run:`): registry down -> 3 retry warnings then
one error annotation and exit 1; flaky -> recovers and exits 0; healthy ->
exits 0 immediately with no retry noise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
backup_current tested `[ -f serverkit.db ]` FIRST and only fell through to the
PostgreSQL branch when no file was there. An install migrated SQLite ->
PostgreSQL keeps its old serverkit.db, and both deploy paths copy it forward on
file-existence alone (deploy_source, deploy_release), so the stale file is
carried into every future slot and never ages out. Such a box backed up the
abandoned file, skipped pg_dump entirely, and left PRE_UPGRADE_DUMP unset --
which migrate_database reads as "${PRE_UPGRADE_DUMP:-}", so its auto-restore
silently did nothing. PostgreSQL is the one engine that migrates IN PLACE and
cannot roll back by switching slots, so that dump is its only way home: the
exact safety net a61eab1/b48256f added was disabled for it.
migrate_database has always keyed off DATABASE_URL (its sqlite check). Test the
declared engine first so the two functions agree.
Deliberately NOT `grep DATABASE_URL=sqlite && [ -f db ]` as the reviewer
suggested: backend/config.py defaults DATABASE_URL when absent, so any install
whose .env lacks the line would fall to the else branch and update with NO
backup at all -- trading a mismatched backup for none. An explicit `postgres`
URL is a positive signal; the absence of a `sqlite` one is not.
Also:
* name the engine when it is one we cannot dump. "No SQLite database ..."
read like a benign nothing-to-do on a MySQL install that in fact had no
safety net. Prints the scheme only -- a DATABASE_URL carries a password and
this text lands in the update log.
* drop the stale "Python 3.11/3.12 is guaranteed by pre-flight" comment,
contradicted by locate_sqlite_python two lines below it (664a5e6).
* discord-notify: skip cleanly when DISCORD_WEBHOOK_URL is absent. Secrets
are not exposed to `pull_request` runs from a fork, so a merged fork PR
failed the job on an empty curl URL. Not switching to pull_request_target
for this -- handing a write-scoped token and the webhook to an
outsider-triggered run is a poor trade for a chat notification.
Tests: a PostgreSQL install carrying a leftover serverkit.db must still dump
and set PRE_UPGRADE_DUMP (fails on the parent commit, rc=1); an undumpable
engine must be named without leaking the password. Neither needs python3, so
both run on all 7 images. Workflow verified end-to-end against a synthetic
event payload: fork PR exits 0 without calling curl, normal PR still builds the
same embed.
Verified on the full CI matrix (ubuntu 22.04/24.04, debian:12, rockylinux:9,
almalinux:9, fedora:40, opensuse/leap:15.5), all six suites: 0 failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Introduces a full-stack Test Sandbox feature to run distro matrix tests in Docker from the UI: new backend API/service, sandbox_runs model + migration, run/log endpoints, and a new frontend page with mode selection, run history, live polling, cancel, and per-distro logs.
Also adds ARM64 scripts CI coverage (ubuntu-24.04-arm) and strengthens install/test automation for minimal images and overlay rebuilds: early curl/bootstrap index refresh, stricter venv validation, release-venv glibc fallback, Node reprovisioning on source fallback, rsync --delete overlay behavior, npm bootstrap in VM install, Windows tar pinning in PowerShell harness, plus updated test docs and distro coverage notes.
Make dependency scanning reliable in CI by adding a blocking pip-audit job for the backend and test harness requirements, while keeping Safety as a non-blocking advisory check because third-party service failures should not block merges. Also bump `click` to `8.3.3` to address the reported vulnerability in a directly pinned dependency.
The vite 8 upgrade (dcf05d1) makes the frontend build require Node ^20.19.0 ||
>=22.12.0 — on anything older npm silently skips rolldown's native binary and
`npm run build` dies with a cryptic MODULE_NOT_FOUND. Servers that build the SPA
on-box (source install + every update) would break their next update. So:
- install.sh node_ready() now gates on the real vite 8 floor (was Node >=18) and
provisions NodeSource 22 LTS when the host Node is too old.
- update.sh gains _node_build_ok(): build_frontend_bundle refuses (returns 1 →
caller rolls back) and the Docker-host rebuild halts, both with an actionable
"upgrade Node" message instead of a rolldown stack trace.
- Dockerfile + CI (release, release-smoke, extensions, scripts-nightly) bumped
node 20 -> 22 so the shipped/pre-built bundle is built on a compatible Node.
Also fixes a pre-existing parity bug the operator caught: update.sh rendered the
systemd unit WITHOUT substituting @BIND_HOST@ (install.sh does), so an update
left the literal placeholder and gunicorn would fail to bind. update.sh now
substitutes it (loopback default, SERVERKIT_BIND_HOST override) like install.sh.
Tests: test_update.sh +3 (node-floor gate, build refusal on old Node,
@BIND_HOST@ substitution) — 48 passed / 0 failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Install the guard the data loss taught us to want: code and its proving tests
died together, so the suite stayed green (2572->2209 passing) while whole
features silently vanished — nobody diffs a green number.
- backend/tests/BASELINE_COUNT (2475) + check_test_count.py: compares pytest's
collected count to the floor; CI (backend-ci.yml new step) fails if it drops.
Lowering the floor requires editing BASELINE_COUNT in the same commit, so a
silent test loss surfaces as a red X instead of a still-green suite. Verified:
passes at baseline, exits 1 on a simulated drop.
- CHANGELOG: document the recovery-parity restore (Cloudflare R2 engine,
email-bounce, DNS-cutover staging, survey routes, cron CSS, 18 test suites)
and the ratchet.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The July 2 outage lived in update.sh's top-level run block — code the
unit matrix never executes. Close the blind spot:
- update-e2e (PR-gated): real install.sh on the runner VM, health
probe, then THIS branch's update.sh --force with self-update
disabled, re-probe; failure dumps /var/log/serverkit + journal
- scripts-nightly.yml: install from the built release tarball (proves
the offline path + shipped-venv relocation), and install latest
published release -> update to current checkout (carry-forward
regressions same-version --force runs can't catch)
- advisory (non-gating) shellcheck --enable=all on install.sh/update.sh
- unit steps now also run test_cli.sh + test_agent_install.sh in both
the ubuntu job and the 7-distro matrix
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The panel frontend is now served straight from nginx/sites-available/*
(static SPA at location /, plus /api + /socket.io proxy). A broken
directive there 502s the whole panel on a real box, and nothing parsed
these configs in CI.
Add scripts/test/test_nginx_conf.sh: stages each vhost (secure +
insecure) against a real nginx, recreating the absolute paths they
reference (a stand-in frontend/dist bundle, the serverkit-locations /
conf.d include dirs, certbot webroot) and a throwaway self-signed cert
for the TLS vhost, then runs `nginx -t`. Skips cleanly (exit 0) where
nginx/openssl/root aren't available, so it's safe on a dev laptop.
Wire it into the Scripts CI `shell` job after the unit suites (installs
nginx + openssl so the step actually runs). shellcheck-clean at warning
severity; bash -n clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hardening plan §7. Catches distro-specific regressions before release.
- scripts/test/vm-update.sh: install → forced local update (full blue/green
deploy/migrate/switch/health) → re-run to confirm the "Already up to date"
version-gate. Same VM harness contract as vm-install.sh (log + status file).
- scripts/test/vm-uninstall.sh: install + seed data → default uninstall asserts
data preserved (DB snapshot in backups, /var/lib survives) → reinstall →
--purge asserts data removed. End-to-end proof of the unified uninstaller.
- scripts-ci.yml: now runs all three serverless suites (update+install+lib),
lints the new lib/ + test/ files (error severity) and keeps scripts/lib/*
warning-clean, and expands the cross-distro unit matrix to Ubuntu 22.04/24.04,
Debian 12, Rocky 9, AlmaLinux 9, Fedora 40, and openSUSE Leap 15.5.
- test_install.sh: +choose_pkg_manager graceful-failure case (7.4); a no-op
docker stub so the uninstall-matrix cases run where docker isn't installed.
- test_lib.sh: neutralize the ambient container=oci env var (RPM/SUSE images)
and force an empty PATH for the "no package manager" case — both bugs the new
matrix surfaced (the tests, not the code).
- README: fixed the stale Fedora box, documented the new VM scripts + suites.
Validated by running the exact CI container logic locally across all seven
images: test_update 13/13, test_install 15-16/0, test_lib 13/13 everywhere.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The updater's logic had zero test coverage on any distro, and ServerKit
targets several Linux families — so a portability regression only ever showed
up on a server. Add a matrix job that runs bash -n + scripts/test/test_update.sh
inside debian:12 and rockylinux:9 containers (Ubuntu is already covered by the
lint job). Verified locally: 9/9 pass on bash 5.2 (Debian) and 5.1 (Rocky).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Harden update.sh against the class of failure that left a box silently stuck
on the old version, and make it testable without a real deploy.
Robustness (cascade):
- L5 loud failures: `set -Eeuo pipefail` (the -E/errtrace makes the ERR trap
fire inside functions too) + a report_failure ERR trap that names the phase,
exit code, line and command, plus always-on logging to
/var/log/serverkit/update-*.log. No more silent drop-to-prompt.
- L4 docker-path rollback: snapshot the previous git SHA, :rollback image tags
and dist.rollback before mutating; if the new backend is not healthy within
90s, revert to the snapshot and halt (was: warn and continue).
- L6 self-heal: pin COMPOSE_PROJECT_NAME (kills the symlink-rename warning) and
reclaim a stale inactive blue/green slot left by a prior failed hybrid run.
Testability:
- Parameterize the system dirs (SERVERKIT_NGINX_DIR / _LETSENCRYPT_DIR /
_SYSTEMD_DIR / _CONFIG_DIR) and add SERVERKIT_NO_LOG, so refresh_config can
run against fixtures instead of the host /etc.
- Make update.sh source-able via a BASH_SOURCE guard so functions can be
exercised directly.
- scripts/test/test_update.sh: plain-bash unit tests (no deps, no server) that
reproduce the missing-serverkit.conf regression and cover the
detection/slot/report logic.
- .github/workflows/scripts-ci.yml: bash -n + shellcheck + the unit tests on
changes to scripts/**, install.sh, serverkit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce a GitHub Actions workflow (release-smoke.yml) that builds the release tarball on pushes/PRs to dev/main for relevant paths, catching release-script issues early. Adjust scripts/build-release.sh rsync excludes to explicitly target backups directories (root/backups and backend instance/dev-data backups) instead of a broader exclude, preventing accidental omission of needed files during release packaging.
The release workflow was publishing Docker assets and source but never
the serverkit-vX.Y.Z-linux-{amd64,arm64}.tar.gz files that install.sh,
update.sh, and 'serverkit update' expect to download. This caused release
updates to fail with 'Failed to download the release tarball.'
- Make scripts/build-release.sh honor BUILD_ARCH for CI matrix builds.
- Add a build-release job that builds tarballs on native amd64 and arm64
runners and uploads them as artifacts.
- Wire the release job to download those artifacts, include them in the
GitHub Release, and checksum them.
- Add the one-line installer to the release notes body.
Mirror the Mapanare release pattern for the ServerKit panel:
- Trigger release on push to main (PR merge), v* tags, and workflow_dispatch
- Add workflow_call to backend-ci.yml so release can use it as a CI gate
- Read VERSION file, create/push v{VERSION} tag when releasing from main
- Build and push multi-arch Docker image jhd3197/serverkit:{VERSION} + latest
- Create GitHub Release with Docker install instructions
Juan Denis