CONTRIBUTING asks contributors to target dev, but every workflow only
triggered on PRs into main, so contributor PRs merged with no checks
and failures surfaced only after the push to dev.
The panel's crontab/nginx code only ever met the fake_subprocess stub —
the 8-distro matrix runs stubs and the container legs run 12 detection
tests, so a syntax the real parser rejects failed on a user's box, not
in CI (the #117 class).
- new `real_binaries` marker (registered in conftest); the tests also
gate on SERVERKIT_REAL_BINARIES=1 so ordinary suite runs never mutate
a machine's crontab by surprise. Local loop (WSL works):
SERVERKIT_REAL_BINARIES=1 pytest tests -m real_binaries
- test_real_crontab_roundtrip: add -> list -> toggle -> remove through
the real `crontab -`/`crontab -l`, operator lines preserved, name
injection blocked; user crontab saved/restored around each test
(verified green on WSL against the real binary)
- test_real_nginx_vhost_syntax: every render_site_config flavor through
a real `nginx -t` in a tmp wrapper conf, plus a garbage probe so the
harness can't go vacuous
- backend-ci gains a real-binaries job (ubuntu runner, apt nginx+cron)
- scripts-ci gains the probe-matrix job: scripts/test/probe-matrix.sh
(real ufw/nginx/firewalld in 4 container shapes) existed since plan
75 but was wired into NO workflow; the ubuntu-root-sbinless leg was
re-verified green locally before wiring
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Seven distro images were green throughout issue #99 because every one of them
runs the unit suites against STUBS. That is the right trade for logic, but it
left the one thing that talks to real package repos completely untested — so a
new Debian release dropping the Python we ask for could only ever be found by a
user.
scripts/test/test_provision_python.sh runs identify_system, choose_pkg_manager
and provision_python for real, then proves the result is usable: in range, can
create a virtualenv, has working pip, and imports ssl/sqlite3/ctypes. It needs
no systemd, nginx or ports, so it fits the plain containers the matrix already
pulls. Reaching the source build counts as a FAILURE, not a pass: a distro that
gets there is one we have not really added support for.
Wired up as a new `python-provisioning` job over the same images, and debian:13
added to both matrices. Adding trixie to the existing matrix alone would have
been decorative — those legs would have gone green on a distro that cannot
install.
Proven in both directions before being relied on: against the pre-fix tree it
reproduced #99 exactly on debian:13 (the reporter's apt errors, the fall
through to a source build, and the abort message), while debian:12 passed 7/7.
Also fixed the suite pinning the bug. T2 asserted `ver_in_range 3.13` must be
FALSE and T1's stub used 3.13 as its "too new" case, so the gate could not be
widened without the tests failing. Both moved to 3.14, plus a direct guard on
PYTHON_MAX — otherwise a regression to 3.12 could be "fixed" by editing the
assertion to match. 122 passed, 0 failed.
full-stack-test.ps1 documents why debian13 is NOT in the VM harness: no trixie
box publishes a Hyper-V provider (generic/debian13 does not exist,
debian/trixie64 is libvirt-only, bento/debian-13 has no hyperv) and Multipass
is Ubuntu-only. Named there with what remains uncovered on trixie — the full
systemd + nginx + API end-to-end — rather than left as a silent gap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These five workflows triggered on both `push: [dev, main]` and
`pull_request: [dev, main]`, so one change was validated up to three times: on
the dev push, again on the dev->main PR, and a third time on the post-merge
push to main.
Narrow them to `push: [dev]` + `pull_request: [main]`:
- `pull_request: [dev]` was dead config. Every PR this repo has had targets
main (checked back through #77, dependabot's included), so it never produced
a run.
- `push: [main]` was redundant for release-smoke: release.yml's build-release
job runs that same scripts/build-release.sh for real, on the same commit,
moments later.
Also drops test-system-utils' `unit-tests` job. It ran
`pytest tests/test_utils_system.py` (44 tests) that Backend CI's bare `pytest`
already collects — there is no pytest.ini, addopts or collect_ignore narrowing
collection. The distro matrix and the raw-subprocess audit stay; those are the
parts Backend CI genuinely cannot do.
Coverage of main is unchanged: a pull_request run tests the merge result, and
release.yml still gates itself on the full backend suite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The retry added in 1f2381c behaved exactly as designed and still failed the
job: Hub was unreachable for the full two minutes ("Client.Timeout exceeded
while awaiting headers" on every attempt), not flaky. Retrying a single
registry harder cannot survive that registry being down -- it only delays the
failure. Two consecutive runs were blocked this way.
Fall back to Google's public Hub mirror, a separate failure domain, and re-tag
to the original name so the run step needs no changes (`docker run` resolves
the local image; default pull policy is "missing"). Hub stays primary.
Ref construction differs by image kind -- `mirror.gcr.io/library/<img>` for
official images, `mirror.gcr.io/<ns>/<img>` for namespaced ones -- and all
seven matrix images were confirmed to resolve on the mirror via
`docker manifest inspect`, opensuse/leap included.
ONLY the pull falls back. Wrapping `docker run` would re-run the suites on
failure too, quietly turning a flaky TEST green.
Verified locally end-to-end against the step text extracted from this YAML,
under `bash -e`, with a docker shim reproducing the runner's exact Hub error:
* Hub down, mirror up -> 3 warnings, REAL pull from mirror.gcr.io, re-tag,
and `docker run <image>` then starts a container with the repo mounted
while Hub is still blocked
* both down -> 6 warnings, one error annotation, exit 1 (fails honestly)
* Hub healthy -> exits 0 on the first pull, no mirror attempt, no noise
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`docker run` pulls implicitly, so a registry timeout ("context deadline
exceeded" reaching registry-1.docker.io) killed the whole matrix leg with exit
125 before a single test ran, and cost a manual re-run. Seven images per matrix
run makes that a recurring tax rather than a one-off.
Pull up front with backoff (10/20/30s) and only then run the suites.
ONLY the pull is retried. Wrapping `docker run` would re-run the suites on
failure too, which would quietly turn a flaky TEST green -- the opposite of
what this job is for.
Left test-system-utils.yml alone: it uses a job-level `container:`, which the
runner pulls itself, so there is nothing to wrap.
Verified against the step text extracted from the YAML, under `bash -e` (the
shell GitHub actually uses for `run:`): registry down -> 3 retry warnings then
one error annotation and exit 1; flaky -> recovers and exits 0; healthy ->
exits 0 immediately with no retry noise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Introduces a full-stack Test Sandbox feature to run distro matrix tests in Docker from the UI: new backend API/service, sandbox_runs model + migration, run/log endpoints, and a new frontend page with mode selection, run history, live polling, cancel, and per-distro logs.
Also adds ARM64 scripts CI coverage (ubuntu-24.04-arm) and strengthens install/test automation for minimal images and overlay rebuilds: early curl/bootstrap index refresh, stricter venv validation, release-venv glibc fallback, Node reprovisioning on source fallback, rsync --delete overlay behavior, npm bootstrap in VM install, Windows tar pinning in PowerShell harness, plus updated test docs and distro coverage notes.
The July 2 outage lived in update.sh's top-level run block — code the
unit matrix never executes. Close the blind spot:
- update-e2e (PR-gated): real install.sh on the runner VM, health
probe, then THIS branch's update.sh --force with self-update
disabled, re-probe; failure dumps /var/log/serverkit + journal
- scripts-nightly.yml: install from the built release tarball (proves
the offline path + shipped-venv relocation), and install latest
published release -> update to current checkout (carry-forward
regressions same-version --force runs can't catch)
- advisory (non-gating) shellcheck --enable=all on install.sh/update.sh
- unit steps now also run test_cli.sh + test_agent_install.sh in both
the ubuntu job and the 7-distro matrix
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The panel frontend is now served straight from nginx/sites-available/*
(static SPA at location /, plus /api + /socket.io proxy). A broken
directive there 502s the whole panel on a real box, and nothing parsed
these configs in CI.
Add scripts/test/test_nginx_conf.sh: stages each vhost (secure +
insecure) against a real nginx, recreating the absolute paths they
reference (a stand-in frontend/dist bundle, the serverkit-locations /
conf.d include dirs, certbot webroot) and a throwaway self-signed cert
for the TLS vhost, then runs `nginx -t`. Skips cleanly (exit 0) where
nginx/openssl/root aren't available, so it's safe on a dev laptop.
Wire it into the Scripts CI `shell` job after the unit suites (installs
nginx + openssl so the step actually runs). shellcheck-clean at warning
severity; bash -n clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hardening plan §7. Catches distro-specific regressions before release.
- scripts/test/vm-update.sh: install → forced local update (full blue/green
deploy/migrate/switch/health) → re-run to confirm the "Already up to date"
version-gate. Same VM harness contract as vm-install.sh (log + status file).
- scripts/test/vm-uninstall.sh: install + seed data → default uninstall asserts
data preserved (DB snapshot in backups, /var/lib survives) → reinstall →
--purge asserts data removed. End-to-end proof of the unified uninstaller.
- scripts-ci.yml: now runs all three serverless suites (update+install+lib),
lints the new lib/ + test/ files (error severity) and keeps scripts/lib/*
warning-clean, and expands the cross-distro unit matrix to Ubuntu 22.04/24.04,
Debian 12, Rocky 9, AlmaLinux 9, Fedora 40, and openSUSE Leap 15.5.
- test_install.sh: +choose_pkg_manager graceful-failure case (7.4); a no-op
docker stub so the uninstall-matrix cases run where docker isn't installed.
- test_lib.sh: neutralize the ambient container=oci env var (RPM/SUSE images)
and force an empty PATH for the "no package manager" case — both bugs the new
matrix surfaced (the tests, not the code).
- README: fixed the stale Fedora box, documented the new VM scripts + suites.
Validated by running the exact CI container logic locally across all seven
images: test_update 13/13, test_install 15-16/0, test_lib 13/13 everywhere.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The updater's logic had zero test coverage on any distro, and ServerKit
targets several Linux families — so a portability regression only ever showed
up on a server. Add a matrix job that runs bash -n + scripts/test/test_update.sh
inside debian:12 and rockylinux:9 containers (Ubuntu is already covered by the
lint job). Verified locally: 9/9 pass on bash 5.2 (Debian) and 5.1 (Rocky).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Harden update.sh against the class of failure that left a box silently stuck
on the old version, and make it testable without a real deploy.
Robustness (cascade):
- L5 loud failures: `set -Eeuo pipefail` (the -E/errtrace makes the ERR trap
fire inside functions too) + a report_failure ERR trap that names the phase,
exit code, line and command, plus always-on logging to
/var/log/serverkit/update-*.log. No more silent drop-to-prompt.
- L4 docker-path rollback: snapshot the previous git SHA, :rollback image tags
and dist.rollback before mutating; if the new backend is not healthy within
90s, revert to the snapshot and halt (was: warn and continue).
- L6 self-heal: pin COMPOSE_PROJECT_NAME (kills the symlink-rename warning) and
reclaim a stale inactive blue/green slot left by a prior failed hybrid run.
Testability:
- Parameterize the system dirs (SERVERKIT_NGINX_DIR / _LETSENCRYPT_DIR /
_SYSTEMD_DIR / _CONFIG_DIR) and add SERVERKIT_NO_LOG, so refresh_config can
run against fixtures instead of the host /etc.
- Make update.sh source-able via a BASH_SOURCE guard so functions can be
exercised directly.
- scripts/test/test_update.sh: plain-bash unit tests (no deps, no server) that
reproduce the missing-serverkit.conf regression and cover the
detection/slot/report logic.
- .github/workflows/scripts-ci.yml: bash -n + shellcheck + the unit tests on
changes to scripts/**, install.sh, serverkit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Juan Denis