Found by the full-stack VM harness, which 3524 unit tests and a 7-image
container matrix had all missed. On a real Ubuntu 24.04 install the log read
"Opening HTTP/HTTPS (80, 443) via nftables" and not one line of
bootstrap_firewall appeared: it returned at its "a firewall is already active"
guard, so the box got its two ports opened and no default-deny policy at all.
The headline feature of that work was a no-op on exactly the population it was
written for, and it looked green the whole way.
The guard asked `[ "$(firewall_detect)" != "none" ]`. firewall_detect answers a
different question — "which backend do I open a port THROUGH" — and returns
`nftables` whenever the nft binary exists and can list a ruleset, including an
empty one. That is true of every modern Linux, so the guard reported every box
as already protected. Verified directly against a stock-image fixture: same
inputs, `firewall_detect` -> nftables, inbound-default-deny -> unprotected.
firewall_inbound_default_deny() asks what the bootstrap actually cares about:
does this host already deny inbound by default? Presence of rules is
deliberately NOT the test -- Docker installs plenty of nft and iptables chains
and protects nothing. A default-deny policy on the input hook is: firewalld
running, ufw active *with* "deny (incoming)", an nft base chain hooked to input
with policy drop or reject, or an iptables INPUT policy that is not ACCEPT.
Unknown answers 1 (unprotected), because the caller's response is to offer
protection and every caller re-checks before acting.
firewall_detect is left alone -- configure_firewall legitimately wants a
backend name for a binary that exists -- and both functions now carry a comment
saying which question they answer, since that is what was conflated.
Six cases pin the behaviour, five of them shapes the old guard got wrong: an
empty nft ruleset must run the bootstrap, Docker's chains must not read as
protection, an active-but-allow-inbound ufw must be fixed rather than
respected, a real nft policy drop must be left alone, and the two predicates
must disagree on a stock image -- that last one asserted at the predicate level
so the distinction survives a refactor of either function. make_stub_stdout is
new because an exit-code stub cannot express "the tool ran fine and reported
nothing", which is the whole stock-image case.
Green on all seven CI images (265/265/265/275/275/275/263, 0 failed).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 73 item 8. The installer opened 80/443 on whatever firewall it found and
shrugged -- "assuming ports 80/443 are already open" -- when there wasn't one,
which is the state of a stock Ubuntu/Debian VPS. That box then ran ServerKit
with every port a deploy maps onto the host (8001-8999, plus database
containers on their stock ports) facing the internet. fail2ban only shipped
with the "full" capacity profile, so the typical small VPS got neither.
The dangerous half is enabling default-deny without allowing the port sshd
actually answers on, over the SSH session running the installer. detect_ssh_ports
therefore never guesses: it asks `sshd -T` (the effective config, which resolves
the Include that Ubuntu 22.04+ ships), then sshd_config plus its drop-ins with
comments stripped, then a live `ss`. Multiple Port lines all come back. 22 is
used only when a config exists and names no port -- sshd's compiled-in default
-- and if nothing answers at all the function fails and no firewall is
installed. An exposed box is recoverable; a locked-out one usually is not.
That interlock had a real bug in its first draft, caught before it shipped and
now pinned by a named regression test: most boxes have no sshd_config.d/, the
unmatched glob makes cat exit non-zero, and under `set -o pipefail` that wiped
a correctly parsed answer and fell through to the 22 default -- returning 22 on
a box whose sshd is on 2222, which is precisely the lockout this function
exists to prevent.
Ordering is the other safety property and is asserted: every allow rule (all
detected SSH ports, 80, 443, and PANEL_PORT when non-standard) lands before
`default deny incoming`, and the policy before `--force enable`. The bootstrap
refuses to act inside a container, when a firewall is already active (it is the
operator's -- opening ports on it is fine, rewriting its default policy is not),
under SERVERKIT_SKIP_FIREWALL=1, and when a previous run bootstrapped but the
firewall is off now, since the operator turned it off deliberately. Non-Debian
installs are byte-identical to before: RHEL-family boots with firewalld already
active, and a firewalld default-deny bootstrap for boxes where it is inactive
is marked as an explicit follow-up rather than guessed at.
fail2ban moves to the standard profile with an sshd jail that watches the same
ports detect_ssh_ports found -- a jail pinned to :22 on a box listening on :2222
bans nobody -- and declares `backend = systemd` when there is no auth.log to
tail, without which the jail refuses to start and fail2ban looks installed while
protecting nothing. certbot stays full-only.
setup-health gains firewall (critical) and fail2ban (recommended) items. The
split is deliberate: this file grades critical by silent breakage, and an
unfirewalled box qualifies -- the operator set it in motion by deploying,
nothing in the UI says the port is public, and the failure mode is compromise
rather than an error. A missing fail2ban thins defence in depth but breaks
nothing silently, since the panel already throttles logins per IP. Both are
gated on actually owning the host -- Linux, /etc/serverkit present, not
containerised -- so a dev checkout or CI runner drops them entirely instead of
reporting a fake ok that would inflate the score.
SetupHealthWidget was orphaned; so, it turns out, was its stylesheet, which
main.scss never imported. Both are wired up, as a plain admin-only card above
the board rather than a placeable widget: an unfirewalled box has to be visible
without the operator first opting into showing it. It collapses to one line
when everything is clean.
Implementing this inverts a shipped test that asserted provision_hardening was
a no-op on the standard profile; it is rewritten to the new contract rather
than deleted, with a minimal-profile case added alongside.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Piped execution leaves BASH_SOURCE unset, so the sourceability guard's bare
${BASH_SOURCE[0]} was an unbound variable under `set -u` and aborted the
install before main() ever ran:
bash: line 2107: BASH_SOURCE[0]: unbound variable
This broke the one-liner README/DEPLOYMENT document, on every distro — not
just the Debian 13 it was reported on (reproduced on debian:13 bash 5.2.37,
ubuntu:22.04 bash 5.1.16, almalinux:9 bash 5.1.8). CI never saw it because
every job runs `bash install.sh` from a checkout, where BASH_SOURCE IS set.
Use the same non-empty-first guard scripts/install.sh already carries, and
add the proving test the panel installer was missing (T41) — the agent
installer has had its twin since the same fix landed there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- install_profile_service: record what install.sh put on the box
(minimal/standard/full), recommended from detected hardware and
overridable; thresholds mirrored by recommend_profile in install.sh
- setup wizard: replace the hard ResourceGate/SetupStepTier with
SetupStepCapacity + SetupStepSecurity
- ResourceAdvisory: inline capacity warning at the point of action
(WordPress/Docker/Services/Deployments), backed by headroom.fits
- RequiresDocker component for docker-dependent surfaces
- install.sh/update.sh: profile detection + tests
(test_install.sh, test_update.sh, test_server_capacity.py)
backup_current tested `[ -f serverkit.db ]` FIRST and only fell through to the
PostgreSQL branch when no file was there. An install migrated SQLite ->
PostgreSQL keeps its old serverkit.db, and both deploy paths copy it forward on
file-existence alone (deploy_source, deploy_release), so the stale file is
carried into every future slot and never ages out. Such a box backed up the
abandoned file, skipped pg_dump entirely, and left PRE_UPGRADE_DUMP unset --
which migrate_database reads as "${PRE_UPGRADE_DUMP:-}", so its auto-restore
silently did nothing. PostgreSQL is the one engine that migrates IN PLACE and
cannot roll back by switching slots, so that dump is its only way home: the
exact safety net a61eab1/b48256f added was disabled for it.
migrate_database has always keyed off DATABASE_URL (its sqlite check). Test the
declared engine first so the two functions agree.
Deliberately NOT `grep DATABASE_URL=sqlite && [ -f db ]` as the reviewer
suggested: backend/config.py defaults DATABASE_URL when absent, so any install
whose .env lacks the line would fall to the else branch and update with NO
backup at all -- trading a mismatched backup for none. An explicit `postgres`
URL is a positive signal; the absence of a `sqlite` one is not.
Also:
* name the engine when it is one we cannot dump. "No SQLite database ..."
read like a benign nothing-to-do on a MySQL install that in fact had no
safety net. Prints the scheme only -- a DATABASE_URL carries a password and
this text lands in the update log.
* drop the stale "Python 3.11/3.12 is guaranteed by pre-flight" comment,
contradicted by locate_sqlite_python two lines below it (664a5e6).
* discord-notify: skip cleanly when DISCORD_WEBHOOK_URL is absent. Secrets
are not exposed to `pull_request` runs from a fork, so a merged fork PR
failed the job on an empty curl URL. Not switching to pull_request_target
for this -- handing a write-scoped token and the webhook to an
outsider-triggered run is a poor trade for a chat notification.
Tests: a PostgreSQL install carrying a leftover serverkit.db must still dump
and set PRE_UPGRADE_DUMP (fails on the parent commit, rc=1); an undumpable
engine must be named without leaking the password. Neither needs python3, so
both run on all 7 images. Workflow verified end-to-end against a synthetic
event payload: fork PR exits 0 without calling curl, normal PR still builds the
same embed.
Verified on the full CI matrix (ubuntu 22.04/24.04, debian:12, rockylinux:9,
almalinux:9, fedora:40, opensuse/leap:15.5), all six suites: 0 failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
copy_sqlite_db and verify_sqlite_db reused locate_python, which enforces
3.11 <= v <= 3.12. That bound is correct for rebuild_virtualenv and
preflight_check, but snapshotting or integrity-checking a SQLite file needs
nothing beyond the stdlib sqlite3 module. Reusing it coupled the DATA-safety
paths to the venv-builder constraint and blocked updates for a reason unrelated
to the data:
* an all-Docker install returns before preflight_check ever runs and does
every real step in the container -- yet backup_docker_db snapshots with the
CONTAINER's python and then verified the extracted file with the HOST's.
On a stock EL9 (python3.9) or Ubuntu 22.04 (python3.10) host that halted
with "Could not obtain a VALID database backup", blaming the backup for a
missing interpreter.
* backup_current halts when it cannot snapshot, so those hosts could never
update at all.
Split out locate_sqlite_python: any candidate that can `import sqlite3` wins,
and the install's own venv interpreter is preferred -- on a real box it is
guaranteed present, guaranteed to carry the module, and independent of PATH.
locate_python keeps the 3.11/3.12 bound where it belongs.
Surfaced by the scripts-ci unit matrix: almalinux:9 and rockylinux:9 ship
python3.9, so the suite's own probe accepted an interpreter locate_python
rejected, and copy_sqlite_db/verify_sqlite_db failed there while passing on
python3.12 runners.
T10i pins the contract: on a box where every candidate reports 3.9 (locate_python
fails), the helpers still resolve an interpreter, prefer the slot's venv, and
round-trip 100 rows. Fails on the parent commit, passes here.
Verified on the full 7-image CI matrix (ubuntu 22.04/24.04, debian:12,
rockylinux:9, almalinux:9, fedora:40, opensuse/leap:15.5), all six shell suites:
0 failed. test_update is 63 passed / 0 failed on every image carrying a python3.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
verify_pg_dump only runs its structural `pg_restore -l` TOC check when the
binary is on PATH. The harness stubs every other external command it touches
(systemctl, nginx, npm, curl, docker, flask, pg_dump) but never pg_restore, so
two assertions were decided by the host:
* no postgresql-client -> branch skipped, check degrades to "non-empty",
and the synthetic `PGDMP fake contents` fixture passes
* postgresql-client present -> the real binary parses the version bytes
after the PGDMP magic, rejects the fixture, verify_pg_dump returns 1
That is the CI failure. T10f failed directly; T10e failed downstream because
backup_current correctly halts on an unverifiable backup. update.sh is right --
the tests leaked.
Add a make_stub_pg_restore factory mimicking the real -l semantics (rejects
missing/empty, rejects anything without the PGDMP magic, else prints a TOC) and
install it in T10e/T10f. Give T10f a non-empty non-archive fixture too: that is
the case separating the two branches inside verify_pg_dump -- the size check
passes it, only the TOC listing catches it -- so the path the test was named
for is now actually exercised.
Verified both with and without a pg_restore on PATH: 59 passed, 0 failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A backup that was never verified is a hope, not a safety net. Closes the
remaining ways the pre-upgrade safety net could silently not exist:
- Every backup is verified immediately after creation: SQLite snapshots via
PRAGMA integrity_check, PostgreSQL dumps via size + pg_restore TOC
listing (when available). Invalid artifacts are deleted and the update
halts — a malformed backup is never mistaken for a good one.
- Partial artifacts from a failed backup are removed (rm on failure), so a
half-written file can never be 'restored' later.
- A failed IN-PLACE PostgreSQL migration now automatically restores the
verified pre-upgrade dump (pg_restore --clean --if-exists) before
halting; without it the half-migrated schema would stay live against
the old code. If the restore itself fails, the halt message carries the
exact manual pg_restore command.
- backup_docker_db tries both python and python3 in the container, and now
verifies the result too — the plain docker cp fallback can no longer
pass off a torn copy as a backup.
- Backup retention capped: default 5 (SERVERKIT_BACKUP_RETENTION
overrides) across trees, .db and .dump files. Full-size DB copies pile
up fast — a real 25G droplet carried 6.3G of stale backups.
Tests: T10f (verification helpers accept/reject correctly), T10g (failed
in-place PG migration auto-restores with the exact pg_restore args),
T10h (retention prunes all three backup kinds to the configured cap).
Postgres panels updated with zero safety net: backup_current found no
SQLite file and just warned, while the migration runs IN PLACE against the
shared external DB — a failed migration left no way back.
- backup_current now pg_dumps (custom format) when .env points at
PostgreSQL, sanitizing SQLAlchemy driver suffixes (postgresql+psycopg2://)
that libpq can't parse.
- Same rule as SQLite: no valid backup, no update — halt when pg_dump is
missing (install postgresql-client) or the dump fails.
Tests: T10e covers the dump invocation + URL sanitization and the
halt-on-failed-dump path.
Root cause of a real-world failed update: backup_current, deploy_source and
deploy_release all plain-cp'd the LIVE panel database. On a busy panel that
races the backend's own writes and the copy can capture half a transaction
— 'database disk image is malformed' — which then sinks the migration in
the new slot and silently poisons the pre-upgrade backup that was supposed
to be the safety net.
- New copy_sqlite_db() uses the SQLite online backup API via python
(guaranteed by pre-flight) and refuses to fall back to a racy cp.
- backup_current halts rather than proceed without a valid safety net.
- deploy_source/deploy_release halt before any switch if the live DB
cannot be snapshotted consistently.
- backup_docker_db prefers a consistent snapshot via docker exec python,
with the plain docker cp as a fallback.
- locate_python rejects candidates whose version probe returns empty
(broken shims pass 'command -v' but can't execute; an empty version
string used to pass the sort-based range check).
Tests: T10d exercises a real snapshot (integrity ok, all rows); a contract
test bans plain-cp of the live DB; T10d fixture python detection validates
the interpreter actually runs.
The SQLite pre-check in migrate_database treated an EMPTY probe result
(unusable venv python) the same as real corruption and halted the update —
a false positive that broke the T10/T24 contract tests. Now:
- probe output != 'ok' -> genuine corruption -> halt with repair guidance
- empty probe output -> probe couldn't run -> warn and let flask surface
any real error
test_update.sh: T10/T24 fixtures stub the venv python (healthy-DB path);
new T10b asserts the corrupt-DB halt fires before flask runs; new T10c
asserts an unrunnable probe warns and proceeds.
Introduces a full-stack Test Sandbox feature to run distro matrix tests in Docker from the UI: new backend API/service, sandbox_runs model + migration, run/log endpoints, and a new frontend page with mode selection, run history, live polling, cancel, and per-distro logs.
Also adds ARM64 scripts CI coverage (ubuntu-24.04-arm) and strengthens install/test automation for minimal images and overlay rebuilds: early curl/bootstrap index refresh, stricter venv validation, release-venv glibc fallback, Node reprovisioning on source fallback, rsync --delete overlay behavior, npm bootstrap in VM install, Windows tar pinning in PowerShell harness, plus updated test docs and distro coverage notes.
The installer's node_ready() now gates on Node ^20.19.0 || >=22.12.0 (was >=18),
so the T3 assertion "node_ready true for node v18" was stale and failed CI.
Rewrite T3 to prove the real floor: accepts 20.19+/22.12+ (with npm), rejects
v16/v18/20.17/20.18/21.x/22.11. (Dropped an npm-absent sub-case that can't be
isolated in-harness — command -v npm still finds the runner's real npm.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make `update.sh` fetch the CI-built release tarball by default instead of
rebuilding the SPA on the server, so a normal update needs no Node/npm on the
box. This keeps updates working on old distros, ARM/Pi and tiny hosts where the
vite 8 toolchain (Node 20.19+/22.12+) can't or shouldn't run — the Node
requirement stays a build-time/CI concern, not a runtime one.
On-box source rebuilds remain available and explicit: --source / --main (main
branch), --branch <name>, or BUILD_FROM_SOURCE=1. --branch now implies source
mode (clears the release default). npm is only required for those paths.
Test: test_update.sh +1 (mode mapping — default=release, --source/--branch/
BUILD_FROM_SOURCE=source). 49 passed / 0 failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The vite 8 upgrade (dcf05d1) makes the frontend build require Node ^20.19.0 ||
>=22.12.0 — on anything older npm silently skips rolldown's native binary and
`npm run build` dies with a cryptic MODULE_NOT_FOUND. Servers that build the SPA
on-box (source install + every update) would break their next update. So:
- install.sh node_ready() now gates on the real vite 8 floor (was Node >=18) and
provisions NodeSource 22 LTS when the host Node is too old.
- update.sh gains _node_build_ok(): build_frontend_bundle refuses (returns 1 →
caller rolls back) and the Docker-host rebuild halts, both with an actionable
"upgrade Node" message instead of a rolldown stack trace.
- Dockerfile + CI (release, release-smoke, extensions, scripts-nightly) bumped
node 20 -> 22 so the shipped/pre-built bundle is built on a compatible Node.
Also fixes a pre-existing parity bug the operator caught: update.sh rendered the
systemd unit WITHOUT substituting @BIND_HOST@ (install.sh does), so an update
left the literal placeholder and gunicorn would fail to bind. update.sh now
substitutes it (loopback default, SERVERKIT_BIND_HOST override) like install.sh.
Tests: test_update.sh +3 (node-floor gate, build refusal on old Node,
@BIND_HOST@ substitution) — 48 passed / 0 failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 1.7.25 -> 1.7.33 update broke on panels that still had the retired
serverkit-workflows extension on disk: preserve_installed_plugins carried
it into the fresh tree, where its reactflow import (removed with plan 45)
sank the whole bundle build and blocked the update.
- RETIRED_PLUGIN_SLUGS: the carry-forward now skips slugs the product
deliberately deleted, and docker in-place mode strips their leftovers
after the hard reset (untracked dirs survive it).
- Quarantine-and-retry: when the bundle build fails and plugins were
carried forward, move them into /var/backups/serverkit/quarantined-
plugins-<ts> and retry once -- fail the plugin, never the update.
The updater self-refreshes from main before running, so this protects
every panel on its next `serverkit update` once merged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live test on the dev box caught 'serverkit list-apps --json' printing
the table: cmd_list_apps' hand-rolled flag mapping dropped everything
but --all, and cmd_list_users/cmd_list_servers dropped all args. All
three now forward "$@" (with -a kept as the legacy --all alias), and
test_cli.sh T20 drives the real executed-mode dispatch through a fake
venv with a recording python stub so any wrapper verb that eats flags
fails the suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
serverkit migrate-db forwarded the wrapper's own name to cli.py, which
only defines db-migrate — the command has failed with 'No such command'
since the rename. Fix the forward and stop the drift class entirely:
- cmd_panel_cli forwards Click-only verbs verbatim (manifest, services,
apps, repair, login-url, support-bundle, db-migrate/status/history;
panel-status/panel-doctor where a native bash name collides)
- serverkit completion prints a bash completion script; install.sh and
update.sh drop it into /etc/bash_completion.d (best-effort), the
uninstall lib removes it
- test_cli.sh T17-T19: migrate-db forwarding, dispatch reachability,
and a two-way parity guard (cli.py verbs <-> dispatch <-> completion);
test_update.sh T29 covers refresh_bash_completion
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Commit 6e32519 (the July 5-7 recovery restore) wrote git-reflog garbage
over scripts/install.sh (the agent installer served at
GET /api/v1/servers/install.sh) and whitespace-mangled content over
scripts/test/full-stack-test.ps1, breaking scripts-ci: shellcheck SC2148
(no shebang) and 15/18 test_agent_install.sh failures. Restore both files
verbatim from 6e32519~1, the last good revision.
Verified: test_agent_install.sh 18/18, test_install.sh 62 passed,
shellcheck --severity=error clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A test box's panel domain kept serving a Jun 25 bundle after the Jul 3
update — the domain's custom vhost still proxied the retired :3847
frontend container, and the old bundle's calls to deleted legacy
marketplace endpoints read as 'Failed to load extensions'. It looked
exactly like a browser/CDN cache problem (it wasn't — cf-cache-status
was DYNAMIC; the origin itself was stale).
- both stock confs: expires -1 on the SPA entrypoint (emits
Cache-Control: no-cache WITHOUT discarding server-level security
headers, unlike add_header-in-location) — a cached index.html
references hashed assets that vanish on the next blue/green switch
- update.sh refresh_config now runs report_stale_panel_vhosts: warns,
naming the file, when any enabled vhost still proxies :3847;
observation-only per R1, silent on a missing sites-enabled dir,
added to the fresh-box loop per policy
- proving tests: entrypoint no-cache asserted for both confs; stale
vhost flagged (and clean one not); missing-dir silence (42 passed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bash suppresses set -e inside any tested-context command list — an
if-condition subshell, or the left side of ||, even with set -Eeuo
pipefail re-armed inside — so multi-statement test bodies only gated
on their LAST statement. Today's cmp-less EL9 CI run proved it: a
byte-identity test passed while its cmp assertions exited 127.
28 blocks across test_update/test_install/test_lib/test_agent_install
converted to explicit && chains so every assertion gates the result.
No test weakened; no vacuous pass was hiding a real script bug (each
newly-gated statement was verified to genuinely succeed).
Verified: Git Bash 171 passed / 0 failed; rockylinux:9 all five
suites 177p/0f; opensuse/leap:15.5 148p/0f (pre-existing python3/tar
skips).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
EL9/Fedora containers ship without diffutils and openSUSE Leap without
tar, which failed 4 of the 7 cross-distro unit jobs:
- files_identical() in stubs.sh: cmp when available, sha256sum
(coreutils, everywhere) otherwise
- the two tarball-fixture tests skip cleanly when tar is absent
(harness's existing skip convention; proven on the other distros)
- the t16/t16r assertion lists are now && chains: bash suppresses
set -e inside an if-condition subshell, so sequential statements
only gated on the LAST one — the cmp-less EL9 run proved it by
passing t16r vacuously while cmp exited 127
Verified in the actual images: rockylinux:9 65p/0f/0s,
opensuse/leap:15.5 61p/0f/4s (was 63p/2f and 61p/2f).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The July 2 outage function had a unit test — with a populated fixture
only. This encodes the missing net:
- fresh-minimal-box fixture loop: every observation/discovery/snapshot/
report function in update.sh (17) and install.sh (16) must survive a
zero-app box under set -Eeuo pipefail; policy comment requires new
functions to be appended
- scripts/test/stubs.sh: shared failing-stub factories (systemctl exit
5, docker fail/empty, curl exit 22, GitHub rate-limit JSON)
- assertion sweep: 43 bare capture-then-compare sites now include exit
status in the assertion
- best-effort contract test: the run block's real APP_BASELINE ->
report chain exits 0 on a fresh box AND on a genuine regression
- fix(update): the loop immediately caught report_failure's trailing
'[ -n $UPDATE_LOG ] && printf' returning 1 when UPDATE_LOG is empty
(same species as the outage) — converted to if, added to the loop
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- version discovery pages agent-v tags (?per_page=100, 3 pages) so
panel releases can no longer push agent tags off page 1; panel
injects its resolved agent version into the served script and its
own releases fetch is paginated too (A1)
- downloaded agent binary is sha256-verified against the release's
checksums.txt: halt on mismatch, warn-and-continue when absent (A2)
- value-taking options error loudly when the value is missing; busybox
adduser fallback; no GNU-only grep -oP; pipe-safe source guard (A3)
- test-routing.sh fixed (PASS++ suicide under set -e, guarded probes,
curl 000000 false-pass) rather than deleted - still live tooling
- new scripts/test/test_agent_install.sh (18 passed); backend server
tests 110 passed
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- update --branch without a value errors loudly instead of dying
silently on shift (L5)
- cmd_start/cmd_restart mirror cmd_stop's systemctl/docker guards (L6)
- cmd_logs degrades past a dead/missing journalctl to docker logs (L7)
- cmd_add_site/remove_site/list_sites detect Debian vs conf.d nginx
layout; reloads are guarded with a manual-reload hint (L9)
- uninstall on a remnants-only box prints exact recovery commands (L4)
- portable port extraction (no grep -oP), read-at-EOF defaults,
compare_versions rejects unparsable as update-available, doctor
exits nonzero on issues (L12); BASH_SOURCE guard for testability
- new scripts/test/test_cli.sh (22 passed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- EUID check runs before any host write; friendly non-root halt (I11)
- get.docker.com staged to temp file with retry + pkg fallback (I13)
- df -Pk on the real install target; free guarded for LXC (I14)
- release tag via releases/latest redirect (no API quota) + loud
source-fallback warning (I15)
- tar/cp failures in fetch_release restore live state and fall back
to source cleanly (I16)
- svc_enable/svc_start init-system abstraction at service call sites (I17)
- stale slot-A collision guard in ensure_install_layout (I18)
- swap verified via /proc/swaps before claiming active; dd/mkswap
failures degrade to warn (I19)
- VERSION clobber, telemetry pipefail, stale SSL_MODE rewrite,
apply_frontend_root end-of-loop && (I22-I24, bonus)
- 22 new proving assertions (61 passed local, 64 on Linux)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- find-based backup pruning; empty/missing BACKUP_DIR is a no-op (U1)
- download_release progress to stderr; stdout is the tarball path only (U2)
- guard bare systemctl stop/start/daemon-reload in forward+rollback paths (U3,U4,U8,U13)
- ROLLING_BACK=1 inside rollback(); never runs twice via the EXIT trap (U5)
- npm preflight for source updates; guarded frontend build pipeline (U6)
- empty-ORIG_ARGS re-exec guard, cleanup_on_exit return 0, df -Pk,
SIGPIPE guards, cd leaks subshelled, local typo (U7,U9-U12,U14)
- 14 new proving assertions in test_update.sh (35 passed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Allow failures when querying GitHub release tag (prevent pipeline from aborting) and add a unit test ensuring discover_app_upstreams tolerates an empty app-locations directory.
Changes:
- scripts/update.sh: make the curl|grep pipeline resilient by appending `|| true` so the command substitution won't fail the script when the GitHub API call returns nothing.
- scripts/test/test_update.sh: add test T16b to verify discover_app_upstreams does not abort under `set -euo pipefail` when the app-locations directory is empty.
These changes prevent hard failures during release detection and add coverage for empty-directory handling.
A panel update deploys a fresh clone/tarball preserving only .env and
the DB, wiping app/plugins/<slug> + src/plugins/<slug> for any
URL/registry/upload-installed extension — the surviving InstalledPlugin
row then flipped to error at next boot. Two-layer fix:
- update.sh: preserve_installed_plugins carries user-installed plugin
dirs from the old tree into the new one (skipping anything the new
tree already ships and __pycache__), before the frontend build so
carried UIs recompile. Covered by harness test T18.
- boot repair pass (plugin_service.repair_missing_plugins, runs before
load_all_plugins): builtin slugs reinstall from builtin-extensions/,
url installs re-download from source_url (force + no hot-load so the
loader registers once), sourceless uploads get a clear re-upload
error. install_from_url/path grew force/hot_load plumbing.
4 new repair tests (16 total in the pipeline suite); update.sh harness
20 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The panel frontend is now served straight from nginx/sites-available/*
(static SPA at location /, plus /api + /socket.io proxy). A broken
directive there 502s the whole panel on a real box, and nothing parsed
these configs in CI.
Add scripts/test/test_nginx_conf.sh: stages each vhost (secure +
insecure) against a real nginx, recreating the absolute paths they
reference (a stand-in frontend/dist bundle, the serverkit-locations /
conf.d include dirs, certbot webroot) and a throwaway self-signed cert
for the TLS vhost, then runs `nginx -t`. Skips cleanly (exit 0) where
nginx/openssl/root aren't available, so it's safe on a dev laptop.
Wire it into the Scripts CI `shell` job after the unit suites (installs
nginx + openssl so the step actually runs). shellcheck-clean at warning
severity; bash -n clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Turns the zero-downtime promise into an actual check. The updater now
snapshots which managed-app upstreams are reachable through nginx
*before* the blue/green switch, re-probes after the graceful reload, and
reports any app that was serving and has since gone dark.
- discover_app_upstreams scans /etc/nginx/serverkit-locations/*.conf for
the per-app proxy_pass upstreams (host:port).
- probe_app_upstreams / snapshot_app_reachability record up/down (any
HTTP status = up; a refused/timed-out connection = down).
- report_app_uptime_regressions diffs before-vs-after and warns for any
up->down app, ignoring apps that were already down. Best-effort: it
never fails an otherwise-healthy update, just surfaces the regression.
- New "Verifying App Uptime" phase wired into the main flow; baseline
captured before the backend stop.
Tests: T16 (upstream discovery), T17 (regression detection both ways).
19 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The panel is now fully off Docker: the React SPA is served as static
files directly by host nginx from frontend/dist, eliminating the
serverkit-frontend container and the double nginx hop (host nginx ->
container nginx -> dist). Docker is now used ONLY for managed app
workloads.
- nginx sites (serverkit.conf + serverkit-insecure.conf): `location /`
now roots at /opt/serverkit/frontend/dist with a try_files SPA
fallback and a long-cache /assets/ block, instead of proxying
127.0.0.1:3847. Root tracks the /opt/serverkit symlink so blue/green
updates swap assets atomically.
- install.sh: launch_services no longer builds/starts a frontend
container; adds apply_frontend_root (rewrites the dist root for a
custom SERVERKIT_DIR) and selinux_label_frontend_dist (httpd_sys_-
content_t so SELinux hosts can read the bundle from /opt).
- update.sh: refresh_config carries the same root rewrite across
upgrades; selinux_label_dist relabels the switched-in slot; dropped
the `docker compose up -d --force-recreate frontend` from both the
forward and rollback paths.
- docker-compose.yml: frontend service moved behind a `legacy-frontend`
profile (opt-in escape hatch); default `up -d` starts nothing.
- docs/DEPLOYMENT.md: documents the off-Docker panel model.
Tests: T14 (refresh_config rewrites dist root for custom install dir),
T15 (shipped nginx configs serve static, never proxy :3847). 16 passed.
Note: live nginx -t / SELinux file-serving not validated on a RHEL box
from this dev host; logic is best-effort guarded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Host nginx is the front door for every managed app (it reverse-proxies
their containers via /etc/nginx/serverkit-locations/*.conf). The updater
used to `systemctl stop nginx` for the whole blue/green switch window,
which blacked out every unrelated hosted site for 10-60s during a panel
update. The slot switch only ever changes nginx config, never anything
nginx serves from the slot, so a stop was never required.
Add reload_nginx_graceful(): validate config (nginx -t) then
`systemctl reload nginx` - keeps listening sockets and in-flight
connections alive, so apps never blink. Starts nginx if it happens to be
down. Wired into both the forward update flow and rollback(); only the
panel backend is now cycled. Dropped the stray frontend-only
`docker compose up -d backend` line.
Tests: T11 reloads-not-stops a running nginx, T12 starts a stopped
nginx, T13 guards against `systemctl stop nginx` reappearing. 13 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ServerKit's backend manages the host it runs on -- it creates app dirs under
/var/serverkit/apps, drives the host Docker daemon (one container per app),
and reloads nginx. The default compose containerized the backend, so it ran
as a non-root user into a root-owned /var/serverkit/apps with no host
docker.sock: WordPress/app creation failed with
`Permission denied: /var/serverkit/apps/...`. It also contradicted install.sh,
which runs the backend on the host via systemd and expects compose to bring
up only the frontend.
- docker-compose.yml: frontend-only; the frontend reaches the host backend via
extra_hosts ["backend:host-gateway"].
- backend/Dockerfile: chown /var/serverkit to appuser (latent EACCES on the
apps dir for dev/demo container use) + header noting it cannot manage host.
- frontend/nginx.conf: document the backend:host-gateway upstream.
- docs/DEPLOYMENT.md: add a Deployment model note; fix build/start + data paths.
- scripts/test/test_install.sh: T12 guards against re-adding a backend service.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hardening plan §7. Catches distro-specific regressions before release.
- scripts/test/vm-update.sh: install → forced local update (full blue/green
deploy/migrate/switch/health) → re-run to confirm the "Already up to date"
version-gate. Same VM harness contract as vm-install.sh (log + status file).
- scripts/test/vm-uninstall.sh: install + seed data → default uninstall asserts
data preserved (DB snapshot in backups, /var/lib survives) → reinstall →
--purge asserts data removed. End-to-end proof of the unified uninstaller.
- scripts-ci.yml: now runs all three serverless suites (update+install+lib),
lints the new lib/ + test/ files (error severity) and keeps scripts/lib/*
warning-clean, and expands the cross-distro unit matrix to Ubuntu 22.04/24.04,
Debian 12, Rocky 9, AlmaLinux 9, Fedora 40, and openSUSE Leap 15.5.
- test_install.sh: +choose_pkg_manager graceful-failure case (7.4); a no-op
docker stub so the uninstall-matrix cases run where docker isn't installed.
- test_lib.sh: neutralize the ambient container=oci env var (RPM/SUSE images)
and force an empty PATH for the "no package manager" case — both bugs the new
matrix surfaced (the tests, not the code).
- README: fixed the stale Fedora box, documented the new VM scripts + suites.
Validated by running the exact CI container logic locally across all seven
images: test_update 13/13, test_install 15-16/0, test_lib 13/13 everywhere.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hardening plan §6. Reduces the distro-specific if/else surface and handles
edge environments:
- Dependency manifest (scripts/deps/manifest.yaml): declarative per-family
package/python/node/docker/init spec. Consumed directly by the Phase 5
Python bootstrap; install.sh mirrors it inline (it can't parse YAML before
Python exists in the curl|bash path).
- New sourceable libs (built in parallel, each shellcheck-clean + unit-tested):
- scripts/lib/pkg.sh — pkg_detect/refresh/install/is_installed across
apt/dnf/yum/zypper/pacman/apk (PKG_MGR_OVERRIDE + PKG_DRY_RUN for tests).
- scripts/lib/init.sh — init_detect + start/stop/enable/disable/reload/
is_active across systemd/openrc/runit/sysvinit/none (INIT_OVERRIDE/DRY_RUN).
- scripts/lib/env.sh — is_container/is_wsl/has_systemd (override-friendly).
- ID_LIKE handling (G5/6.5): install.sh's os_family_from() (now a pure,
testable function reading ${SERVERKIT_OS_RELEASE}) maps unknown IDs via
ID_LIKE to debian/rhel/fedora/suse/arch/alpine — rhel before fedora so RHEL
clones pick the RHEL Docker repo.
- zypper + pacman + apk (G6/6.6): choose_pkg_manager/refresh_pkg_index/pkg_add,
provision_docker and provision_python gained suse/arch/alpine branches;
Docker enable/start is now init-aware (systemd OR OpenRC) so it no longer
aborts on Alpine.
- Templated systemd unit (6.7): serverkit-backend.service → templates/
serverkit-backend.service.in, rendered (render_service_unit) at install and
in refresh_config so a custom SERVERKIT_DIR/venv/log path is honored.
install_service warns-and-skips enable when has_systemd is false (G5).
- nginx TLS floor as a reversible conf.d snippet (6.8): install.sh + update.sh
prefer /etc/nginx/conf.d/serverkit-tls.conf when nginx.conf doesn't already
declare ssl_protocols (RHEL/Fedora/SUSE), falling back to the in-place edit
on Debian/Ubuntu to avoid a 'duplicate ssl_protocols' error. uninstall
removes only that snippet.
Tests: new scripts/test/test_lib.sh (13) + 4 new test_install.sh cases
(os_family_from, render_service_unit, harden_global_tls A/B). 13+15+10 pass;
shellcheck error+warning clean across all new/changed shell.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updater robustness from the hardening plan §5:
- Version comparison before updating (G3): is_already_current compares the
local VERSION to the target release tag (release mode) or the local checkout
HEAD to the remote branch HEAD via `git ls-remote` (branch/main mode). When
current, both the hybrid and docker flows print "Already up to date" and exit
0 — unless --force. Offline/indeterminate always proceeds rather than wrongly
blocking.
- Dry-run fidelity: new run_in_dir reports the working directory a command
would run in, replacing the `run_or_dry cd X && run_or_dry compose ...` chain
that printed compose commands with no cwd context.
- Rollback hardening: migrate_database now runs against the NEW slot's DB copy
(slot-absolute path) instead of the /opt/serverkit symlink that still
resolves to the live old slot — so a rollback hands the old code a
byte-for-byte-unchanged database (PostgreSQL/shared DBs migrate directly, as
before). rollback_docker guards `git reset` on a non-empty SHA with a clear
warning. Both rollback paths now run a post-rollback health check and say
plainly when even the rollback is unhealthy.
- Telemetry/logging: halt() always prints the full log path; both summaries
gained an "old → new", mode (release/branch/main, +docker), and duration
line, plus the log path.
Tests: test_update.sh gains versions_equal, is_already_current force/offline
short-circuits, and a flask-stub proof that migrate_database targets the new
slot's DB (T10). 10 passed / 2 skipped locally; shellcheck error+warning clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Critical install/update/uninstall hotfixes from the hardening plan
(docs/plans/INSTALLER_UPDATER_UNINSTALLER_HARDENING_PLAN.md §4):
- Firewall automation (G1): new sourceable scripts/lib/firewall.sh detects
firewalld/ufw/nftables/iptables and opens 80/443; install.sh runs
configure_firewall, update.sh re-ensures it idempotently. Exactly what we
open is recorded in /etc/serverkit/install-state.json (scripts/lib/state.sh)
so uninstall undoes only our rules. Fixes the #1 "installed but unreachable"
failure on fresh RHEL-family boxes.
- Accept Python 3.11 OR 3.12 (G2): locate_python now iterates
python3.12/3.11/3 and provision_python installs the distro package
(python3.11 on Debian 12, 3.12 on Ubuntu/Fedora, 3.11/3.12 on RHEL) before
ever falling back to a source compile.
- Use $PYTHON_BIN for secret generation instead of a bare python3.
- Distro-first Node 20 (replacing unconditional NodeSource): install the
distro nodejs+npm, fall back to NodeSource only when it's missing or < 18,
and validate node>=18 before continuing.
- Unify uninstall (G4): new scripts/lib/uninstall.sh is the single canonical
teardown routine shared by uninstall.sh and `serverkit uninstall` (they had
drifted — one ran `down -v`, one didn't). Default preserves user data
(volumes kept, DB snapshotted to backups); --purge/--delete-volumes removes
volumes+DB+data dirs; --keep-data preserves /etc + /var/lib + backups;
--yes/--dry-run added; honors SERVERKIT_DIR. Removes the apt lock-wait
drop-in on uninstall.
Tests: new scripts/test/test_install.sh (11 cases, source-able install.sh)
covers python/node detection, firewall dry-run + state, and the uninstall
data-handling matrix. Existing test_update.sh still green. shellcheck clean
(error + warning) on all new/changed shell.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two reliability gaps remained after the failure cascade:
1. Stale-updater trap. The updater that runs is whatever is installed, so an
old update.sh fails in ways the current code already handles and can't fix
itself — exactly what left a box stuck. maybe_reexec_latest_updater fetches
the newest update.sh for the target ref and re-execs into it (guarded by
SERVERKIT_UPDATER_REEXECED, skipped offline/dry-run/opt-out, only after a
bash -n check). From this version on, 'serverkit update' is reliable no
matter how stale the box is.
2. No concurrency guard. acquire_update_lock takes an flock so two updates
can't clobber the same blue/green slot or race docker compose; the lock
auto-releases on exit and degrades gracefully where flock/the path is
unavailable.
Both are covered by scripts/test/test_update.sh (guard no-ops; lock
refuses a held lock, succeeds when free).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Harden update.sh against the class of failure that left a box silently stuck
on the old version, and make it testable without a real deploy.
Robustness (cascade):
- L5 loud failures: `set -Eeuo pipefail` (the -E/errtrace makes the ERR trap
fire inside functions too) + a report_failure ERR trap that names the phase,
exit code, line and command, plus always-on logging to
/var/log/serverkit/update-*.log. No more silent drop-to-prompt.
- L4 docker-path rollback: snapshot the previous git SHA, :rollback image tags
and dist.rollback before mutating; if the new backend is not healthy within
90s, revert to the snapshot and halt (was: warn and continue).
- L6 self-heal: pin COMPOSE_PROJECT_NAME (kills the symlink-rename warning) and
reclaim a stale inactive blue/green slot left by a prior failed hybrid run.
Testability:
- Parameterize the system dirs (SERVERKIT_NGINX_DIR / _LETSENCRYPT_DIR /
_SYSTEMD_DIR / _CONFIG_DIR) and add SERVERKIT_NO_LOG, so refresh_config can
run against fixtures instead of the host /etc.
- Make update.sh source-able via a BASH_SOURCE guard so functions can be
exercised directly.
- scripts/test/test_update.sh: plain-bash unit tests (no deps, no server) that
reproduce the missing-serverkit.conf regression and cover the
detection/slot/report logic.
- .github/workflows/scripts-ci.yml: bash -n + shellcheck + the unit tests on
changes to scripts/**, install.sh, serverkit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Go agent has moved to its own repository:
https://github.com/jhd3197/serverkit-agent
Changes:
- Delete agent/ directory and all source, build, and packaging files
- Delete .github/workflows/agent-ci.yml and agent-release.yml
- Delete scripts/test/agent-test.ps1 (agent now builds in separate repo)
- Remove committed dist/serverkit-agent-windows-amd64.exe build artifact
- Update docs/INSTALLATION.md and docs/README.md links to point to the new repo
- Add migration notes to docs/AGENT_AUDIT.md, docs/REMOTE_ACCESS_ROADMAP.md,
SECURITY_AUDIT.md, and docs/plans/AGENT_FLEET_PHASES_3-4.md
Panel agent API endpoints under /api/v1/servers/agent/ and /api/v1/agent/
are retained — they are the control-plane contract with the standalone agent.
Verified:
- backend: pytest (371 passed, 12 skipped)
- frontend: npm run lint (0 errors)
- Vagrantfile: don't `dnf install rsync/curl` on RHEL boxes — it upgrades
openssl-libs and breaks the box's sshd (same ABI issue as install.sh).
rsync/curl ship preinstalled on generic/rocky9; fail loudly if a future
box lacks them rather than silently corrupting the run.
- full-stack-test.ps1: drop fedora from the default distro set (Vagrant
Cloud's generic/fedora is EOL at f39; opt in via -Only fedora).
- vm-install.sh (new): run install.sh from uploaded local source, overlay
the working tree, rebuild, and probe health — so the harness tests the
exact local code (incl. uncommitted) with sshd/firewall snapshots on every
exit path for post-mortems. Recovered from a local stash.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
install.sh:
- Detect rocky/almalinux/rhel/centos as OS_FAMILY=rhel.
- The Fedora system-deps block now also runs for rhel; auto-install
epel-release on RHEL-family.
- Docker install on Fedora/RHEL no longer pipes get.docker.com. That
script recently added docker-model-plugin to its package list, which
doesn't exist in docker-ce repos and aborts the install on Fedora 39.
Replace it with explicit `dnf config-manager --add-repo` +
`dnf install docker-ce docker-ce-cli containerd.io
docker-compose-plugin docker-buildx-plugin`.
- Node.js and nginx installs route through dnf for rhel too.
- SELinux setsebool runs for rhel too.
scripts/test/full-stack-test.ps1:
- fedora image bumped from fedora39 (EOL Nov 2024) to fedora40 (current).
- Test-VagrantInstall + Test-VmInstall: defensive read of status file.
Previously `(Get-Content -Raw).Trim()` crashed with InvokeMethodOnNull
when the status file was missing/empty (which happens when the VM's
install script crashed before producing it). Now falls back to 'FAIL'
cleanly so the run summary still works.
Found by full 5-distro suite: Ubuntu 22/24 + Debian 12 passed; Fedora 39
failed on docker-model-plugin; Rocky 9 failed silently due to apt-vs-dnf
+ status NPE. All three fixed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous commit added Tee-Object so vagrant up output was visible
in the terminal, but Tee-Object's pipeline output then leaked into
Test-VagrantInstall's return stream — the function returned vagrant's
entire log instead of "OK", so the final summary reported every passing
Vagrant VM as a failure.
Append `| Out-Host` after Tee-Object to consume the pipeline. The
operator still sees the live output; the function's return stays clean.
The previous Debian 12 run was actually all-green (10/10 pytest passed,
0 plugin-loader errors) — only the summary line was wrong.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
