Both the installer and the updater treated a missing checksums.txt as
"skip verification" and installed the tarball anyway. Every release
publishes one, so its absence means a broken mirror or a tampered
download. The installer now falls back to a source build and the
updater refuses; SERVERKIT_ALLOW_UNVERIFIED=1 overrides for mirrors
that do not host checksums.
The openSUSE Leap image ships without tar, so deploy_release could not
unpack the fixture and T20b failed its deploy assertions there. The
test now skips without tar, like its neighbours skip without
sha256sum.
On blue/green installs the previous slot is the code rollback, so the
serverkit-tree-* copy of it duplicated ~1 GB per update. With
SERVERKIT_BACKUP_RETENTION unset, disks under 50 GiB now keep one
database snapshot instead of three.
download_release runs in a command substitution, so its mktemp dir
was never removed: every update stranded the ~70 MB tarball in /tmp
(11 dirs, 644 MB on a 25 GB box). The download and unpack dirs are
now removed on success and on abort, and old leftovers are swept.
- Harness: bootstrap find/awk in distro-compatibility.py (AL2023,
Tumbleweed, SLES BCI minimal images lack them; the source-level
suites legitimately use both)
- Python gate 3.11-3.14: Arch/Manjaro rolling ship only python 3.14
with no versioned fallback packages (same trap as Debian 13/#99);
align scripts/update.sh, which had drifted to a 3.11-3.12 gate
- Gentoo: add emerge as a supported installer package-manager path
(install.sh, scripts/lib/pkg.sh, distro.py family, deps manifest);
provision via slotted dev-lang/python:3.13 -> 3.12
- Tests: move out-of-range stubs above the new ceiling, shadow all
locate_python candidate names (real python3.14/3.13 in Manjaro/
Gentoo images were winning over fixtures), cover the 7th manager
- Catalog: Gentoo is no longer a documented expected failure
Verified: test_lib 32/32, test_install 124/124, test_update 76/76,
pytest test_serverkit_installer 46/46, and all 6 previously failing
images now pass both quick and provision container probes (12/12).
A backup retention cap already existed (SERVERKIT_BACKUP_RETENTION,
default 5), yet a box was found holding six snapshots and 0 bytes free.
Three separate defects, each sufficient on its own:
1. cleanup() is the LAST phase of the update, so retention never ran on
a run that died earlier -- including the run that died BECAUSE the
disk was full. Retention that only executes on success cannot bound a
failure loop. backup_current now trims BEFORE writing, which also
makes peak usage `keep` snapshots rather than `keep + 1`.
2. The preflight required a flat 2 GiB, but the update is about to write
two full copies of the database. With an 800 MB database a box at
2.1 GiB free passed the check and then filled mid-update. The
requirement is now 2 GiB + 2x the database, it trims stale snapshots
and re-checks before giving up, and the halt message points at
`serverkit disk`.
3. A count cap is not a disk guarantee: five snapshots of an 800 MB
database is 8 GB, a third of a 25 GB droplet. The default drops to 3
and enforce_backup_budget caps backups at a share of the filesystem
(SERVERKIT_BACKUP_MAX_PERCENT, default 15), dropping oldest-first and
always keeping at least one restore point -- an update with no way
back is worse than a full disk.
BACKUP_DIR also becomes SERVERKIT_BACKUP_DIR-overridable so it can agree
with the reclaim service, which scans the same directory.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- install_profile_service: record what install.sh put on the box
(minimal/standard/full), recommended from detected hardware and
overridable; thresholds mirrored by recommend_profile in install.sh
- setup wizard: replace the hard ResourceGate/SetupStepTier with
SetupStepCapacity + SetupStepSecurity
- ResourceAdvisory: inline capacity warning at the point of action
(WordPress/Docker/Services/Deployments), backed by headroom.fits
- RequiresDocker component for docker-dependent surfaces
- install.sh/update.sh: profile detection + tests
(test_install.sh, test_update.sh, test_server_capacity.py)
backup_current tested `[ -f serverkit.db ]` FIRST and only fell through to the
PostgreSQL branch when no file was there. An install migrated SQLite ->
PostgreSQL keeps its old serverkit.db, and both deploy paths copy it forward on
file-existence alone (deploy_source, deploy_release), so the stale file is
carried into every future slot and never ages out. Such a box backed up the
abandoned file, skipped pg_dump entirely, and left PRE_UPGRADE_DUMP unset --
which migrate_database reads as "${PRE_UPGRADE_DUMP:-}", so its auto-restore
silently did nothing. PostgreSQL is the one engine that migrates IN PLACE and
cannot roll back by switching slots, so that dump is its only way home: the
exact safety net a61eab1/b48256f added was disabled for it.
migrate_database has always keyed off DATABASE_URL (its sqlite check). Test the
declared engine first so the two functions agree.
Deliberately NOT `grep DATABASE_URL=sqlite && [ -f db ]` as the reviewer
suggested: backend/config.py defaults DATABASE_URL when absent, so any install
whose .env lacks the line would fall to the else branch and update with NO
backup at all -- trading a mismatched backup for none. An explicit `postgres`
URL is a positive signal; the absence of a `sqlite` one is not.
Also:
* name the engine when it is one we cannot dump. "No SQLite database ..."
read like a benign nothing-to-do on a MySQL install that in fact had no
safety net. Prints the scheme only -- a DATABASE_URL carries a password and
this text lands in the update log.
* drop the stale "Python 3.11/3.12 is guaranteed by pre-flight" comment,
contradicted by locate_sqlite_python two lines below it (664a5e6).
* discord-notify: skip cleanly when DISCORD_WEBHOOK_URL is absent. Secrets
are not exposed to `pull_request` runs from a fork, so a merged fork PR
failed the job on an empty curl URL. Not switching to pull_request_target
for this -- handing a write-scoped token and the webhook to an
outsider-triggered run is a poor trade for a chat notification.
Tests: a PostgreSQL install carrying a leftover serverkit.db must still dump
and set PRE_UPGRADE_DUMP (fails on the parent commit, rc=1); an undumpable
engine must be named without leaking the password. Neither needs python3, so
both run on all 7 images. Workflow verified end-to-end against a synthetic
event payload: fork PR exits 0 without calling curl, normal PR still builds the
same embed.
Verified on the full CI matrix (ubuntu 22.04/24.04, debian:12, rockylinux:9,
almalinux:9, fedora:40, opensuse/leap:15.5), all six suites: 0 failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
copy_sqlite_db and verify_sqlite_db reused locate_python, which enforces
3.11 <= v <= 3.12. That bound is correct for rebuild_virtualenv and
preflight_check, but snapshotting or integrity-checking a SQLite file needs
nothing beyond the stdlib sqlite3 module. Reusing it coupled the DATA-safety
paths to the venv-builder constraint and blocked updates for a reason unrelated
to the data:
* an all-Docker install returns before preflight_check ever runs and does
every real step in the container -- yet backup_docker_db snapshots with the
CONTAINER's python and then verified the extracted file with the HOST's.
On a stock EL9 (python3.9) or Ubuntu 22.04 (python3.10) host that halted
with "Could not obtain a VALID database backup", blaming the backup for a
missing interpreter.
* backup_current halts when it cannot snapshot, so those hosts could never
update at all.
Split out locate_sqlite_python: any candidate that can `import sqlite3` wins,
and the install's own venv interpreter is preferred -- on a real box it is
guaranteed present, guaranteed to carry the module, and independent of PATH.
locate_python keeps the 3.11/3.12 bound where it belongs.
Surfaced by the scripts-ci unit matrix: almalinux:9 and rockylinux:9 ship
python3.9, so the suite's own probe accepted an interpreter locate_python
rejected, and copy_sqlite_db/verify_sqlite_db failed there while passing on
python3.12 runners.
T10i pins the contract: on a box where every candidate reports 3.9 (locate_python
fails), the helpers still resolve an interpreter, prefer the slot's venv, and
round-trip 100 rows. Fails on the parent commit, passes here.
Verified on the full 7-image CI matrix (ubuntu 22.04/24.04, debian:12,
rockylinux:9, almalinux:9, fedora:40, opensuse/leap:15.5), all six shell suites:
0 failed. test_update is 63 passed / 0 failed on every image carrying a python3.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
verify_pg_dump only runs its structural `pg_restore -l` TOC check when the
binary is on PATH. The harness stubs every other external command it touches
(systemctl, nginx, npm, curl, docker, flask, pg_dump) but never pg_restore, so
two assertions were decided by the host:
* no postgresql-client -> branch skipped, check degrades to "non-empty",
and the synthetic `PGDMP fake contents` fixture passes
* postgresql-client present -> the real binary parses the version bytes
after the PGDMP magic, rejects the fixture, verify_pg_dump returns 1
That is the CI failure. T10f failed directly; T10e failed downstream because
backup_current correctly halts on an unverifiable backup. update.sh is right --
the tests leaked.
Add a make_stub_pg_restore factory mimicking the real -l semantics (rejects
missing/empty, rejects anything without the PGDMP magic, else prints a TOC) and
install it in T10e/T10f. Give T10f a non-empty non-archive fixture too: that is
the case separating the two branches inside verify_pg_dump -- the size check
passes it, only the TOC listing catches it -- so the path the test was named
for is now actually exercised.
Verified both with and without a pg_restore on PATH: 59 passed, 0 failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A backup that was never verified is a hope, not a safety net. Closes the
remaining ways the pre-upgrade safety net could silently not exist:
- Every backup is verified immediately after creation: SQLite snapshots via
PRAGMA integrity_check, PostgreSQL dumps via size + pg_restore TOC
listing (when available). Invalid artifacts are deleted and the update
halts — a malformed backup is never mistaken for a good one.
- Partial artifacts from a failed backup are removed (rm on failure), so a
half-written file can never be 'restored' later.
- A failed IN-PLACE PostgreSQL migration now automatically restores the
verified pre-upgrade dump (pg_restore --clean --if-exists) before
halting; without it the half-migrated schema would stay live against
the old code. If the restore itself fails, the halt message carries the
exact manual pg_restore command.
- backup_docker_db tries both python and python3 in the container, and now
verifies the result too — the plain docker cp fallback can no longer
pass off a torn copy as a backup.
- Backup retention capped: default 5 (SERVERKIT_BACKUP_RETENTION
overrides) across trees, .db and .dump files. Full-size DB copies pile
up fast — a real 25G droplet carried 6.3G of stale backups.
Tests: T10f (verification helpers accept/reject correctly), T10g (failed
in-place PG migration auto-restores with the exact pg_restore args),
T10h (retention prunes all three backup kinds to the configured cap).
Postgres panels updated with zero safety net: backup_current found no
SQLite file and just warned, while the migration runs IN PLACE against the
shared external DB — a failed migration left no way back.
- backup_current now pg_dumps (custom format) when .env points at
PostgreSQL, sanitizing SQLAlchemy driver suffixes (postgresql+psycopg2://)
that libpq can't parse.
- Same rule as SQLite: no valid backup, no update — halt when pg_dump is
missing (install postgresql-client) or the dump fails.
Tests: T10e covers the dump invocation + URL sanitization and the
halt-on-failed-dump path.
Root cause of a real-world failed update: backup_current, deploy_source and
deploy_release all plain-cp'd the LIVE panel database. On a busy panel that
races the backend's own writes and the copy can capture half a transaction
— 'database disk image is malformed' — which then sinks the migration in
the new slot and silently poisons the pre-upgrade backup that was supposed
to be the safety net.
- New copy_sqlite_db() uses the SQLite online backup API via python
(guaranteed by pre-flight) and refuses to fall back to a racy cp.
- backup_current halts rather than proceed without a valid safety net.
- deploy_source/deploy_release halt before any switch if the live DB
cannot be snapshotted consistently.
- backup_docker_db prefers a consistent snapshot via docker exec python,
with the plain docker cp as a fallback.
- locate_python rejects candidates whose version probe returns empty
(broken shims pass 'command -v' but can't execute; an empty version
string used to pass the sort-based range check).
Tests: T10d exercises a real snapshot (integrity ok, all rows); a contract
test bans plain-cp of the live DB; T10d fixture python detection validates
the interpreter actually runs.
The SQLite pre-check in migrate_database treated an EMPTY probe result
(unusable venv python) the same as real corruption and halted the update —
a false positive that broke the T10/T24 contract tests. Now:
- probe output != 'ok' -> genuine corruption -> halt with repair guidance
- empty probe output -> probe couldn't run -> warn and let flask surface
any real error
test_update.sh: T10/T24 fixtures stub the venv python (healthy-DB path);
new T10b asserts the corrupt-DB halt fires before flask runs; new T10c
asserts an unrunnable probe warns and proceeds.
Make `update.sh` fetch the CI-built release tarball by default instead of
rebuilding the SPA on the server, so a normal update needs no Node/npm on the
box. This keeps updates working on old distros, ARM/Pi and tiny hosts where the
vite 8 toolchain (Node 20.19+/22.12+) can't or shouldn't run — the Node
requirement stays a build-time/CI concern, not a runtime one.
On-box source rebuilds remain available and explicit: --source / --main (main
branch), --branch <name>, or BUILD_FROM_SOURCE=1. --branch now implies source
mode (clears the release default). npm is only required for those paths.
Test: test_update.sh +1 (mode mapping — default=release, --source/--branch/
BUILD_FROM_SOURCE=source). 49 passed / 0 failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The vite 8 upgrade (dcf05d1) makes the frontend build require Node ^20.19.0 ||
>=22.12.0 — on anything older npm silently skips rolldown's native binary and
`npm run build` dies with a cryptic MODULE_NOT_FOUND. Servers that build the SPA
on-box (source install + every update) would break their next update. So:
- install.sh node_ready() now gates on the real vite 8 floor (was Node >=18) and
provisions NodeSource 22 LTS when the host Node is too old.
- update.sh gains _node_build_ok(): build_frontend_bundle refuses (returns 1 →
caller rolls back) and the Docker-host rebuild halts, both with an actionable
"upgrade Node" message instead of a rolldown stack trace.
- Dockerfile + CI (release, release-smoke, extensions, scripts-nightly) bumped
node 20 -> 22 so the shipped/pre-built bundle is built on a compatible Node.
Also fixes a pre-existing parity bug the operator caught: update.sh rendered the
systemd unit WITHOUT substituting @BIND_HOST@ (install.sh does), so an update
left the literal placeholder and gunicorn would fail to bind. update.sh now
substitutes it (loopback default, SERVERKIT_BIND_HOST override) like install.sh.
Tests: test_update.sh +3 (node-floor gate, build refusal on old Node,
@BIND_HOST@ substitution) — 48 passed / 0 failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 1.7.25 -> 1.7.33 update broke on panels that still had the retired
serverkit-workflows extension on disk: preserve_installed_plugins carried
it into the fresh tree, where its reactflow import (removed with plan 45)
sank the whole bundle build and blocked the update.
- RETIRED_PLUGIN_SLUGS: the carry-forward now skips slugs the product
deliberately deleted, and docker in-place mode strips their leftovers
after the hard reset (untracked dirs survive it).
- Quarantine-and-retry: when the bundle build fails and plugins were
carried forward, move them into /var/backups/serverkit/quarantined-
plugins-<ts> and retry once -- fail the plugin, never the update.
The updater self-refreshes from main before running, so this protects
every panel on its next `serverkit update` once merged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
serverkit migrate-db forwarded the wrapper's own name to cli.py, which
only defines db-migrate — the command has failed with 'No such command'
since the rename. Fix the forward and stop the drift class entirely:
- cmd_panel_cli forwards Click-only verbs verbatim (manifest, services,
apps, repair, login-url, support-bundle, db-migrate/status/history;
panel-status/panel-doctor where a native bash name collides)
- serverkit completion prints a bash completion script; install.sh and
update.sh drop it into /etc/bash_completion.d (best-effort), the
uninstall lib removes it
- test_cli.sh T17-T19: migrate-db forwarding, dispatch reachability,
and a two-way parity guard (cli.py verbs <-> dispatch <-> completion);
test_update.sh T29 covers refresh_bash_completion
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A test box's panel domain kept serving a Jun 25 bundle after the Jul 3
update — the domain's custom vhost still proxied the retired :3847
frontend container, and the old bundle's calls to deleted legacy
marketplace endpoints read as 'Failed to load extensions'. It looked
exactly like a browser/CDN cache problem (it wasn't — cf-cache-status
was DYNAMIC; the origin itself was stale).
- both stock confs: expires -1 on the SPA entrypoint (emits
Cache-Control: no-cache WITHOUT discarding server-level security
headers, unlike add_header-in-location) — a cached index.html
references hashed assets that vanish on the next blue/green switch
- update.sh refresh_config now runs report_stale_panel_vhosts: warns,
naming the file, when any enabled vhost still proxies :3847;
observation-only per R1, silent on a missing sites-enabled dir,
added to the fresh-box loop per policy
- proving tests: entrypoint no-cache asserted for both confs; stale
vhost flagged (and clean one not); missing-dir silence (42 passed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bash suppresses set -e inside any tested-context command list — an
if-condition subshell, or the left side of ||, even with set -Eeuo
pipefail re-armed inside — so multi-statement test bodies only gated
on their LAST statement. Today's cmp-less EL9 CI run proved it: a
byte-identity test passed while its cmp assertions exited 127.
28 blocks across test_update/test_install/test_lib/test_agent_install
converted to explicit && chains so every assertion gates the result.
No test weakened; no vacuous pass was hiding a real script bug (each
newly-gated statement was verified to genuinely succeed).
Verified: Git Bash 171 passed / 0 failed; rockylinux:9 all five
suites 177p/0f; opensuse/leap:15.5 148p/0f (pre-existing python3/tar
skips).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The July 2 outage function had a unit test — with a populated fixture
only. This encodes the missing net:
- fresh-minimal-box fixture loop: every observation/discovery/snapshot/
report function in update.sh (17) and install.sh (16) must survive a
zero-app box under set -Eeuo pipefail; policy comment requires new
functions to be appended
- scripts/test/stubs.sh: shared failing-stub factories (systemctl exit
5, docker fail/empty, curl exit 22, GitHub rate-limit JSON)
- assertion sweep: 43 bare capture-then-compare sites now include exit
status in the assertion
- best-effort contract test: the run block's real APP_BASELINE ->
report chain exits 0 on a fresh box AND on a genuine regression
- fix(update): the loop immediately caught report_failure's trailing
'[ -n $UPDATE_LOG ] && printf' returning 1 when UPDATE_LOG is empty
(same species as the outage) — converted to if, added to the loop
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- find-based backup pruning; empty/missing BACKUP_DIR is a no-op (U1)
- download_release progress to stderr; stdout is the tarball path only (U2)
- guard bare systemctl stop/start/daemon-reload in forward+rollback paths (U3,U4,U8,U13)
- ROLLING_BACK=1 inside rollback(); never runs twice via the EXIT trap (U5)
- npm preflight for source updates; guarded frontend build pipeline (U6)
- empty-ORIG_ARGS re-exec guard, cleanup_on_exit return 0, df -Pk,
SIGPIPE guards, cd leaks subshelled, local typo (U7,U9-U12,U14)
- 14 new proving assertions in test_update.sh (35 passed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Allow failures when querying GitHub release tag (prevent pipeline from aborting) and add a unit test ensuring discover_app_upstreams tolerates an empty app-locations directory.
Changes:
- scripts/update.sh: make the curl|grep pipeline resilient by appending `|| true` so the command substitution won't fail the script when the GitHub API call returns nothing.
- scripts/test/test_update.sh: add test T16b to verify discover_app_upstreams does not abort under `set -euo pipefail` when the app-locations directory is empty.
These changes prevent hard failures during release detection and add coverage for empty-directory handling.
A panel update deploys a fresh clone/tarball preserving only .env and
the DB, wiping app/plugins/<slug> + src/plugins/<slug> for any
URL/registry/upload-installed extension — the surviving InstalledPlugin
row then flipped to error at next boot. Two-layer fix:
- update.sh: preserve_installed_plugins carries user-installed plugin
dirs from the old tree into the new one (skipping anything the new
tree already ships and __pycache__), before the frontend build so
carried UIs recompile. Covered by harness test T18.
- boot repair pass (plugin_service.repair_missing_plugins, runs before
load_all_plugins): builtin slugs reinstall from builtin-extensions/,
url installs re-download from source_url (force + no hot-load so the
loader registers once), sourceless uploads get a clear re-upload
error. install_from_url/path grew force/hot_load plumbing.
4 new repair tests (16 total in the pipeline suite); update.sh harness
20 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Turns the zero-downtime promise into an actual check. The updater now
snapshots which managed-app upstreams are reachable through nginx
*before* the blue/green switch, re-probes after the graceful reload, and
reports any app that was serving and has since gone dark.
- discover_app_upstreams scans /etc/nginx/serverkit-locations/*.conf for
the per-app proxy_pass upstreams (host:port).
- probe_app_upstreams / snapshot_app_reachability record up/down (any
HTTP status = up; a refused/timed-out connection = down).
- report_app_uptime_regressions diffs before-vs-after and warns for any
up->down app, ignoring apps that were already down. Best-effort: it
never fails an otherwise-healthy update, just surfaces the regression.
- New "Verifying App Uptime" phase wired into the main flow; baseline
captured before the backend stop.
Tests: T16 (upstream discovery), T17 (regression detection both ways).
19 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The panel is now fully off Docker: the React SPA is served as static
files directly by host nginx from frontend/dist, eliminating the
serverkit-frontend container and the double nginx hop (host nginx ->
container nginx -> dist). Docker is now used ONLY for managed app
workloads.
- nginx sites (serverkit.conf + serverkit-insecure.conf): `location /`
now roots at /opt/serverkit/frontend/dist with a try_files SPA
fallback and a long-cache /assets/ block, instead of proxying
127.0.0.1:3847. Root tracks the /opt/serverkit symlink so blue/green
updates swap assets atomically.
- install.sh: launch_services no longer builds/starts a frontend
container; adds apply_frontend_root (rewrites the dist root for a
custom SERVERKIT_DIR) and selinux_label_frontend_dist (httpd_sys_-
content_t so SELinux hosts can read the bundle from /opt).
- update.sh: refresh_config carries the same root rewrite across
upgrades; selinux_label_dist relabels the switched-in slot; dropped
the `docker compose up -d --force-recreate frontend` from both the
forward and rollback paths.
- docker-compose.yml: frontend service moved behind a `legacy-frontend`
profile (opt-in escape hatch); default `up -d` starts nothing.
- docs/DEPLOYMENT.md: documents the off-Docker panel model.
Tests: T14 (refresh_config rewrites dist root for custom install dir),
T15 (shipped nginx configs serve static, never proxy :3847). 16 passed.
Note: live nginx -t / SELinux file-serving not validated on a RHEL box
from this dev host; logic is best-effort guarded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Host nginx is the front door for every managed app (it reverse-proxies
their containers via /etc/nginx/serverkit-locations/*.conf). The updater
used to `systemctl stop nginx` for the whole blue/green switch window,
which blacked out every unrelated hosted site for 10-60s during a panel
update. The slot switch only ever changes nginx config, never anything
nginx serves from the slot, so a stop was never required.
Add reload_nginx_graceful(): validate config (nginx -t) then
`systemctl reload nginx` - keeps listening sockets and in-flight
connections alive, so apps never blink. Starts nginx if it happens to be
down. Wired into both the forward update flow and rollback(); only the
panel backend is now cycled. Dropped the stray frontend-only
`docker compose up -d backend` line.
Tests: T11 reloads-not-stops a running nginx, T12 starts a stopped
nginx, T13 guards against `systemctl stop nginx` reappearing. 13 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updater robustness from the hardening plan §5:
- Version comparison before updating (G3): is_already_current compares the
local VERSION to the target release tag (release mode) or the local checkout
HEAD to the remote branch HEAD via `git ls-remote` (branch/main mode). When
current, both the hybrid and docker flows print "Already up to date" and exit
0 — unless --force. Offline/indeterminate always proceeds rather than wrongly
blocking.
- Dry-run fidelity: new run_in_dir reports the working directory a command
would run in, replacing the `run_or_dry cd X && run_or_dry compose ...` chain
that printed compose commands with no cwd context.
- Rollback hardening: migrate_database now runs against the NEW slot's DB copy
(slot-absolute path) instead of the /opt/serverkit symlink that still
resolves to the live old slot — so a rollback hands the old code a
byte-for-byte-unchanged database (PostgreSQL/shared DBs migrate directly, as
before). rollback_docker guards `git reset` on a non-empty SHA with a clear
warning. Both rollback paths now run a post-rollback health check and say
plainly when even the rollback is unhealthy.
- Telemetry/logging: halt() always prints the full log path; both summaries
gained an "old → new", mode (release/branch/main, +docker), and duration
line, plus the log path.
Tests: test_update.sh gains versions_equal, is_already_current force/offline
short-circuits, and a flask-stub proof that migrate_database targets the new
slot's DB (T10). 10 passed / 2 skipped locally; shellcheck error+warning clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two reliability gaps remained after the failure cascade:
1. Stale-updater trap. The updater that runs is whatever is installed, so an
old update.sh fails in ways the current code already handles and can't fix
itself — exactly what left a box stuck. maybe_reexec_latest_updater fetches
the newest update.sh for the target ref and re-execs into it (guarded by
SERVERKIT_UPDATER_REEXECED, skipped offline/dry-run/opt-out, only after a
bash -n check). From this version on, 'serverkit update' is reliable no
matter how stale the box is.
2. No concurrency guard. acquire_update_lock takes an flock so two updates
can't clobber the same blue/green slot or race docker compose; the lock
auto-releases on exit and degrades gracefully where flock/the path is
unavailable.
Both are covered by scripts/test/test_update.sh (guard no-ops; lock
refuses a held lock, succeeds when free).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Harden update.sh against the class of failure that left a box silently stuck
on the old version, and make it testable without a real deploy.
Robustness (cascade):
- L5 loud failures: `set -Eeuo pipefail` (the -E/errtrace makes the ERR trap
fire inside functions too) + a report_failure ERR trap that names the phase,
exit code, line and command, plus always-on logging to
/var/log/serverkit/update-*.log. No more silent drop-to-prompt.
- L4 docker-path rollback: snapshot the previous git SHA, :rollback image tags
and dist.rollback before mutating; if the new backend is not healthy within
90s, revert to the snapshot and halt (was: warn and continue).
- L6 self-heal: pin COMPOSE_PROJECT_NAME (kills the symlink-rename warning) and
reclaim a stale inactive blue/green slot left by a prior failed hybrid run.
Testability:
- Parameterize the system dirs (SERVERKIT_NGINX_DIR / _LETSENCRYPT_DIR /
_SYSTEMD_DIR / _CONFIG_DIR) and add SERVERKIT_NO_LOG, so refresh_config can
run against fixtures instead of the host /etc.
- Make update.sh source-able via a BASH_SOURCE guard so functions can be
exercised directly.
- scripts/test/test_update.sh: plain-bash unit tests (no deps, no server) that
reproduce the missing-serverkit.conf regression and cover the
detection/slot/report logic.
- .github/workflows/scripts-ci.yml: bash -n + shellcheck + the unit tests on
changes to scripts/**, install.sh, serverkit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Juan Denis