bash suppresses set -e inside any tested-context command list — an
if-condition subshell, or the left side of ||, even with set -Eeuo
pipefail re-armed inside — so multi-statement test bodies only gated
on their LAST statement. Today's cmp-less EL9 CI run proved it: a
byte-identity test passed while its cmp assertions exited 127.
28 blocks across test_update/test_install/test_lib/test_agent_install
converted to explicit && chains so every assertion gates the result.
No test weakened; no vacuous pass was hiding a real script bug (each
newly-gated statement was verified to genuinely succeed).
Verified: Git Bash 171 passed / 0 failed; rockylinux:9 all five
suites 177p/0f; opensuse/leap:15.5 148p/0f (pre-existing python3/tar
skips).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
EL9/Fedora containers ship without diffutils and openSUSE Leap without
tar, which failed 4 of the 7 cross-distro unit jobs:
- files_identical() in stubs.sh: cmp when available, sha256sum
(coreutils, everywhere) otherwise
- the two tarball-fixture tests skip cleanly when tar is absent
(harness's existing skip convention; proven on the other distros)
- the t16/t16r assertion lists are now && chains: bash suppresses
set -e inside an if-condition subshell, so sequential statements
only gated on the LAST one — the cmp-less EL9 run proved it by
passing t16r vacuously while cmp exited 127
Verified in the actual images: rockylinux:9 65p/0f/0s,
opensuse/leap:15.5 61p/0f/4s (was 63p/2f and 61p/2f).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The July 2 outage function had a unit test — with a populated fixture
only. This encodes the missing net:
- fresh-minimal-box fixture loop: every observation/discovery/snapshot/
report function in update.sh (17) and install.sh (16) must survive a
zero-app box under set -Eeuo pipefail; policy comment requires new
functions to be appended
- scripts/test/stubs.sh: shared failing-stub factories (systemctl exit
5, docker fail/empty, curl exit 22, GitHub rate-limit JSON)
- assertion sweep: 43 bare capture-then-compare sites now include exit
status in the assertion
- best-effort contract test: the run block's real APP_BASELINE ->
report chain exits 0 on a fresh box AND on a genuine regression
- fix(update): the loop immediately caught report_failure's trailing
'[ -n $UPDATE_LOG ] && printf' returning 1 when UPDATE_LOG is empty
(same species as the outage) — converted to if, added to the loop
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- EUID check runs before any host write; friendly non-root halt (I11)
- get.docker.com staged to temp file with retry + pkg fallback (I13)
- df -Pk on the real install target; free guarded for LXC (I14)
- release tag via releases/latest redirect (no API quota) + loud
source-fallback warning (I15)
- tar/cp failures in fetch_release restore live state and fall back
to source cleanly (I16)
- svc_enable/svc_start init-system abstraction at service call sites (I17)
- stale slot-A collision guard in ensure_install_layout (I18)
- swap verified via /proc/swaps before claiming active; dd/mkswap
failures degrade to warn (I19)
- VERSION clobber, telemetry pipefail, stale SSL_MODE rewrite,
apply_frontend_root end-of-loop && (I22-I24, bonus)
- 22 new proving assertions (61 passed local, 64 on Linux)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ServerKit's backend manages the host it runs on -- it creates app dirs under
/var/serverkit/apps, drives the host Docker daemon (one container per app),
and reloads nginx. The default compose containerized the backend, so it ran
as a non-root user into a root-owned /var/serverkit/apps with no host
docker.sock: WordPress/app creation failed with
`Permission denied: /var/serverkit/apps/...`. It also contradicted install.sh,
which runs the backend on the host via systemd and expects compose to bring
up only the frontend.
- docker-compose.yml: frontend-only; the frontend reaches the host backend via
extra_hosts ["backend:host-gateway"].
- backend/Dockerfile: chown /var/serverkit to appuser (latent EACCES on the
apps dir for dev/demo container use) + header noting it cannot manage host.
- frontend/nginx.conf: document the backend:host-gateway upstream.
- docs/DEPLOYMENT.md: add a Deployment model note; fix build/start + data paths.
- scripts/test/test_install.sh: T12 guards against re-adding a backend service.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hardening plan §7. Catches distro-specific regressions before release.
- scripts/test/vm-update.sh: install → forced local update (full blue/green
deploy/migrate/switch/health) → re-run to confirm the "Already up to date"
version-gate. Same VM harness contract as vm-install.sh (log + status file).
- scripts/test/vm-uninstall.sh: install + seed data → default uninstall asserts
data preserved (DB snapshot in backups, /var/lib survives) → reinstall →
--purge asserts data removed. End-to-end proof of the unified uninstaller.
- scripts-ci.yml: now runs all three serverless suites (update+install+lib),
lints the new lib/ + test/ files (error severity) and keeps scripts/lib/*
warning-clean, and expands the cross-distro unit matrix to Ubuntu 22.04/24.04,
Debian 12, Rocky 9, AlmaLinux 9, Fedora 40, and openSUSE Leap 15.5.
- test_install.sh: +choose_pkg_manager graceful-failure case (7.4); a no-op
docker stub so the uninstall-matrix cases run where docker isn't installed.
- test_lib.sh: neutralize the ambient container=oci env var (RPM/SUSE images)
and force an empty PATH for the "no package manager" case — both bugs the new
matrix surfaced (the tests, not the code).
- README: fixed the stale Fedora box, documented the new VM scripts + suites.
Validated by running the exact CI container logic locally across all seven
images: test_update 13/13, test_install 15-16/0, test_lib 13/13 everywhere.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hardening plan §6. Reduces the distro-specific if/else surface and handles
edge environments:
- Dependency manifest (scripts/deps/manifest.yaml): declarative per-family
package/python/node/docker/init spec. Consumed directly by the Phase 5
Python bootstrap; install.sh mirrors it inline (it can't parse YAML before
Python exists in the curl|bash path).
- New sourceable libs (built in parallel, each shellcheck-clean + unit-tested):
- scripts/lib/pkg.sh — pkg_detect/refresh/install/is_installed across
apt/dnf/yum/zypper/pacman/apk (PKG_MGR_OVERRIDE + PKG_DRY_RUN for tests).
- scripts/lib/init.sh — init_detect + start/stop/enable/disable/reload/
is_active across systemd/openrc/runit/sysvinit/none (INIT_OVERRIDE/DRY_RUN).
- scripts/lib/env.sh — is_container/is_wsl/has_systemd (override-friendly).
- ID_LIKE handling (G5/6.5): install.sh's os_family_from() (now a pure,
testable function reading ${SERVERKIT_OS_RELEASE}) maps unknown IDs via
ID_LIKE to debian/rhel/fedora/suse/arch/alpine — rhel before fedora so RHEL
clones pick the RHEL Docker repo.
- zypper + pacman + apk (G6/6.6): choose_pkg_manager/refresh_pkg_index/pkg_add,
provision_docker and provision_python gained suse/arch/alpine branches;
Docker enable/start is now init-aware (systemd OR OpenRC) so it no longer
aborts on Alpine.
- Templated systemd unit (6.7): serverkit-backend.service → templates/
serverkit-backend.service.in, rendered (render_service_unit) at install and
in refresh_config so a custom SERVERKIT_DIR/venv/log path is honored.
install_service warns-and-skips enable when has_systemd is false (G5).
- nginx TLS floor as a reversible conf.d snippet (6.8): install.sh + update.sh
prefer /etc/nginx/conf.d/serverkit-tls.conf when nginx.conf doesn't already
declare ssl_protocols (RHEL/Fedora/SUSE), falling back to the in-place edit
on Debian/Ubuntu to avoid a 'duplicate ssl_protocols' error. uninstall
removes only that snippet.
Tests: new scripts/test/test_lib.sh (13) + 4 new test_install.sh cases
(os_family_from, render_service_unit, harden_global_tls A/B). 13+15+10 pass;
shellcheck error+warning clean across all new/changed shell.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Critical install/update/uninstall hotfixes from the hardening plan
(docs/plans/INSTALLER_UPDATER_UNINSTALLER_HARDENING_PLAN.md §4):
- Firewall automation (G1): new sourceable scripts/lib/firewall.sh detects
firewalld/ufw/nftables/iptables and opens 80/443; install.sh runs
configure_firewall, update.sh re-ensures it idempotently. Exactly what we
open is recorded in /etc/serverkit/install-state.json (scripts/lib/state.sh)
so uninstall undoes only our rules. Fixes the #1 "installed but unreachable"
failure on fresh RHEL-family boxes.
- Accept Python 3.11 OR 3.12 (G2): locate_python now iterates
python3.12/3.11/3 and provision_python installs the distro package
(python3.11 on Debian 12, 3.12 on Ubuntu/Fedora, 3.11/3.12 on RHEL) before
ever falling back to a source compile.
- Use $PYTHON_BIN for secret generation instead of a bare python3.
- Distro-first Node 20 (replacing unconditional NodeSource): install the
distro nodejs+npm, fall back to NodeSource only when it's missing or < 18,
and validate node>=18 before continuing.
- Unify uninstall (G4): new scripts/lib/uninstall.sh is the single canonical
teardown routine shared by uninstall.sh and `serverkit uninstall` (they had
drifted — one ran `down -v`, one didn't). Default preserves user data
(volumes kept, DB snapshotted to backups); --purge/--delete-volumes removes
volumes+DB+data dirs; --keep-data preserves /etc + /var/lib + backups;
--yes/--dry-run added; honors SERVERKIT_DIR. Removes the apt lock-wait
drop-in on uninstall.
Tests: new scripts/test/test_install.sh (11 cases, source-able install.sh)
covers python/node detection, firewall dry-run + state, and the uninstall
data-handling matrix. Existing test_update.sh still green. shellcheck clean
(error + warning) on all new/changed shell.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
