A test box's panel domain kept serving a Jun 25 bundle after the Jul 3
update — the domain's custom vhost still proxied the retired :3847
frontend container, and the old bundle's calls to deleted legacy
marketplace endpoints read as 'Failed to load extensions'. It looked
exactly like a browser/CDN cache problem (it wasn't — cf-cache-status
was DYNAMIC; the origin itself was stale).
- both stock confs: expires -1 on the SPA entrypoint (emits
Cache-Control: no-cache WITHOUT discarding server-level security
headers, unlike add_header-in-location) — a cached index.html
references hashed assets that vanish on the next blue/green switch
- update.sh refresh_config now runs report_stale_panel_vhosts: warns,
naming the file, when any enabled vhost still proxies :3847;
observation-only per R1, silent on a missing sites-enabled dir,
added to the fresh-box loop per policy
- proving tests: entrypoint no-cache asserted for both confs; stale
vhost flagged (and clean one not); missing-dir silence (42 passed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- PULL_REQUEST_TEMPLATE gains a scripts-only checklist condensing the
failure-semantics rules (guarded observations, no &&-tail returns,
capture-safe stdout, errexit-immune helpers, fresh-box loop policy,
proving test in the same commit)
- record the self-update channel decision in the bootstrap header:
keep fetching from main, gated by the update-e2e CI job, preserving
one-push-fixes-the-fleet
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The July 2 outage function had a unit test — with a populated fixture
only. This encodes the missing net:
- fresh-minimal-box fixture loop: every observation/discovery/snapshot/
report function in update.sh (17) and install.sh (16) must survive a
zero-app box under set -Eeuo pipefail; policy comment requires new
functions to be appended
- scripts/test/stubs.sh: shared failing-stub factories (systemctl exit
5, docker fail/empty, curl exit 22, GitHub rate-limit JSON)
- assertion sweep: 43 bare capture-then-compare sites now include exit
status in the assertion
- best-effort contract test: the run block's real APP_BASELINE ->
report chain exits 0 on a fresh box AND on a genuine regression
- fix(update): the loop immediately caught report_failure's trailing
'[ -n $UPDATE_LOG ] && printf' returning 1 when UPDATE_LOG is empty
(same species as the outage) — converted to if, added to the loop
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- find-based backup pruning; empty/missing BACKUP_DIR is a no-op (U1)
- download_release progress to stderr; stdout is the tarball path only (U2)
- guard bare systemctl stop/start/daemon-reload in forward+rollback paths (U3,U4,U8,U13)
- ROLLING_BACK=1 inside rollback(); never runs twice via the EXIT trap (U5)
- npm preflight for source updates; guarded frontend build pipeline (U6)
- empty-ORIG_ARGS re-exec guard, cleanup_on_exit return 0, df -Pk,
SIGPIPE guards, cd leaks subshelled, local typo (U7,U9-U12,U14)
- 14 new proving assertions in test_update.sh (35 passed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Allow failures when querying GitHub release tag (prevent pipeline from aborting) and add a unit test ensuring discover_app_upstreams tolerates an empty app-locations directory.
Changes:
- scripts/update.sh: make the curl|grep pipeline resilient by appending `|| true` so the command substitution won't fail the script when the GitHub API call returns nothing.
- scripts/test/test_update.sh: add test T16b to verify discover_app_upstreams does not abort under `set -euo pipefail` when the app-locations directory is empty.
These changes prevent hard failures during release detection and add coverage for empty-directory handling.
Suppress a non-zero exit from the pipeline in discover_app_upstreams by appending `|| true` after the sort. This prevents the script from exiting with an error when no matching proxy_pass entries are found in APP_LOCATIONS_DIR (e.g., empty or missing files). Improves robustness of scripts/update.sh in scenarios with no app upstreams configured.
A panel update deploys a fresh clone/tarball preserving only .env and
the DB, wiping app/plugins/<slug> + src/plugins/<slug> for any
URL/registry/upload-installed extension — the surviving InstalledPlugin
row then flipped to error at next boot. Two-layer fix:
- update.sh: preserve_installed_plugins carries user-installed plugin
dirs from the old tree into the new one (skipping anything the new
tree already ships and __pycache__), before the frontend build so
carried UIs recompile. Covered by harness test T18.
- boot repair pass (plugin_service.repair_missing_plugins, runs before
load_all_plugins): builtin slugs reinstall from builtin-extensions/,
url installs re-download from source_url (force + no hot-load so the
loader registers once), sourceless uploads get a clear re-upload
error. install_from_url/path grew force/hot_load plumbing.
4 new repair tests (16 total in the pipeline suite); update.sh harness
20 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Turns the zero-downtime promise into an actual check. The updater now
snapshots which managed-app upstreams are reachable through nginx
*before* the blue/green switch, re-probes after the graceful reload, and
reports any app that was serving and has since gone dark.
- discover_app_upstreams scans /etc/nginx/serverkit-locations/*.conf for
the per-app proxy_pass upstreams (host:port).
- probe_app_upstreams / snapshot_app_reachability record up/down (any
HTTP status = up; a refused/timed-out connection = down).
- report_app_uptime_regressions diffs before-vs-after and warns for any
up->down app, ignoring apps that were already down. Best-effort: it
never fails an otherwise-healthy update, just surfaces the regression.
- New "Verifying App Uptime" phase wired into the main flow; baseline
captured before the backend stop.
Tests: T16 (upstream discovery), T17 (regression detection both ways).
19 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The panel is now fully off Docker: the React SPA is served as static
files directly by host nginx from frontend/dist, eliminating the
serverkit-frontend container and the double nginx hop (host nginx ->
container nginx -> dist). Docker is now used ONLY for managed app
workloads.
- nginx sites (serverkit.conf + serverkit-insecure.conf): `location /`
now roots at /opt/serverkit/frontend/dist with a try_files SPA
fallback and a long-cache /assets/ block, instead of proxying
127.0.0.1:3847. Root tracks the /opt/serverkit symlink so blue/green
updates swap assets atomically.
- install.sh: launch_services no longer builds/starts a frontend
container; adds apply_frontend_root (rewrites the dist root for a
custom SERVERKIT_DIR) and selinux_label_frontend_dist (httpd_sys_-
content_t so SELinux hosts can read the bundle from /opt).
- update.sh: refresh_config carries the same root rewrite across
upgrades; selinux_label_dist relabels the switched-in slot; dropped
the `docker compose up -d --force-recreate frontend` from both the
forward and rollback paths.
- docker-compose.yml: frontend service moved behind a `legacy-frontend`
profile (opt-in escape hatch); default `up -d` starts nothing.
- docs/DEPLOYMENT.md: documents the off-Docker panel model.
Tests: T14 (refresh_config rewrites dist root for custom install dir),
T15 (shipped nginx configs serve static, never proxy :3847). 16 passed.
Note: live nginx -t / SELinux file-serving not validated on a RHEL box
from this dev host; logic is best-effort guarded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Host nginx is the front door for every managed app (it reverse-proxies
their containers via /etc/nginx/serverkit-locations/*.conf). The updater
used to `systemctl stop nginx` for the whole blue/green switch window,
which blacked out every unrelated hosted site for 10-60s during a panel
update. The slot switch only ever changes nginx config, never anything
nginx serves from the slot, so a stop was never required.
Add reload_nginx_graceful(): validate config (nginx -t) then
`systemctl reload nginx` - keeps listening sockets and in-flight
connections alive, so apps never blink. Starts nginx if it happens to be
down. Wired into both the forward update flow and rollback(); only the
panel backend is now cycled. Dropped the stray frontend-only
`docker compose up -d backend` line.
Tests: T11 reloads-not-stops a running nginx, T12 starts a stopped
nginx, T13 guards against `systemctl stop nginx` reappearing. 13 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hardening plan §6. Reduces the distro-specific if/else surface and handles
edge environments:
- Dependency manifest (scripts/deps/manifest.yaml): declarative per-family
package/python/node/docker/init spec. Consumed directly by the Phase 5
Python bootstrap; install.sh mirrors it inline (it can't parse YAML before
Python exists in the curl|bash path).
- New sourceable libs (built in parallel, each shellcheck-clean + unit-tested):
- scripts/lib/pkg.sh — pkg_detect/refresh/install/is_installed across
apt/dnf/yum/zypper/pacman/apk (PKG_MGR_OVERRIDE + PKG_DRY_RUN for tests).
- scripts/lib/init.sh — init_detect + start/stop/enable/disable/reload/
is_active across systemd/openrc/runit/sysvinit/none (INIT_OVERRIDE/DRY_RUN).
- scripts/lib/env.sh — is_container/is_wsl/has_systemd (override-friendly).
- ID_LIKE handling (G5/6.5): install.sh's os_family_from() (now a pure,
testable function reading ${SERVERKIT_OS_RELEASE}) maps unknown IDs via
ID_LIKE to debian/rhel/fedora/suse/arch/alpine — rhel before fedora so RHEL
clones pick the RHEL Docker repo.
- zypper + pacman + apk (G6/6.6): choose_pkg_manager/refresh_pkg_index/pkg_add,
provision_docker and provision_python gained suse/arch/alpine branches;
Docker enable/start is now init-aware (systemd OR OpenRC) so it no longer
aborts on Alpine.
- Templated systemd unit (6.7): serverkit-backend.service → templates/
serverkit-backend.service.in, rendered (render_service_unit) at install and
in refresh_config so a custom SERVERKIT_DIR/venv/log path is honored.
install_service warns-and-skips enable when has_systemd is false (G5).
- nginx TLS floor as a reversible conf.d snippet (6.8): install.sh + update.sh
prefer /etc/nginx/conf.d/serverkit-tls.conf when nginx.conf doesn't already
declare ssl_protocols (RHEL/Fedora/SUSE), falling back to the in-place edit
on Debian/Ubuntu to avoid a 'duplicate ssl_protocols' error. uninstall
removes only that snippet.
Tests: new scripts/test/test_lib.sh (13) + 4 new test_install.sh cases
(os_family_from, render_service_unit, harden_global_tls A/B). 13+15+10 pass;
shellcheck error+warning clean across all new/changed shell.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updater robustness from the hardening plan §5:
- Version comparison before updating (G3): is_already_current compares the
local VERSION to the target release tag (release mode) or the local checkout
HEAD to the remote branch HEAD via `git ls-remote` (branch/main mode). When
current, both the hybrid and docker flows print "Already up to date" and exit
0 — unless --force. Offline/indeterminate always proceeds rather than wrongly
blocking.
- Dry-run fidelity: new run_in_dir reports the working directory a command
would run in, replacing the `run_or_dry cd X && run_or_dry compose ...` chain
that printed compose commands with no cwd context.
- Rollback hardening: migrate_database now runs against the NEW slot's DB copy
(slot-absolute path) instead of the /opt/serverkit symlink that still
resolves to the live old slot — so a rollback hands the old code a
byte-for-byte-unchanged database (PostgreSQL/shared DBs migrate directly, as
before). rollback_docker guards `git reset` on a non-empty SHA with a clear
warning. Both rollback paths now run a post-rollback health check and say
plainly when even the rollback is unhealthy.
- Telemetry/logging: halt() always prints the full log path; both summaries
gained an "old → new", mode (release/branch/main, +docker), and duration
line, plus the log path.
Tests: test_update.sh gains versions_equal, is_already_current force/offline
short-circuits, and a flask-stub proof that migrate_database targets the new
slot's DB (T10). 10 passed / 2 skipped locally; shellcheck error+warning clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Critical install/update/uninstall hotfixes from the hardening plan
(docs/plans/INSTALLER_UPDATER_UNINSTALLER_HARDENING_PLAN.md §4):
- Firewall automation (G1): new sourceable scripts/lib/firewall.sh detects
firewalld/ufw/nftables/iptables and opens 80/443; install.sh runs
configure_firewall, update.sh re-ensures it idempotently. Exactly what we
open is recorded in /etc/serverkit/install-state.json (scripts/lib/state.sh)
so uninstall undoes only our rules. Fixes the #1 "installed but unreachable"
failure on fresh RHEL-family boxes.
- Accept Python 3.11 OR 3.12 (G2): locate_python now iterates
python3.12/3.11/3 and provision_python installs the distro package
(python3.11 on Debian 12, 3.12 on Ubuntu/Fedora, 3.11/3.12 on RHEL) before
ever falling back to a source compile.
- Use $PYTHON_BIN for secret generation instead of a bare python3.
- Distro-first Node 20 (replacing unconditional NodeSource): install the
distro nodejs+npm, fall back to NodeSource only when it's missing or < 18,
and validate node>=18 before continuing.
- Unify uninstall (G4): new scripts/lib/uninstall.sh is the single canonical
teardown routine shared by uninstall.sh and `serverkit uninstall` (they had
drifted — one ran `down -v`, one didn't). Default preserves user data
(volumes kept, DB snapshotted to backups); --purge/--delete-volumes removes
volumes+DB+data dirs; --keep-data preserves /etc + /var/lib + backups;
--yes/--dry-run added; honors SERVERKIT_DIR. Removes the apt lock-wait
drop-in on uninstall.
Tests: new scripts/test/test_install.sh (11 cases, source-able install.sh)
covers python/node detection, firewall dry-run + state, and the uninstall
data-handling matrix. Existing test_update.sh still green. shellcheck clean
(error + warning) on all new/changed shell.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two reliability gaps remained after the failure cascade:
1. Stale-updater trap. The updater that runs is whatever is installed, so an
old update.sh fails in ways the current code already handles and can't fix
itself — exactly what left a box stuck. maybe_reexec_latest_updater fetches
the newest update.sh for the target ref and re-execs into it (guarded by
SERVERKIT_UPDATER_REEXECED, skipped offline/dry-run/opt-out, only after a
bash -n check). From this version on, 'serverkit update' is reliable no
matter how stale the box is.
2. No concurrency guard. acquire_update_lock takes an flock so two updates
can't clobber the same blue/green slot or race docker compose; the lock
auto-releases on exit and degrades gracefully where flock/the path is
unavailable.
Both are covered by scripts/test/test_update.sh (guard no-ops; lock
refuses a held lock, succeeds when free).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Harden update.sh against the class of failure that left a box silently stuck
on the old version, and make it testable without a real deploy.
Robustness (cascade):
- L5 loud failures: `set -Eeuo pipefail` (the -E/errtrace makes the ERR trap
fire inside functions too) + a report_failure ERR trap that names the phase,
exit code, line and command, plus always-on logging to
/var/log/serverkit/update-*.log. No more silent drop-to-prompt.
- L4 docker-path rollback: snapshot the previous git SHA, :rollback image tags
and dist.rollback before mutating; if the new backend is not healthy within
90s, revert to the snapshot and halt (was: warn and continue).
- L6 self-heal: pin COMPOSE_PROJECT_NAME (kills the symlink-rename warning) and
reclaim a stale inactive blue/green slot left by a prior failed hybrid run.
Testability:
- Parameterize the system dirs (SERVERKIT_NGINX_DIR / _LETSENCRYPT_DIR /
_SYSTEMD_DIR / _CONFIG_DIR) and add SERVERKIT_NO_LOG, so refresh_config can
run against fixtures instead of the host /etc.
- Make update.sh source-able via a BASH_SOURCE guard so functions can be
exercised directly.
- scripts/test/test_update.sh: plain-bash unit tests (no deps, no server) that
reproduce the missing-serverkit.conf regression and cover the
detection/slot/report logic.
- .github/workflows/scripts-ci.yml: bash -n + shellcheck + the unit tests on
changes to scripts/**, install.sh, serverkit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The panel-domain recovery greps /etc/nginx/sites-available/serverkit.conf,
but HTTP-only boxes never have that file. Under `set -euo pipefail` the
no-match/missing-file grep (exit 1/2) propagated and silently killed the
updater at the "Refreshing Configuration" phase — before the atomic switch
and before PREVIOUS_DIR is set, so no error and no rollback. The update
appeared to "do nothing" and the version never advanced.
Guard the command substitution with `|| true` (matching the idiom used a
few lines below for panel-domain).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
serverkit update (update.sh) assumed the hybrid install shape (backend in a
host venv under systemd, frontend in Docker) and unconditionally ran
`source $INSTALL_DIR/venv/bin/activate`. On an all-Docker deployment (backend
AND frontend as compose services, no host venv) that crashes with
"venv/bin/activate: No such file or directory" -- and the preceding "stop
services" step leaves the box down with no restart.
Add is_docker_deployment() (compose file present + no host venv + a
serverkit-backend container exists) that routes to update_docker_compose():
in-place git sync, optional host frontend build when ./frontend/dist is
bind-mounted, `docker compose build` (the running version keeps serving until
the build succeeds), `up -d` (backend auto-migrates on boot), then a
container-health check. The hybrid venv/systemd/blue-green path is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make email handling case-insensitive (queries + model normalization) to prevent duplicate accounts. Add SiteDomainService.panel_origin() helper and use it across services to build public URLs (Gitea, WordPress, Roundcube) and Django ALLOWED_HOSTS. Add optional custom-domain support when creating WordPress sites and propagate domain to Roundcube install. Improve environment domain fallback to prefer panel/base domains over .localhost. Small Dockerfile healthcheck tweak (use 127.0.0.1). Update install/update/build scripts to always (re)create virtualenv locally for release installs. Remove unused frontend CreateSiteModal and expose domain field in WordPress page.
KIMI's optional-SSL commit hardened ssl_protocols/ssl_ciphers per-vhost but
left the system nginx.conf http{} defaults untouched, so the default server
and any non-ServerKit vhost could still negotiate TLS 1.0/1.1 and weak ciphers
(the scan's "insecure TLS versions" / "weak ciphers in TLS 1.2" findings).
- install.sh: harden_global_tls() rewrites (or injects) ssl_protocols and
ssl_ciphers in /etc/nginx/nginx.conf to TLS1.2/1.3 + AEAD-only. Editing the
line in place avoids nginx's duplicate-directive error on Debian/Ubuntu,
which already set ssl_protocols in http{}.
- update.sh: re-applies the floor so existing installs harden on upgrade too.
- Persist the panel domain to /etc/serverkit/panel-domain; update.sh recovers
it from there (or the live cert path) instead of the commented-out .env
SERVERKIT_PUBLIC_URL grep, which never matched -> secure installs were
getting the YOUR_DOMAIN placeholder re-applied and nginx -t failing on update.
- update.sh now degrades to the HTTP config when a secure cert is missing,
rather than leaving nginx unable to reload.
Global TLS settings are inert in HTTP-only mode, so optional SSL is unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add secure nginx config with TLS 1.2+, HSTS preload, CSP, and security headers
- Add insecure HTTP fallback config for users who cannot or do not want HTTPS
- Make installer best-effort SSL: try Let's Encrypt, fall back to HTTP without failing
- Preserve secure/insecure SSL choice across updates via /etc/serverkit/ssl-mode
- Support SERVERKIT_SKIP_SSL=1 to disable HTTPS entirely
- Add CAA record to default DNS zone template
- Harden generated per-app nginx configs with HSTS preload, CSP, and security headers
- Align Flask security middleware headers with nginx edge config
