Turns the zero-downtime promise into an actual check. The updater now
snapshots which managed-app upstreams are reachable through nginx
*before* the blue/green switch, re-probes after the graceful reload, and
reports any app that was serving and has since gone dark.
- discover_app_upstreams scans /etc/nginx/serverkit-locations/*.conf for
the per-app proxy_pass upstreams (host:port).
- probe_app_upstreams / snapshot_app_reachability record up/down (any
HTTP status = up; a refused/timed-out connection = down).
- report_app_uptime_regressions diffs before-vs-after and warns for any
up->down app, ignoring apps that were already down. Best-effort: it
never fails an otherwise-healthy update, just surfaces the regression.
- New "Verifying App Uptime" phase wired into the main flow; baseline
captured before the backend stop.
Tests: T16 (upstream discovery), T17 (regression detection both ways).
19 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The panel is now fully off Docker: the React SPA is served as static
files directly by host nginx from frontend/dist, eliminating the
serverkit-frontend container and the double nginx hop (host nginx ->
container nginx -> dist). Docker is now used ONLY for managed app
workloads.
- nginx sites (serverkit.conf + serverkit-insecure.conf): `location /`
now roots at /opt/serverkit/frontend/dist with a try_files SPA
fallback and a long-cache /assets/ block, instead of proxying
127.0.0.1:3847. Root tracks the /opt/serverkit symlink so blue/green
updates swap assets atomically.
- install.sh: launch_services no longer builds/starts a frontend
container; adds apply_frontend_root (rewrites the dist root for a
custom SERVERKIT_DIR) and selinux_label_frontend_dist (httpd_sys_-
content_t so SELinux hosts can read the bundle from /opt).
- update.sh: refresh_config carries the same root rewrite across
upgrades; selinux_label_dist relabels the switched-in slot; dropped
the `docker compose up -d --force-recreate frontend` from both the
forward and rollback paths.
- docker-compose.yml: frontend service moved behind a `legacy-frontend`
profile (opt-in escape hatch); default `up -d` starts nothing.
- docs/DEPLOYMENT.md: documents the off-Docker panel model.
Tests: T14 (refresh_config rewrites dist root for custom install dir),
T15 (shipped nginx configs serve static, never proxy :3847). 16 passed.
Note: live nginx -t / SELinux file-serving not validated on a RHEL box
from this dev host; logic is best-effort guarded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Host nginx is the front door for every managed app (it reverse-proxies
their containers via /etc/nginx/serverkit-locations/*.conf). The updater
used to `systemctl stop nginx` for the whole blue/green switch window,
which blacked out every unrelated hosted site for 10-60s during a panel
update. The slot switch only ever changes nginx config, never anything
nginx serves from the slot, so a stop was never required.
Add reload_nginx_graceful(): validate config (nginx -t) then
`systemctl reload nginx` - keeps listening sockets and in-flight
connections alive, so apps never blink. Starts nginx if it happens to be
down. Wired into both the forward update flow and rollback(); only the
panel backend is now cycled. Dropped the stray frontend-only
`docker compose up -d backend` line.
Tests: T11 reloads-not-stops a running nginx, T12 starts a stopped
nginx, T13 guards against `systemctl stop nginx` reappearing. 13 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hardening plan §6. Reduces the distro-specific if/else surface and handles
edge environments:
- Dependency manifest (scripts/deps/manifest.yaml): declarative per-family
package/python/node/docker/init spec. Consumed directly by the Phase 5
Python bootstrap; install.sh mirrors it inline (it can't parse YAML before
Python exists in the curl|bash path).
- New sourceable libs (built in parallel, each shellcheck-clean + unit-tested):
- scripts/lib/pkg.sh — pkg_detect/refresh/install/is_installed across
apt/dnf/yum/zypper/pacman/apk (PKG_MGR_OVERRIDE + PKG_DRY_RUN for tests).
- scripts/lib/init.sh — init_detect + start/stop/enable/disable/reload/
is_active across systemd/openrc/runit/sysvinit/none (INIT_OVERRIDE/DRY_RUN).
- scripts/lib/env.sh — is_container/is_wsl/has_systemd (override-friendly).
- ID_LIKE handling (G5/6.5): install.sh's os_family_from() (now a pure,
testable function reading ${SERVERKIT_OS_RELEASE}) maps unknown IDs via
ID_LIKE to debian/rhel/fedora/suse/arch/alpine — rhel before fedora so RHEL
clones pick the RHEL Docker repo.
- zypper + pacman + apk (G6/6.6): choose_pkg_manager/refresh_pkg_index/pkg_add,
provision_docker and provision_python gained suse/arch/alpine branches;
Docker enable/start is now init-aware (systemd OR OpenRC) so it no longer
aborts on Alpine.
- Templated systemd unit (6.7): serverkit-backend.service → templates/
serverkit-backend.service.in, rendered (render_service_unit) at install and
in refresh_config so a custom SERVERKIT_DIR/venv/log path is honored.
install_service warns-and-skips enable when has_systemd is false (G5).
- nginx TLS floor as a reversible conf.d snippet (6.8): install.sh + update.sh
prefer /etc/nginx/conf.d/serverkit-tls.conf when nginx.conf doesn't already
declare ssl_protocols (RHEL/Fedora/SUSE), falling back to the in-place edit
on Debian/Ubuntu to avoid a 'duplicate ssl_protocols' error. uninstall
removes only that snippet.
Tests: new scripts/test/test_lib.sh (13) + 4 new test_install.sh cases
(os_family_from, render_service_unit, harden_global_tls A/B). 13+15+10 pass;
shellcheck error+warning clean across all new/changed shell.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updater robustness from the hardening plan §5:
- Version comparison before updating (G3): is_already_current compares the
local VERSION to the target release tag (release mode) or the local checkout
HEAD to the remote branch HEAD via `git ls-remote` (branch/main mode). When
current, both the hybrid and docker flows print "Already up to date" and exit
0 — unless --force. Offline/indeterminate always proceeds rather than wrongly
blocking.
- Dry-run fidelity: new run_in_dir reports the working directory a command
would run in, replacing the `run_or_dry cd X && run_or_dry compose ...` chain
that printed compose commands with no cwd context.
- Rollback hardening: migrate_database now runs against the NEW slot's DB copy
(slot-absolute path) instead of the /opt/serverkit symlink that still
resolves to the live old slot — so a rollback hands the old code a
byte-for-byte-unchanged database (PostgreSQL/shared DBs migrate directly, as
before). rollback_docker guards `git reset` on a non-empty SHA with a clear
warning. Both rollback paths now run a post-rollback health check and say
plainly when even the rollback is unhealthy.
- Telemetry/logging: halt() always prints the full log path; both summaries
gained an "old → new", mode (release/branch/main, +docker), and duration
line, plus the log path.
Tests: test_update.sh gains versions_equal, is_already_current force/offline
short-circuits, and a flask-stub proof that migrate_database targets the new
slot's DB (T10). 10 passed / 2 skipped locally; shellcheck error+warning clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Critical install/update/uninstall hotfixes from the hardening plan
(docs/plans/INSTALLER_UPDATER_UNINSTALLER_HARDENING_PLAN.md §4):
- Firewall automation (G1): new sourceable scripts/lib/firewall.sh detects
firewalld/ufw/nftables/iptables and opens 80/443; install.sh runs
configure_firewall, update.sh re-ensures it idempotently. Exactly what we
open is recorded in /etc/serverkit/install-state.json (scripts/lib/state.sh)
so uninstall undoes only our rules. Fixes the #1 "installed but unreachable"
failure on fresh RHEL-family boxes.
- Accept Python 3.11 OR 3.12 (G2): locate_python now iterates
python3.12/3.11/3 and provision_python installs the distro package
(python3.11 on Debian 12, 3.12 on Ubuntu/Fedora, 3.11/3.12 on RHEL) before
ever falling back to a source compile.
- Use $PYTHON_BIN for secret generation instead of a bare python3.
- Distro-first Node 20 (replacing unconditional NodeSource): install the
distro nodejs+npm, fall back to NodeSource only when it's missing or < 18,
and validate node>=18 before continuing.
- Unify uninstall (G4): new scripts/lib/uninstall.sh is the single canonical
teardown routine shared by uninstall.sh and `serverkit uninstall` (they had
drifted — one ran `down -v`, one didn't). Default preserves user data
(volumes kept, DB snapshotted to backups); --purge/--delete-volumes removes
volumes+DB+data dirs; --keep-data preserves /etc + /var/lib + backups;
--yes/--dry-run added; honors SERVERKIT_DIR. Removes the apt lock-wait
drop-in on uninstall.
Tests: new scripts/test/test_install.sh (11 cases, source-able install.sh)
covers python/node detection, firewall dry-run + state, and the uninstall
data-handling matrix. Existing test_update.sh still green. shellcheck clean
(error + warning) on all new/changed shell.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two reliability gaps remained after the failure cascade:
1. Stale-updater trap. The updater that runs is whatever is installed, so an
old update.sh fails in ways the current code already handles and can't fix
itself — exactly what left a box stuck. maybe_reexec_latest_updater fetches
the newest update.sh for the target ref and re-execs into it (guarded by
SERVERKIT_UPDATER_REEXECED, skipped offline/dry-run/opt-out, only after a
bash -n check). From this version on, 'serverkit update' is reliable no
matter how stale the box is.
2. No concurrency guard. acquire_update_lock takes an flock so two updates
can't clobber the same blue/green slot or race docker compose; the lock
auto-releases on exit and degrades gracefully where flock/the path is
unavailable.
Both are covered by scripts/test/test_update.sh (guard no-ops; lock
refuses a held lock, succeeds when free).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Harden update.sh against the class of failure that left a box silently stuck
on the old version, and make it testable without a real deploy.
Robustness (cascade):
- L5 loud failures: `set -Eeuo pipefail` (the -E/errtrace makes the ERR trap
fire inside functions too) + a report_failure ERR trap that names the phase,
exit code, line and command, plus always-on logging to
/var/log/serverkit/update-*.log. No more silent drop-to-prompt.
- L4 docker-path rollback: snapshot the previous git SHA, :rollback image tags
and dist.rollback before mutating; if the new backend is not healthy within
90s, revert to the snapshot and halt (was: warn and continue).
- L6 self-heal: pin COMPOSE_PROJECT_NAME (kills the symlink-rename warning) and
reclaim a stale inactive blue/green slot left by a prior failed hybrid run.
Testability:
- Parameterize the system dirs (SERVERKIT_NGINX_DIR / _LETSENCRYPT_DIR /
_SYSTEMD_DIR / _CONFIG_DIR) and add SERVERKIT_NO_LOG, so refresh_config can
run against fixtures instead of the host /etc.
- Make update.sh source-able via a BASH_SOURCE guard so functions can be
exercised directly.
- scripts/test/test_update.sh: plain-bash unit tests (no deps, no server) that
reproduce the missing-serverkit.conf regression and cover the
detection/slot/report logic.
- .github/workflows/scripts-ci.yml: bash -n + shellcheck + the unit tests on
changes to scripts/**, install.sh, serverkit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The panel-domain recovery greps /etc/nginx/sites-available/serverkit.conf,
but HTTP-only boxes never have that file. Under `set -euo pipefail` the
no-match/missing-file grep (exit 1/2) propagated and silently killed the
updater at the "Refreshing Configuration" phase — before the atomic switch
and before PREVIOUS_DIR is set, so no error and no rollback. The update
appeared to "do nothing" and the version never advanced.
Guard the command substitution with `|| true` (matching the idiom used a
few lines below for panel-domain).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
serverkit update (update.sh) assumed the hybrid install shape (backend in a
host venv under systemd, frontend in Docker) and unconditionally ran
`source $INSTALL_DIR/venv/bin/activate`. On an all-Docker deployment (backend
AND frontend as compose services, no host venv) that crashes with
"venv/bin/activate: No such file or directory" -- and the preceding "stop
services" step leaves the box down with no restart.
Add is_docker_deployment() (compose file present + no host venv + a
serverkit-backend container exists) that routes to update_docker_compose():
in-place git sync, optional host frontend build when ./frontend/dist is
bind-mounted, `docker compose build` (the running version keeps serving until
the build succeeds), `up -d` (backend auto-migrates on boot), then a
container-health check. The hybrid venv/systemd/blue-green path is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make email handling case-insensitive (queries + model normalization) to prevent duplicate accounts. Add SiteDomainService.panel_origin() helper and use it across services to build public URLs (Gitea, WordPress, Roundcube) and Django ALLOWED_HOSTS. Add optional custom-domain support when creating WordPress sites and propagate domain to Roundcube install. Improve environment domain fallback to prefer panel/base domains over .localhost. Small Dockerfile healthcheck tweak (use 127.0.0.1). Update install/update/build scripts to always (re)create virtualenv locally for release installs. Remove unused frontend CreateSiteModal and expose domain field in WordPress page.
KIMI's optional-SSL commit hardened ssl_protocols/ssl_ciphers per-vhost but
left the system nginx.conf http{} defaults untouched, so the default server
and any non-ServerKit vhost could still negotiate TLS 1.0/1.1 and weak ciphers
(the scan's "insecure TLS versions" / "weak ciphers in TLS 1.2" findings).
- install.sh: harden_global_tls() rewrites (or injects) ssl_protocols and
ssl_ciphers in /etc/nginx/nginx.conf to TLS1.2/1.3 + AEAD-only. Editing the
line in place avoids nginx's duplicate-directive error on Debian/Ubuntu,
which already set ssl_protocols in http{}.
- update.sh: re-applies the floor so existing installs harden on upgrade too.
- Persist the panel domain to /etc/serverkit/panel-domain; update.sh recovers
it from there (or the live cert path) instead of the commented-out .env
SERVERKIT_PUBLIC_URL grep, which never matched -> secure installs were
getting the YOUR_DOMAIN placeholder re-applied and nginx -t failing on update.
- update.sh now degrades to the HTTP config when a secure cert is missing,
rather than leaving nginx unable to reload.
Global TLS settings are inert in HTTP-only mode, so optional SSL is unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add secure nginx config with TLS 1.2+, HSTS preload, CSP, and security headers
- Add insecure HTTP fallback config for users who cannot or do not want HTTPS
- Make installer best-effort SSL: try Let's Encrypt, fall back to HTTP without failing
- Preserve secure/insecure SSL choice across updates via /etc/serverkit/ssl-mode
- Support SERVERKIT_SKIP_SSL=1 to disable HTTPS entirely
- Add CAA record to default DNS zone template
- Harden generated per-app nginx configs with HSTS preload, CSP, and security headers
- Align Flask security middleware headers with nginx edge config
