Both the installer and the updater treated a missing checksums.txt as
"skip verification" and installed the tarball anyway. Every release
publishes one, so its absence means a broken mirror or a tampered
download. The installer now falls back to a source build and the
updater refuses; SERVERKIT_ALLOW_UNVERIFIED=1 overrides for mirrors
that do not host checksums.
The panel runs async_mode='threading' under a plain threaded gunicorn
worker; nothing imports gevent. It was still installed everywhere, and
on Python 3.14 (no wheel yet) its source build is what broke installs
in issue #148. backend/Dockerfile was unreferenced and still launched
the gevent-websocket worker that double-answers the WS handshake; the
run.py hint recommended the same worker.
On blue/green installs the previous slot is the code rollback, so the
serverkit-tree-* copy of it duplicated ~1 GB per update. With
SERVERKIT_BACKUP_RETENTION unset, disks under 50 GiB now keep one
database snapshot instead of three.
download_release runs in a command substitution, so its mktemp dir
was never removed: every update stranded the ~70 MB tarball in /tmp
(11 dirs, 644 MB on a 25 GB box). The download and unpack dirs are
now removed on success and on abort, and old leftovers are swept.
The tree backup rsync copied backend/instance/serverkit.db even though the
same database had just been written as the pre-upgrade copy, and rollback
only ever restores from that copy. On a 450 MB database that made every
update cost ~900 MB of backups, which is how small VPSes kept filling up.
Exclude the SQLite files from the tree snapshot. Retention is unchanged
(SERVERKIT_BACKUP_RETENTION, default 3, plus the 15% disk budget).
Add a test that every shipped nginx vhost forces a JavaScript MIME type for
/serverkit-vendor/*.mjs and that the regex cannot capture the
/api/v1/plugins/<slug>/assets/... route.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NWc84WDsWR7M55Mn7qtpie
- Harness: bootstrap find/awk in distro-compatibility.py (AL2023,
Tumbleweed, SLES BCI minimal images lack them; the source-level
suites legitimately use both)
- Python gate 3.11-3.14: Arch/Manjaro rolling ship only python 3.14
with no versioned fallback packages (same trap as Debian 13/#99);
align scripts/update.sh, which had drifted to a 3.11-3.12 gate
- Gentoo: add emerge as a supported installer package-manager path
(install.sh, scripts/lib/pkg.sh, distro.py family, deps manifest);
provision via slotted dev-lang/python:3.13 -> 3.12
- Tests: move out-of-range stubs above the new ceiling, shadow all
locate_python candidate names (real python3.14/3.13 in Manjaro/
Gentoo images were winning over fixtures), cover the 7th manager
- Catalog: Gentoo is no longer a documented expected failure
Verified: test_lib 32/32, test_install 124/124, test_update 76/76,
pytest test_serverkit_installer 46/46, and all 6 previously failing
images now pass both quick and provision container probes (12/12).
A backup retention cap already existed (SERVERKIT_BACKUP_RETENTION,
default 5), yet a box was found holding six snapshots and 0 bytes free.
Three separate defects, each sufficient on its own:
1. cleanup() is the LAST phase of the update, so retention never ran on
a run that died earlier -- including the run that died BECAUSE the
disk was full. Retention that only executes on success cannot bound a
failure loop. backup_current now trims BEFORE writing, which also
makes peak usage `keep` snapshots rather than `keep + 1`.
2. The preflight required a flat 2 GiB, but the update is about to write
two full copies of the database. With an 800 MB database a box at
2.1 GiB free passed the check and then filled mid-update. The
requirement is now 2 GiB + 2x the database, it trims stale snapshots
and re-checks before giving up, and the halt message points at
`serverkit disk`.
3. A count cap is not a disk guarantee: five snapshots of an 800 MB
database is 8 GB, a third of a 25 GB droplet. The default drops to 3
and enforce_backup_budget caps backups at a share of the filesystem
(SERVERKIT_BACKUP_MAX_PERCENT, default 15), dropping oldest-first and
always keeping at least one restore point -- an update with no way
back is worse than a full disk.
BACKUP_DIR also becomes SERVERKIT_BACKUP_DIR-overridable so it can agree
with the reclaim service, which scans the same directory.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- install_profile_service: record what install.sh put on the box
(minimal/standard/full), recommended from detected hardware and
overridable; thresholds mirrored by recommend_profile in install.sh
- setup wizard: replace the hard ResourceGate/SetupStepTier with
SetupStepCapacity + SetupStepSecurity
- ResourceAdvisory: inline capacity warning at the point of action
(WordPress/Docker/Services/Deployments), backed by headroom.fits
- RequiresDocker component for docker-dependent surfaces
- install.sh/update.sh: profile detection + tests
(test_install.sh, test_update.sh, test_server_capacity.py)
backup_current tested `[ -f serverkit.db ]` FIRST and only fell through to the
PostgreSQL branch when no file was there. An install migrated SQLite ->
PostgreSQL keeps its old serverkit.db, and both deploy paths copy it forward on
file-existence alone (deploy_source, deploy_release), so the stale file is
carried into every future slot and never ages out. Such a box backed up the
abandoned file, skipped pg_dump entirely, and left PRE_UPGRADE_DUMP unset --
which migrate_database reads as "${PRE_UPGRADE_DUMP:-}", so its auto-restore
silently did nothing. PostgreSQL is the one engine that migrates IN PLACE and
cannot roll back by switching slots, so that dump is its only way home: the
exact safety net a61eab1/b48256f added was disabled for it.
migrate_database has always keyed off DATABASE_URL (its sqlite check). Test the
declared engine first so the two functions agree.
Deliberately NOT `grep DATABASE_URL=sqlite && [ -f db ]` as the reviewer
suggested: backend/config.py defaults DATABASE_URL when absent, so any install
whose .env lacks the line would fall to the else branch and update with NO
backup at all -- trading a mismatched backup for none. An explicit `postgres`
URL is a positive signal; the absence of a `sqlite` one is not.
Also:
* name the engine when it is one we cannot dump. "No SQLite database ..."
read like a benign nothing-to-do on a MySQL install that in fact had no
safety net. Prints the scheme only -- a DATABASE_URL carries a password and
this text lands in the update log.
* drop the stale "Python 3.11/3.12 is guaranteed by pre-flight" comment,
contradicted by locate_sqlite_python two lines below it (664a5e6).
* discord-notify: skip cleanly when DISCORD_WEBHOOK_URL is absent. Secrets
are not exposed to `pull_request` runs from a fork, so a merged fork PR
failed the job on an empty curl URL. Not switching to pull_request_target
for this -- handing a write-scoped token and the webhook to an
outsider-triggered run is a poor trade for a chat notification.
Tests: a PostgreSQL install carrying a leftover serverkit.db must still dump
and set PRE_UPGRADE_DUMP (fails on the parent commit, rc=1); an undumpable
engine must be named without leaking the password. Neither needs python3, so
both run on all 7 images. Workflow verified end-to-end against a synthetic
event payload: fork PR exits 0 without calling curl, normal PR still builds the
same embed.
Verified on the full CI matrix (ubuntu 22.04/24.04, debian:12, rockylinux:9,
almalinux:9, fedora:40, opensuse/leap:15.5), all six suites: 0 failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
copy_sqlite_db and verify_sqlite_db reused locate_python, which enforces
3.11 <= v <= 3.12. That bound is correct for rebuild_virtualenv and
preflight_check, but snapshotting or integrity-checking a SQLite file needs
nothing beyond the stdlib sqlite3 module. Reusing it coupled the DATA-safety
paths to the venv-builder constraint and blocked updates for a reason unrelated
to the data:
* an all-Docker install returns before preflight_check ever runs and does
every real step in the container -- yet backup_docker_db snapshots with the
CONTAINER's python and then verified the extracted file with the HOST's.
On a stock EL9 (python3.9) or Ubuntu 22.04 (python3.10) host that halted
with "Could not obtain a VALID database backup", blaming the backup for a
missing interpreter.
* backup_current halts when it cannot snapshot, so those hosts could never
update at all.
Split out locate_sqlite_python: any candidate that can `import sqlite3` wins,
and the install's own venv interpreter is preferred -- on a real box it is
guaranteed present, guaranteed to carry the module, and independent of PATH.
locate_python keeps the 3.11/3.12 bound where it belongs.
Surfaced by the scripts-ci unit matrix: almalinux:9 and rockylinux:9 ship
python3.9, so the suite's own probe accepted an interpreter locate_python
rejected, and copy_sqlite_db/verify_sqlite_db failed there while passing on
python3.12 runners.
T10i pins the contract: on a box where every candidate reports 3.9 (locate_python
fails), the helpers still resolve an interpreter, prefer the slot's venv, and
round-trip 100 rows. Fails on the parent commit, passes here.
Verified on the full 7-image CI matrix (ubuntu 22.04/24.04, debian:12,
rockylinux:9, almalinux:9, fedora:40, opensuse/leap:15.5), all six shell suites:
0 failed. test_update is 63 passed / 0 failed on every image carrying a python3.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A backup that was never verified is a hope, not a safety net. Closes the
remaining ways the pre-upgrade safety net could silently not exist:
- Every backup is verified immediately after creation: SQLite snapshots via
PRAGMA integrity_check, PostgreSQL dumps via size + pg_restore TOC
listing (when available). Invalid artifacts are deleted and the update
halts — a malformed backup is never mistaken for a good one.
- Partial artifacts from a failed backup are removed (rm on failure), so a
half-written file can never be 'restored' later.
- A failed IN-PLACE PostgreSQL migration now automatically restores the
verified pre-upgrade dump (pg_restore --clean --if-exists) before
halting; without it the half-migrated schema would stay live against
the old code. If the restore itself fails, the halt message carries the
exact manual pg_restore command.
- backup_docker_db tries both python and python3 in the container, and now
verifies the result too — the plain docker cp fallback can no longer
pass off a torn copy as a backup.
- Backup retention capped: default 5 (SERVERKIT_BACKUP_RETENTION
overrides) across trees, .db and .dump files. Full-size DB copies pile
up fast — a real 25G droplet carried 6.3G of stale backups.
Tests: T10f (verification helpers accept/reject correctly), T10g (failed
in-place PG migration auto-restores with the exact pg_restore args),
T10h (retention prunes all three backup kinds to the configured cap).
Postgres panels updated with zero safety net: backup_current found no
SQLite file and just warned, while the migration runs IN PLACE against the
shared external DB — a failed migration left no way back.
- backup_current now pg_dumps (custom format) when .env points at
PostgreSQL, sanitizing SQLAlchemy driver suffixes (postgresql+psycopg2://)
that libpq can't parse.
- Same rule as SQLite: no valid backup, no update — halt when pg_dump is
missing (install postgresql-client) or the dump fails.
Tests: T10e covers the dump invocation + URL sanitization and the
halt-on-failed-dump path.
The venv binding check only matched the quoted VIRTUAL_ENV="/path" form;
venvs created on some platforms write 'export VIRTUAL_ENV=/path' unquoted,
so the binding read as 'unknown path' and always triggered a rebuild.
Now matches both forms (and skips cygpath-style indirection), so fresh
installs keep the fast prebuilt-venv path.
Root cause of a real-world failed update: backup_current, deploy_source and
deploy_release all plain-cp'd the LIVE panel database. On a busy panel that
races the backend's own writes and the copy can capture half a transaction
— 'database disk image is malformed' — which then sinks the migration in
the new slot and silently poisons the pre-upgrade backup that was supposed
to be the safety net.
- New copy_sqlite_db() uses the SQLite online backup API via python
(guaranteed by pre-flight) and refuses to fall back to a racy cp.
- backup_current halts rather than proceed without a valid safety net.
- deploy_source/deploy_release halt before any switch if the live DB
cannot be snapshotted consistently.
- backup_docker_db prefers a consistent snapshot via docker exec python,
with the plain docker cp as a fallback.
- locate_python rejects candidates whose version probe returns empty
(broken shims pass 'command -v' but can't execute; an empty version
string used to pass the sort-based range check).
Tests: T10d exercises a real snapshot (integrity ok, all rows); a contract
test bans plain-cp of the live DB; T10d fixture python detection validates
the interpreter actually runs.
The SQLite pre-check in migrate_database treated an EMPTY probe result
(unusable venv python) the same as real corruption and halted the update —
a false positive that broke the T10/T24 contract tests. Now:
- probe output != 'ok' -> genuine corruption -> halt with repair guidance
- empty probe output -> probe couldn't run -> warn and let flask surface
any real error
test_update.sh: T10/T24 fixtures stub the venv python (healthy-DB path);
new T10b asserts the corrupt-DB halt fires before flask runs; new T10c
asserts an unrunnable probe warns and proceeds.
- create_app(): skip background daemons (queue consumers, metrics, job
system, analytics flush, linked-panel client) when loaded by a Flask CLI
one-shot such as 'flask db upgrade', or when SERVERKIT_SKIP_BACKGROUND=1.
They query the DB before migrations run, so any error there (corrupt DB,
pre-migration schema) aborted the migration and sank the whole update.
- update.sh: set SERVERKIT_SKIP_BACKGROUND=1 for both flask db upgrade
invocations as an explicit contract.
- update.sh: fail fast with an actionable message when the SQLite slot copy
fails PRAGMA integrity_check, instead of dying mid-boot with 'database
disk image is malformed'.
- update.sh: rebuild the release tarball's prebuilt venv when its baked
VIRTUAL_ENV path resolves to the old slot (it is baked as
/opt/serverkit/venv), which previously ran new code on the old slot's
dependencies during migration. Resolved-path comparison keeps the fast
prebuilt path for fresh installs.
Make `update.sh` fetch the CI-built release tarball by default instead of
rebuilding the SPA on the server, so a normal update needs no Node/npm on the
box. This keeps updates working on old distros, ARM/Pi and tiny hosts where the
vite 8 toolchain (Node 20.19+/22.12+) can't or shouldn't run — the Node
requirement stays a build-time/CI concern, not a runtime one.
On-box source rebuilds remain available and explicit: --source / --main (main
branch), --branch <name>, or BUILD_FROM_SOURCE=1. --branch now implies source
mode (clears the release default). npm is only required for those paths.
Test: test_update.sh +1 (mode mapping — default=release, --source/--branch/
BUILD_FROM_SOURCE=source). 49 passed / 0 failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The vite 8 upgrade (dcf05d1) makes the frontend build require Node ^20.19.0 ||
>=22.12.0 — on anything older npm silently skips rolldown's native binary and
`npm run build` dies with a cryptic MODULE_NOT_FOUND. Servers that build the SPA
on-box (source install + every update) would break their next update. So:
- install.sh node_ready() now gates on the real vite 8 floor (was Node >=18) and
provisions NodeSource 22 LTS when the host Node is too old.
- update.sh gains _node_build_ok(): build_frontend_bundle refuses (returns 1 →
caller rolls back) and the Docker-host rebuild halts, both with an actionable
"upgrade Node" message instead of a rolldown stack trace.
- Dockerfile + CI (release, release-smoke, extensions, scripts-nightly) bumped
node 20 -> 22 so the shipped/pre-built bundle is built on a compatible Node.
Also fixes a pre-existing parity bug the operator caught: update.sh rendered the
systemd unit WITHOUT substituting @BIND_HOST@ (install.sh does), so an update
left the literal placeholder and gunicorn would fail to bind. update.sh now
substitutes it (loopback default, SERVERKIT_BIND_HOST override) like install.sh.
Tests: test_update.sh +3 (node-floor gate, build refusal on old Node,
@BIND_HOST@ substitution) — 48 passed / 0 failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two gaps for existing installs:
- The tint ran only in install.sh, so `serverkit update` (which has its own
build path in scripts/update.sh) would rebuild with the DEFAULT color and a
fixed favicon.ico — existing installs would never get a unique mark.
- Re-tinting on every rebuild would also repaint the box on each update.
Fix: assign the color ONCE and persist it in /etc/serverkit/favicon-color
(survives updates), then re-apply the SAME color from both install and update.
update.sh applies it to the staged dist unconditionally (after the guarded
build), so it works whether dist was rebuilt (source mode) or shipped prebuilt
(release). An install that predates the feature picks up a color on its first
update and keeps it thereafter — a car keeps its paint.
Verified: bash -n on both scripts; persistence confirmed (re-run reuses the
stored color).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 1.7.25 -> 1.7.33 update broke on panels that still had the retired
serverkit-workflows extension on disk: preserve_installed_plugins carried
it into the fresh tree, where its reactflow import (removed with plan 45)
sank the whole bundle build and blocked the update.
- RETIRED_PLUGIN_SLUGS: the carry-forward now skips slugs the product
deliberately deleted, and docker in-place mode strips their leftovers
after the hard reset (untracked dirs survive it).
- Quarantine-and-retry: when the bundle build fails and plugins were
carried forward, move them into /var/backups/serverkit/quarantined-
plugins-<ts> and retry once -- fail the plugin, never the update.
The updater self-refreshes from main before running, so this protects
every panel on its next `serverkit update` once merged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
serverkit migrate-db forwarded the wrapper's own name to cli.py, which
only defines db-migrate — the command has failed with 'No such command'
since the rename. Fix the forward and stop the drift class entirely:
- cmd_panel_cli forwards Click-only verbs verbatim (manifest, services,
apps, repair, login-url, support-bundle, db-migrate/status/history;
panel-status/panel-doctor where a native bash name collides)
- serverkit completion prints a bash completion script; install.sh and
update.sh drop it into /etc/bash_completion.d (best-effort), the
uninstall lib removes it
- test_cli.sh T17-T19: migrate-db forwarding, dispatch reachability,
and a two-way parity guard (cli.py verbs <-> dispatch <-> completion);
test_update.sh T29 covers refresh_bash_completion
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A test box's panel domain kept serving a Jun 25 bundle after the Jul 3
update — the domain's custom vhost still proxied the retired :3847
frontend container, and the old bundle's calls to deleted legacy
marketplace endpoints read as 'Failed to load extensions'. It looked
exactly like a browser/CDN cache problem (it wasn't — cf-cache-status
was DYNAMIC; the origin itself was stale).
- both stock confs: expires -1 on the SPA entrypoint (emits
Cache-Control: no-cache WITHOUT discarding server-level security
headers, unlike add_header-in-location) — a cached index.html
references hashed assets that vanish on the next blue/green switch
- update.sh refresh_config now runs report_stale_panel_vhosts: warns,
naming the file, when any enabled vhost still proxies :3847;
observation-only per R1, silent on a missing sites-enabled dir,
added to the fresh-box loop per policy
- proving tests: entrypoint no-cache asserted for both confs; stale
vhost flagged (and clean one not); missing-dir silence (42 passed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- PULL_REQUEST_TEMPLATE gains a scripts-only checklist condensing the
failure-semantics rules (guarded observations, no &&-tail returns,
capture-safe stdout, errexit-immune helpers, fresh-box loop policy,
proving test in the same commit)
- record the self-update channel decision in the bootstrap header:
keep fetching from main, gated by the update-e2e CI job, preserving
one-push-fixes-the-fleet
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The July 2 outage function had a unit test — with a populated fixture
only. This encodes the missing net:
- fresh-minimal-box fixture loop: every observation/discovery/snapshot/
report function in update.sh (17) and install.sh (16) must survive a
zero-app box under set -Eeuo pipefail; policy comment requires new
functions to be appended
- scripts/test/stubs.sh: shared failing-stub factories (systemctl exit
5, docker fail/empty, curl exit 22, GitHub rate-limit JSON)
- assertion sweep: 43 bare capture-then-compare sites now include exit
status in the assertion
- best-effort contract test: the run block's real APP_BASELINE ->
report chain exits 0 on a fresh box AND on a genuine regression
- fix(update): the loop immediately caught report_failure's trailing
'[ -n $UPDATE_LOG ] && printf' returning 1 when UPDATE_LOG is empty
(same species as the outage) — converted to if, added to the loop
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- find-based backup pruning; empty/missing BACKUP_DIR is a no-op (U1)
- download_release progress to stderr; stdout is the tarball path only (U2)
- guard bare systemctl stop/start/daemon-reload in forward+rollback paths (U3,U4,U8,U13)
- ROLLING_BACK=1 inside rollback(); never runs twice via the EXIT trap (U5)
- npm preflight for source updates; guarded frontend build pipeline (U6)
- empty-ORIG_ARGS re-exec guard, cleanup_on_exit return 0, df -Pk,
SIGPIPE guards, cd leaks subshelled, local typo (U7,U9-U12,U14)
- 14 new proving assertions in test_update.sh (35 passed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Allow failures when querying GitHub release tag (prevent pipeline from aborting) and add a unit test ensuring discover_app_upstreams tolerates an empty app-locations directory.
Changes:
- scripts/update.sh: make the curl|grep pipeline resilient by appending `|| true` so the command substitution won't fail the script when the GitHub API call returns nothing.
- scripts/test/test_update.sh: add test T16b to verify discover_app_upstreams does not abort under `set -euo pipefail` when the app-locations directory is empty.
These changes prevent hard failures during release detection and add coverage for empty-directory handling.
Suppress a non-zero exit from the pipeline in discover_app_upstreams by appending `|| true` after the sort. This prevents the script from exiting with an error when no matching proxy_pass entries are found in APP_LOCATIONS_DIR (e.g., empty or missing files). Improves robustness of scripts/update.sh in scenarios with no app upstreams configured.
A panel update deploys a fresh clone/tarball preserving only .env and
the DB, wiping app/plugins/<slug> + src/plugins/<slug> for any
URL/registry/upload-installed extension — the surviving InstalledPlugin
row then flipped to error at next boot. Two-layer fix:
- update.sh: preserve_installed_plugins carries user-installed plugin
dirs from the old tree into the new one (skipping anything the new
tree already ships and __pycache__), before the frontend build so
carried UIs recompile. Covered by harness test T18.
- boot repair pass (plugin_service.repair_missing_plugins, runs before
load_all_plugins): builtin slugs reinstall from builtin-extensions/,
url installs re-download from source_url (force + no hot-load so the
loader registers once), sourceless uploads get a clear re-upload
error. install_from_url/path grew force/hot_load plumbing.
4 new repair tests (16 total in the pipeline suite); update.sh harness
20 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Turns the zero-downtime promise into an actual check. The updater now
snapshots which managed-app upstreams are reachable through nginx
*before* the blue/green switch, re-probes after the graceful reload, and
reports any app that was serving and has since gone dark.
- discover_app_upstreams scans /etc/nginx/serverkit-locations/*.conf for
the per-app proxy_pass upstreams (host:port).
- probe_app_upstreams / snapshot_app_reachability record up/down (any
HTTP status = up; a refused/timed-out connection = down).
- report_app_uptime_regressions diffs before-vs-after and warns for any
up->down app, ignoring apps that were already down. Best-effort: it
never fails an otherwise-healthy update, just surfaces the regression.
- New "Verifying App Uptime" phase wired into the main flow; baseline
captured before the backend stop.
Tests: T16 (upstream discovery), T17 (regression detection both ways).
19 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The panel is now fully off Docker: the React SPA is served as static
files directly by host nginx from frontend/dist, eliminating the
serverkit-frontend container and the double nginx hop (host nginx ->
container nginx -> dist). Docker is now used ONLY for managed app
workloads.
- nginx sites (serverkit.conf + serverkit-insecure.conf): `location /`
now roots at /opt/serverkit/frontend/dist with a try_files SPA
fallback and a long-cache /assets/ block, instead of proxying
127.0.0.1:3847. Root tracks the /opt/serverkit symlink so blue/green
updates swap assets atomically.
- install.sh: launch_services no longer builds/starts a frontend
container; adds apply_frontend_root (rewrites the dist root for a
custom SERVERKIT_DIR) and selinux_label_frontend_dist (httpd_sys_-
content_t so SELinux hosts can read the bundle from /opt).
- update.sh: refresh_config carries the same root rewrite across
upgrades; selinux_label_dist relabels the switched-in slot; dropped
the `docker compose up -d --force-recreate frontend` from both the
forward and rollback paths.
- docker-compose.yml: frontend service moved behind a `legacy-frontend`
profile (opt-in escape hatch); default `up -d` starts nothing.
- docs/DEPLOYMENT.md: documents the off-Docker panel model.
Tests: T14 (refresh_config rewrites dist root for custom install dir),
T15 (shipped nginx configs serve static, never proxy :3847). 16 passed.
Note: live nginx -t / SELinux file-serving not validated on a RHEL box
from this dev host; logic is best-effort guarded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Host nginx is the front door for every managed app (it reverse-proxies
their containers via /etc/nginx/serverkit-locations/*.conf). The updater
used to `systemctl stop nginx` for the whole blue/green switch window,
which blacked out every unrelated hosted site for 10-60s during a panel
update. The slot switch only ever changes nginx config, never anything
nginx serves from the slot, so a stop was never required.
Add reload_nginx_graceful(): validate config (nginx -t) then
`systemctl reload nginx` - keeps listening sockets and in-flight
connections alive, so apps never blink. Starts nginx if it happens to be
down. Wired into both the forward update flow and rollback(); only the
panel backend is now cycled. Dropped the stray frontend-only
`docker compose up -d backend` line.
Tests: T11 reloads-not-stops a running nginx, T12 starts a stopped
nginx, T13 guards against `systemctl stop nginx` reappearing. 13 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hardening plan §6. Reduces the distro-specific if/else surface and handles
edge environments:
- Dependency manifest (scripts/deps/manifest.yaml): declarative per-family
package/python/node/docker/init spec. Consumed directly by the Phase 5
Python bootstrap; install.sh mirrors it inline (it can't parse YAML before
Python exists in the curl|bash path).
- New sourceable libs (built in parallel, each shellcheck-clean + unit-tested):
- scripts/lib/pkg.sh — pkg_detect/refresh/install/is_installed across
apt/dnf/yum/zypper/pacman/apk (PKG_MGR_OVERRIDE + PKG_DRY_RUN for tests).
- scripts/lib/init.sh — init_detect + start/stop/enable/disable/reload/
is_active across systemd/openrc/runit/sysvinit/none (INIT_OVERRIDE/DRY_RUN).
- scripts/lib/env.sh — is_container/is_wsl/has_systemd (override-friendly).
- ID_LIKE handling (G5/6.5): install.sh's os_family_from() (now a pure,
testable function reading ${SERVERKIT_OS_RELEASE}) maps unknown IDs via
ID_LIKE to debian/rhel/fedora/suse/arch/alpine — rhel before fedora so RHEL
clones pick the RHEL Docker repo.
- zypper + pacman + apk (G6/6.6): choose_pkg_manager/refresh_pkg_index/pkg_add,
provision_docker and provision_python gained suse/arch/alpine branches;
Docker enable/start is now init-aware (systemd OR OpenRC) so it no longer
aborts on Alpine.
- Templated systemd unit (6.7): serverkit-backend.service → templates/
serverkit-backend.service.in, rendered (render_service_unit) at install and
in refresh_config so a custom SERVERKIT_DIR/venv/log path is honored.
install_service warns-and-skips enable when has_systemd is false (G5).
- nginx TLS floor as a reversible conf.d snippet (6.8): install.sh + update.sh
prefer /etc/nginx/conf.d/serverkit-tls.conf when nginx.conf doesn't already
declare ssl_protocols (RHEL/Fedora/SUSE), falling back to the in-place edit
on Debian/Ubuntu to avoid a 'duplicate ssl_protocols' error. uninstall
removes only that snippet.
Tests: new scripts/test/test_lib.sh (13) + 4 new test_install.sh cases
(os_family_from, render_service_unit, harden_global_tls A/B). 13+15+10 pass;
shellcheck error+warning clean across all new/changed shell.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updater robustness from the hardening plan §5:
- Version comparison before updating (G3): is_already_current compares the
local VERSION to the target release tag (release mode) or the local checkout
HEAD to the remote branch HEAD via `git ls-remote` (branch/main mode). When
current, both the hybrid and docker flows print "Already up to date" and exit
0 — unless --force. Offline/indeterminate always proceeds rather than wrongly
blocking.
- Dry-run fidelity: new run_in_dir reports the working directory a command
would run in, replacing the `run_or_dry cd X && run_or_dry compose ...` chain
that printed compose commands with no cwd context.
- Rollback hardening: migrate_database now runs against the NEW slot's DB copy
(slot-absolute path) instead of the /opt/serverkit symlink that still
resolves to the live old slot — so a rollback hands the old code a
byte-for-byte-unchanged database (PostgreSQL/shared DBs migrate directly, as
before). rollback_docker guards `git reset` on a non-empty SHA with a clear
warning. Both rollback paths now run a post-rollback health check and say
plainly when even the rollback is unhealthy.
- Telemetry/logging: halt() always prints the full log path; both summaries
gained an "old → new", mode (release/branch/main, +docker), and duration
line, plus the log path.
Tests: test_update.sh gains versions_equal, is_already_current force/offline
short-circuits, and a flask-stub proof that migrate_database targets the new
slot's DB (T10). 10 passed / 2 skipped locally; shellcheck error+warning clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Critical install/update/uninstall hotfixes from the hardening plan
(docs/plans/INSTALLER_UPDATER_UNINSTALLER_HARDENING_PLAN.md §4):
- Firewall automation (G1): new sourceable scripts/lib/firewall.sh detects
firewalld/ufw/nftables/iptables and opens 80/443; install.sh runs
configure_firewall, update.sh re-ensures it idempotently. Exactly what we
open is recorded in /etc/serverkit/install-state.json (scripts/lib/state.sh)
so uninstall undoes only our rules. Fixes the #1 "installed but unreachable"
failure on fresh RHEL-family boxes.
- Accept Python 3.11 OR 3.12 (G2): locate_python now iterates
python3.12/3.11/3 and provision_python installs the distro package
(python3.11 on Debian 12, 3.12 on Ubuntu/Fedora, 3.11/3.12 on RHEL) before
ever falling back to a source compile.
- Use $PYTHON_BIN for secret generation instead of a bare python3.
- Distro-first Node 20 (replacing unconditional NodeSource): install the
distro nodejs+npm, fall back to NodeSource only when it's missing or < 18,
and validate node>=18 before continuing.
- Unify uninstall (G4): new scripts/lib/uninstall.sh is the single canonical
teardown routine shared by uninstall.sh and `serverkit uninstall` (they had
drifted — one ran `down -v`, one didn't). Default preserves user data
(volumes kept, DB snapshotted to backups); --purge/--delete-volumes removes
volumes+DB+data dirs; --keep-data preserves /etc + /var/lib + backups;
--yes/--dry-run added; honors SERVERKIT_DIR. Removes the apt lock-wait
drop-in on uninstall.
Tests: new scripts/test/test_install.sh (11 cases, source-able install.sh)
covers python/node detection, firewall dry-run + state, and the uninstall
data-handling matrix. Existing test_update.sh still green. shellcheck clean
(error + warning) on all new/changed shell.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two reliability gaps remained after the failure cascade:
1. Stale-updater trap. The updater that runs is whatever is installed, so an
old update.sh fails in ways the current code already handles and can't fix
itself — exactly what left a box stuck. maybe_reexec_latest_updater fetches
the newest update.sh for the target ref and re-execs into it (guarded by
SERVERKIT_UPDATER_REEXECED, skipped offline/dry-run/opt-out, only after a
bash -n check). From this version on, 'serverkit update' is reliable no
matter how stale the box is.
2. No concurrency guard. acquire_update_lock takes an flock so two updates
can't clobber the same blue/green slot or race docker compose; the lock
auto-releases on exit and degrades gracefully where flock/the path is
unavailable.
Both are covered by scripts/test/test_update.sh (guard no-ops; lock
refuses a held lock, succeeds when free).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Harden update.sh against the class of failure that left a box silently stuck
on the old version, and make it testable without a real deploy.
Robustness (cascade):
- L5 loud failures: `set -Eeuo pipefail` (the -E/errtrace makes the ERR trap
fire inside functions too) + a report_failure ERR trap that names the phase,
exit code, line and command, plus always-on logging to
/var/log/serverkit/update-*.log. No more silent drop-to-prompt.
- L4 docker-path rollback: snapshot the previous git SHA, :rollback image tags
and dist.rollback before mutating; if the new backend is not healthy within
90s, revert to the snapshot and halt (was: warn and continue).
- L6 self-heal: pin COMPOSE_PROJECT_NAME (kills the symlink-rename warning) and
reclaim a stale inactive blue/green slot left by a prior failed hybrid run.
Testability:
- Parameterize the system dirs (SERVERKIT_NGINX_DIR / _LETSENCRYPT_DIR /
_SYSTEMD_DIR / _CONFIG_DIR) and add SERVERKIT_NO_LOG, so refresh_config can
run against fixtures instead of the host /etc.
- Make update.sh source-able via a BASH_SOURCE guard so functions can be
exercised directly.
- scripts/test/test_update.sh: plain-bash unit tests (no deps, no server) that
reproduce the missing-serverkit.conf regression and cover the
detection/slot/report logic.
- .github/workflows/scripts-ci.yml: bash -n + shellcheck + the unit tests on
changes to scripts/**, install.sh, serverkit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The panel-domain recovery greps /etc/nginx/sites-available/serverkit.conf,
but HTTP-only boxes never have that file. Under `set -euo pipefail` the
no-match/missing-file grep (exit 1/2) propagated and silently killed the
updater at the "Refreshing Configuration" phase — before the atomic switch
and before PREVIOUS_DIR is set, so no error and no rollback. The update
appeared to "do nothing" and the version never advanced.
Guard the command substitution with `|| true` (matching the idiom used a
few lines below for panel-domain).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
serverkit update (update.sh) assumed the hybrid install shape (backend in a
host venv under systemd, frontend in Docker) and unconditionally ran
`source $INSTALL_DIR/venv/bin/activate`. On an all-Docker deployment (backend
AND frontend as compose services, no host venv) that crashes with
"venv/bin/activate: No such file or directory" -- and the preceding "stop
services" step leaves the box down with no restart.
Add is_docker_deployment() (compose file present + no host venv + a
serverkit-backend container exists) that routes to update_docker_compose():
in-place git sync, optional host frontend build when ./frontend/dist is
bind-mounted, `docker compose build` (the running version keeps serving until
the build succeeds), `up -d` (backend auto-migrates on boot), then a
container-health check. The hybrid venv/systemd/blue-green path is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make email handling case-insensitive (queries + model normalization) to prevent duplicate accounts. Add SiteDomainService.panel_origin() helper and use it across services to build public URLs (Gitea, WordPress, Roundcube) and Django ALLOWED_HOSTS. Add optional custom-domain support when creating WordPress sites and propagate domain to Roundcube install. Improve environment domain fallback to prefer panel/base domains over .localhost. Small Dockerfile healthcheck tweak (use 127.0.0.1). Update install/update/build scripts to always (re)create virtualenv locally for release installs. Remove unused frontend CreateSiteModal and expose domain field in WordPress page.
KIMI's optional-SSL commit hardened ssl_protocols/ssl_ciphers per-vhost but
left the system nginx.conf http{} defaults untouched, so the default server
and any non-ServerKit vhost could still negotiate TLS 1.0/1.1 and weak ciphers
(the scan's "insecure TLS versions" / "weak ciphers in TLS 1.2" findings).
- install.sh: harden_global_tls() rewrites (or injects) ssl_protocols and
ssl_ciphers in /etc/nginx/nginx.conf to TLS1.2/1.3 + AEAD-only. Editing the
line in place avoids nginx's duplicate-directive error on Debian/Ubuntu,
which already set ssl_protocols in http{}.
- update.sh: re-applies the floor so existing installs harden on upgrade too.
- Persist the panel domain to /etc/serverkit/panel-domain; update.sh recovers
it from there (or the live cert path) instead of the commented-out .env
SERVERKIT_PUBLIC_URL grep, which never matched -> secure installs were
getting the YOUR_DOMAIN placeholder re-applied and nginx -t failing on update.
- update.sh now degrades to the HTTP config when a secure cert is missing,
rather than leaving nginx unable to reload.
Global TLS settings are inert in HTTP-only mode, so optional SSL is unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add secure nginx config with TLS 1.2+, HSTS preload, CSP, and security headers
- Add insecure HTTP fallback config for users who cannot or do not want HTTPS
- Make installer best-effort SSL: try Let's Encrypt, fall back to HTTP without failing
- Preserve secure/insecure SSL choice across updates via /etc/serverkit/ssl-mode
- Support SERVERKIT_SKIP_SSL=1 to disable HTTPS entirely
- Add CAA record to default DNS zone template
- Harden generated per-app nginx configs with HSTS preload, CSP, and security headers
- Align Flask security middleware headers with nginx edge config
Juan Denis