Persist the databases ServerKit provisions (and let you adopt existing ones) so
backups, connection strings, and the UI have durable state -- beside, not
replacing, the live-introspection explorer. Not a DBaaS (no pooling/replicas).
- ManagedDatabase model (engine/name/host/port/origin/owner; admin secret
Fernet-encrypted, unique per engine+host+name) + migration 044.
- managed_database_service: record_provisioned (additive on the create routes),
adopt (idempotent), build_connection_uri (real URI, masked by default),
sync_state (drift detection), delete (optional drop), protect.
- Backup FK fix (the headline): a managed-DB BackupPolicy uses
target_id = managed_databases.id plus a `managed` marker, and _resolve_target
resolves the descriptor (incl. decrypted secret) from the row. Legacy
JSON-descriptor policies keep resolving unchanged (backward compatible).
- API under /api/v1/databases/managed (list/create/get+sync/delete?drop=/adopt/
connection-uri/protect); the connection-uri reveal is admin-only and audited.
- Frontend: a "Managed" panel on the Database Explorer (connection-string
reveal/copy, one-click Protect, Untrack/Drop).
- Tests: provisioning persists a row, adopt idempotent, connection URI per
engine + masking, sync_state drift, protect resolves via the real FK, legacy
policy still resolves, API masking/admin-gating/audited reveal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Give a service tracked, named persistent storage that survives redeploys,
replacing fragile relative bind mounts (./mysql-data:/var/lib/mysql).
- AppVolume model (namespaced serverkit-app-{id}-{slug}, unique per mount path)
+ migration 043.
- volume_service: create (docker volume + row, dup-path refusal), list_for_app
(rows + live inspect), delete (row-only by default; wipe only when the app is
stopped and no running container uses it), measure, convert_bind_mount,
run_args, compose_fragment, host_paths_for_app.
- Wired into the single-container deploy (_deploy_docker passes -v specs to
docker run) and new compose stacks (create_docker_app gains named_volumes ->
top-level volumes: block).
- API under /api/v1/apps/<id>/volumes (list/attach/detach?wipe=/convert), ACL
via ResourceGrantService.
- Frontend: Storage panel in Service detail Settings (attach/detach with an
off-by-default "also delete data" checkbox, disabled while running).
- Tests: create + namespaced volume, dup-path refusal, wipe safety, single-
container -v proving test, compose top-level named-volume generation, convert,
API CRUD.
Scoping: volumes are backup-addressable (mountpoints exposed; protect via the
files target) rather than force-folded into the app's single-parent incremental
tar chain, which would break restore. Live compose files (esp. the WP multidev
stack) are not rewritten -- convert_bind_mount is the opt-in migration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Store per-registry credentials (Fernet-encrypted) and run `docker login`
before pulling a private image — GHCR, Docker Hub, GitLab, ECR, or any
generic registry. Public/anonymous pulls are unchanged.
- ContainerRegistry model (secret encrypted at rest; to_dict never returns it)
+ nullable applications.registry_id; migration 042.
- container_registry_service: CRUD + login/logout/test_connection (secret piped
via stdin, never on argv) + lazy ECR token exchange.
- DockerService.pull_image gains a registry= arg (login -> pull -> logout in a
try/finally). Wired into the deploy path (_deploy_docker) and the compose
path (start_app / apply_image_update): a docker login authenticates both
`docker pull` and `docker compose pull`.
- API under /api/v1/connections/registries (list=auth, writes=admin); registries
join the unified /connections list.
- Frontend: Connections hub "Container registry" card (add/test/remove with a
provider selector) + a per-app registry picker in Container Ops.
- Tests: encrypt/decrypt, stdin-not-argv, login->pull->logout ordering incl.
finally-on-failure, deploy + compose auth-before-pull, API CRUD/masking/admin.
Deviation from the plan's "single injection point": docker apps run via compose,
so login was wired into both the compose and run_container paths; remote-agent
compose is a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Register WordPress plugins from GitHub repos or local paths in a global
library, then install or update them across any managed WP site from one
place. The per-site Plugins tab gains a "managed" badge and an
"update from library" action.
- models: WordPressCustomPlugin (library) + WordPressSitePlugin (installs)
- service: WordPressPluginLibraryService - sync, install, update, scan,
with a per-slug cache dir under paths.SERVERKIT_CACHE_DIR
- api: /plugins/library CRUD + sync/install/bulk-update on wordpress_bp
- migration 041_wordpress_plugin_library
- frontend: WordPressPluginLibrary page + "Plugin Library" tab; managed
badges and library-update action in PluginsTab
- tests: test_wp_plugin_library
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bring the multi-base picker to the generic app/service publish flow, matching
the WordPress create flow. The service Settings → "Give it a subdomain" modal
now loads the registered base domains and, when more than one exists, shows a
"Base domain" dropdown (default marked, HTTPS flagged); the live suffix and the
wildcard/per-site hint follow the selection, and the choice is passed through to
give-subdomain so the app publishes at <label>.<chosen-base>. A single-base
install is unchanged (no dropdown).
api.suggestSubdomain / giveSubdomain take an optional base; new
api.getSiteBaseDomains() reads GET /domains/base-domains.
Proven: suggest-subdomain honours ?base (test_site_base_domains.py); frontend
lints clean (0 errors) and builds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Turn the single-base "Managed Sites Domain & HTTPS" card into a registry
manager. Operators can now register several base domains, mark a default,
remove one, flip each between wildcard/per-site DNS, and set up per-base
wildcard HTTPS with a chosen DNS provider — all from the UI.
- Each registered base renders as a row with Default / HTTPS badges, a DNS-mode
select, a "Set up / Renew HTTPS" action (using the selected provider, base
passed through), plus Make default / Remove.
- Back-compat: an install with no registry rows shows its single legacy base as
an editable in-place row (saves sites_base_domain); adding a second domain
seeds that legacy value into a real row server-side, and the table takes over.
A fresh install with no base shows an empty state + the Add form.
- Server public IP and DNS provider are shared across bases.
- New admin PATCH /admin/sites-https/base-domains/<domain> for DNS-mode edits;
api service gains add/remove/setDefault/update helpers and setupSitesHttps now
takes a base.
Proven: test_site_base_domains.py adds a PATCH-dns-mode API test (13 pass);
frontend lints clean (0 errors) and builds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
With several base domains registered, the create modal now lets you choose
which one the site publishes under. A "Publish under" dropdown lists the
available bases (default first, marked, showing which have HTTPS) and previews
the exact live host — <slug>.<base> — as you type the name. The choice rides
through as baseDomain in the existing createSite payload. When no base domain
is configured the field explains the site will stay on localhost until one is
set. The Custom Domain field is clarified as an override of the base choice.
wordpressApi.listBaseDomains() reads GET /api/v1/domains/base-domains; the base
list loads when the modal opens and preselects the returned default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the base-domain registry into HTTP + the cert flow:
- SitesHttpsService.setup(provider_id, email, base) now issues the wildcard cert
for a specific base domain and persists HTTPS state to THAT base's registry row
(recording the provider used), so example.com on Cloudflare and toto.com on
Route53 each get their own cert; falls back to the legacy sites_* settings for
a single-base install. status() now also returns the full `bases` registry.
- Admin API: POST/DELETE /admin/sites-https/base-domains[/<domain>] and
.../<domain>/default to register, remove, and set the default base; setup
accepts an optional `base`.
- GET /api/v1/domains/base-domains lists the bases a new site can use (registry,
or the single legacy base) with the pre-selected default — the create-flow
picker's data source. give-subdomain + suggest-subdomain accept a `base`.
- WordPress create accepts baseDomain end to end (API -> service -> subdomain_for),
and picks the covering base's HTTPS state for the baked-in site URL.
Proven in test_site_base_domains.py: the list endpoint (registry + legacy),
admin add/set-default/remove, and per-base HTTPS setup persisting to one row
while the other base stays HTTP. WordPress suite (66) still green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Managed sites had exactly one base domain (the sites_base_domain setting), so
every site was <slug>.<that-one>. Add a registry (site_base_domains) letting an
operator register several — example.com, toto.com — each with its own DNS mode,
HTTPS state, and DNS provider connection, so a new site can be published under
any of them (a.example.com or a.toto.com), defaulting to the one marked default.
- New SiteBaseDomain model + migration 040 (own dns_provider_config_id + cert
state per base; exactly one is_default).
- SiteBaseDomainService: add/remove/set_default/update, and ensure_seeded which
materialises the legacy single sites_base_domain setting into the default row
the first time a second domain is registered — so old and new never disagree.
- SiteDomainService is now base-aware and back-compatible: subdomain_for(name,
base), https_enabled(base), dns_mode(base), wildcard_cert_paths(base) and
give_subdomain(app, base) take an optional base; covering_base(host) picks the
base (longest match) whose wildcard cert covers a host so vhosts get the right
per-base cert; base_domain() returns the default and still falls back to the
legacy setting / SITES_BASE_DOMAIN config for un-migrated & dev installs.
Existing single-base installs are untouched (zero rows -> setting fallback);
45 existing site-domain tests still green. New coverage in
test_site_base_domains.py: seeding, default promotion/demotion, per-base
https/dns-mode/cert, longest-match cover, and give_subdomain under a chosen base.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A managed-sites base domain that contains the panel's own hostname is a
wildcard footgun: *.<base> also matches the panel host, so it keeps resolving
only while the panel's explicit DNS record out-specifies the wildcard (or, when
base == panel domain, the wildcard setup's apex record repoints the panel
itself). Add a fourth publishing gap that detects this and nudges admins.
base_domain_overlaps_panel() flags panel host == base and panel host as a
direct single-label child of base; a deeper descendant (a.b.<base>) is safe
because a wildcard is single-label. It reads the panel's genuinely-configured
domain (canonical_domain / PUBLIC_URL), not panel_origin's base-domain
fallback — that fallback would equate panel with base and both mask real
overlaps and break the subdomain-vs-apex suggestion.
Proven in test_sites_publishing_gaps.py: equal-domain and direct-child overlap,
deep-descendant and scoped-label (apps.<domain>) non-overlap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Creating a site while the managed-sites config is only partly done (base
domain set but not the DNS/HTTPS half, or nothing set at all) left the site
quietly on localhost with no signal to finish setup. Now each create checks
readiness and drops an in-app nudge to admins for any open gap:
- no base domain -> site is localhost-only; suggests one to set
- base domain, HTTPS off -> HTTP-only (gentle/optional, HTTPS stays best-effort)
- per-site DNS mode, no server IP -> A records can't auto-create
The suggested base domain is derived from the panel's OWN domain, so it fits
the install shape: an apex install (example.com) suggests apps.example.com,
while a subdomain install (panel.example.com) suggests a sibling
apps.example.com that won't collide with the panel's record. Nudges are
deduped against an already-open unread notice, so a gap is raised once, not on
every create. Wired into WordPress create and the generic give_subdomain
publish path; best-effort so a notification failure never affects the create.
Proven in test_sites_publishing_gaps.py (each gap, the apex-vs-subdomain
suggestion, and the dedup).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After creating a WordPress site the post-create banner showed only the admin
credentials, so users had no signal that the site was already reachable at
<slug>.<base> — the exact "preview URL before DNS" gap roadmap #21 tracked,
now unblocked by the base-domain + Phase-5 wildcard-cert stack.
The banner now surfaces the live link on every successful create: a clickable
<slug>.<base> (noting HTTPS when the wildcard cert is set up), or, when the
site only came up on localhost, a nudge to set a managed-sites base domain in
Settings to publish it publicly. Shown even when the automated install did not
finalize (no captured credentials).
Mark #21 done in the roadmap: the wildcard base subdomain is the working,
TLS-valid preview a user relies on while pointing a custom domain, so the
separate *.preview.<panel-domain> scheme is unnecessary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
give_subdomain only wrote an nginx vhost when app_type=='docker', so managed
PHP/static/python/flask/django apps got a Domain row but no route and stayed
reachable only at localhost:<port>. NginxService.create_site already had the
right templates for all of them.
Add a shared SiteDomainService.write_app_vhost(app, force_type=None) +
_vhost_create_kwargs mapper (docker/wordpress -> docker proxy,
flask/django/python -> proxy+static, php/static -> filesystem root; same
wildcard-cert selection as before) and route give_subdomain through it.
Collapse WordPressService._write_app_vhost into a thin delegate
(force_type='docker'), removing the duplicate with no behavior change.
Proven in test_dns_give_subdomain.py: php/static route from root, python
routes by port, and an unroutable app surfaces a warning instead of
silently faking success.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a 'Related Projects' section linking Faro (desktop SFTP/SSH/S3 client) to the main README and translated docs (Spanish, Portuguese, Chinese). Explains Faro's features and relationship to ServerKit, and adds a link to the latest releases. This provides users a recommended desktop companion for file transfers and agent bridge functionality.
The panel frontend is now served straight from nginx/sites-available/*
(static SPA at location /, plus /api + /socket.io proxy). A broken
directive there 502s the whole panel on a real box, and nothing parsed
these configs in CI.
Add scripts/test/test_nginx_conf.sh: stages each vhost (secure +
insecure) against a real nginx, recreating the absolute paths they
reference (a stand-in frontend/dist bundle, the serverkit-locations /
conf.d include dirs, certbot webroot) and a throwaway self-signed cert
for the TLS vhost, then runs `nginx -t`. Skips cleanly (exit 0) where
nginx/openssl/root aren't available, so it's safe on a dev laptop.
Wire it into the Scripts CI `shell` job after the unit suites (installs
nginx + openssl so the step actually runs). shellcheck-clean at warning
severity; bash -n clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Turns the zero-downtime promise into an actual check. The updater now
snapshots which managed-app upstreams are reachable through nginx
*before* the blue/green switch, re-probes after the graceful reload, and
reports any app that was serving and has since gone dark.
- discover_app_upstreams scans /etc/nginx/serverkit-locations/*.conf for
the per-app proxy_pass upstreams (host:port).
- probe_app_upstreams / snapshot_app_reachability record up/down (any
HTTP status = up; a refused/timed-out connection = down).
- report_app_uptime_regressions diffs before-vs-after and warns for any
up->down app, ignoring apps that were already down. Best-effort: it
never fails an otherwise-healthy update, just surfaces the regression.
- New "Verifying App Uptime" phase wired into the main flow; baseline
captured before the backend stop.
Tests: T16 (upstream discovery), T17 (regression detection both ways).
19 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The panel is now fully off Docker: the React SPA is served as static
files directly by host nginx from frontend/dist, eliminating the
serverkit-frontend container and the double nginx hop (host nginx ->
container nginx -> dist). Docker is now used ONLY for managed app
workloads.
- nginx sites (serverkit.conf + serverkit-insecure.conf): `location /`
now roots at /opt/serverkit/frontend/dist with a try_files SPA
fallback and a long-cache /assets/ block, instead of proxying
127.0.0.1:3847. Root tracks the /opt/serverkit symlink so blue/green
updates swap assets atomically.
- install.sh: launch_services no longer builds/starts a frontend
container; adds apply_frontend_root (rewrites the dist root for a
custom SERVERKIT_DIR) and selinux_label_frontend_dist (httpd_sys_-
content_t so SELinux hosts can read the bundle from /opt).
- update.sh: refresh_config carries the same root rewrite across
upgrades; selinux_label_dist relabels the switched-in slot; dropped
the `docker compose up -d --force-recreate frontend` from both the
forward and rollback paths.
- docker-compose.yml: frontend service moved behind a `legacy-frontend`
profile (opt-in escape hatch); default `up -d` starts nothing.
- docs/DEPLOYMENT.md: documents the off-Docker panel model.
Tests: T14 (refresh_config rewrites dist root for custom install dir),
T15 (shipped nginx configs serve static, never proxy :3847). 16 passed.
Note: live nginx -t / SELinux file-serving not validated on a RHEL box
from this dev host; logic is best-effort guarded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Host nginx is the front door for every managed app (it reverse-proxies
their containers via /etc/nginx/serverkit-locations/*.conf). The updater
used to `systemctl stop nginx` for the whole blue/green switch window,
which blacked out every unrelated hosted site for 10-60s during a panel
update. The slot switch only ever changes nginx config, never anything
nginx serves from the slot, so a stop was never required.
Add reload_nginx_graceful(): validate config (nginx -t) then
`systemctl reload nginx` - keeps listening sockets and in-flight
connections alive, so apps never blink. Starts nginx if it happens to be
down. Wired into both the forward update flow and rollback(); only the
panel backend is now cycled. Dropped the stray frontend-only
`docker compose up -d backend` line.
Tests: T11 reloads-not-stops a running nginx, T12 starts a stopped
nginx, T13 guards against `systemctl stop nginx` reappearing. 13 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ServerKit's backend manages the host it runs on -- it creates app dirs under
/var/serverkit/apps, drives the host Docker daemon (one container per app),
and reloads nginx. The default compose containerized the backend, so it ran
as a non-root user into a root-owned /var/serverkit/apps with no host
docker.sock: WordPress/app creation failed with
`Permission denied: /var/serverkit/apps/...`. It also contradicted install.sh,
which runs the backend on the host via systemd and expects compose to bring
up only the frontend.
- docker-compose.yml: frontend-only; the frontend reaches the host backend via
extra_hosts ["backend:host-gateway"].
- backend/Dockerfile: chown /var/serverkit to appuser (latent EACCES on the
apps dir for dev/demo container use) + header noting it cannot manage host.
- frontend/nginx.conf: document the backend:host-gateway upstream.
- docs/DEPLOYMENT.md: add a Deployment model note; fix build/start + data paths.
- scripts/test/test_install.sh: T12 guards against re-adding a backend service.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hardening plan §8. A stdlib-only, import-safe, unit-tested Python package that
ports the stabilized bash install logic into testable orchestration. Per the
plan's migration path this is ADDITIVE — install.sh stays canonical and this is
NOT wired into it yet; it is the validated foundation a future thin-bootstrap
install.sh can call.
backend/serverkit_installer/:
- _proc.py — sudo-aware subprocess helpers, mirroring app/utils/system.py
WITHOUT importing app (the bootstrap runs before the venv
exists, so it must not pull in Flask).
- distro.py — OS family mapping (a faithful port of install.sh's
os_family_from, rhel-before-fedora) + container/WSL/systemd
predicates (mirroring scripts/lib/env.sh).
- deps.py — loads scripts/deps/manifest.yaml (its primary consumer) and
resolves base/python/node/docker/pkg-manager per family.
- firewall.py — detect + pure open/close command builders (port of
scripts/lib/firewall.sh).
- service.py — init detect + pure start/stop/enable/reload builders (port of
scripts/lib/init.sh).
- main.py — composes a READ-ONLY install "plan" + `python -m
serverkit_installer plan|detect` CLI. No host mutation.
Tests: backend/tests/test_serverkit_installer.py — 46 pure-logic cases
(family mapping vs os_family_from, real-manifest resolution for all six
families, firewall/service command builders, env predicates, composed plan).
46 passed; py_compile + pyflakes clean. The CLI proves the manifest is genuinely
consumed (Debian -> apt + python3.11 + base packages), not dead config.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hardening plan §7. Catches distro-specific regressions before release.
- scripts/test/vm-update.sh: install → forced local update (full blue/green
deploy/migrate/switch/health) → re-run to confirm the "Already up to date"
version-gate. Same VM harness contract as vm-install.sh (log + status file).
- scripts/test/vm-uninstall.sh: install + seed data → default uninstall asserts
data preserved (DB snapshot in backups, /var/lib survives) → reinstall →
--purge asserts data removed. End-to-end proof of the unified uninstaller.
- scripts-ci.yml: now runs all three serverless suites (update+install+lib),
lints the new lib/ + test/ files (error severity) and keeps scripts/lib/*
warning-clean, and expands the cross-distro unit matrix to Ubuntu 22.04/24.04,
Debian 12, Rocky 9, AlmaLinux 9, Fedora 40, and openSUSE Leap 15.5.
- test_install.sh: +choose_pkg_manager graceful-failure case (7.4); a no-op
docker stub so the uninstall-matrix cases run where docker isn't installed.
- test_lib.sh: neutralize the ambient container=oci env var (RPM/SUSE images)
and force an empty PATH for the "no package manager" case — both bugs the new
matrix surfaced (the tests, not the code).
- README: fixed the stale Fedora box, documented the new VM scripts + suites.
Validated by running the exact CI container logic locally across all seven
images: test_update 13/13, test_install 15-16/0, test_lib 13/13 everywhere.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hardening plan §6. Reduces the distro-specific if/else surface and handles
edge environments:
- Dependency manifest (scripts/deps/manifest.yaml): declarative per-family
package/python/node/docker/init spec. Consumed directly by the Phase 5
Python bootstrap; install.sh mirrors it inline (it can't parse YAML before
Python exists in the curl|bash path).
- New sourceable libs (built in parallel, each shellcheck-clean + unit-tested):
- scripts/lib/pkg.sh — pkg_detect/refresh/install/is_installed across
apt/dnf/yum/zypper/pacman/apk (PKG_MGR_OVERRIDE + PKG_DRY_RUN for tests).
- scripts/lib/init.sh — init_detect + start/stop/enable/disable/reload/
is_active across systemd/openrc/runit/sysvinit/none (INIT_OVERRIDE/DRY_RUN).
- scripts/lib/env.sh — is_container/is_wsl/has_systemd (override-friendly).
- ID_LIKE handling (G5/6.5): install.sh's os_family_from() (now a pure,
testable function reading ${SERVERKIT_OS_RELEASE}) maps unknown IDs via
ID_LIKE to debian/rhel/fedora/suse/arch/alpine — rhel before fedora so RHEL
clones pick the RHEL Docker repo.
- zypper + pacman + apk (G6/6.6): choose_pkg_manager/refresh_pkg_index/pkg_add,
provision_docker and provision_python gained suse/arch/alpine branches;
Docker enable/start is now init-aware (systemd OR OpenRC) so it no longer
aborts on Alpine.
- Templated systemd unit (6.7): serverkit-backend.service → templates/
serverkit-backend.service.in, rendered (render_service_unit) at install and
in refresh_config so a custom SERVERKIT_DIR/venv/log path is honored.
install_service warns-and-skips enable when has_systemd is false (G5).
- nginx TLS floor as a reversible conf.d snippet (6.8): install.sh + update.sh
prefer /etc/nginx/conf.d/serverkit-tls.conf when nginx.conf doesn't already
declare ssl_protocols (RHEL/Fedora/SUSE), falling back to the in-place edit
on Debian/Ubuntu to avoid a 'duplicate ssl_protocols' error. uninstall
removes only that snippet.
Tests: new scripts/test/test_lib.sh (13) + 4 new test_install.sh cases
(os_family_from, render_service_unit, harden_global_tls A/B). 13+15+10 pass;
shellcheck error+warning clean across all new/changed shell.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Updater robustness from the hardening plan §5:
- Version comparison before updating (G3): is_already_current compares the
local VERSION to the target release tag (release mode) or the local checkout
HEAD to the remote branch HEAD via `git ls-remote` (branch/main mode). When
current, both the hybrid and docker flows print "Already up to date" and exit
0 — unless --force. Offline/indeterminate always proceeds rather than wrongly
blocking.
- Dry-run fidelity: new run_in_dir reports the working directory a command
would run in, replacing the `run_or_dry cd X && run_or_dry compose ...` chain
that printed compose commands with no cwd context.
- Rollback hardening: migrate_database now runs against the NEW slot's DB copy
(slot-absolute path) instead of the /opt/serverkit symlink that still
resolves to the live old slot — so a rollback hands the old code a
byte-for-byte-unchanged database (PostgreSQL/shared DBs migrate directly, as
before). rollback_docker guards `git reset` on a non-empty SHA with a clear
warning. Both rollback paths now run a post-rollback health check and say
plainly when even the rollback is unhealthy.
- Telemetry/logging: halt() always prints the full log path; both summaries
gained an "old → new", mode (release/branch/main, +docker), and duration
line, plus the log path.
Tests: test_update.sh gains versions_equal, is_already_current force/offline
short-circuits, and a flask-stub proof that migrate_database targets the new
slot's DB (T10). 10 passed / 2 skipped locally; shellcheck error+warning clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Critical install/update/uninstall hotfixes from the hardening plan
(docs/plans/INSTALLER_UPDATER_UNINSTALLER_HARDENING_PLAN.md §4):
- Firewall automation (G1): new sourceable scripts/lib/firewall.sh detects
firewalld/ufw/nftables/iptables and opens 80/443; install.sh runs
configure_firewall, update.sh re-ensures it idempotently. Exactly what we
open is recorded in /etc/serverkit/install-state.json (scripts/lib/state.sh)
so uninstall undoes only our rules. Fixes the #1 "installed but unreachable"
failure on fresh RHEL-family boxes.
- Accept Python 3.11 OR 3.12 (G2): locate_python now iterates
python3.12/3.11/3 and provision_python installs the distro package
(python3.11 on Debian 12, 3.12 on Ubuntu/Fedora, 3.11/3.12 on RHEL) before
ever falling back to a source compile.
- Use $PYTHON_BIN for secret generation instead of a bare python3.
- Distro-first Node 20 (replacing unconditional NodeSource): install the
distro nodejs+npm, fall back to NodeSource only when it's missing or < 18,
and validate node>=18 before continuing.
- Unify uninstall (G4): new scripts/lib/uninstall.sh is the single canonical
teardown routine shared by uninstall.sh and `serverkit uninstall` (they had
drifted — one ran `down -v`, one didn't). Default preserves user data
(volumes kept, DB snapshotted to backups); --purge/--delete-volumes removes
volumes+DB+data dirs; --keep-data preserves /etc + /var/lib + backups;
--yes/--dry-run added; honors SERVERKIT_DIR. Removes the apt lock-wait
drop-in on uninstall.
Tests: new scripts/test/test_install.sh (11 cases, source-able install.sh)
covers python/node detection, firewall dry-run + state, and the uninstall
data-handling matrix. Existing test_update.sh still green. shellcheck clean
(error + warning) on all new/changed shell.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The updater's logic had zero test coverage on any distro, and ServerKit
targets several Linux families — so a portability regression only ever showed
up on a server. Add a matrix job that runs bash -n + scripts/test/test_update.sh
inside debian:12 and rockylinux:9 containers (Ubuntu is already covered by the
lint job). Verified locally: 9/9 pass on bash 5.2 (Debian) and 5.1 (Rocky).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two reliability gaps remained after the failure cascade:
1. Stale-updater trap. The updater that runs is whatever is installed, so an
old update.sh fails in ways the current code already handles and can't fix
itself — exactly what left a box stuck. maybe_reexec_latest_updater fetches
the newest update.sh for the target ref and re-execs into it (guarded by
SERVERKIT_UPDATER_REEXECED, skipped offline/dry-run/opt-out, only after a
bash -n check). From this version on, 'serverkit update' is reliable no
matter how stale the box is.
2. No concurrency guard. acquire_update_lock takes an flock so two updates
can't clobber the same blue/green slot or race docker compose; the lock
auto-releases on exit and degrades gracefully where flock/the path is
unavailable.
Both are covered by scripts/test/test_update.sh (guard no-ops; lock
refuses a held lock, succeeds when free).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Harden update.sh against the class of failure that left a box silently stuck
on the old version, and make it testable without a real deploy.
Robustness (cascade):
- L5 loud failures: `set -Eeuo pipefail` (the -E/errtrace makes the ERR trap
fire inside functions too) + a report_failure ERR trap that names the phase,
exit code, line and command, plus always-on logging to
/var/log/serverkit/update-*.log. No more silent drop-to-prompt.
- L4 docker-path rollback: snapshot the previous git SHA, :rollback image tags
and dist.rollback before mutating; if the new backend is not healthy within
90s, revert to the snapshot and halt (was: warn and continue).
- L6 self-heal: pin COMPOSE_PROJECT_NAME (kills the symlink-rename warning) and
reclaim a stale inactive blue/green slot left by a prior failed hybrid run.
Testability:
- Parameterize the system dirs (SERVERKIT_NGINX_DIR / _LETSENCRYPT_DIR /
_SYSTEMD_DIR / _CONFIG_DIR) and add SERVERKIT_NO_LOG, so refresh_config can
run against fixtures instead of the host /etc.
- Make update.sh source-able via a BASH_SOURCE guard so functions can be
exercised directly.
- scripts/test/test_update.sh: plain-bash unit tests (no deps, no server) that
reproduce the missing-serverkit.conf regression and cover the
detection/slot/report logic.
- .github/workflows/scripts-ci.yml: bash -n + shellcheck + the unit tests on
changes to scripts/**, install.sh, serverkit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The panel-domain recovery greps /etc/nginx/sites-available/serverkit.conf,
but HTTP-only boxes never have that file. Under `set -euo pipefail` the
no-match/missing-file grep (exit 1/2) propagated and silently killed the
updater at the "Refreshing Configuration" phase — before the atomic switch
and before PREVIOUS_DIR is set, so no error and no rollback. The update
appeared to "do nothing" and the version never advanced.
Guard the command substitution with `|| true` (matching the idiom used a
few lines below for panel-domain).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace spinners with skeleton placeholders across page/section loaders
(EmptyState, DataTable, and ~page surfaces), add a reusable Skeleton
component + _skeleton.scss, rework AppLoader into a branded "server
powering on" loader, and refresh the favicon set (incl. SVG icon link).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backend:
- The monitoring / metrics / telemetry(events) / uptime / fleet / status-page
read surfaces are re-mounted under /api/v1/observability/<domain> as true
aliases (same blueprints, distinct names). Original prefixes stay; the public
status page route /api/v1/status/public/<slug> is untouched.
Frontend:
- The Monitoring tab group becomes "Observability": Monitoring · Events
(the SystemEvent stream, formerly the standalone "Telemetry" page) · Status
Pages, sharing one PageTopbar. /observability lands on Monitoring.
- The standalone Telemetry route + sidebar item are folded into the group
(sidebar "Monitoring" → "Observability"; Events covered by matchPrefixes;
removed from ADVANCED_ITEM_IDS). Command palette updated.
- Fleet Monitor stays under Servers (server-scoped) and GPU stays its own
hardware page — intentional, per the plan.
Deferred (plan §4 deep merge, documented): the unified events/metrics/checks/
alerts/thresholds tables + collector migration into the job system. The
read-surface namespace + IA consolidation ship now; the schema merge is a large
data migration best done against a live metrics store.
Verified: 3 backend namespace tests + telemetry suite pass (22); frontend build
+ eslint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive foundation + one read namespace for the three parallel deploy records
(versioned build Deployment, webhook GitDeployment, DeploymentJob executions).
Backend:
- DeploymentJob gains deployment_id / git_deployment_id / webhook_id FKs +
commit_hash / image_tag / container_id, so it can be the canonical execution
record pointing at the release ledgers it produces (surfaced in to_dict).
Migration 039 (idempotent, additive).
- New /api/v1/deployments surface:
GET /apps/:id/history?source=build|webhook|all → federated, source-tagged
GET /:id → build Deployment detail
and the DeploymentJob surface re-mounted at /api/v1/deployments/jobs (alias of
/deployment-jobs). Per-system blueprints (/builds, /deploy, /git) unchanged.
Deliberately NOT done (plan §3 Phase 1/2 deep merge, with rationale): folding
GitDeployment into Deployment+DeploymentJob and async-ifying the currently
synchronous build/webhook deploys. Recon confirmed those paths are synchronous
with distinct failure/timeout semantics and GitDeployment carries rollback state
(snapshot_data, rolled_back_to_id); flipping them to the queue can't be verified
without a live build/runner, so it's deferred behind the additive links above.
Verified: 6 new tests + previews/config-snapshot regression pass (35).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive API consolidation (plan §7 Phases 3–4) that exposes a WordPress site
through its Application id, treating WordPressSite as the 1:1 extension table it
already is:
- GET /apps/:id/wordpress → {is_wordpress, site_id, env_type, ...}
- GET /apps/:id/db-snapshots → WP DB snapshots (was §8-deferred)
- POST /apps/:id/db-snapshots → create
- DELETE /apps/:id/db-snapshots/:id → delete
All grant-aware via the apps.py access seam (ResourceGrantService).
Deliberately NOT done (plan §7 Phase 1/2, with rationale): blanket-setting
app_type='wordpress' on WordPressSite-linked apps. Recon showed many WP sites are
app_type='docker', and 14+ docker-compose/nginx/scale/sleep code paths key on
app_type=='docker' — changing it would silently break start/stop/restart/logs/
status/domain config for containerized WP. The codebase already treats the
WordPressSite record (not app_type) as the source of truth (wordpress.py
_is_wp_app), which is the correct architecture; flipping app_type would regress
it. Frontend WordPress→Services merge likewise deferred (the /wordpress/:id route
is site-id-keyed).
Verified: 6 new surface tests + WP/resource-grant regression suites pass (23).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Unifies "SSL Certificates" and "Advanced SSL" into one lifecycle surface.
Backend:
- The advanced cert operations (wildcard DNS-01, SAN, custom upload, TLS
profiles, health, expiry alerts) now mount under /api/v1/ssl alongside the
basic certbot routes — no path collisions. /api/v1/ssl/advanced kept as a
deprecated alias (same blueprint, second registration).
Frontend:
- api client advanced methods repointed to the unified /ssl/* paths.
- SSLCertificates.jsx (already the single SSL page) gains the two cert-creation
paths that were previously UI-inaccessible: a "Wildcard (DNS-01)" mode in the
New Certificate modal (Cloudflare token / Route 53 keys) and an "Upload"
action + modal for bring-your-own PEM cert/key/chain.
Deferred (documented follow-up): a unified Certificate registry model + disk
scan, and surfacing health/profiles panels — the API is unified and the client
methods exist; these are read-side enhancements.
Verified: 3 backend route-unification tests + existing SSL suites pass (19);
frontend build + eslint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the email unification: the outbound relay is now one smtp
EmailProviderConnection flagged uses_relay, so a single table drives both the
Notification Bus and the Postfix relay. The legacy single-row email_relay_config
is folded in on first boot and kept only as a read fallback.
- EmailRelayService rewritten over the unified table: get/save/disable/test all
operate on the uses_relay smtp connection; password stays Fernet-encrypted in
credentials_json; masked round-trips preserve the stored secret. Postfix apply
unchanged (no-op when Postfix absent).
- migrate_legacy_config(): idempotent fold of email_relay_config → connection
(password copied verbatim — same Fernet key); wired into the boot-time legacy
migration block.
- API: PUT /notifications/admin/email-providers/:id toggles usage flags
(uses_notifications/uses_relay/relay_priority); uses_relay honored for SMTP only.
Verified: 6 new relay tests + usage-flag + notification-bus suites pass (30 passed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Foundation for folding the single-row email_relay_config into the multi-row
EmailProviderConnection so one table drives notifications and the Postfix relay.
- EmailProviderConnection gains uses_notifications (default true), uses_relay
(default false), relay_priority (default 0); surfaced in to_dict.
- Notification selection (default_provider) now restricted to uses_notifications
rows, so relay-only connections are never used to send notifications.
- New relay_provider() selects the active SMTP connection by relay_priority.
- add_provider() accepts the flags; uses_relay honored only for provider=smtp
(API providers can't be a Postfix smarthost).
- Migration 038 (idempotent additive columns).
Additive — existing providers default to uses_notifications=true / no relay.
Verified: 4 new tests + notification-bus suite pass (24 passed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the §8 API consolidation: one admin surface drives a BackupPolicy for
ANY target — application, wordpress_site, database, files, server — by
(target_type, target_id):
GET/PUT /api/v1/backups/policies/<type>/<id> (PUT accepts
target_subtype + target_meta to describe database/files targets)
POST /api/v1/backups/policies/<type>/<id>/run
GET /api/v1/backups/policies/<type>/<id>/runs
POST .../runs/<run_id>/restore | /verify
DELETE .../runs/<run_id>
This is the only way to drive the database/files/server targets (which have no
per-resource page). The grant-aware per-resource mounts
(/apps/:id/backup-policy, /wordpress/sites/:id/backup-policy) are unchanged.
Deferred to §7 (WordPress-as-app): surfacing WP DatabaseSnapshot under
/apps/:id/db-snapshots, and the frontend Backups.jsx rewrite, both of which
depend on the app↔site unification.
Verified: 5 new tests; full backup/snapshot suite green (63 passed, 2 skipped).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Recorded §8 decision: a deployment config snapshot is shown as a "Config
Checkpoint" in the UI to end the overloaded "snapshot" wording. Distinct from
data-protection backups and from WordPress DB snapshots, which are untouched.
Backend:
- snapshots blueprint serves every route under /apps/:id/config-checkpoints
in addition to /apps/:id/snapshots (dual @route — both resolve to one view).
Frontend:
- DeployTab section "Deployment Timeline" → "Config Checkpoints" with reworded
hint; ConfigDiffModal + DeploymentTimeline empty/peer copy use "checkpoint".
- api/snapshots.js gains getConfigCheckpoint(s)/Diff/restore aliases.
- ProtectionStatusCard: "Snapshots run on the configured schedule" → "Backups"
(that card describes backups, not config checkpoints).
Verified: 2 alias tests + existing config-snapshot suite pass; frontend build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the cross-target queries the unified Backups page needs:
- GET /api/v1/backups/policies (filters: target_type, enabled) — replaces the
legacy filesystem schedule list with one view of every BackupPolicy.
- GET /api/v1/backups/runs (filters: target_type, status, policy_id, limit≤500)
— every BackupRun across all policies, each augmented with its target's
type/id/subtype so the UI filters without a second lookup.
Both admin-guarded like the rest of /backups.
Verified: 6 new API tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
