Juan DenisandClaude Opus 5
3c30d2f537
Plan 73 item 8. The installer opened 80/443 on whatever firewall it found and shrugged -- "assuming ports 80/443 are already open" -- when there wasn't one, which is the state of a stock Ubuntu/Debian VPS. That box then ran ServerKit with every port a deploy maps onto the host (8001-8999, plus database containers on their stock ports) facing the internet. fail2ban only shipped with the "full" capacity profile, so the typical small VPS got neither. The dangerous half is enabling default-deny without allowing the port sshd actually answers on, over the SSH session running the installer. detect_ssh_ports therefore never guesses: it asks `sshd -T` (the effective config, which resolves the Include that Ubuntu 22.04+ ships), then sshd_config plus its drop-ins with comments stripped, then a live `ss`. Multiple Port lines all come back. 22 is used only when a config exists and names no port -- sshd's compiled-in default -- and if nothing answers at all the function fails and no firewall is installed. An exposed box is recoverable; a locked-out one usually is not. That interlock had a real bug in its first draft, caught before it shipped and now pinned by a named regression test: most boxes have no sshd_config.d/, the unmatched glob makes cat exit non-zero, and under `set -o pipefail` that wiped a correctly parsed answer and fell through to the 22 default -- returning 22 on a box whose sshd is on 2222, which is precisely the lockout this function exists to prevent. Ordering is the other safety property and is asserted: every allow rule (all detected SSH ports, 80, 443, and PANEL_PORT when non-standard) lands before `default deny incoming`, and the policy before `--force enable`. The bootstrap refuses to act inside a container, when a firewall is already active (it is the operator's -- opening ports on it is fine, rewriting its default policy is not), under SERVERKIT_SKIP_FIREWALL=1, and when a previous run bootstrapped but the firewall is off now, since the operator turned it off deliberately. Non-Debian installs are byte-identical to before: RHEL-family boots with firewalld already active, and a firewalld default-deny bootstrap for boxes where it is inactive is marked as an explicit follow-up rather than guessed at. fail2ban moves to the standard profile with an sshd jail that watches the same ports detect_ssh_ports found -- a jail pinned to :22 on a box listening on :2222 bans nobody -- and declares `backend = systemd` when there is no auth.log to tail, without which the jail refuses to start and fail2ban looks installed while protecting nothing. certbot stays full-only. setup-health gains firewall (critical) and fail2ban (recommended) items. The split is deliberate: this file grades critical by silent breakage, and an unfirewalled box qualifies -- the operator set it in motion by deploying, nothing in the UI says the port is public, and the failure mode is compromise rather than an error. A missing fail2ban thins defence in depth but breaks nothing silently, since the panel already throttles logins per IP. Both are gated on actually owning the host -- Linux, /etc/serverkit present, not containerised -- so a dev checkout or CI runner drops them entirely instead of reporting a fake ok that would inflate the score. SetupHealthWidget was orphaned; so, it turns out, was its stylesheet, which main.scss never imported. Both are wired up, as a plain admin-only card above the board rather than a placeable widget: an unfirewalled box has to be visible without the operator first opting into showing it. It collapses to one line when everything is clean. Implementing this inverts a shipped test that asserted provision_hardening was a no-op on the standard profile; it is rewritten to the new contract rather than deleted, with a minimal-profile case added alongside. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ServerKit E2E Test Harness
One-click full-stack test that spins up fresh Linux VMs on your Windows box, installs ServerKit from your local working tree (uncommitted changes included), and runs an API harness against the live panel. Aggregates everything into a single HTML report.
Designed for the "I'm about to merge a huge PR and want to be sure the installer actually works on multiple distros" use case.
Prerequisites (one-time)
- Multipass — Ubuntu 22.04 + 24.04 VMs
winget install Canonical.Multipass - Vagrant — Debian 12, Fedora, Rocky 9 VMs (uses Hyper-V provider, no VirtualBox)
Note: Vagrant + Hyper-V requires running the script from an elevated PowerShell (right-click → Run as Administrator). If you run as a normal user, the script will tell you and exit cleanly. Multipass-only runs (Ubuntu) don't need admin.
winget install Hashicorp.Vagrant - Python 3 on PATH (for the HTML report generator; tests run inside the VMs).
- Hyper-V enabled (both Multipass and Vagrant use it on Windows).
You can install only Multipass if Ubuntu coverage is enough — the harness auto-detects which backends are available and skips distros it can't run.
Usage
From the repo root, in PowerShell:
.\scripts\test\full-stack-test.ps1
That's it. Goes and makes coffee. ~1-2 hours on first run (cloud images download), 15-30 min after that. When it finishes, your browser opens to the HTML report.
Options
# Full default suite: ubuntu22, ubuntu24, debian12, rocky9
.\scripts\test\full-stack-test.ps1
# Subset
.\scripts\test\full-stack-test.ps1 -Only "ubuntu24,debian12"
# Opt-in extras (fedora40 box is community-pinned; alma9/debian11/opensuse15
# widen homelab coverage)
.\scripts\test\full-stack-test.ps1 -Only "fedora,alma9,debian11,opensuse15"
# Keep VMs running so you can shell in and poke around
.\scripts\test\full-stack-test.ps1 -Keep
# Bigger VMs (default 2 CPU / 4 GB RAM / 15 GB disk)
.\scripts\test\full-stack-test.ps1 -Cpus 4 -MemoryGB 8
# Re-run harness against an already-installed Multipass VM (fast iteration)
.\scripts\test\full-stack-test.ps1 -ReuseVm sk-test-ubuntu24-<id>
Distro coverage
| Distro | Backend | Box / Image | In default suite? |
|---|---|---|---|
| Ubuntu 22.04 | Multipass | 22.04 |
✅ |
| Ubuntu 24.04 | Multipass | 24.04 |
✅ |
| Debian 12 | Vagrant | generic/debian12 |
✅ |
| Rocky 9 | Vagrant | generic/rocky9 |
✅ |
| Fedora 40 | Vagrant | generic/fedora40 |
opt-in (-Only fedora) |
| AlmaLinux 9 | Vagrant | generic/alma9 |
opt-in |
| Debian 11 | Vagrant | generic/debian11 |
opt-in (legacy homelab) |
| openSUSE Leap 15 | Vagrant | generic/opensuse15 |
opt-in |
Agent pairing test (optional)
After running with -Keep:
.\scripts\test\agent-test.ps1
Exercises the agent <-> panel pairing API end-to-end (enroll → claim).
What it does
- Tar your local repo (excluding
.git,node_modules, venvs,dist/). multipass launchthree VMs in parallel: Ubuntu 22.04, Ubuntu 24.04, Debian 12.- On each VM, in parallel:
- Upload the tarball +
vm-install.sh. - Run
install.sh(real public installer — clones from GitHub, installs Python, Node, Docker, nginx, builds frontend, starts systemd unit). - Overlay your local working tree on top of
/opt/serverkit, rebuild frontend, restartserverkitsystemd unit. This is what makes us test your code, not what's onmain. - Wait for
/api/v1/system/healthto return 200. - Push the pytest harness and run it against the live panel.
- Capture install log + journalctl regardless of outcome.
- Upload the tarball +
- Generate one self-contained
report.htmlwith green/red per VM, per test, plus full logs inline. - Tear down VMs (unless
-Keep).
Output
scripts/test/output/<run-id>/
report.html <- open this
serverkit-src.tar.gz
sk-test-ubuntu22-<id>/
install.log
vm-install.log
journalctl.log
install-status ("OK" or "FAIL")
pytest.log
pytest-report.json
sk-test-ubuntu24-<id>/ ...
sk-test-debian12-<id>/ ...
Update & uninstall coverage
Alongside vm-install.sh, two more "run inside the VM" scripts exercise the
other lifecycle stages end-to-end (same contract: log to
/var/log/serverkit-test-*.log, write OK/FAIL to /tmp/serverkit-*-status):
vm-update.sh— installs ServerKit, runs the localscripts/update.shwith--force(full blue/green deploy → migrate → switch → health), then re-runs without--forceto confirm the "Already up to date" version-gate.vm-uninstall.sh— installs + seeds sample data, runs the default uninstall and asserts user data is preserved (DB snapshot in/var/backups/serverkit,/var/lib/serverkitsurvives), then reinstalls and asserts--purgeremoves the data dirs.
Run them by uploading to a -Keep VM and sudo bash /tmp/vm-update.sh, or wire
them into the orchestrator the same way vm-install.sh is invoked.
Fast, serverless unit tests
Most regressions are caught long before a VM is needed. The source-able
scripts/test/test_{update,install,lib}.sh suites exercise the updater,
installer, and the scripts/lib/* abstractions against fixtures and PATH stubs
— no server, seconds to run, and gated in CI (.github/workflows/scripts-ci.yml)
across Ubuntu 22.04/24.04, Debian 12, Rocky 9, AlmaLinux 9, Fedora 40, and
openSUSE Leap 15.5 containers.
bash scripts/test/test_update.sh
bash scripts/test/test_install.sh
bash scripts/test/test_lib.sh
Extending the harness
Add new tests under harness/test_*.py — they're plain pytest using a
session-scoped admin_token fixture and a base_url fixture. The orchestrator
auto-copies every file in harness/ to each VM, so just dropping in a new
test_05_whatever.py is enough.
Currently covered:
test_01_health.py— backend health, frontend reachabletest_02_auth.py— setup-status, register, login, JWT-authed requesttest_03_plugins.py— list plugins (install-from-URL test is@skipif'd until a stable test plugin repo exists; flip when ready)test_04_smoke.py— sample of authed endpoints must not return 5xx
Limitations
- Doesn't test bare-metal-only stuff (hardware drivers, real partitioning).
- Agent ARM64 MSI installer isn't exercised (no ARM hardware).
- UI smoke (Playwright) is not wired up yet — easy to add as a follow-up if the API surface stops catching everything.
- Fedora isn't in the default distro list (the generic/fedora40 Hyper-V box
is community-pinned); opt in with
-Only fedora. alma9 / debian11 / opensuse15 are also opt-in extras. - ARM (Raspberry Pi etc.) can't be virtualized here — Hyper-V on Windows is
x86_64 only. ARM64 coverage comes from the CI container matrix on the
ubuntu-24.04-armrunner (see.github/workflows/scripts-ci.yml), which approximates 64-bit Raspberry Pi OS / Ubuntu ARM homelabs; real-hardware validation still needs a physical Pi.
Troubleshooting
"multipass: command not found" — install Multipass and restart PowerShell.
Launch fails with timeout — first launch downloads ~600 MB per distro
image; let it run. If it times out repeatedly, increase Multipass timeout:
multipass set local.driver=hyperv.
Health check never passes — open report.html, expand "Install log" and
"journalctl" for that VM. Most common: missing system package on a distro the
installer doesn't handle, or frontend build OOM (bump -MemoryGB).
Want to debug a failing VM — re-run with -Keep, then:
multipass shell sk-test-ubuntu24-<id>
sudo journalctl -u serverkit -f