ServerKit/scripts/test

ServerKit E2E Test Harness

One-click full-stack test that spins up fresh Linux VMs on your Windows box, installs ServerKit from your local working tree (uncommitted changes included), and runs an API harness against the live panel. Aggregates everything into a single HTML report.

Designed for the "I'm about to merge a huge PR and want to be sure the installer actually works on multiple distros" use case.

Prerequisites (one-time)

1. Multipass — Ubuntu 22.04 + 24.04 VMs

   winget install Canonical.Multipass
   

2. Vagrant — Debian 12, Fedora, Rocky 9 VMs (uses Hyper-V provider, no VirtualBox)

   winget install Hashicorp.Vagrant
   

   Note: Vagrant + Hyper-V requires running the script from an elevated PowerShell (right-click → Run as Administrator). If you run as a normal user, the script will tell you and exit cleanly. Multipass-only runs (Ubuntu) don't need admin.
3. Python 3 on PATH (for the HTML report generator; tests run inside the VMs).
4. Hyper-V enabled (both Multipass and Vagrant use it on Windows).

You can install only Multipass if Ubuntu coverage is enough — the harness auto-detects which backends are available and skips distros it can't run.

Usage

From the repo root, in PowerShell:

.\scripts\test\full-stack-test.ps1

That's it. Goes and makes coffee. ~1-2 hours on first run (cloud images download), 15-30 min after that. When it finishes, your browser opens to the HTML report.

Options

# Full default suite: ubuntu22, ubuntu24, debian12, rocky9
.\scripts\test\full-stack-test.ps1

# Subset
.\scripts\test\full-stack-test.ps1 -Only "ubuntu24,debian12"

# Opt-in extras (fedora40 box is community-pinned; alma9/debian11/opensuse15
# widen homelab coverage)
.\scripts\test\full-stack-test.ps1 -Only "fedora,alma9,debian11,opensuse15"

# Keep VMs running so you can shell in and poke around
.\scripts\test\full-stack-test.ps1 -Keep

# Bigger VMs (default 2 CPU / 4 GB RAM / 15 GB disk)
.\scripts\test\full-stack-test.ps1 -Cpus 4 -MemoryGB 8

# Re-run harness against an already-installed Multipass VM (fast iteration)
.\scripts\test\full-stack-test.ps1 -ReuseVm sk-test-ubuntu24-<id>

Distro coverage

Distro	Backend	Box / Image	In default suite?
Ubuntu 22.04	Multipass	22.04	✅
Ubuntu 24.04	Multipass	24.04	✅
Debian 12	Vagrant	generic/debian12	✅
Rocky 9	Vagrant	generic/rocky9	✅
Fedora 40	Vagrant	generic/fedora40	opt-in (-Only fedora)
AlmaLinux 9	Vagrant	generic/alma9	opt-in
Debian 11	Vagrant	generic/debian11	opt-in (legacy homelab)
openSUSE Leap 15	Vagrant	generic/opensuse15	opt-in

Agent pairing test (optional)

After running with -Keep:

.\scripts\test\agent-test.ps1

Exercises the agent <-> panel pairing API end-to-end (enroll → claim).

What it does

1. Tar your local repo (excluding .git, node_modules, venvs, dist/).
2. multipass launch three VMs in parallel: Ubuntu 22.04, Ubuntu 24.04, Debian 12.
3. On each VM, in parallel:
   • Upload the tarball + vm-install.sh.
   • Run install.sh (real public installer — clones from GitHub, installs Python, Node, Docker, nginx, builds frontend, starts systemd unit).
   • Overlay your local working tree on top of /opt/serverkit, rebuild frontend, restart serverkit systemd unit. This is what makes us test your code, not what's on main.
   • Wait for /api/v1/system/health to return 200.
   • Push the pytest harness and run it against the live panel.
   • Capture install log + journalctl regardless of outcome.
4. Generate one self-contained report.html with green/red per VM, per test, plus full logs inline.
5. Tear down VMs (unless -Keep).

Output

scripts/test/output/<run-id>/
  report.html                     <- open this
  serverkit-src.tar.gz
  sk-test-ubuntu22-<id>/
    install.log
    vm-install.log
    journalctl.log
    install-status                ("OK" or "FAIL")
    pytest.log
    pytest-report.json
  sk-test-ubuntu24-<id>/ ...
  sk-test-debian12-<id>/ ...

Update & uninstall coverage

Alongside vm-install.sh, two more "run inside the VM" scripts exercise the other lifecycle stages end-to-end (same contract: log to /var/log/serverkit-test-*.log, write OK/FAIL to /tmp/serverkit-*-status):

• vm-update.sh — installs ServerKit, runs the local scripts/update.sh with --force (full blue/green deploy → migrate → switch → health), then re-runs without --force to confirm the "Already up to date" version-gate.
• vm-uninstall.sh — installs + seeds sample data, runs the default uninstall and asserts user data is preserved (DB snapshot in /var/backups/serverkit, /var/lib/serverkit survives), then reinstalls and asserts --purge removes the data dirs.

Run them by uploading to a -Keep VM and sudo bash /tmp/vm-update.sh, or wire them into the orchestrator the same way vm-install.sh is invoked.

Fast, serverless unit tests

Most regressions are caught long before a VM is needed. The source-able scripts/test/test_{update,install,lib}.sh suites exercise the updater, installer, and the scripts/lib/* abstractions against fixtures and PATH stubs — no server, seconds to run, and gated in CI (.github/workflows/scripts-ci.yml) across Ubuntu 22.04/24.04, Debian 12, Rocky 9, AlmaLinux 9, Fedora 40, and openSUSE Leap 15.5 containers.

bash scripts/test/test_update.sh
bash scripts/test/test_install.sh
bash scripts/test/test_lib.sh

Extending the harness

Add new tests under harness/test_*.py — they're plain pytest using a session-scoped admin_token fixture and a base_url fixture. The orchestrator auto-copies every file in harness/ to each VM, so just dropping in a new test_05_whatever.py is enough.

Currently covered:

• test_01_health.py — backend health, frontend reachable
• test_02_auth.py — setup-status, register, login, JWT-authed request
• test_03_plugins.py — list plugins (install-from-URL test is @skipif'd until a stable test plugin repo exists; flip when ready)
• test_04_smoke.py — sample of authed endpoints must not return 5xx

Limitations

• Doesn't test bare-metal-only stuff (hardware drivers, real partitioning).
• Agent ARM64 MSI installer isn't exercised (no ARM hardware).
• UI smoke (Playwright) is not wired up yet — easy to add as a follow-up if the API surface stops catching everything.
• Fedora isn't in the default distro list (the generic/fedora40 Hyper-V box is community-pinned); opt in with -Only fedora. alma9 / debian11 / opensuse15 are also opt-in extras.
• ARM (Raspberry Pi etc.) can't be virtualized here — Hyper-V on Windows is x86_64 only. ARM64 coverage comes from the CI container matrix on the ubuntu-24.04-arm runner (see .github/workflows/scripts-ci.yml), which approximates 64-bit Raspberry Pi OS / Ubuntu ARM homelabs; real-hardware validation still needs a physical Pi.

Troubleshooting

"multipass: command not found" — install Multipass and restart PowerShell.

Launch fails with timeout — first launch downloads ~600 MB per distro image; let it run. If it times out repeatedly, increase Multipass timeout: multipass set local.driver=hyperv.

Health check never passes — open report.html, expand "Install log" and "journalctl" for that VM. Most common: missing system package on a distro the installer doesn't handle, or frontend build OOM (bump -MemoryGB).

Want to debug a failing VM — re-run with -Keep, then:

multipass shell sk-test-ubuntu24-<id>
sudo journalctl -u serverkit -f