mirror of
https://github.com/jhd3197/ServerKit.git
synced 2026-11-09 05:24:49 +09:00
Juan Denis
b1bbd1e67b
Four docker legs exercising detect/status paths against real binaries —
probe-layer smoke, not provisioning: ubuntu root with an sbin-less PATH
(the plan 74 unit shape), ubuntu non-root with sudo (where the sbin bug
cannot reproduce — now proven, not reasoned), Rocky 9 (firewalld
paths), and capability-dropped ubuntu (docker's default cap set drops
NET_ADMIN — the unprivileged-LXC shape, where firewall commands exist,
are on PATH, and still fail).
The matrix pinned a live bug on first run: _check_ufw returned
active=False for a failed status probe in the capability-restricted
leg — "probe failed" indistinguishable from "firewall off" (fixed in
2f002c59).
234 lines
9.3 KiB
Markdown
234 lines
9.3 KiB
Markdown
# ServerKit E2E Test Harness
|
|
|
|
One-click full-stack test that spins up fresh Linux VMs on your Windows box,
|
|
installs ServerKit from your **local working tree** (uncommitted changes
|
|
included), and runs an API harness against the live panel. Aggregates
|
|
everything into a single HTML report.
|
|
|
|
Designed for the "I'm about to merge a huge PR and want to be sure the
|
|
installer actually works on multiple distros" use case.
|
|
|
|
## Prerequisites (one-time)
|
|
|
|
1. **Multipass** — Ubuntu 22.04 + 24.04 VMs
|
|
```powershell
|
|
winget install Canonical.Multipass
|
|
```
|
|
2. **Vagrant** — Debian 12, Fedora, Rocky 9 VMs (uses Hyper-V provider, no VirtualBox)
|
|
```powershell
|
|
winget install Hashicorp.Vagrant
|
|
```
|
|
**Note:** Vagrant + Hyper-V requires running the script from an
|
|
**elevated** PowerShell (right-click → Run as Administrator). If you
|
|
run as a normal user, the script will tell you and exit cleanly.
|
|
Multipass-only runs (Ubuntu) don't need admin.
|
|
3. **Python 3** on PATH (for the HTML report generator; tests run inside the VMs).
|
|
4. **Hyper-V** enabled (both Multipass and Vagrant use it on Windows).
|
|
|
|
You can install only Multipass if Ubuntu coverage is enough — the harness
|
|
auto-detects which backends are available and skips distros it can't run.
|
|
|
|
## Usage
|
|
|
|
From the repo root, in PowerShell:
|
|
|
|
```powershell
|
|
.\scripts\test\full-stack-test.ps1
|
|
```
|
|
|
|
That's it. Goes and makes coffee. ~1-2 hours on first run (cloud images
|
|
download), 15-30 min after that. When it finishes, your browser opens to the
|
|
HTML report.
|
|
|
|
### Options
|
|
|
|
```powershell
|
|
# Full default suite: ubuntu22, ubuntu24, debian12, rocky9
|
|
.\scripts\test\full-stack-test.ps1
|
|
|
|
# Subset
|
|
.\scripts\test\full-stack-test.ps1 -Only "ubuntu24,debian12"
|
|
|
|
# Opt-in extras (fedora40 box is community-pinned; alma9/debian11/opensuse15
|
|
# widen homelab coverage)
|
|
.\scripts\test\full-stack-test.ps1 -Only "fedora,alma9,debian11,opensuse15"
|
|
|
|
# Keep VMs running so you can shell in and poke around
|
|
.\scripts\test\full-stack-test.ps1 -Keep
|
|
|
|
# Bigger VMs (default 2 CPU / 4 GB RAM / 15 GB disk)
|
|
.\scripts\test\full-stack-test.ps1 -Cpus 4 -MemoryGB 8
|
|
|
|
# Re-run harness against an already-installed Multipass VM (fast iteration)
|
|
.\scripts\test\full-stack-test.ps1 -ReuseVm sk-test-ubuntu24-<id>
|
|
```
|
|
|
|
### Distro coverage
|
|
|
|
| Distro | Backend | Box / Image | In default suite? |
|
|
|---|---|---|---|
|
|
| Ubuntu 22.04 | Multipass | `22.04` | ✅ |
|
|
| Ubuntu 24.04 | Multipass | `24.04` | ✅ |
|
|
| Debian 12 | Vagrant | `generic/debian12` | ✅ |
|
|
| Rocky 9 | Vagrant | `generic/rocky9` | ✅ |
|
|
| Fedora 40 | Vagrant | `generic/fedora40` | opt-in (`-Only fedora`) |
|
|
| AlmaLinux 9 | Vagrant | `generic/alma9` | opt-in |
|
|
| Debian 11 | Vagrant | `generic/debian11` | opt-in (legacy homelab) |
|
|
| openSUSE Leap 15 | Vagrant | `generic/opensuse15` | opt-in |
|
|
|
|
### Agent pairing test (optional)
|
|
|
|
After running with `-Keep`:
|
|
|
|
```powershell
|
|
.\scripts\test\agent-test.ps1
|
|
```
|
|
|
|
Exercises the agent <-> panel pairing API end-to-end (enroll → claim).
|
|
|
|
## What it does
|
|
|
|
1. Tar your local repo (excluding `.git`, `node_modules`, venvs, `dist/`).
|
|
2. `multipass launch` three VMs in parallel: Ubuntu 22.04, Ubuntu 24.04, Debian 12.
|
|
3. On each VM, in parallel:
|
|
- Upload the tarball + `vm-install.sh`.
|
|
- Run `install.sh` (real public installer — clones from GitHub, installs
|
|
Python, Node, Docker, nginx, builds frontend, starts systemd unit).
|
|
- Overlay your local working tree on top of `/opt/serverkit`, rebuild
|
|
frontend, restart `serverkit` systemd unit. This is what makes us test
|
|
**your code**, not what's on `main`.
|
|
- Wait for `/api/v1/system/health` to return 200.
|
|
- Push the pytest harness and run it against the live panel.
|
|
- Capture install log + journalctl regardless of outcome.
|
|
4. Generate one self-contained `report.html` with green/red per VM, per test,
|
|
plus full logs inline.
|
|
5. Tear down VMs (unless `-Keep`).
|
|
|
|
## Output
|
|
|
|
```
|
|
scripts/test/output/<run-id>/
|
|
report.html <- open this
|
|
serverkit-src.tar.gz
|
|
sk-test-ubuntu22-<id>/
|
|
install.log
|
|
vm-install.log
|
|
journalctl.log
|
|
install-status ("OK" or "FAIL")
|
|
pytest.log
|
|
pytest-report.json
|
|
sk-test-ubuntu24-<id>/ ...
|
|
sk-test-debian12-<id>/ ...
|
|
```
|
|
|
|
## Update & uninstall coverage
|
|
|
|
Alongside `vm-install.sh`, two more "run inside the VM" scripts exercise the
|
|
other lifecycle stages end-to-end (same contract: log to
|
|
`/var/log/serverkit-test-*.log`, write `OK`/`FAIL` to `/tmp/serverkit-*-status`):
|
|
|
|
- **`vm-update.sh`** — installs ServerKit, runs the local `scripts/update.sh`
|
|
with `--force` (full blue/green deploy → migrate → switch → health), then
|
|
re-runs without `--force` to confirm the "Already up to date" version-gate.
|
|
- **`vm-uninstall.sh`** — installs + seeds sample data, runs the default
|
|
uninstall and asserts user data is preserved (DB snapshot in
|
|
`/var/backups/serverkit`, `/var/lib/serverkit` survives), then reinstalls and
|
|
asserts `--purge` removes the data dirs.
|
|
|
|
Run them by uploading to a `-Keep` VM and `sudo bash /tmp/vm-update.sh`, or wire
|
|
them into the orchestrator the same way `vm-install.sh` is invoked.
|
|
|
|
## Fast, serverless unit tests
|
|
|
|
Most regressions are caught long before a VM is needed. The source-able
|
|
`scripts/test/test_{update,install,lib}.sh` suites exercise the updater,
|
|
installer, and the `scripts/lib/*` abstractions against fixtures and PATH stubs
|
|
— no server, seconds to run, and gated in CI (`.github/workflows/scripts-ci.yml`)
|
|
across Ubuntu 22.04/24.04, Debian 12, Rocky 9, AlmaLinux 9, Fedora 40, and
|
|
openSUSE Leap 15.5 containers.
|
|
|
|
```bash
|
|
bash scripts/test/test_update.sh
|
|
bash scripts/test/test_install.sh
|
|
bash scripts/test/test_lib.sh
|
|
```
|
|
|
|
## Probe environment matrix (plan 75 §D)
|
|
|
|
`scripts/test/probe-matrix.sh` runs `backend/tests/test_probe_env_integration.py`
|
|
— mock-free probe-honesty tests (firewall detect/status paths only, no
|
|
provisioning) — in docker containers shaped like the install environments the
|
|
plan 74 sbin fix could not verify on its single Ubuntu 22.04 root box. One
|
|
line on why each leg matters:
|
|
|
|
- **ubuntu-root-sbinless** — the plan 74 shape itself: pytest runs with
|
|
`PATH=/usr/local/bin:/usr/bin:/bin` (the panel unit's PATH), proving
|
|
detection never reports "not installed" because $PATH lacked sbin.
|
|
- **ubuntu-nonroot-sudo** — non-root install: `_needs_sudo()` is True and
|
|
probes route through sudo's secure_path, so the sbin bug can't reproduce;
|
|
a root-only verification proves nothing about it.
|
|
- **rocky-root** — RHEL family: firewalld instead of ufw, rpm/dnf instead of
|
|
dpkg/apt, usrmerge paths (firewall-cmd lives in /usr/bin).
|
|
- **ubuntu-capdrop** — unprivileged-LXC shape: `--cap-drop=NET_ADMIN`. ufw
|
|
exists and resolves, but `ufw status` fails on permissions — the failure
|
|
must surface as error/unknown, never a silent "not installed" or a
|
|
fabricated "inactive".
|
|
|
|
```bash
|
|
scripts/test/probe-matrix.sh # all four legs, summary table
|
|
scripts/test/probe-matrix.sh rocky # single leg (substring match)
|
|
scripts/test/probe-matrix.sh --list # leg names
|
|
```
|
|
|
|
The repo is mounted read-only; each leg installs python3+pytest+the firewall
|
|
package and exits non-zero if any leg fails. The tests auto-detect their
|
|
environment and skip with explicit reasons when an invariant doesn't hold, so
|
|
the same file also runs cleanly on a dev box (WSL included).
|
|
|
|
## Extending the harness
|
|
|
|
Add new tests under `harness/test_*.py` — they're plain pytest using a
|
|
session-scoped `admin_token` fixture and a `base_url` fixture. The orchestrator
|
|
auto-copies every file in `harness/` to each VM, so just dropping in a new
|
|
`test_05_whatever.py` is enough.
|
|
|
|
Currently covered:
|
|
- `test_01_health.py` — backend health, frontend reachable
|
|
- `test_02_auth.py` — setup-status, register, login, JWT-authed request
|
|
- `test_03_plugins.py` — list plugins (install-from-URL test is `@skipif`'d
|
|
until a stable test plugin repo exists; flip when ready)
|
|
- `test_04_smoke.py` — sample of authed endpoints must not return 5xx
|
|
|
|
## Limitations
|
|
|
|
- Doesn't test bare-metal-only stuff (hardware drivers, real partitioning).
|
|
- Agent ARM64 MSI installer isn't exercised (no ARM hardware).
|
|
- UI smoke (Playwright) is not wired up yet — easy to add as a follow-up if
|
|
the API surface stops catching everything.
|
|
- Fedora isn't in the default distro list (the generic/fedora40 Hyper-V box
|
|
is community-pinned); opt in with `-Only fedora`. alma9 / debian11 /
|
|
opensuse15 are also opt-in extras.
|
|
- ARM (Raspberry Pi etc.) can't be virtualized here — Hyper-V on Windows is
|
|
x86_64 only. ARM64 coverage comes from the CI container matrix on the
|
|
`ubuntu-24.04-arm` runner (see `.github/workflows/scripts-ci.yml`), which
|
|
approximates 64-bit Raspberry Pi OS / Ubuntu ARM homelabs; real-hardware
|
|
validation still needs a physical Pi.
|
|
|
|
## Troubleshooting
|
|
|
|
**"multipass: command not found"** — install Multipass and restart PowerShell.
|
|
|
|
**Launch fails with timeout** — first launch downloads ~600 MB per distro
|
|
image; let it run. If it times out repeatedly, increase Multipass timeout:
|
|
`multipass set local.driver=hyperv`.
|
|
|
|
**Health check never passes** — open `report.html`, expand "Install log" and
|
|
"journalctl" for that VM. Most common: missing system package on a distro the
|
|
installer doesn't handle, or `frontend build` OOM (bump `-MemoryGB`).
|
|
|
|
**Want to debug a failing VM** — re-run with `-Keep`, then:
|
|
```powershell
|
|
multipass shell sk-test-ubuntu24-<id>
|
|
sudo journalctl -u serverkit -f
|
|
```
|