# ServerKit E2E Test Harness

One-click full-stack test that spins up fresh Linux VMs on your Windows box,
installs ServerKit from your **local working tree** (uncommitted changes
included), and runs an API harness against the live panel. Aggregates
everything into a single HTML report.

Designed for the "I'm about to merge a huge PR and want to be sure the
installer actually works on multiple distros" use case.

## Prerequisites (one-time)

1. **Multipass** — Ubuntu 22.04 + 24.04 VMs
   ```powershell
   winget install Canonical.Multipass
   ```
2. **Vagrant** — Debian 12, Fedora, Rocky 9 VMs (uses Hyper-V provider, no VirtualBox)
   ```powershell
   winget install Hashicorp.Vagrant
   ```
   **Note:** Vagrant + Hyper-V requires running the script from an
   **elevated** PowerShell (right-click → Run as Administrator). If you
   run as a normal user, the script will tell you and exit cleanly.
   Multipass-only runs (Ubuntu) don't need admin.
3. **Python 3** on PATH (for the HTML report generator; tests run inside the VMs).
4. **Hyper-V** enabled (both Multipass and Vagrant use it on Windows).

You can install only Multipass if Ubuntu coverage is enough — the harness
auto-detects which backends are available and skips distros it can't run.

## Usage

From the repo root, in PowerShell:

```powershell
.\scripts\test\full-stack-test.ps1
```

That's it. Goes and makes coffee. ~1-2 hours on first run (cloud images
download), 15-30 min after that. When it finishes, your browser opens to the
HTML report.

### Options

```powershell
# Full default suite: ubuntu22, ubuntu24, debian12, rocky9
.\scripts\test\full-stack-test.ps1

# Subset
.\scripts\test\full-stack-test.ps1 -Only "ubuntu24,debian12"

# Opt-in extras (fedora40 box is community-pinned; alma9/debian11/opensuse15
# widen homelab coverage)
.\scripts\test\full-stack-test.ps1 -Only "fedora,alma9,debian11,opensuse15"

# Keep VMs running so you can shell in and poke around
.\scripts\test\full-stack-test.ps1 -Keep

# Bigger VMs (default 2 CPU / 4 GB RAM / 15 GB disk)
.\scripts\test\full-stack-test.ps1 -Cpus 4 -MemoryGB 8

# Re-run harness against an already-installed Multipass VM (fast iteration)
.\scripts\test\full-stack-test.ps1 -ReuseVm sk-test-ubuntu24-<id>
```

### Distro coverage

| Distro | Backend | Box / Image | In default suite? |
|---|---|---|---|
| Ubuntu 22.04 | Multipass | `22.04` | ✅ |
| Ubuntu 24.04 | Multipass | `24.04` | ✅ |
| Debian 12    | Vagrant   | `generic/debian12` | ✅ |
| Rocky 9      | Vagrant   | `generic/rocky9` | ✅ |
| Fedora 40    | Vagrant   | `generic/fedora40` | opt-in (`-Only fedora`) |
| AlmaLinux 9  | Vagrant   | `generic/alma9` | opt-in |
| Debian 11    | Vagrant   | `generic/debian11` | opt-in (legacy homelab) |
| openSUSE Leap 15 | Vagrant | `generic/opensuse15` | opt-in |

### Agent pairing test (optional)

After running with `-Keep`:

```powershell
.\scripts\test\agent-test.ps1
```

Exercises the agent <-> panel pairing API end-to-end (enroll → claim).

## What it does

1. Tar your local repo (excluding `.git`, `node_modules`, venvs, `dist/`).
2. `multipass launch` three VMs in parallel: Ubuntu 22.04, Ubuntu 24.04, Debian 12.
3. On each VM, in parallel:
   - Upload the tarball + `vm-install.sh`.
   - Run `install.sh` (real public installer — clones from GitHub, installs
     Python, Node, Docker, nginx, builds frontend, starts systemd unit).
   - Overlay your local working tree on top of `/opt/serverkit`, rebuild
     frontend, restart `serverkit` systemd unit. This is what makes us test
     **your code**, not what's on `main`.
   - Wait for `/api/v1/system/health` to return 200.
   - Push the pytest harness and run it against the live panel.
   - Capture install log + journalctl regardless of outcome.
4. Generate one self-contained `report.html` with green/red per VM, per test,
   plus full logs inline.
5. Tear down VMs (unless `-Keep`).

## Output

```
scripts/test/output/<run-id>/
  report.html                     <- open this
  serverkit-src.tar.gz
  sk-test-ubuntu22-<id>/
    install.log
    vm-install.log
    journalctl.log
    install-status                ("OK" or "FAIL")
    pytest.log
    pytest-report.json
  sk-test-ubuntu24-<id>/ ...
  sk-test-debian12-<id>/ ...
```

## Update & uninstall coverage

Alongside `vm-install.sh`, two more "run inside the VM" scripts exercise the
other lifecycle stages end-to-end (same contract: log to
`/var/log/serverkit-test-*.log`, write `OK`/`FAIL` to `/tmp/serverkit-*-status`):

- **`vm-update.sh`** — installs ServerKit, runs the local `scripts/update.sh`
  with `--force` (full blue/green deploy → migrate → switch → health), then
  re-runs without `--force` to confirm the "Already up to date" version-gate.
- **`vm-uninstall.sh`** — installs + seeds sample data, runs the default
  uninstall and asserts user data is preserved (DB snapshot in
  `/var/backups/serverkit`, `/var/lib/serverkit` survives), then reinstalls and
  asserts `--purge` removes the data dirs.

Run them by uploading to a `-Keep` VM and `sudo bash /tmp/vm-update.sh`, or wire
them into the orchestrator the same way `vm-install.sh` is invoked.

## Fast, serverless unit tests

Most regressions are caught long before a VM is needed. The source-able
`scripts/test/test_{update,install,lib}.sh` suites exercise the updater,
installer, and the `scripts/lib/*` abstractions against fixtures and PATH stubs
— no server, seconds to run, and gated in CI (`.github/workflows/scripts-ci.yml`)
across Ubuntu 22.04/24.04, Debian 12, Rocky 9, AlmaLinux 9, Fedora 40, and
openSUSE Leap 15.5 containers.

```bash
bash scripts/test/test_update.sh
bash scripts/test/test_install.sh
bash scripts/test/test_lib.sh
```

## Probe environment matrix (plan 75 §D)

`scripts/test/probe-matrix.sh` runs `backend/tests/test_probe_env_integration.py`
— mock-free probe-honesty tests (firewall detect/status paths only, no
provisioning) — in docker containers shaped like the install environments the
plan 74 sbin fix could not verify on its single Ubuntu 22.04 root box. One
line on why each leg matters:

- **ubuntu-root-sbinless** — the plan 74 shape itself: pytest runs with
  `PATH=/usr/local/bin:/usr/bin:/bin` (the panel unit's PATH), proving
  detection never reports "not installed" because $PATH lacked sbin.
- **ubuntu-nonroot-sudo** — non-root install: `_needs_sudo()` is True and
  probes route through sudo's secure_path, so the sbin bug can't reproduce;
  a root-only verification proves nothing about it.
- **rocky-root** — RHEL family: firewalld instead of ufw, rpm/dnf instead of
  dpkg/apt, usrmerge paths (firewall-cmd lives in /usr/bin).
- **ubuntu-capdrop** — unprivileged-LXC shape: `--cap-drop=NET_ADMIN`. ufw
  exists and resolves, but `ufw status` fails on permissions — the failure
  must surface as error/unknown, never a silent "not installed" or a
  fabricated "inactive".

```bash
scripts/test/probe-matrix.sh            # all four legs, summary table
scripts/test/probe-matrix.sh rocky      # single leg (substring match)
scripts/test/probe-matrix.sh --list     # leg names
```

The repo is mounted read-only; each leg installs python3+pytest+the firewall
package and exits non-zero if any leg fails. The tests auto-detect their
environment and skip with explicit reasons when an invariant doesn't hold, so
the same file also runs cleanly on a dev box (WSL included).

## Extending the harness

Add new tests under `harness/test_*.py` — they're plain pytest using a
session-scoped `admin_token` fixture and a `base_url` fixture. The orchestrator
auto-copies every file in `harness/` to each VM, so just dropping in a new
`test_05_whatever.py` is enough.

Currently covered:
- `test_01_health.py` — backend health, frontend reachable
- `test_02_auth.py` — setup-status, register, login, JWT-authed request
- `test_03_plugins.py` — list plugins (install-from-URL test is `@skipif`'d
  until a stable test plugin repo exists; flip when ready)
- `test_04_smoke.py` — sample of authed endpoints must not return 5xx

## Limitations

- Doesn't test bare-metal-only stuff (hardware drivers, real partitioning).
- Agent ARM64 MSI installer isn't exercised (no ARM hardware).
- UI smoke (Playwright) is not wired up yet — easy to add as a follow-up if
  the API surface stops catching everything.
- Fedora isn't in the default distro list (the generic/fedora40 Hyper-V box
  is community-pinned); opt in with `-Only fedora`. alma9 / debian11 /
  opensuse15 are also opt-in extras.
- ARM (Raspberry Pi etc.) can't be virtualized here — Hyper-V on Windows is
  x86_64 only. ARM64 coverage comes from the CI container matrix on the
  `ubuntu-24.04-arm` runner (see `.github/workflows/scripts-ci.yml`), which
  approximates 64-bit Raspberry Pi OS / Ubuntu ARM homelabs; real-hardware
  validation still needs a physical Pi.

## Troubleshooting

**"multipass: command not found"** — install Multipass and restart PowerShell.

**Launch fails with timeout** — first launch downloads ~600 MB per distro
image; let it run. If it times out repeatedly, increase Multipass timeout:
`multipass set local.driver=hyperv`.

**Health check never passes** — open `report.html`, expand "Install log" and
"journalctl" for that VM. Most common: missing system package on a distro the
installer doesn't handle, or `frontend build` OOM (bump `-MemoryGB`).

**Want to debug a failing VM** — re-run with `-Keep`, then:
```powershell
multipass shell sk-test-ubuntu24-<id>
sudo journalctl -u serverkit -f
```