Full VM installs and Test Sandbox full mode now record a short
post-install runtime observation (60s settle + 120s sampling) with an
explicit measured/unavailable/failed/interrupted status that never
rewrites install success. Adds the standalone scripts/measure-runtime.py
sampler, the measure-installed-runtime.py harness wrapper, E2E report
rendering, results-API summaries, CI coverage and METRICS.md accounting
boundaries.
Seven distro images were green throughout issue #99 because every one of them
runs the unit suites against STUBS. That is the right trade for logic, but it
left the one thing that talks to real package repos completely untested — so a
new Debian release dropping the Python we ask for could only ever be found by a
user.
scripts/test/test_provision_python.sh runs identify_system, choose_pkg_manager
and provision_python for real, then proves the result is usable: in range, can
create a virtualenv, has working pip, and imports ssl/sqlite3/ctypes. It needs
no systemd, nginx or ports, so it fits the plain containers the matrix already
pulls. Reaching the source build counts as a FAILURE, not a pass: a distro that
gets there is one we have not really added support for.
Wired up as a new `python-provisioning` job over the same images, and debian:13
added to both matrices. Adding trixie to the existing matrix alone would have
been decorative — those legs would have gone green on a distro that cannot
install.
Proven in both directions before being relied on: against the pre-fix tree it
reproduced #99 exactly on debian:13 (the reporter's apt errors, the fall
through to a source build, and the abort message), while debian:12 passed 7/7.
Also fixed the suite pinning the bug. T2 asserted `ver_in_range 3.13` must be
FALSE and T1's stub used 3.13 as its "too new" case, so the gate could not be
widened without the tests failing. Both moved to 3.14, plus a direct guard on
PYTHON_MAX — otherwise a regression to 3.12 could be "fixed" by editing the
assertion to match. 122 passed, 0 failed.
full-stack-test.ps1 documents why debian13 is NOT in the VM harness: no trixie
box publishes a Hyper-V provider (generic/debian13 does not exist,
debian/trixie64 is libvirt-only, bento/debian-13 has no hyperv) and Multipass
is Ubuntu-only. Named there with what remains uncovered on trixie — the full
systemd + nginx + API end-to-end — rather than left as a silent gap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Introduces a full-stack Test Sandbox feature to run distro matrix tests in Docker from the UI: new backend API/service, sandbox_runs model + migration, run/log endpoints, and a new frontend page with mode selection, run history, live polling, cancel, and per-distro logs.
Also adds ARM64 scripts CI coverage (ubuntu-24.04-arm) and strengthens install/test automation for minimal images and overlay rebuilds: early curl/bootstrap index refresh, stricter venv validation, release-venv glibc fallback, Node reprovisioning on source fallback, rsync --delete overlay behavior, npm bootstrap in VM install, Windows tar pinning in PowerShell harness, plus updated test docs and distro coverage notes.
Commit 6e32519 (the July 5-7 recovery restore) wrote git-reflog garbage
over scripts/install.sh (the agent installer served at
GET /api/v1/servers/install.sh) and whitespace-mangled content over
scripts/test/full-stack-test.ps1, breaking scripts-ci: shellcheck SC2148
(no shebang) and 15/18 test_agent_install.sh failures. Restore both files
verbatim from 6e32519~1, the last good revision.
Verified: test_agent_install.sh 18/18, test_install.sh 62 passed,
shellcheck --severity=error clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Vagrantfile: don't `dnf install rsync/curl` on RHEL boxes — it upgrades
openssl-libs and breaks the box's sshd (same ABI issue as install.sh).
rsync/curl ship preinstalled on generic/rocky9; fail loudly if a future
box lacks them rather than silently corrupting the run.
- full-stack-test.ps1: drop fedora from the default distro set (Vagrant
Cloud's generic/fedora is EOL at f39; opt in via -Only fedora).
- vm-install.sh (new): run install.sh from uploaded local source, overlay
the working tree, rebuild, and probe health — so the harness tests the
exact local code (incl. uncommitted) with sshd/firewall snapshots on every
exit path for post-mortems. Recovered from a local stash.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
install.sh:
- Detect rocky/almalinux/rhel/centos as OS_FAMILY=rhel.
- The Fedora system-deps block now also runs for rhel; auto-install
epel-release on RHEL-family.
- Docker install on Fedora/RHEL no longer pipes get.docker.com. That
script recently added docker-model-plugin to its package list, which
doesn't exist in docker-ce repos and aborts the install on Fedora 39.
Replace it with explicit `dnf config-manager --add-repo` +
`dnf install docker-ce docker-ce-cli containerd.io
docker-compose-plugin docker-buildx-plugin`.
- Node.js and nginx installs route through dnf for rhel too.
- SELinux setsebool runs for rhel too.
scripts/test/full-stack-test.ps1:
- fedora image bumped from fedora39 (EOL Nov 2024) to fedora40 (current).
- Test-VagrantInstall + Test-VmInstall: defensive read of status file.
Previously `(Get-Content -Raw).Trim()` crashed with InvokeMethodOnNull
when the status file was missing/empty (which happens when the VM's
install script crashed before producing it). Now falls back to 'FAIL'
cleanly so the run summary still works.
Found by full 5-distro suite: Ubuntu 22/24 + Debian 12 passed; Fedora 39
failed on docker-model-plugin; Rocky 9 failed silently due to apt-vs-dnf
+ status NPE. All three fixed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous commit added Tee-Object so vagrant up output was visible
in the terminal, but Tee-Object's pipeline output then leaked into
Test-VagrantInstall's return stream — the function returned vagrant's
entire log instead of "OK", so the final summary reported every passing
Vagrant VM as a failure.
Append `| Out-Host` after Tee-Object to consume the pipeline. The
operator still sees the live output; the function's return stays clean.
The previous Debian 12 run was actually all-green (10/10 pytest passed,
0 plugin-loader errors) — only the summary line was wrong.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously the orchestrator redirected vagrant up's stdout+stderr to a
log file, which meant the box download progress was invisible and any
interactive prompts (like "choose a Hyper-V switch") were silently
swallowed while vagrant blocked on stdin. The operator had no signal
the run was hung.
Use Tee-Object so the operator sees download progress + prompts live
in the terminal, while the log file still gets a copy for later
inspection.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Vagrant + Hyper-V requires the process itself to run elevated; being in
the Hyper-V Administrators group is necessary but not sufficient. Without
this guard, the script would launch, pack the source, then hit a wall on
'vagrant up' with a permission error mid-run.
Now it checks early: if any requested distro uses the Vagrant backend
and the script isn't running elevated, print exact instructions and exit.
Multipass-only runs (Ubuntu) still work without admin.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Multipass on Windows only ships Ubuntu images. To cover Debian 12,
Fedora, and Rocky 9 too, add a parallel Vagrant runner that uses the
Hyper-V provider (no VirtualBox conflict with Multipass).
scripts/test/
- vagrant/Vagrantfile: parameterized via SK_BOX, SK_VM_NAME env vars;
Hyper-V provider with linked clones; bootstraps rsync + curl.
- full-stack-test.ps1:
- DistroMap replaces ImageMap; each entry tags backend (multipass/
vagrant), image/box name, login user.
- Default Distros now: ubuntu22, ubuntu24, debian12, fedora, rocky9.
- Backend auto-detection: if Vagrant isn't installed, vagrant distros
silently skip rather than failing. Same for Multipass.
- Test-VagrantInstall mirrors Test-VmInstall using `vagrant upload` /
`vagrant ssh -c`, per-VM VAGRANT_CWD under OutDir for state.
- Teardown dispatches by backend (multipass delete vs vagrant destroy).
- README updated with the new prereqs and distro coverage table.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two fixes for the race where the browser opens report.html before any
VM has registered:
- Orchestrator seeds state.json with {"running":true,"vms":{}} the
instant OutDir is created, so a refresh before the first VM is added
still has a "running" signal.
- report.py treats missing state.json as RUNNING (not FAIL), and a VM
with a non-done stage but no install-status file as RUNNING in its
per-VM badge.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
backend: move load_all_plugins(app) inside the with app.app_context()
block, after MigrationService.check_and_prepare(app). The loader queries
the installed_plugins table; previously it ran before migrations created
that table, causing "no such table: installed_plugins" errors in journal
on every cold start. Surfaced by the new E2E harness (visible in the
journalctl pane of the report).
scripts/test: live HTML report.
- Orchestrator writes state.json after each phase + regenerates
report.html, so opening the page shows real-time progress.
- Browser auto-opens the report when the run starts; page auto-refreshes
every 8s while a run is in progress.
- Added per-VM 'Stage' field (pending -> transferring source ->
installing -> running pytest -> done).
- Fixed empty "Details" cells rendering as black boxes; now show "-".
- Pulsing dot indicator in the live banner.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes a cascade of PowerShell + Multipass quirks found during the first
real run:
- Rewrote orchestrator sequentially. Start-Job's separate runspace was
hiding the real errors below behind opaque "method on null" failures.
- Removed `function Mp` helper. PowerShell ships a built-in alias `mp`
for Move-ItemProperty that silently shadows same-named functions and
turned multipass calls into Move-ItemProperty calls. Inline `& $MpExe`
everywhere instead.
- Lowered ErrorActionPreference from Stop to Continue. Multipass writes
harmless "cannot set permissions" warnings to stderr when transferring
to NTFS; Stop turned those into fatal aborts.
- Stopped piping `multipass exec` stdout through Tee-Object. The pipe
could stay open indefinitely after the remote command exited, hanging
the orchestrator for 45+ min on a finished install. Now redirects to a
file on the VM and only checks the exit code.
- Tightened tar excludes. Tarball dropped from 305 MB to 20 MB.
- Added `npm ci` in vm-install.sh after the local-code overlay. The
original installer ran `npm ci` from origin/main, then the overlay
dropped in a dev package.json that needed different deps, causing a
spurious frontend build failure.
- Added -ReuseVm flag for re-running just the harness against an
already-installed VM (skips the 20-min launch cost).
First successful end-to-end run on Ubuntu 24.04: 10/10 tests passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PowerShell 5.1 reads .ps1 files as ANSI by default; em-dashes and check
marks were mangled into bytes that broke the parser. Replaced with ASCII
equivalents so the script runs under stock Windows PowerShell without
needing pwsh.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One-click PowerShell entry point that spins up Ubuntu/Debian VMs in
parallel, installs ServerKit from the local working tree (not origin/main),
runs a pytest API harness against the live panel, and aggregates results
into a self-contained HTML report.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Juan Denis