scripts/sign-extension.mjs signs a bundle with an ed25519 author key;
EXTENSIONS.md documents the format, key model, and verdict behaviors.
Generated private keys live in gitignored scripts/keys/.
- install_profile_service: record what install.sh put on the box
(minimal/standard/full), recommended from detected hardware and
overridable; thresholds mirrored by recommend_profile in install.sh
- setup wizard: replace the hard ResourceGate/SetupStepTier with
SetupStepCapacity + SetupStepSecurity
- ResourceAdvisory: inline capacity warning at the point of action
(WordPress/Docker/Services/Deployments), backed by headroom.fits
- RequiresDocker component for docker-dependent surfaces
- install.sh/update.sh: profile detection + tests
(test_install.sh, test_update.sh, test_server_capacity.py)
backup_current tested `[ -f serverkit.db ]` FIRST and only fell through to the
PostgreSQL branch when no file was there. An install migrated SQLite ->
PostgreSQL keeps its old serverkit.db, and both deploy paths copy it forward on
file-existence alone (deploy_source, deploy_release), so the stale file is
carried into every future slot and never ages out. Such a box backed up the
abandoned file, skipped pg_dump entirely, and left PRE_UPGRADE_DUMP unset --
which migrate_database reads as "${PRE_UPGRADE_DUMP:-}", so its auto-restore
silently did nothing. PostgreSQL is the one engine that migrates IN PLACE and
cannot roll back by switching slots, so that dump is its only way home: the
exact safety net a61eab1/b48256f added was disabled for it.
migrate_database has always keyed off DATABASE_URL (its sqlite check). Test the
declared engine first so the two functions agree.
Deliberately NOT `grep DATABASE_URL=sqlite && [ -f db ]` as the reviewer
suggested: backend/config.py defaults DATABASE_URL when absent, so any install
whose .env lacks the line would fall to the else branch and update with NO
backup at all -- trading a mismatched backup for none. An explicit `postgres`
URL is a positive signal; the absence of a `sqlite` one is not.
Also:
* name the engine when it is one we cannot dump. "No SQLite database ..."
read like a benign nothing-to-do on a MySQL install that in fact had no
safety net. Prints the scheme only -- a DATABASE_URL carries a password and
this text lands in the update log.
* drop the stale "Python 3.11/3.12 is guaranteed by pre-flight" comment,
contradicted by locate_sqlite_python two lines below it (664a5e6).
* discord-notify: skip cleanly when DISCORD_WEBHOOK_URL is absent. Secrets
are not exposed to `pull_request` runs from a fork, so a merged fork PR
failed the job on an empty curl URL. Not switching to pull_request_target
for this -- handing a write-scoped token and the webhook to an
outsider-triggered run is a poor trade for a chat notification.
Tests: a PostgreSQL install carrying a leftover serverkit.db must still dump
and set PRE_UPGRADE_DUMP (fails on the parent commit, rc=1); an undumpable
engine must be named without leaking the password. Neither needs python3, so
both run on all 7 images. Workflow verified end-to-end against a synthetic
event payload: fork PR exits 0 without calling curl, normal PR still builds the
same embed.
Verified on the full CI matrix (ubuntu 22.04/24.04, debian:12, rockylinux:9,
almalinux:9, fedora:40, opensuse/leap:15.5), all six suites: 0 failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
copy_sqlite_db and verify_sqlite_db reused locate_python, which enforces
3.11 <= v <= 3.12. That bound is correct for rebuild_virtualenv and
preflight_check, but snapshotting or integrity-checking a SQLite file needs
nothing beyond the stdlib sqlite3 module. Reusing it coupled the DATA-safety
paths to the venv-builder constraint and blocked updates for a reason unrelated
to the data:
* an all-Docker install returns before preflight_check ever runs and does
every real step in the container -- yet backup_docker_db snapshots with the
CONTAINER's python and then verified the extracted file with the HOST's.
On a stock EL9 (python3.9) or Ubuntu 22.04 (python3.10) host that halted
with "Could not obtain a VALID database backup", blaming the backup for a
missing interpreter.
* backup_current halts when it cannot snapshot, so those hosts could never
update at all.
Split out locate_sqlite_python: any candidate that can `import sqlite3` wins,
and the install's own venv interpreter is preferred -- on a real box it is
guaranteed present, guaranteed to carry the module, and independent of PATH.
locate_python keeps the 3.11/3.12 bound where it belongs.
Surfaced by the scripts-ci unit matrix: almalinux:9 and rockylinux:9 ship
python3.9, so the suite's own probe accepted an interpreter locate_python
rejected, and copy_sqlite_db/verify_sqlite_db failed there while passing on
python3.12 runners.
T10i pins the contract: on a box where every candidate reports 3.9 (locate_python
fails), the helpers still resolve an interpreter, prefer the slot's venv, and
round-trip 100 rows. Fails on the parent commit, passes here.
Verified on the full 7-image CI matrix (ubuntu 22.04/24.04, debian:12,
rockylinux:9, almalinux:9, fedora:40, opensuse/leap:15.5), all six shell suites:
0 failed. test_update is 63 passed / 0 failed on every image carrying a python3.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
verify_pg_dump only runs its structural `pg_restore -l` TOC check when the
binary is on PATH. The harness stubs every other external command it touches
(systemctl, nginx, npm, curl, docker, flask, pg_dump) but never pg_restore, so
two assertions were decided by the host:
* no postgresql-client -> branch skipped, check degrades to "non-empty",
and the synthetic `PGDMP fake contents` fixture passes
* postgresql-client present -> the real binary parses the version bytes
after the PGDMP magic, rejects the fixture, verify_pg_dump returns 1
That is the CI failure. T10f failed directly; T10e failed downstream because
backup_current correctly halts on an unverifiable backup. update.sh is right --
the tests leaked.
Add a make_stub_pg_restore factory mimicking the real -l semantics (rejects
missing/empty, rejects anything without the PGDMP magic, else prints a TOC) and
install it in T10e/T10f. Give T10f a non-empty non-archive fixture too: that is
the case separating the two branches inside verify_pg_dump -- the size check
passes it, only the TOC listing catches it -- so the path the test was named
for is now actually exercised.
Verified both with and without a pg_restore on PATH: 59 passed, 0 failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A backup that was never verified is a hope, not a safety net. Closes the
remaining ways the pre-upgrade safety net could silently not exist:
- Every backup is verified immediately after creation: SQLite snapshots via
PRAGMA integrity_check, PostgreSQL dumps via size + pg_restore TOC
listing (when available). Invalid artifacts are deleted and the update
halts — a malformed backup is never mistaken for a good one.
- Partial artifacts from a failed backup are removed (rm on failure), so a
half-written file can never be 'restored' later.
- A failed IN-PLACE PostgreSQL migration now automatically restores the
verified pre-upgrade dump (pg_restore --clean --if-exists) before
halting; without it the half-migrated schema would stay live against
the old code. If the restore itself fails, the halt message carries the
exact manual pg_restore command.
- backup_docker_db tries both python and python3 in the container, and now
verifies the result too — the plain docker cp fallback can no longer
pass off a torn copy as a backup.
- Backup retention capped: default 5 (SERVERKIT_BACKUP_RETENTION
overrides) across trees, .db and .dump files. Full-size DB copies pile
up fast — a real 25G droplet carried 6.3G of stale backups.
Tests: T10f (verification helpers accept/reject correctly), T10g (failed
in-place PG migration auto-restores with the exact pg_restore args),
T10h (retention prunes all three backup kinds to the configured cap).
Postgres panels updated with zero safety net: backup_current found no
SQLite file and just warned, while the migration runs IN PLACE against the
shared external DB — a failed migration left no way back.
- backup_current now pg_dumps (custom format) when .env points at
PostgreSQL, sanitizing SQLAlchemy driver suffixes (postgresql+psycopg2://)
that libpq can't parse.
- Same rule as SQLite: no valid backup, no update — halt when pg_dump is
missing (install postgresql-client) or the dump fails.
Tests: T10e covers the dump invocation + URL sanitization and the
halt-on-failed-dump path.
The venv binding check only matched the quoted VIRTUAL_ENV="/path" form;
venvs created on some platforms write 'export VIRTUAL_ENV=/path' unquoted,
so the binding read as 'unknown path' and always triggered a rebuild.
Now matches both forms (and skips cygpath-style indirection), so fresh
installs keep the fast prebuilt-venv path.
Root cause of a real-world failed update: backup_current, deploy_source and
deploy_release all plain-cp'd the LIVE panel database. On a busy panel that
races the backend's own writes and the copy can capture half a transaction
— 'database disk image is malformed' — which then sinks the migration in
the new slot and silently poisons the pre-upgrade backup that was supposed
to be the safety net.
- New copy_sqlite_db() uses the SQLite online backup API via python
(guaranteed by pre-flight) and refuses to fall back to a racy cp.
- backup_current halts rather than proceed without a valid safety net.
- deploy_source/deploy_release halt before any switch if the live DB
cannot be snapshotted consistently.
- backup_docker_db prefers a consistent snapshot via docker exec python,
with the plain docker cp as a fallback.
- locate_python rejects candidates whose version probe returns empty
(broken shims pass 'command -v' but can't execute; an empty version
string used to pass the sort-based range check).
Tests: T10d exercises a real snapshot (integrity ok, all rows); a contract
test bans plain-cp of the live DB; T10d fixture python detection validates
the interpreter actually runs.
The SQLite pre-check in migrate_database treated an EMPTY probe result
(unusable venv python) the same as real corruption and halted the update —
a false positive that broke the T10/T24 contract tests. Now:
- probe output != 'ok' -> genuine corruption -> halt with repair guidance
- empty probe output -> probe couldn't run -> warn and let flask surface
any real error
test_update.sh: T10/T24 fixtures stub the venv python (healthy-DB path);
new T10b asserts the corrupt-DB halt fires before flask runs; new T10c
asserts an unrunnable probe warns and proceeds.
- create_app(): skip background daemons (queue consumers, metrics, job
system, analytics flush, linked-panel client) when loaded by a Flask CLI
one-shot such as 'flask db upgrade', or when SERVERKIT_SKIP_BACKGROUND=1.
They query the DB before migrations run, so any error there (corrupt DB,
pre-migration schema) aborted the migration and sank the whole update.
- update.sh: set SERVERKIT_SKIP_BACKGROUND=1 for both flask db upgrade
invocations as an explicit contract.
- update.sh: fail fast with an actionable message when the SQLite slot copy
fails PRAGMA integrity_check, instead of dying mid-boot with 'database
disk image is malformed'.
- update.sh: rebuild the release tarball's prebuilt venv when its baked
VIRTUAL_ENV path resolves to the old slot (it is baked as
/opt/serverkit/venv), which previously ran new code on the old slot's
dependencies during migration. Resolved-path comparison keeps the fast
prebuilt path for fresh installs.
Plan 55 Phase 1. The sweep found zero vulnerabilities: every raw-HTML sink is
safe by construction (escape-first SQL/file highlighters, sanitizeSvgInner for
extension icons, the escape-then-allowlist markdown renderer) and the backend
has no |safe/Markup(/render_template_string.
To keep it that way: scripts/check-html-sinks.mjs (wired into npm run lint) and a
Python mirror test (test_html_sink_sweep.py, runs in the backend suite) fail on
any raw-HTML sink that neither references an allowlisted sanitizer nor carries a
'sink-safe:' annotation. Standardized annotations on the three variable-fed sinks
(markdown, SQL editor, marketplace icon) and documented the rule in SECURITY.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduces a full-stack Test Sandbox feature to run distro matrix tests in Docker from the UI: new backend API/service, sandbox_runs model + migration, run/log endpoints, and a new frontend page with mode selection, run history, live polling, cancel, and per-distro logs.
Also adds ARM64 scripts CI coverage (ubuntu-24.04-arm) and strengthens install/test automation for minimal images and overlay rebuilds: early curl/bootstrap index refresh, stricter venv validation, release-venv glibc fallback, Node reprovisioning on source fallback, rsync --delete overlay behavior, npm bootstrap in VM install, Windows tar pinning in PowerShell harness, plus updated test docs and distro coverage notes.
Adds USDT (TRC-20 and ERC-20), BTC and SOL to the Support section of all four
READMEs, with the network in its own column — USDT exists on several chains
whose addresses look alike, and the network is what tells a donor which one to
send on.
QR codes come from scripts/generate-funding-qr.mjs rather than a web generator,
since an online tool sits in a position to encode a different address than the
one entered and the resulting image looks perfectly normal. The script holds the
addresses as the single source of truth and checksum-validates each before
encoding — base58check for Tron, EIP-55 casing for Ethereum, bech32 for Bitcoin,
32-byte base58 for Solana — refusing to emit anything if one fails. Every code
was decoded back and diffed against its source string. `--check` re-verifies the
committed PNGs so an address edit with a stale QR cannot pass unnoticed.
These are self-custody wallets rather than exchange deposit addresses, which can
be rotated out from under a published README and turn third-party donations into
an AML review on the receiving account.
Adds CODEOWNERS over the READMEs, the QR script, the funding images,
FUNDING.yml, LICENSE and SECURITY.md. Swapping a donation address through an
innocuous-looking docs PR is a known attack on open-source projects; this
requests maintainer review on every path that carries one. It takes effect once
"Require review from Code Owners" is on in branch protection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Generated manifests now contribute both `<slug>` and `<slug>/:tab`, forwarding
every sub-path to the same component. Adding a tab to a scaffolded extension
becomes a one-line change to TABS in frontend/index.jsx instead of a manifest
edit plus a route entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Analytics, CrowdSec, DNS Server, GPU Monitor, Kubernetes, Mail Server and
Automations (tramo) now live in their own repositories and install from the
registry as versioned, sha256-pinned zips. Their sources leave
builtin-extensions/, their pre-bundled frontends leave frontend/src/plugins/,
and registry_index.json gains a `bundled: false` entry for each so the
Marketplace offers them as normal registry installs.
The panel keeps only the wiring: comments at the former mount points explain
where each surface now comes from, and the gpu auto-install gate in
extension_migration.py is dropped since a host-probe can no longer decide to
re-vendor an extension that is not in this tree.
Also untracks backend/app/plugins/serverkit-{gpu,k8s,tramo}/, which were the
panel's own install artifacts rather than source. Committing those shipped a
second copy of code that now lives in three other repos, guaranteed to drift.
.gitignore now excludes the install path for every extracted slug on both the
backend and frontend side, so a local install can never be committed back.
Four tests from test_tramo_extension.py exercise panel core rather than the
extension (the retired /api/v1/workflows blueprint, the plan-45 event emitters
ported to EventService, and the docker_service double-emit fix); they move to
test_tramo_panel_core.py and stay here. The rest of the extension suites go
with their repos.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The installer's node_ready() now gates on Node ^20.19.0 || >=22.12.0 (was >=18),
so the T3 assertion "node_ready true for node v18" was stale and failed CI.
Rewrite T3 to prove the real floor: accepts 20.19+/22.12+ (with npm), rejects
v16/v18/20.17/20.18/21.x/22.11. (Dropped an npm-absent sub-case that can't be
isolated in-harness — command -v npm still finds the runner's real npm.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make `update.sh` fetch the CI-built release tarball by default instead of
rebuilding the SPA on the server, so a normal update needs no Node/npm on the
box. This keeps updates working on old distros, ARM/Pi and tiny hosts where the
vite 8 toolchain (Node 20.19+/22.12+) can't or shouldn't run — the Node
requirement stays a build-time/CI concern, not a runtime one.
On-box source rebuilds remain available and explicit: --source / --main (main
branch), --branch <name>, or BUILD_FROM_SOURCE=1. --branch now implies source
mode (clears the release default). npm is only required for those paths.
Test: test_update.sh +1 (mode mapping — default=release, --source/--branch/
BUILD_FROM_SOURCE=source). 49 passed / 0 failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The vite 8 upgrade (dcf05d1) makes the frontend build require Node ^20.19.0 ||
>=22.12.0 — on anything older npm silently skips rolldown's native binary and
`npm run build` dies with a cryptic MODULE_NOT_FOUND. Servers that build the SPA
on-box (source install + every update) would break their next update. So:
- install.sh node_ready() now gates on the real vite 8 floor (was Node >=18) and
provisions NodeSource 22 LTS when the host Node is too old.
- update.sh gains _node_build_ok(): build_frontend_bundle refuses (returns 1 →
caller rolls back) and the Docker-host rebuild halts, both with an actionable
"upgrade Node" message instead of a rolldown stack trace.
- Dockerfile + CI (release, release-smoke, extensions, scripts-nightly) bumped
node 20 -> 22 so the shipped/pre-built bundle is built on a compatible Node.
Also fixes a pre-existing parity bug the operator caught: update.sh rendered the
systemd unit WITHOUT substituting @BIND_HOST@ (install.sh does), so an update
left the literal placeholder and gunicorn would fail to bind. update.sh now
substitutes it (loopback default, SERVERKIT_BIND_HOST override) like install.sh.
Tests: test_update.sh +3 (node-floor gate, build refusal on old Node,
@BIND_HOST@ substitution) — 48 passed / 0 failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two gaps for existing installs:
- The tint ran only in install.sh, so `serverkit update` (which has its own
build path in scripts/update.sh) would rebuild with the DEFAULT color and a
fixed favicon.ico — existing installs would never get a unique mark.
- Re-tinting on every rebuild would also repaint the box on each update.
Fix: assign the color ONCE and persist it in /etc/serverkit/favicon-color
(survives updates), then re-apply the SAME color from both install and update.
update.sh applies it to the staged dist unconditionally (after the guarded
build), so it works whether dist was rebuilt (source mode) or shipped prebuilt
(release). An install that predates the feature picks up a color on its first
update and keeps it thereafter — a car keeps its paint.
Verified: bash -n on both scripts; persistence confirmed (re-run reuses the
stored color).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 1.7.25 -> 1.7.33 update broke on panels that still had the retired
serverkit-workflows extension on disk: preserve_installed_plugins carried
it into the fresh tree, where its reactflow import (removed with plan 45)
sank the whole bundle build and blocked the update.
- RETIRED_PLUGIN_SLUGS: the carry-forward now skips slugs the product
deliberately deleted, and docker in-place mode strips their leftovers
after the hard reset (untracked dirs survive it).
- Quarantine-and-retry: when the bundle build fails and plugins were
carried forward, move them into /var/backups/serverkit/quarantined-
plugins-<ts> and retry once -- fail the plugin, never the update.
The updater self-refreshes from main before running, so this protects
every panel on its next `serverkit update` once merged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live test on the dev box caught 'serverkit list-apps --json' printing
the table: cmd_list_apps' hand-rolled flag mapping dropped everything
but --all, and cmd_list_users/cmd_list_servers dropped all args. All
three now forward "$@" (with -a kept as the legacy --all alias), and
test_cli.sh T20 drives the real executed-mode dispatch through a fake
venv with a recording python stub so any wrapper verb that eats flags
fails the suite.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
serverkit migrate-db forwarded the wrapper's own name to cli.py, which
only defines db-migrate — the command has failed with 'No such command'
since the rename. Fix the forward and stop the drift class entirely:
- cmd_panel_cli forwards Click-only verbs verbatim (manifest, services,
apps, repair, login-url, support-bundle, db-migrate/status/history;
panel-status/panel-doctor where a native bash name collides)
- serverkit completion prints a bash completion script; install.sh and
update.sh drop it into /etc/bash_completion.d (best-effort), the
uninstall lib removes it
- test_cli.sh T17-T19: migrate-db forwarding, dispatch reachability,
and a two-way parity guard (cli.py verbs <-> dispatch <-> completion);
test_update.sh T29 covers refresh_bash_completion
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
build-release.sh excluded *.png/*.jpg/*.jpeg globally — fine when the only
images were screenshots, fatal once ExtensionIcons.js (8bd9c1a) started
importing frontend/src/assets/extension-icons/*.png as build inputs: the
/tmp build tree lost them and rollup failed to resolve. The blanket rule
was also silently stripping frontend/public/favicon-*.png from every
release build.
Replace with targeted excludes: docs/screenshots, frontend/dist,
.codex-screenshots, template-icons, and (via --no-wildcards-match-slash)
root-level ./*.png|jpg|jpeg strays + dev/*.png. Verified by listing the
tar: 14 extension icons + 4 favicons included, zero stray images, *.log/
*.tmp/*.pyc still excluded; shellcheck clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scripts/export-registry-entries.mjs reads builtin-extensions/*/plugin.json
and emits index-v2 bundled entries for the serverkit-extensions registry.
Normalizes panel categories onto the registry enum and optionally attaches
repo-relative logo paths when art exists. Run and paste into index.json via
PR whenever the bundled set changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Commit 6e32519 (the July 5-7 recovery restore) wrote git-reflog garbage
over scripts/install.sh (the agent installer served at
GET /api/v1/servers/install.sh) and whitespace-mangled content over
scripts/test/full-stack-test.ps1, breaking scripts-ci: shellcheck SC2148
(no shebang) and 15/18 test_agent_install.sh failures. Restore both files
verbatim from 6e32519~1, the last good revision.
Verified: test_agent_install.sh 18/18, test_install.sh 62 passed,
shellcheck --severity=error clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The staging verify's migration-head check grepped for Alembic hex hashes
([0-9a-f]{8,}), but ServerKit revision IDs are descriptive slugs (e.g.
074_cf_ops_changes). The empty grep exited non-zero and, unguarded under
set -e, aborted verify before the VERDICT line ever printed. Take the first
token of `flask db current`/`heads` instead, and guard with `|| true`.
Surfaced driving plan-38 Phase 5 (staged deploy of the recovered dev to the DO
test box via Faro): deploy migrated cleanly to head 074 but verify crashed
pre-verdict. With this fix, `verify --full` emits a green VERDICT (pass:true).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A test box's panel domain kept serving a Jun 25 bundle after the Jul 3
update — the domain's custom vhost still proxied the retired :3847
frontend container, and the old bundle's calls to deleted legacy
marketplace endpoints read as 'Failed to load extensions'. It looked
exactly like a browser/CDN cache problem (it wasn't — cf-cache-status
was DYNAMIC; the origin itself was stale).
- both stock confs: expires -1 on the SPA entrypoint (emits
Cache-Control: no-cache WITHOUT discarding server-level security
headers, unlike add_header-in-location) — a cached index.html
references hashed assets that vanish on the next blue/green switch
- update.sh refresh_config now runs report_stale_panel_vhosts: warns,
naming the file, when any enabled vhost still proxies :3847;
observation-only per R1, silent on a missing sites-enabled dir,
added to the fresh-box loop per policy
- proving tests: entrypoint no-cache asserted for both confs; stale
vhost flagged (and clean one not); missing-dir silence (42 passed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bash suppresses set -e inside any tested-context command list — an
if-condition subshell, or the left side of ||, even with set -Eeuo
pipefail re-armed inside — so multi-statement test bodies only gated
on their LAST statement. Today's cmp-less EL9 CI run proved it: a
byte-identity test passed while its cmp assertions exited 127.
28 blocks across test_update/test_install/test_lib/test_agent_install
converted to explicit && chains so every assertion gates the result.
No test weakened; no vacuous pass was hiding a real script bug (each
newly-gated statement was verified to genuinely succeed).
Verified: Git Bash 171 passed / 0 failed; rockylinux:9 all five
suites 177p/0f; opensuse/leap:15.5 148p/0f (pre-existing python3/tar
skips).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
EL9/Fedora containers ship without diffutils and openSUSE Leap without
tar, which failed 4 of the 7 cross-distro unit jobs:
- files_identical() in stubs.sh: cmp when available, sha256sum
(coreutils, everywhere) otherwise
- the two tarball-fixture tests skip cleanly when tar is absent
(harness's existing skip convention; proven on the other distros)
- the t16/t16r assertion lists are now && chains: bash suppresses
set -e inside an if-condition subshell, so sequential statements
only gated on the LAST one — the cmp-less EL9 run proved it by
passing t16r vacuously while cmp exited 127
Verified in the actual images: rockylinux:9 65p/0f/0s,
opensuse/leap:15.5 61p/0f/4s (was 63p/2f and 61p/2f).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- PULL_REQUEST_TEMPLATE gains a scripts-only checklist condensing the
failure-semantics rules (guarded observations, no &&-tail returns,
capture-safe stdout, errexit-immune helpers, fresh-box loop policy,
proving test in the same commit)
- record the self-update channel decision in the bootstrap header:
keep fetching from main, gated by the update-e2e CI job, preserving
one-push-fixes-the-fleet
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The July 2 outage function had a unit test — with a populated fixture
only. This encodes the missing net:
- fresh-minimal-box fixture loop: every observation/discovery/snapshot/
report function in update.sh (17) and install.sh (16) must survive a
zero-app box under set -Eeuo pipefail; policy comment requires new
functions to be appended
- scripts/test/stubs.sh: shared failing-stub factories (systemctl exit
5, docker fail/empty, curl exit 22, GitHub rate-limit JSON)
- assertion sweep: 43 bare capture-then-compare sites now include exit
status in the assertion
- best-effort contract test: the run block's real APP_BASELINE ->
report chain exits 0 on a fresh box AND on a genuine regression
- fix(update): the loop immediately caught report_failure's trailing
'[ -n $UPDATE_LOG ] && printf' returning 1 when UPDATE_LOG is empty
(same species as the outage) — converted to if, added to the loop
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- version discovery pages agent-v tags (?per_page=100, 3 pages) so
panel releases can no longer push agent tags off page 1; panel
injects its resolved agent version into the served script and its
own releases fetch is paginated too (A1)
- downloaded agent binary is sha256-verified against the release's
checksums.txt: halt on mismatch, warn-and-continue when absent (A2)
- value-taking options error loudly when the value is missing; busybox
adduser fallback; no GNU-only grep -oP; pipe-safe source guard (A3)
- test-routing.sh fixed (PASS++ suicide under set -e, guarded probes,
curl 000000 false-pass) rather than deleted - still live tooling
- new scripts/test/test_agent_install.sh (18 passed); backend server
tests 110 passed
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- update --branch without a value errors loudly instead of dying
silently on shift (L5)
- cmd_start/cmd_restart mirror cmd_stop's systemctl/docker guards (L6)
- cmd_logs degrades past a dead/missing journalctl to docker logs (L7)
- cmd_add_site/remove_site/list_sites detect Debian vs conf.d nginx
layout; reloads are guarded with a manual-reload hint (L9)
- uninstall on a remnants-only box prints exact recovery commands (L4)
- portable port extraction (no grep -oP), read-at-EOF defaults,
compare_versions rejects unparsable as update-available, doctor
exits nonzero on issues (L12); BASH_SOURCE guard for testability
- new scripts/test/test_cli.sh (22 passed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- EUID check runs before any host write; friendly non-root halt (I11)
- get.docker.com staged to temp file with retry + pkg fallback (I13)
- df -Pk on the real install target; free guarded for LXC (I14)
- release tag via releases/latest redirect (no API quota) + loud
source-fallback warning (I15)
- tar/cp failures in fetch_release restore live state and fall back
to source cleanly (I16)
- svc_enable/svc_start init-system abstraction at service call sites (I17)
- stale slot-A collision guard in ensure_install_layout (I18)
- swap verified via /proc/swaps before claiming active; dd/mkswap
failures degrade to warn (I19)
- VERSION clobber, telemetry pipefail, stale SSL_MODE rewrite,
apply_frontend_root end-of-loop && (I22-I24, bonus)
- 22 new proving assertions (61 passed local, 64 on Linux)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- find-based backup pruning; empty/missing BACKUP_DIR is a no-op (U1)
- download_release progress to stderr; stdout is the tarball path only (U2)
- guard bare systemctl stop/start/daemon-reload in forward+rollback paths (U3,U4,U8,U13)
- ROLLING_BACK=1 inside rollback(); never runs twice via the EXIT trap (U5)
- npm preflight for source updates; guarded frontend build pipeline (U6)
- empty-ORIG_ARGS re-exec guard, cleanup_on_exit return 0, df -Pk,
SIGPIPE guards, cd leaks subshelled, local typo (U7,U9-U12,U14)
- 14 new proving assertions in test_update.sh (35 passed)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Allow failures when querying GitHub release tag (prevent pipeline from aborting) and add a unit test ensuring discover_app_upstreams tolerates an empty app-locations directory.
Changes:
- scripts/update.sh: make the curl|grep pipeline resilient by appending `|| true` so the command substitution won't fail the script when the GitHub API call returns nothing.
- scripts/test/test_update.sh: add test T16b to verify discover_app_upstreams does not abort under `set -euo pipefail` when the app-locations directory is empty.
These changes prevent hard failures during release detection and add coverage for empty-directory handling.
Suppress a non-zero exit from the pipeline in discover_app_upstreams by appending `|| true` after the sort. This prevents the script from exiting with an error when no matching proxy_pass entries are found in APP_LOCATIONS_DIR (e.g., empty or missing files). Improves robustness of scripts/update.sh in scenarios with no app upstreams configured.
Extensions can now contribute Settings sections: a settings.section
PluginSlot renders below the active Settings tab with the tab id as
context. _validate_manifest grew shape checks for everything the
platform consumes declaratively — module:attr refs (entry_point,
socket_entry, models, lifecycle hooks, job handlers), jobs/schedules
shapes, and per-kind required contribution keys (nav/routes/tabs/
palette/widgets/layouts) — so a malformed block fails the install with
a message naming each problem instead of being silently dropped at
runtime. Unknown contribution kinds only warn (forward compat). The
scaffolder gained --validate <path> mirroring the same rules for
authors, wired into the conversion recipe in docs/EXTENSIONS.md.
3 new lint tests incl. one asserting every shipped builtin manifest
passes; full backend suite 1227 green; lint 0 errors; build clean;
--validate smoke-tested on a good and a bad manifest.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A panel update deploys a fresh clone/tarball preserving only .env and
the DB, wiping app/plugins/<slug> + src/plugins/<slug> for any
URL/registry/upload-installed extension — the surviving InstalledPlugin
row then flipped to error at next boot. Two-layer fix:
- update.sh: preserve_installed_plugins carries user-installed plugin
dirs from the old tree into the new one (skipping anything the new
tree already ships and __pycache__), before the frontend build so
carried UIs recompile. Covered by harness test T18.
- boot repair pass (plugin_service.repair_missing_plugins, runs before
load_all_plugins): builtin slugs reinstall from builtin-extensions/,
url installs re-download from source_url (force + no hot-load so the
loader registers once), sourceless uploads get a clear re-upload
error. install_from_url/path grew force/hot_load plumbing.
4 new repair tests (16 total in the pipeline suite); update.sh harness
20 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The panel frontend is now served straight from nginx/sites-available/*
(static SPA at location /, plus /api + /socket.io proxy). A broken
directive there 502s the whole panel on a real box, and nothing parsed
these configs in CI.
Add scripts/test/test_nginx_conf.sh: stages each vhost (secure +
insecure) against a real nginx, recreating the absolute paths they
reference (a stand-in frontend/dist bundle, the serverkit-locations /
conf.d include dirs, certbot webroot) and a throwaway self-signed cert
for the TLS vhost, then runs `nginx -t`. Skips cleanly (exit 0) where
nginx/openssl/root aren't available, so it's safe on a dev laptop.
Wire it into the Scripts CI `shell` job after the unit suites (installs
nginx + openssl so the step actually runs). shellcheck-clean at warning
severity; bash -n clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Juan Denis