A name used before it exists only fails when that line runs, so a
rarely-taken branch ships broken. Two routes 500'd on every call and
Stop crashed on compose-slot apps, both found by pyflakes in seconds.
The check now runs in the ratchet job and in preflight.
The panel runs a plain threaded gunicorn worker and gunicorn is already
in requirements.txt; the extra gevent and gevent-websocket installs were
left over from the old worker class.
Both the installer and the updater treated a missing checksums.txt as
"skip verification" and installed the tarball anyway. Every release
publishes one, so its absence means a broken mirror or a tampered
download. The installer now falls back to a source build and the
updater refuses; SERVERKIT_ALLOW_UNVERIFIED=1 overrides for mirrors
that do not host checksums.
The panel runs async_mode='threading' under a plain threaded gunicorn
worker; nothing imports gevent. It was still installed everywhere, and
on Python 3.14 (no wheel yet) its source build is what broke installs
in issue #148. backend/Dockerfile was unreferenced and still launched
the gevent-websocket worker that double-answers the WS handshake; the
run.py hint recommended the same worker.
update, uninstall and cleanup-apps silently dropped any flag they did
not know, so 'serverkit update --help' ran a real update. --help now
prints usage and an unknown flag fails before anything runs.
The openSUSE Leap image ships without tar, so deploy_release could not
unpack the fixture and T20b failed its deploy assertions there. The
test now skips without tar, like its neighbours skip without
sha256sum.
On blue/green installs the previous slot is the code rollback, so the
serverkit-tree-* copy of it duplicated ~1 GB per update. With
SERVERKIT_BACKUP_RETENTION unset, disks under 50 GiB now keep one
database snapshot instead of three.
download_release runs in a command substitution, so its mktemp dir
was never removed: every update stranded the ~70 MB tarball in /tmp
(11 dirs, 644 MB on a 25 GB box). The download and unpack dirs are
now removed on success and on abort, and old leftovers are swept.
Full VM installs and Test Sandbox full mode now record a short
post-install runtime observation (60s settle + 120s sampling) with an
explicit measured/unavailable/failed/interrupted status that never
rewrites install success. Adds the standalone scripts/measure-runtime.py
sampler, the measure-installed-runtime.py harness wrapper, E2E report
rendering, results-API summaries, CI coverage and METRICS.md accounting
boundaries.
The tree backup rsync copied backend/instance/serverkit.db even though the
same database had just been written as the pre-upgrade copy, and rollback
only ever restores from that copy. On a 450 MB database that made every
update cost ~900 MB of backups, which is how small VPSes kept filling up.
Exclude the SQLite files from the tree snapshot. Retention is unchanged
(SERVERKIT_BACKUP_RETENTION, default 3, plus the 15% disk budget).
Add a test that every shipped nginx vhost forces a JavaScript MIME type for
/serverkit-vendor/*.mjs and that the regex cannot capture the
/api/v1/plugins/<slug>/assets/... route.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NWc84WDsWR7M55Mn7qtpie
Generate all four README tables from reviewed source and build snapshots, distinguish collection and asset sizes from runtime results, and remove unsupported footprint claims. Add a consistency check in CI and document how to reproduce profiling and measurements.
Centralize backup schedule calculations and shared request/form flows, encode query values, and move repository finalization into its service. Add browser CI coverage, repair stale test fixtures, and provide opt-in request and build measurement tools.
Schedule dependency and extension scans, audit production frontend dependencies, and replace category suppressions with reviewed function-specific exceptions. Preserve full reports and test the exception gate.
backend/cli.py gained `connect` (with status/doctor/disconnect) for the
ServerKit Cloud pairing but the `serverkit` wrapper never learned it, so
`serverkit connect` fell through to "unknown command" and the wrapper
dispatch census failed the Scripts CI suite. Add it to the help, the
dispatch case and bash completion; the passthrough test covers it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LGmHp7mZ2wSeA6z8BXZVxh
build-release.sh deleted every png/jpg in the staged tree except under
frontend/dist. That emptied frontend/src/assets/extension-icons in the
shipped /opt/serverkit, so the tree could no longer rebuild its own
frontend: ExtensionIcons.js imports 18 of those files and any `npm run
build` from the release (which `serverkit doctor --fix` triggered through
the Dockerfile) died on unresolved imports. Leave the whole frontend/ tree
alone; the tar excludes already keep docs and dev screenshots out.
Guarded in test_release_workflow.sh.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LGmHp7mZ2wSeA6z8BXZVxh
`serverkit doctor` assumed the frontend is the `serverkit-frontend`
container. On a host install that container never exists: install.sh has
nginx serve frontend/dist from disk, and the container only lives under the
opt-in `legacy-frontend` compose profile. Doctor flagged a phantom issue on
every host install, and `--fix` "repaired" it with a bare `docker compose
up -d` in /opt/serverkit, which does not start that container at all: it
builds the whole all-in-one panel image, which would then collide with the
systemd backend on :5000. start/stop/restart/status/logs carried the same
assumption.
Add `frontend_mode`, which reads the layout off the serverkit vhost (a
`root .../frontend/dist` is static, a `proxy_pass` on / is the legacy
container, neither is unknown), and make every command key off it. Static
installs are checked for the bundle nginx actually serves; when it is
missing doctor names the command that restores it (`serverkit update`)
instead of promising an auto-fix. The legacy path targets the profile's
service (`docker compose --profile legacy-frontend up -d frontend`), never
the panel image.
Tests: static install passes with no compose call, missing bundle under
--fix hints at update and still never calls compose, frontend_mode
classifies all three layouts, and an awk guard fails the suite if a bare
compose call from $INSTALL_DIR comes back.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LGmHp7mZ2wSeA6z8BXZVxh
Extensions can now contribute walkthroughs via contributions.walkthroughs
in plugin.json. The backend validates definitions against a shared
contract (route, signal, manual, and check completion types), transports
them through the dynamic contribution envelope, and persists custom
walkthroughs from the Studio. Includes the JSON Schema, SDK version gate,
and extension scaffold support.
The Extensions CI drift check (sync-builtin-frontends.mjs --check) had
no local counterpart, so a builtin-extensions/ manifest edit without a
re-sync reached CI unrun — exactly how the serverkit-git split went red
on the next PR. Same contract as the migration-inventory gate: a stale
copy is re-synced on the spot and flagged so the diff gets committed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Harness: bootstrap find/awk in distro-compatibility.py (AL2023,
Tumbleweed, SLES BCI minimal images lack them; the source-level
suites legitimately use both)
- Python gate 3.11-3.14: Arch/Manjaro rolling ship only python 3.14
with no versioned fallback packages (same trap as Debian 13/#99);
align scripts/update.sh, which had drifted to a 3.11-3.12 gate
- Gentoo: add emerge as a supported installer package-manager path
(install.sh, scripts/lib/pkg.sh, distro.py family, deps manifest);
provision via slotted dev-lang/python:3.13 -> 3.12
- Tests: move out-of-range stubs above the new ceiling, shadow all
locate_python candidate names (real python3.14/3.13 in Manjaro/
Gentoo images were winning over fixtures), cover the 7th manager
- Catalog: Gentoo is no longer a documented expected failure
Verified: test_lib 32/32, test_install 124/124, test_update 76/76,
pytest test_serverkit_installer 46/46, and all 6 previously failing
images now pass both quick and provision container probes (12/12).
Adjust release cleanup/packaging to stop stripping image files from frontend/dist, which broke Vite hashed asset references at runtime. The script now excludes dist from image deletion, removes image-wide tar excludes, and adds a bundle integrity check that fails the build if referenced /assets files are missing.
Five red CI rounds in one day, none of them a broken feature: a floor measured
on a dev box, a fingerprint baseline left unregenerated after a refactor, a
ceiling that moved with the code, a generated doc gone stale, and a text-grep
guard that matched a docstring. Every one is an AST or text scan over the tree
that needs no shard and no push to run.
preflight.sh runs the test-count ratchet, every backend ratchet/census/guard
test (matched by filename, so a new ratchet is covered the day it lands rather
than the day someone remembers this script), the frontend boundary check, and
the migration-inventory freshness check. About two minutes against a five-minute
push-and-wait.
It resolves node.exe through WSL interop, so the one WSL invocation in the
header comment covers both halves on this dev box. The inventory check compares
against a snapshot of the file rather than against HEAD, so an uncommitted but
correct regeneration is a pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two more shards were red on generated artifacts nobody regenerated:
- api_controller_boundary_baseline.json: the attachment round rewrote
_persist_user_message to build the row before adding it, so the fingerprint
moved from `db.session.add(AiMessage(...))` to `db.session.add(message)`.
Same crossing, same category, count unchanged at 509 -- one line.
- docs/MIGRATION_INVENTORY.md: three measured numbers had moved without a
regeneration -- bare @jwt_required() routes 577 -> 576, and the two frontend
rows whose totals come from maps inside check-frontend-boundaries.mjs
(setInterval pollers 10 -> 9, hex colour literals 151 -> 150). All three are
deterministic AST/constant counts, so these are CI's numbers, not this box's.
The reason nobody caught the inventory locally is the third fix here: the
generator hardcoded `node`, and under WSL the Windows nvm directory is on PATH
but exposes only `node.exe`, so test_migration_inventory.py skipped on the dev
box. Both the generator and the test now resolve $NODE / node / node.exe, and
the test runs instead of skipping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A backup retention cap already existed (SERVERKIT_BACKUP_RETENTION,
default 5), yet a box was found holding six snapshots and 0 bytes free.
Three separate defects, each sufficient on its own:
1. cleanup() is the LAST phase of the update, so retention never ran on
a run that died earlier -- including the run that died BECAUSE the
disk was full. Retention that only executes on success cannot bound a
failure loop. backup_current now trims BEFORE writing, which also
makes peak usage `keep` snapshots rather than `keep + 1`.
2. The preflight required a flat 2 GiB, but the update is about to write
two full copies of the database. With an 800 MB database a box at
2.1 GiB free passed the check and then filled mid-update. The
requirement is now 2 GiB + 2x the database, it trims stale snapshots
and re-checks before giving up, and the halt message points at
`serverkit disk`.
3. A count cap is not a disk guarantee: five snapshots of an 800 MB
database is 8 GB, a third of a 25 GB droplet. The default drops to 3
and enforce_backup_budget caps backups at a share of the filesystem
(SERVERKIT_BACKUP_MAX_PERCENT, default 15), dropping oldest-first and
always keeping at least one restore point -- an update with no way
back is worse than a full disk.
BACKUP_DIR also becomes SERVERKIT_BACKUP_DIR-overridable so it can agree
with the reclaim service, which scans the same directory.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docs/MIGRATION_INVENTORY.md is generated from the live censuses and the
frontend boundary checker's new --inventory JSON mode — no hand-typed
numbers, so the plan cannot claim completion while a door is bypassed. A
freshness test fails whenever a ratchet number moves without regenerating
the document, making the regeneration part of every migration commit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four docker legs exercising detect/status paths against real binaries —
probe-layer smoke, not provisioning: ubuntu root with an sbin-less PATH
(the plan 74 unit shape), ubuntu non-root with sudo (where the sbin bug
cannot reproduce — now proven, not reasoned), Rocky 9 (firewalld
paths), and capability-dropped ubuntu (docker's default cap set drops
NET_ADMIN — the unprivileged-LXC shape, where firewall commands exist,
are on PATH, and still fail).
The matrix pinned a live bug on first run: _check_ufw returned
active=False for a failed status probe in the capability-restricted
leg — "probe failed" indistinguishable from "firewall off" (fixed in
2f002c59).
Picks up serverkit-agent 219e75e, which points the usage and --help examples at
/api/v1/servers/install.{sh,ps1} instead of the domain root. Produced by
scripts/sync-agent-installers.sh, not hand-edited — scripts/test/check-installer-drift.sh
fails on any divergence and confirms these match the canonical copies again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
backend/app/data/registry_index.json is a mirror of serverkit-extensions'
index.json, and nothing automated the mirror -- so every registry publish left
the panel's offline fallback stale until CI complained. That is how
`serverkit-localkit` sat missing from it: an offline panel offered a different
extension set than an online one.
Same shape as scripts/sync-agent-installers.sh: prefer a sibling checkout so it
works offline and before the registry PR is merged, else the published branch,
resolving the default branch from the API rather than hardcoding a name.
Two things it is deliberately careful about:
- It vendors the RAW index, never https://serverkit.ai/ext/index.json. The
panel fetches through that proxy at runtime and the proxy rewrites every
relative `logo` path to an absolute serverkit.ai URL, while the bundled
loader resolves relative paths itself. Vendoring the proxied copy would bake
in the wrong paths, and the tuple the tests compare (version/source/sha256)
is identical either way -- nothing would have caught it.
- It keeps the local `note` (which does not exist upstream) and writes with
indent=2/ensure_ascii=True/trailing newline, which round-trips the existing
file byte-for-byte. An in-sync run is a true no-op, not a reformat.
It refuses to write an index with no entries, so a broken or truncated canonical
source cannot blank the Marketplace fallback.
Verified: no-op when in sync; reproducing the real bug (dropping the localkit
entry) and re-running restores the file byte-identical to HEAD; repins are
reported; the empty-canonical guard exits non-zero leaving the file untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The vendoring and drift-check scripts hardcoded a branch name for the fallback
fetch. That was wrong twice in one day: first `main` (the agent repo used
`master`, so the fetch 404'd), then `master` (which is being renamed to `main`).
A hardcoded branch turns any rename into a 404 that surfaces as "cannot reach
the canonical copy" rather than "wrong branch name" — a confusing way to fail
for a check whose whole job is to be believed.
They now ask the GitHub API for the repo's declared default_branch, falling back
to probing main then master if the API is unreachable or rate-limited. AGENT_REF
still pins an explicit ref (a tag, say) when you want one. Verified: the API
currently reports `master`, and it will report `main` on its own once the rename
lands — no follow-up edit here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The vendoring and drift-check scripts defaulted AGENT_REF to `main`, which this
repo uses but the agent repo does not -- it has no `main` branch at all.
Verified: raw.githubusercontent.com/jhd3197/serverkit-agent/main/install.sh is
404, /master/install.sh is 200.
The bug was invisible locally because both scripts prefer a sibling
../serverkit-agent checkout and only fall back to fetching. It would have shown
up first in nightly CI, as "could not fetch the canonical copy" -- which reads
as an outage rather than a wrong branch name, so it is worth the comment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scripts/install.sh and scripts/install.ps1 are now VENDORED copies. The agent
repo owns them; this repo carries a copy only because the panel serves them off
its own disk at GET /api/v1/servers/install.sh|.ps1 and cannot link to another
repository.
Two copies of one file is what caused the second half of #101. When the agent
moved out of this monorepo it took its installer with it, and from then on there
were two -- actually four: the agent repo also carried a third, older pair under
its own scripts/ that nothing referenced. Each looked fine in isolation, nothing
ever compared them, and the panel went on serving one that downloaded from
`agent-v*` tags which exist in no repository. Unit tests are structurally blind
to this: they exercise whichever copy lives in their own repo and have no idea
another exists.
So:
scripts/sync-agent-installers.sh re-vendors from the agent repo (a sibling
checkout if present, else the published
branch), normalising to LF so this repo's
.gitattributes can apply its own EOL rule
scripts/test/check-installer-drift.sh fails, with the diff and the fix, if a
vendored copy differs from canonical
(EOL-insensitive: *.ps1 is checked out
CRLF here and LF there, which is not drift)
The drift check runs nightly alongside the release-chain verification, since
both need to reach GitHub. Both scripts run locally against a sibling checkout,
so this works offline and before an agent change is published.
Both installers carry a provenance header naming the canonical location, so
anyone who opens the vendored copy to patch it is told where to patch instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Issue #101 shipped because release.yml went straight from `build` to `push`
with nothing in between. The image was missing scripts/, so every containerised
panel 404'd its own enrollment URL, while the entire test suite stayed green --
pytest runs against the source tree, where the file always exists. Only booting
the artifact catches that class of bug.
scripts/test/smoke-docker-image.sh builds (or takes) an image, asserts the files
the panel reads at request time are present, boots it, and requests every route
that serves a file off disk. release.yml now builds amd64, runs it, and only
then pushes multi-arch (layers are cached, so the push reuses the build).
release-smoke.yml gets the same job on dev and PRs, and its path filters now
include Dockerfile, .dockerignore and scripts/ -- none of which used to trigger
it.
scripts/test/verify-agent-release-chain.sh covers the other half, which unit
tests structurally cannot: they stub curl, so they only ever prove the installer
is self-consistent, never that the URL it builds resolves to a real file. This
one reads the coordinates back out of the script, asks GitHub, downloads the
archive, verifies the checksum and confirms the tarball entry name the installer
moves. It runs nightly, and is also the tripwire for the reverse case -- the
agent repo changing its release naming and breaking the panel from the other
side of the repo split.
Both are plain scripts so the same check runs locally:
bash scripts/test/smoke-docker-image.sh
bash scripts/test/verify-agent-release-chain.sh
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When the Go agent moved out of this monorepo into jhd3197/serverkit-agent, the
installers did not follow it. They still asked jhd3197/ServerKit for `agent-v*`
tags -- coordinates that exist in no repository: jhd3197/ServerKit has zero
agent-v* releases, and the binaries are published as vX.Y.Z in the agent repo.
Version discovery therefore failed outright, and the download URL was a 404.
Everything downstream was already correct and is unchanged: the asset name
serverkit-agent-<ver>-linux-<arch>.tar.gz, the entry inside the tarball
(serverkit-agent-linux-<arch>) and the checksums.txt entry names all match the
real v1.2.0 release. Only the repo and the tag prefix had rotted.
Version discovery now requires a digit after the "v". The agent repo carries a
malformed release tagged literally "v", which would otherwise parse to an empty
version and build .../releases/download/v/serverkit-agent--linux-amd64.tar.gz.
test_agent_install.sh encoded the dead scheme in its fixtures (agent-v0.3.2), so
it stayed green throughout. It now pins the release coordinates as a contract:
the repo, the /v${VERSION}/ URL scheme, the absence of any live agent-v
reference, and the checksums URL exercised through the curl stub. All four fail
against the pre-fix script.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The python-provisioning job failed on ubuntu:22.04 with every apt lookup coming
back empty — software-properties-common not found, so the deadsnakes fallback
could not even be attempted and provision_python fell through to the source
build.
The installer was not broken. main()'s order is
identify_system -> choose_pkg_manager -> ensure_bootstrap_tools -> ... -> provision_python
and ensure_bootstrap_tools is what refreshes the package index — its own comment
says minimal images "ship with empty package indexes". This test called
provision_python without it, so on a fresh container the apt lists were empty
and the FIRST `apt-get install` had no candidate for anything. A test that calls
production functions out of order invents failures; "fixing" install.sh to
satisfy it would have made the real installer worse.
Why it was not caught locally: my verification ran `apt-get update` before the
test, while the CI step is `command -v bash || { apt-get update && install bash; }`
— Ubuntu already ships bash, so that update never runs. I had verified a machine
state CI does not have. Re-verified by running the CI command VERBATIM.
All eight matrix images, exact CI invocation, 7 passed / 0 failed each:
ubuntu:22.04 -> 3.12 (deadsnakes) debian:12 -> 3.11
ubuntu:24.04 -> 3.12 debian:13 -> 3.13
rockylinux:9 -> 3.12 almalinux:9 -> 3.12
fedora:40 -> 3.12 opensuse/leap:15.5 -> 3.11
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
write_config mints its secrets with `openssl rand -hex 32` and a python3
one-liner. The cross-distro containers install ONLY bash — ubuntu:24.04 and
debian:12 ship neither binary — so under `set -e` the failed command
substitution aborted write_config before it wrote a line, and three assertions
read an empty .env:
✘ stock install produced: []
✘ an exposed backend port still trusted X-Forwarded-For: []
✘ external proxy did not set 2 hops: []
The three T44 cases that passed are exactly the ones taking write_config's
existing-.env early return, which never reaches key generation — that split is
what identified the cause.
My bug, and the avoidable kind: T44 was the one part of this suite that leaned
on real binaries, in a file whose whole premise is that it stubs every external
tool. It passed locally only because a dev box happens to carry openssl and
python on PATH.
Stub both inside the three affected subshells. Shell functions take precedence
over PATH lookup, including through `"${PYTHON_BIN:-python3}"`, so no PATH
juggling is needed. The stub values are literals rather than built with
seq/tr/head — those are precisely the tools a stripped base image turns out not
to have, which is the bug being fixed.
Verified in the exact failing image: ubuntu:24.04 goes from 121 passed / 3
failed to 124 passed / 0 failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`apt-get install -y` answers apt's own prompts but NOT debconf's. tzdata,
keyboard-configuration and friends still open an interactive dialog and block
forever. pkg_add captures apt's output, so the installer hangs in complete
silence — no prompt, no progress, nothing to diagnose.
Observed, not theorised: ubuntu:22.04 sat for 60 minutes on 39 bytes of output,
stuck in tzdata.postinst -> debconf frontend. Real Debian/Ubuntu servers are
exposed to the same stall; a container is just where it showed up. Under
`curl | bash` it is worse still, because stdin is the installer script itself,
so debconf can consume installer source as its answers.
DEBIAN_FRONTEND was set NOWHERE in install.sh, scripts/lib/pkg.sh or update.sh.
Every other manager was already guarded (dnf/yum -y, zypper --non-interactive,
pacman --noconfirm) — apt was the only unguarded one, and pkg.sh's own
docstring already claimed the whole function was non-interactive.
In pkg.sh it goes in ARGV as `env DEBIAN_FRONTEND=noninteractive …`, not an
export: _pkg_sudo runs plain `sudo "$@"` and sudo's env_reset would drop an
exported variable before apt saw it. --force-confold/--force-confdef cover the
other prompt class (modified conffiles), which DEBIAN_FRONTEND does not
suppress.
Tests: pkg_install must be non-interactive for every manager, and pkg_add's apt
branch is guarded statically. Three existing assertions pinned the exact apt
argv (`install -y <pkg>`) and now match the package instead — a flag added to
pkg_add should not fail tests whose subject is which package got installed.
Verified across the whole distro matrix: ubuntu:22.04 went from a 60-minute
hang to 7 passed in minutes; debian:13/12, ubuntu:24.04, fedora:40,
rockylinux:9 and opensuse/leap:15.5 all 7 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven distro images were green throughout issue #99 because every one of them
runs the unit suites against STUBS. That is the right trade for logic, but it
left the one thing that talks to real package repos completely untested — so a
new Debian release dropping the Python we ask for could only ever be found by a
user.
scripts/test/test_provision_python.sh runs identify_system, choose_pkg_manager
and provision_python for real, then proves the result is usable: in range, can
create a virtualenv, has working pip, and imports ssl/sqlite3/ctypes. It needs
no systemd, nginx or ports, so it fits the plain containers the matrix already
pulls. Reaching the source build counts as a FAILURE, not a pass: a distro that
gets there is one we have not really added support for.
Wired up as a new `python-provisioning` job over the same images, and debian:13
added to both matrices. Adding trixie to the existing matrix alone would have
been decorative — those legs would have gone green on a distro that cannot
install.
Proven in both directions before being relied on: against the pre-fix tree it
reproduced #99 exactly on debian:13 (the reporter's apt errors, the fall
through to a source build, and the abort message), while debian:12 passed 7/7.
Also fixed the suite pinning the bug. T2 asserted `ver_in_range 3.13` must be
FALSE and T1's stub used 3.13 as its "too new" case, so the gate could not be
widened without the tests failing. Both moved to 3.14, plus a direct guard on
PYTHON_MAX — otherwise a regression to 3.12 could be "fixed" by editing the
assertion to match. 122 passed, 0 failed.
full-stack-test.ps1 documents why debian13 is NOT in the VM harness: no trixie
box publishes a Hyper-V provider (generic/debian13 does not exist,
debian/trixie64 is libvirt-only, bento/debian-13 has no hyperv) and Multipass
is Ubuntu-only. Named there with what remains uncovered on trixie — the full
systemd + nginx + API end-to-end — rather than left as a silent gap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
install.sh wrote no TRUST_PROXY_HEADERS line at all, so the code default (False)
applied while every host install serves the panel through its own nginx. Per-IP
rate limits, the login brute-force throttle and every audit-log entry therefore
keyed on 127.0.0.1, which quietly weakened all three -- one abusive client shared
a bucket with everyone else. .env.example claimed true, so the two disagreed and
it looked configured.
Trusting X-Forwarded-For is only safe when nothing can reach gunicorn without
traversing a proxy we control, so the switch follows that condition rather than
assuming it. On a host install it holds: render_service_unit binds gunicorn to
127.0.0.1 and bootstrap_firewall opens SSH + 80/443 + PANEL_PORT and never the
backend port. backend_bind_host() is now the single source of truth for both the
systemd unit and the trust decision so the two cannot drift, and
backend_is_loopback_only() gates the setting.
- stock install: TRUST_PROXY_HEADERS=true, TRUSTED_PROXY_HOPS=1
- SERVERKIT_EXTERNAL_PROXY=1: true with 2 hops, unchanged
- SERVERKIT_BIND_HOST not loopback: false plus a warning. The port is directly
reachable there, so the header is attacker-chosen, and a forged IP is worse
than a wrong one because it defeats the throttles that consume it.
Re-runs backfill it into installs that predate this, via _env_default_key, which
fills a gap but never overrides an uncommented value -- flipping someone's
deliberate false back on would be the worst kind of surprise.
.env.example moves to false on purpose: that file is the Docker/manual path and
docker-compose.yml publishes 5000 on all interfaces, so true was the unsafe
default there. Host installs never read it. SECURITY.md claimed "the shipped
nginx deploy sets it", which was untrue until now.
T44 covers all six behaviours, each mutation-checked by line number: disabling
the branch, inverting it to trust an exposed port, and disabling the backfill
each fail exactly one assertion and no others.
Shell suite 121 passed. Backend suite 3542 passed, 48 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Make "someone else terminates TLS in front of the panel" a supported install
shape instead of something you reverse-engineer from this script (issue #96).
SERVERKIT_EXTERNAL_PROXY=1 implies SERVERKIT_SKIP_SSL=1, so the plain-HTTP nginx
site is selected and there is no 301 for the external proxy to follow back into
itself; it also writes TRUST_PROXY_HEADERS=true and TRUSTED_PROXY_HOPS=2 (their
proxy appends the real client, our nginx appends their proxy) and asks for
SERVERKIT_PUBLIC_URL, deriving it from PANEL_DOMAIN when it can and warning
rather than blocking when there is no tty. Without that URL the Socket.IO
handshake is rejected on origin and the UI loads but never live-updates.
It applies on re-runs too, via a new _env_set_key helper. "Convert my install to
sit behind Caddy" is a re-run and lands in the existing-.env branch, which
previously only refreshed the ssl mode and profile -- so the documented
conversion path would have silently done nothing. A plain re-run never removes
proxy settings someone configured by hand.
SERVERKIT_CONFIG names a KEY=VALUE file of defaults for the same variables. It
is PARSED, not sourced: sourcing would let file assignments overwrite the
environment, inverting the documented precedence, and would execute anything
else in the file as root. Applying keys one at a time and skipping names already
set gives the stated semantics safely.
scripts/test/test_install.sh T43 covers all of it, including that a command
placed in the config file does not run, that the converted .env flips
SERVERKIT_SSL_MODE to insecure (config resolves the env var before the file, so
writing /etc/serverkit/ssl-mode alone would leave HSTS on), and that a default
install is untouched. T12 gained guards for the compose and Dockerfile
regressions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by the full-stack VM harness, which 3524 unit tests and a 7-image
container matrix had all missed. On a real Ubuntu 24.04 install the log read
"Opening HTTP/HTTPS (80, 443) via nftables" and not one line of
bootstrap_firewall appeared: it returned at its "a firewall is already active"
guard, so the box got its two ports opened and no default-deny policy at all.
The headline feature of that work was a no-op on exactly the population it was
written for, and it looked green the whole way.
The guard asked `[ "$(firewall_detect)" != "none" ]`. firewall_detect answers a
different question — "which backend do I open a port THROUGH" — and returns
`nftables` whenever the nft binary exists and can list a ruleset, including an
empty one. That is true of every modern Linux, so the guard reported every box
as already protected. Verified directly against a stock-image fixture: same
inputs, `firewall_detect` -> nftables, inbound-default-deny -> unprotected.
firewall_inbound_default_deny() asks what the bootstrap actually cares about:
does this host already deny inbound by default? Presence of rules is
deliberately NOT the test -- Docker installs plenty of nft and iptables chains
and protects nothing. A default-deny policy on the input hook is: firewalld
running, ufw active *with* "deny (incoming)", an nft base chain hooked to input
with policy drop or reject, or an iptables INPUT policy that is not ACCEPT.
Unknown answers 1 (unprotected), because the caller's response is to offer
protection and every caller re-checks before acting.
firewall_detect is left alone -- configure_firewall legitimately wants a
backend name for a binary that exists -- and both functions now carry a comment
saying which question they answer, since that is what was conflated.
Six cases pin the behaviour, five of them shapes the old guard got wrong: an
empty nft ruleset must run the bootstrap, Docker's chains must not read as
protection, an active-but-allow-inbound ufw must be fixed rather than
respected, a real nft policy drop must be left alone, and the two predicates
must disagree on a stock image -- that last one asserted at the predicate level
so the distinction survives a refactor of either function. make_stub_stdout is
new because an exit-code stub cannot express "the tool ran fine and reported
nothing", which is the whole stock-image case.
Green on all seven CI images (265/265/265/275/275/275/263, 0 failed).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 73 item 8. The installer opened 80/443 on whatever firewall it found and
shrugged -- "assuming ports 80/443 are already open" -- when there wasn't one,
which is the state of a stock Ubuntu/Debian VPS. That box then ran ServerKit
with every port a deploy maps onto the host (8001-8999, plus database
containers on their stock ports) facing the internet. fail2ban only shipped
with the "full" capacity profile, so the typical small VPS got neither.
The dangerous half is enabling default-deny without allowing the port sshd
actually answers on, over the SSH session running the installer. detect_ssh_ports
therefore never guesses: it asks `sshd -T` (the effective config, which resolves
the Include that Ubuntu 22.04+ ships), then sshd_config plus its drop-ins with
comments stripped, then a live `ss`. Multiple Port lines all come back. 22 is
used only when a config exists and names no port -- sshd's compiled-in default
-- and if nothing answers at all the function fails and no firewall is
installed. An exposed box is recoverable; a locked-out one usually is not.
That interlock had a real bug in its first draft, caught before it shipped and
now pinned by a named regression test: most boxes have no sshd_config.d/, the
unmatched glob makes cat exit non-zero, and under `set -o pipefail` that wiped
a correctly parsed answer and fell through to the 22 default -- returning 22 on
a box whose sshd is on 2222, which is precisely the lockout this function
exists to prevent.
Ordering is the other safety property and is asserted: every allow rule (all
detected SSH ports, 80, 443, and PANEL_PORT when non-standard) lands before
`default deny incoming`, and the policy before `--force enable`. The bootstrap
refuses to act inside a container, when a firewall is already active (it is the
operator's -- opening ports on it is fine, rewriting its default policy is not),
under SERVERKIT_SKIP_FIREWALL=1, and when a previous run bootstrapped but the
firewall is off now, since the operator turned it off deliberately. Non-Debian
installs are byte-identical to before: RHEL-family boots with firewalld already
active, and a firewalld default-deny bootstrap for boxes where it is inactive
is marked as an explicit follow-up rather than guessed at.
fail2ban moves to the standard profile with an sshd jail that watches the same
ports detect_ssh_ports found -- a jail pinned to :22 on a box listening on :2222
bans nobody -- and declares `backend = systemd` when there is no auth.log to
tail, without which the jail refuses to start and fail2ban looks installed while
protecting nothing. certbot stays full-only.
setup-health gains firewall (critical) and fail2ban (recommended) items. The
split is deliberate: this file grades critical by silent breakage, and an
unfirewalled box qualifies -- the operator set it in motion by deploying,
nothing in the UI says the port is public, and the failure mode is compromise
rather than an error. A missing fail2ban thins defence in depth but breaks
nothing silently, since the panel already throttles logins per IP. Both are
gated on actually owning the host -- Linux, /etc/serverkit present, not
containerised -- so a dev checkout or CI runner drops them entirely instead of
reporting a fake ok that would inflate the score.
SetupHealthWidget was orphaned; so, it turns out, was its stylesheet, which
main.scss never imported. Both are wired up, as a plain admin-only card above
the board rather than a placeable widget: an unfirewalled box has to be visible
without the operator first opting into showing it. It collapses to one line
when everything is clean.
Implementing this inverts a shipped test that asserted provision_hardening was
a no-op on the standard profile; it is rewritten to the new contract rather
than deleted, with a minimal-profile case added alongside.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Piped execution leaves BASH_SOURCE unset, so the sourceability guard's bare
${BASH_SOURCE[0]} was an unbound variable under `set -u` and aborted the
install before main() ever ran:
bash: line 2107: BASH_SOURCE[0]: unbound variable
This broke the one-liner README/DEPLOYMENT document, on every distro — not
just the Debian 13 it was reported on (reproduced on debian:13 bash 5.2.37,
ubuntu:22.04 bash 5.1.16, almalinux:9 bash 5.1.8). CI never saw it because
every job runs `bash install.sh` from a checkout, where BASH_SOURCE IS set.
Use the same non-empty-first guard scripts/install.sh already carries, and
add the proving test the panel installer was missing (T41) — the agent
installer has had its twin since the same fix landed there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Juan Denis