The floor was --update'd from a dev box again, so CI collected 4516 against
a floor of 4530 and the ratchet went red for a suite that lost nothing.
Same 14 cases as last time (4431 -> 4417): test_builtin_extension_drift.py
parametrises over the on-disk intersection of builtin-extensions/<ext>/backend
and app/plugins/<ext>, and four live copies here are gitignored --
cloud-provision (3), ftp (3), remote-access (5), status (3).
Fixing the number alone would queue up a third occurrence, so the
machine-dependence is closed instead: check_test_count.py now collects with
SERVERKIT_CLEAN_COLLECT=1, and under that flag the drift test pairs only
git-TRACKED live copies. Ordinary runs are untouched and still compare the
gitignored copies on the machine that has them.
Verified with the WSL venv: the ratchet now reports 4518 locally, exactly
CI's 4516 plus the two tests added here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
check-frontend-boundaries failed on dev: _ai-assistant.scss carried 11
colour literals against a legacy baseline of 10.
The extra one came in with the attachment composer as
`var(--warning, <amber literal>)`. `--warning` is not a defined token
anywhere in the tree, so that declaration always fell through to the
literal -- a hardcoded amber that runtime skins could never recolour,
which is precisely what the boundary check exists to catch. (_doctor.scss
reaches for the same undefined token with a *different* fallback, so the
two rendered slightly different ambers.)
$warning resolves to var(--amber, …), the real token. The adjacent line
already uses $warning-raw, so the variables are in scope; $warning itself
must not be fed to fade() since it expands to a var() rather than a
compile-time colour.
File back to 10 literals, matching the baseline without moving it.
Verified: boundary check, theme-token check, npm run lint and npm run
build all pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ratchet only fails on a DROP, so a floor left un-bumped never
complains -- it just quietly stops protecting anything. At 4417 against
4530 collected, 113 tests could have been deleted with CI still green.
113 new tests since the floor was last set: ~35 from the disk-space work
(test_disk_reclaim.py plus the disk --json case) and the rest from the
branch merged into dev.
Verified trustworthy before committing: `pytest tests --collect-only`
(what backend-ci.yml runs) and the bare `pytest` that check_test_count.py
uses both report 4530 -- backend/dev-data/ errors out of collection
rather than adding to it, so the two agree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The raw subprocess census caught disk_reclaim_service as a 25th call
site against a ceiling of 24. The ceiling only goes down, and this is a
case where the shared door is genuinely the better call rather than a
box to tick.
_run's own `except OSError` turned a missing binary into an empty
result, which is exactly the sbin-PATH failure mode: a unit whose $PATH
omits sbin cannot exec a bare `journalctl`, and this sweep would have
reported "nothing to reclaim" on a host that had plenty to reclaim --
a false fact about the operator's disk rather than a visible error.
run_checked resolves argv[0] the way privileged_cmd does, and separates
"never ran" (returncode None) from "answered no". _run keeps its
(ok, stdout, stderr) shape, so callers and tests are unchanged; a
missing binary still degrades to a zero-sized candidate instead of
aborting the scan, but now the reason reaches stderr.
Census back to 24/24; all six ratchets pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings the disk-space work (serverkit disk, telemetry retention, updater
hardening) plus the AI attachment composer, operations run consolidation
and remote-access pickers already on the branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A backup retention cap already existed (SERVERKIT_BACKUP_RETENTION,
default 5), yet a box was found holding six snapshots and 0 bytes free.
Three separate defects, each sufficient on its own:
1. cleanup() is the LAST phase of the update, so retention never ran on
a run that died earlier -- including the run that died BECAUSE the
disk was full. Retention that only executes on success cannot bound a
failure loop. backup_current now trims BEFORE writing, which also
makes peak usage `keep` snapshots rather than `keep + 1`.
2. The preflight required a flat 2 GiB, but the update is about to write
two full copies of the database. With an 800 MB database a box at
2.1 GiB free passed the check and then filled mid-update. The
requirement is now 2 GiB + 2x the database, it trims stale snapshots
and re-checks before giving up, and the halt message points at
`serverkit disk`.
3. A count cap is not a disk guarantee: five snapshots of an 800 MB
database is 8 GB, a third of a 25 GB droplet. The default drops to 3
and enforce_backup_budget caps backups at a share of the filesystem
(SERVERKIT_BACKUP_MAX_PERCENT, default 15), dropping oldest-first and
always keeping at least one restore point -- an update with no way
back is worse than a full disk.
BACKUP_DIR also becomes SERVERKIT_BACKUP_DIR-overridable so it can agree
with the reclaim service, which scans the same directory.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`jobs` has had retention since day one, but the three tables that grow
alongside it never did. `queue_messages` and `system_events` gain a row
per scheduler tick and `api_usage_logs` one per request; measured on a
single-app box that is ~25k rows/day, or ~11 MB/day of database forever.
That is what filled a 25 GB host from nothing but routine updates: the
database reached 809 MB in two months, and every update copies it twice
as its safety net (a pre-upgrade snapshot plus a tree backup containing
it), so each update cost ~1.6 GB.
Adds builtin.telemetry_retention on a 6h tick, keeping
`telemetry.retention_days` (default 30); 0 disables it. It reuses the
prune from disk_reclaim_service, so the same guards apply -- terminal
rows only, and a queue message still referenced by a surviving job is
kept.
It deliberately does NOT vacuum. VACUUM needs an exclusive lock and free
space equal to the whole database, which is never acceptable on a
background tick. Steady-state pruning leaves the freed pages on SQLite's
freelist for reuse, so the file stops growing; `serverkit disk` remains
what actually shrinks it after a backlog.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A panel host filled to 100% (0 bytes free) purely from routine
`serverkit update` runs, and recovering it meant hand-running du and
sqlite. `serverkit disk` measures every reclaimable item and offers them
for selection: numbers, "safe", or "all".
Upgrade snapshots are grouped by timestamp and offered individually with
age and size, since one update writes both a tree backup and a database
copy under one stamp -- half a snapshot is not a restore point. They can
be picked by number, by age (--older-than N, or "older-than N" in the
prompt), or left to the "all-but-newest" default.
Two ordering rules are enforced regardless of selection order:
* filesystem reclaimers run BEFORE the database VACUUM. VACUUM rewrites
the database into a temp file beside it, so it needs free space equal
to the database -- on a full disk it is the one step that cannot go
first. It refuses with a clear message rather than failing halfway.
* `jobs` rows are deleted before `queue_messages`, and any message still
referenced by a surviving job is kept, so jobs.queue_message_id never
dangles.
in_flight messages and pending/failed jobs are never touched. /tmp is
only swept for entries matching mktemp naming, owned by the invoking
user, over 24h old AND holding an unpacked ServerKit tree. Docker prunes
dangling images and build cache only -- never tagged images or volumes.
Sizes are measured rather than estimated wherever measurement is
possible: docker reports its own reclaimed tally (shared layers make
dangling-image sizes overlap), and the journal and package caches report
before/after deltas -- apt-get clean empties the cache before a naive
count could see it, which would otherwise report 0 freed for real work.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
