1577 Commits
Author SHA1 Message Date
Juan Denis 2932006c70 Merge pull request #84 from jhd3197/dev
Make the updater's pre-upgrade backup an actual safety net
v1.7.74
2026-07-27 00:40:20 -04:00
github-actions[bot] 63b835c355 chore: bump version to 1.7.74 [skip ci] 2026-07-27 04:34:40 +00:00
Juan Denis 862b093ffc Merge branch 'dev' of https://github.com/jhd3197/ServerKit into dev 2026-07-27 00:34:31 -04:00
Juan DenisandClaude Opus 5 42905c3572 ci: fall back to a Hub mirror when Docker Hub is unreachable
The retry added in 1f2381c behaved exactly as designed and still failed the
job: Hub was unreachable for the full two minutes ("Client.Timeout exceeded
while awaiting headers" on every attempt), not flaky. Retrying a single
registry harder cannot survive that registry being down -- it only delays the
failure. Two consecutive runs were blocked this way.

Fall back to Google's public Hub mirror, a separate failure domain, and re-tag
to the original name so the run step needs no changes (`docker run` resolves
the local image; default pull policy is "missing"). Hub stays primary.

Ref construction differs by image kind -- `mirror.gcr.io/library/<img>` for
official images, `mirror.gcr.io/<ns>/<img>` for namespaced ones -- and all
seven matrix images were confirmed to resolve on the mirror via
`docker manifest inspect`, opensuse/leap included.

ONLY the pull falls back. Wrapping `docker run` would re-run the suites on
failure too, quietly turning a flaky TEST green.

Verified locally end-to-end against the step text extracted from this YAML,
under `bash -e`, with a docker shim reproducing the runner's exact Hub error:
  * Hub down, mirror up -> 3 warnings, REAL pull from mirror.gcr.io, re-tag,
    and `docker run <image>` then starts a container with the repo mounted
    while Hub is still blocked
  * both down -> 6 warnings, one error annotation, exit 1 (fails honestly)
  * Hub healthy -> exits 0 on the first pull, no mirror attempt, no noise

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 00:29:15 -04:00
github-actions[bot] 67cbb10c05 chore: bump version to 1.7.73 [skip ci] 2026-07-27 04:11:56 +00:00
Juan DenisandClaude Opus 5 1f2381caa8 ci: retry the distro-image pull instead of failing the job on a Hub hiccup
`docker run` pulls implicitly, so a registry timeout ("context deadline
exceeded" reaching registry-1.docker.io) killed the whole matrix leg with exit
125 before a single test ran, and cost a manual re-run. Seven images per matrix
run makes that a recurring tax rather than a one-off.

Pull up front with backoff (10/20/30s) and only then run the suites.

ONLY the pull is retried. Wrapping `docker run` would re-run the suites on
failure too, which would quietly turn a flaky TEST green -- the opposite of
what this job is for.

Left test-system-utils.yml alone: it uses a job-level `container:`, which the
runner pulls itself, so there is nothing to wrap.

Verified against the step text extracted from the YAML, under `bash -e` (the
shell GitHub actually uses for `run:`): registry down -> 3 retry warnings then
one error annotation and exit 1; flaky -> recovers and exits 0; healthy ->
exits 0 immediately with no retry noise.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 00:11:14 -04:00
github-actions[bot] 45ebb8e878 chore: bump version to 1.7.72 [skip ci] 2026-07-27 04:07:05 +00:00
Juan DenisandClaude Opus 5 5bf5e50f62 fix(update): pick the backup engine by DATABASE_URL, not by which files exist
backup_current tested `[ -f serverkit.db ]` FIRST and only fell through to the
PostgreSQL branch when no file was there. An install migrated SQLite ->
PostgreSQL keeps its old serverkit.db, and both deploy paths copy it forward on
file-existence alone (deploy_source, deploy_release), so the stale file is
carried into every future slot and never ages out. Such a box backed up the
abandoned file, skipped pg_dump entirely, and left PRE_UPGRADE_DUMP unset --
which migrate_database reads as "${PRE_UPGRADE_DUMP:-}", so its auto-restore
silently did nothing. PostgreSQL is the one engine that migrates IN PLACE and
cannot roll back by switching slots, so that dump is its only way home: the
exact safety net a61eab1/b48256f added was disabled for it.

migrate_database has always keyed off DATABASE_URL (its sqlite check). Test the
declared engine first so the two functions agree.

Deliberately NOT `grep DATABASE_URL=sqlite && [ -f db ]` as the reviewer
suggested: backend/config.py defaults DATABASE_URL when absent, so any install
whose .env lacks the line would fall to the else branch and update with NO
backup at all -- trading a mismatched backup for none. An explicit `postgres`
URL is a positive signal; the absence of a `sqlite` one is not.

Also:
  * name the engine when it is one we cannot dump. "No SQLite database ..."
    read like a benign nothing-to-do on a MySQL install that in fact had no
    safety net. Prints the scheme only -- a DATABASE_URL carries a password and
    this text lands in the update log.
  * drop the stale "Python 3.11/3.12 is guaranteed by pre-flight" comment,
    contradicted by locate_sqlite_python two lines below it (664a5e6).
  * discord-notify: skip cleanly when DISCORD_WEBHOOK_URL is absent. Secrets
    are not exposed to `pull_request` runs from a fork, so a merged fork PR
    failed the job on an empty curl URL. Not switching to pull_request_target
    for this -- handing a write-scoped token and the webhook to an
    outsider-triggered run is a poor trade for a chat notification.

Tests: a PostgreSQL install carrying a leftover serverkit.db must still dump
and set PRE_UPGRADE_DUMP (fails on the parent commit, rc=1); an undumpable
engine must be named without leaking the password. Neither needs python3, so
both run on all 7 images. Workflow verified end-to-end against a synthetic
event payload: fork PR exits 0 without calling curl, normal PR still builds the
same embed.

Verified on the full CI matrix (ubuntu 22.04/24.04, debian:12, rockylinux:9,
almalinux:9, fedora:40, opensuse/leap:15.5), all six suites: 0 failed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 00:06:12 -04:00
github-actions[bot] e73e6fc6b4 chore: bump version to 1.7.71 [skip ci] 2026-07-27 03:40:09 +00:00
Juan DenisandClaude Opus 5 664a5e62bd fix(update): don't gate SQLite backup/verify on a 3.11/3.12 interpreter
copy_sqlite_db and verify_sqlite_db reused locate_python, which enforces
3.11 <= v <= 3.12. That bound is correct for rebuild_virtualenv and
preflight_check, but snapshotting or integrity-checking a SQLite file needs
nothing beyond the stdlib sqlite3 module. Reusing it coupled the DATA-safety
paths to the venv-builder constraint and blocked updates for a reason unrelated
to the data:

  * an all-Docker install returns before preflight_check ever runs and does
    every real step in the container -- yet backup_docker_db snapshots with the
    CONTAINER's python and then verified the extracted file with the HOST's.
    On a stock EL9 (python3.9) or Ubuntu 22.04 (python3.10) host that halted
    with "Could not obtain a VALID database backup", blaming the backup for a
    missing interpreter.
  * backup_current halts when it cannot snapshot, so those hosts could never
    update at all.

Split out locate_sqlite_python: any candidate that can `import sqlite3` wins,
and the install's own venv interpreter is preferred -- on a real box it is
guaranteed present, guaranteed to carry the module, and independent of PATH.
locate_python keeps the 3.11/3.12 bound where it belongs.

Surfaced by the scripts-ci unit matrix: almalinux:9 and rockylinux:9 ship
python3.9, so the suite's own probe accepted an interpreter locate_python
rejected, and copy_sqlite_db/verify_sqlite_db failed there while passing on
python3.12 runners.

T10i pins the contract: on a box where every candidate reports 3.9 (locate_python
fails), the helpers still resolve an interpreter, prefer the slot's venv, and
round-trip 100 rows. Fails on the parent commit, passes here.

Verified on the full 7-image CI matrix (ubuntu 22.04/24.04, debian:12,
rockylinux:9, almalinux:9, fedora:40, opensuse/leap:15.5), all six shell suites:
0 failed. test_update is 63 passed / 0 failed on every image carrying a python3.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 23:33:51 -04:00
github-actions[bot] 69b6ffb94a chore: bump version to 1.7.70 [skip ci] 2026-07-27 03:19:39 +00:00
Juan DenisandClaude Opus 5 e97b5dee3b fix(test): stub pg_restore so the pg-dump assertions stop reading host state
verify_pg_dump only runs its structural `pg_restore -l` TOC check when the
binary is on PATH. The harness stubs every other external command it touches
(systemctl, nginx, npm, curl, docker, flask, pg_dump) but never pg_restore, so
two assertions were decided by the host:

  * no postgresql-client -> branch skipped, check degrades to "non-empty",
    and the synthetic `PGDMP fake contents` fixture passes
  * postgresql-client present -> the real binary parses the version bytes
    after the PGDMP magic, rejects the fixture, verify_pg_dump returns 1

That is the CI failure. T10f failed directly; T10e failed downstream because
backup_current correctly halts on an unverifiable backup. update.sh is right --
the tests leaked.

Add a make_stub_pg_restore factory mimicking the real -l semantics (rejects
missing/empty, rejects anything without the PGDMP magic, else prints a TOC) and
install it in T10e/T10f. Give T10f a non-empty non-archive fixture too: that is
the case separating the two branches inside verify_pg_dump -- the size check
passes it, only the TOC listing catches it -- so the path the test was named
for is now actually exercised.

Verified both with and without a pg_restore on PATH: 59 passed, 0 failed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 23:18:21 -04:00
github-actions[bot] 4cb986ecb1 chore: bump version to 1.7.69 [skip ci] 2026-07-27 03:10:47 +00:00
Juan Denis c52927e31f ci: post merged PRs to Discord #dev-updates 2026-07-26 23:03:46 -04:00
Juan Denis b48256fba2 fix(update): dump-proof the backup layer — verify, auto-restore, cap retention
A backup that was never verified is a hope, not a safety net. Closes the
remaining ways the pre-upgrade safety net could silently not exist:

- Every backup is verified immediately after creation: SQLite snapshots via
  PRAGMA integrity_check, PostgreSQL dumps via size + pg_restore TOC
  listing (when available). Invalid artifacts are deleted and the update
  halts — a malformed backup is never mistaken for a good one.
- Partial artifacts from a failed backup are removed (rm on failure), so a
  half-written file can never be 'restored' later.
- A failed IN-PLACE PostgreSQL migration now automatically restores the
  verified pre-upgrade dump (pg_restore --clean --if-exists) before
  halting; without it the half-migrated schema would stay live against
  the old code. If the restore itself fails, the halt message carries the
  exact manual pg_restore command.
- backup_docker_db tries both python and python3 in the container, and now
  verifies the result too — the plain docker cp fallback can no longer
  pass off a torn copy as a backup.
- Backup retention capped: default 5 (SERVERKIT_BACKUP_RETENTION
  overrides) across trees, .db and .dump files. Full-size DB copies pile
  up fast — a real 25G droplet carried 6.3G of stale backups.

Tests: T10f (verification helpers accept/reject correctly), T10g (failed
in-place PG migration auto-restores with the exact pg_restore args),
T10h (retention prunes all three backup kinds to the configured cap).
2026-07-26 19:07:17 -04:00
Juan Denis a61eab1509 fix(update): pg_dump pre-upgrade backup for PostgreSQL installs
Postgres panels updated with zero safety net: backup_current found no
SQLite file and just warned, while the migration runs IN PLACE against the
shared external DB — a failed migration left no way back.

- backup_current now pg_dumps (custom format) when .env points at
  PostgreSQL, sanitizing SQLAlchemy driver suffixes (postgresql+psycopg2://)
  that libpq can't parse.
- Same rule as SQLite: no valid backup, no update — halt when pg_dump is
  missing (install postgresql-client) or the dump fails.

Tests: T10e covers the dump invocation + URL sanitization and the
halt-on-failed-dump path.
2026-07-26 18:56:50 -04:00
Juan Denis 9edb5deb81 fix(update): detect unquoted VIRTUAL_ENV binding in prebuilt venvs
The venv binding check only matched the quoted VIRTUAL_ENV="/path" form;
venvs created on some platforms write 'export VIRTUAL_ENV=/path' unquoted,
so the binding read as 'unknown path' and always triggered a rebuild.
Now matches both forms (and skips cygpath-style indirection), so fresh
installs keep the fast prebuilt-venv path.
2026-07-26 18:42:10 -04:00
Juan Denis c0b9245102 fix(update): snapshot the live SQLite DB consistently (online backup API)
Root cause of a real-world failed update: backup_current, deploy_source and
deploy_release all plain-cp'd the LIVE panel database. On a busy panel that
races the backend's own writes and the copy can capture half a transaction
— 'database disk image is malformed' — which then sinks the migration in
the new slot and silently poisons the pre-upgrade backup that was supposed
to be the safety net.

- New copy_sqlite_db() uses the SQLite online backup API via python
  (guaranteed by pre-flight) and refuses to fall back to a racy cp.
- backup_current halts rather than proceed without a valid safety net.
- deploy_source/deploy_release halt before any switch if the live DB
  cannot be snapshotted consistently.
- backup_docker_db prefers a consistent snapshot via docker exec python,
  with the plain docker cp as a fallback.
- locate_python rejects candidates whose version probe returns empty
  (broken shims pass 'command -v' but can't execute; an empty version
  string used to pass the sort-based range check).

Tests: T10d exercises a real snapshot (integrity ok, all rows); a contract
test bans plain-cp of the live DB; T10d fixture python detection validates
the interpreter actually runs.
2026-07-26 18:35:21 -04:00
Juan Denis 54ad479561 Merge pull request #83 from jhd3197/dev
Enforce File Manager permissions and stop shelling out as root
v1.7.68
2026-07-26 17:18:21 -04:00
github-actions[bot] 27e8004ce9 chore: bump version to 1.7.68 [skip ci] 2026-07-26 20:36:05 +00:00
Juan Denis 66b3252b17 Merge branch 'dev' of https://github.com/jhd3197/ServerKit into dev 2026-07-26 16:35:56 -04:00
Juan Denis 146113bd82 fix(update): integrity probe distinguishes corruption from probe failure
The SQLite pre-check in migrate_database treated an EMPTY probe result
(unusable venv python) the same as real corruption and halted the update —
a false positive that broke the T10/T24 contract tests. Now:

- probe output != 'ok'  -> genuine corruption -> halt with repair guidance
- empty probe output    -> probe couldn't run -> warn and let flask surface
                          any real error

test_update.sh: T10/T24 fixtures stub the venv python (healthy-DB path);
new T10b asserts the corrupt-DB halt fires before flask runs; new T10c
asserts an unrunnable probe warns and proceeds.
2026-07-26 16:35:39 -04:00
github-actions[bot] 0129d7a13a chore: bump version to 1.7.67 [skip ci] 2026-07-26 20:28:42 +00:00
Juan Denis e3c4fa778d fix(files): address PR review — literal grep match, tail clamp, extend RBAC coverage
- log_service: grep the service/unit filter with -F so names like
  'nginx.service' are matched literally instead of as a regex.
- log_service: clamp the in-Python tail like 'tail -n' — lines=0 sliced
  as [-0:] == all lines, negatives kept nearly everything.
- test_files_rbac: add files/upload and files/s3/upload to WRITE_CASES,
  and disk-usage/disk-mounts/analyze/type-breakdown to READ_CASES, so the
  permission gates on the full /api/v1/files surface are covered.
2026-07-26 16:25:12 -04:00
Juan Denis 88295ab000 fix(update): harden migration step against boot side-effects and slot venv mismatch
- create_app(): skip background daemons (queue consumers, metrics, job
  system, analytics flush, linked-panel client) when loaded by a Flask CLI
  one-shot such as 'flask db upgrade', or when SERVERKIT_SKIP_BACKGROUND=1.
  They query the DB before migrations run, so any error there (corrupt DB,
  pre-migration schema) aborted the migration and sank the whole update.
- update.sh: set SERVERKIT_SKIP_BACKGROUND=1 for both flask db upgrade
  invocations as an explicit contract.
- update.sh: fail fast with an actionable message when the SQLite slot copy
  fails PRAGMA integrity_check, instead of dying mid-boot with 'database
  disk image is malformed'.
- update.sh: rebuild the release tarball's prebuilt venv when its baked
  VIRTUAL_ENV path resolves to the old slot (it is baked as
  /opt/serverkit/venv), which previously ran new code on the old slot's
  dependencies during migration. Resolved-path comparison keeps the fast
  prebuilt path for fresh installs.
2026-07-26 16:18:37 -04:00
github-actions[bot] b6f689d572 chore: bump version to 1.7.66 [skip ci] 2026-07-26 19:45:24 +00:00
Juan Denis e4d35bffae feat(files): hide File Manager nav when files.read is revoked
Now that every /api/v1/files endpoint enforces the files.read
permission, a user with that permission revoked would only hit 403
toasts. Filter the Files item out of the sidebar and the command
palette (usePaletteAuthz) instead, using the resolved permissions
already returned by /api/v1/auth/me.
2026-07-26 15:40:22 -04:00
Juan Denis ddf2e948d1 fix(security): enforce files.read on reads; de-shell syslog unit filter
Follow-ups from the GHSA-4wqh-7f4f-5qmx / GHSA-mc93-rc3x-fpgq review:

- File Manager read endpoints (browse/info/read/search/disk-*/analyze/
  type-breakdown/download + s3/browse, s3/read, s3/download-url) only
  required a valid JWT, so a user whose files.read permission is revoked
  via custom permissions could still read and download files. They now
  require permission_required('files', 'read'), matching the write side.

- LogService._read_syslog() interpolated the unit name into a bash -c
  string using subprocess.list2cmdline(), which is cmd.exe-style quoting
  and does not stop $(...)/backtick expansion under bash. grep now runs
  as a plain argv list (with --) and the tail happens in Python.
  Reachable only on non-systemd hosts by admins, but the same bug class
  as the Postfix advisory.

Regression tests: revoked files.read 403s every read endpoint; the
syslog filter never invokes a shell and passes payloads verbatim.
2026-07-26 15:40:14 -04:00
Juan DenisandAruvasgaChithan 696226002b fix(security): enforce files.write permission on File Manager mutations (GHSA-4wqh-7f4f-5qmx)
Mutating File Manager endpoints (write/create/mkdir/delete/rename/copy/
move/chmod/upload and the S3 equivalents) only required a valid JWT, so a
viewer account (files.write=false) could write and delete files. They now
require the files.write permission via permission_required(). Read
endpoints are unchanged. Adds regression tests covering both advisories.

Co-authored-by: AruvasgaChithan <AruvasgaChithan@users.noreply.github.com>
2026-07-26 15:19:22 -04:00
Juan Denisandtonghuaroot 20e79ff363 fix(security): prevent command injection via hostname in Postfix install (GHSA-mc93-rc3x-fpgq)
The hostname parameter was concatenated into a bash -c string executed as
root via run_privileged(). Validate the hostname against a strict charset
and pipe debconf selections via stdin so no user input ever reaches a
shell command line.

Co-authored-by: tonghuaroot <tonghuaroot@gmail.com>
2026-07-26 15:19:08 -04:00
Juan DenisandClaude Opus 5 2f34ba578d fix(deps): unpin postcss override so security patches can resolve
The exact `overrides.postcss: "8.5.16"` pin was the sole constraint
holding postcss at a vulnerable version -- vite@8.1.4 (the only
dependent) already declares `postcss: ^8.5.16`, so the override was
redundant. It also made Dependabot fail with
`security_update_not_possible`, reporting the pin back as though vite
required it.

Removing it resolves postcss to 8.5.23, above the 8.5.18 patched
floor for GHSA-r28c-9q8g-f849 (path traversal in previous-source-map
auto-loading).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 12:07:39 -04:00
Juan Denis 0be785fcaf Merge pull request #82 from jhd3197/dev
Rebuild the dashboard and databases, and open the panel to extensions
v1.7.65
2026-07-26 11:55:14 -04:00
github-actions[bot] d83d812a02 chore: bump version to 1.7.65 [skip ci] 2026-07-26 15:17:47 +00:00
Juan DenisandClaude Opus 5 e172289850 docs(screenshots): show the app, not just the page
The dashboard shots were clipped to `.dashboard-page`, which cropped the
sidebar out and made them the only images in the set without the app chrome
around them — next to Services and Docker in the README it read as a bug.

The clip was there for a reason: `.main-content` owns the scroll, so a plain
shot stops at the viewport and `fullPage` does not reach the bottom widgets.
The fix is a per-section viewport instead — a taller frame captures the whole
board WITH the navigation, and `capture.mjs` now honours `section.viewport`
for any page with the same internal-scroll shape.

Also stops the shots claiming work is in flight. The dashboard fixtures
answered the deploy pill with running jobs, so every capture carried a
"2 deploys running" pill over the sidebar. The pill is correct behaviour; it
just has nothing to do with a screenshot of an idle board, so the fixture now
reports settled jobs rather than the CSS hiding a true statement.

31/31 sections still render.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 11:16:44 -04:00
github-actions[bot] 30f5ca71da chore: bump version to 1.7.64 [skip ci] 2026-07-26 14:34:50 +00:00
Juan Denis f935fd4b6e Merge branch 'dev' of https://github.com/jhd3197/ServerKit into dev 2026-07-26 10:34:35 -04:00
Juan DenisandClaude Opus 5 640e7e827e docs(screenshots): recapture, and fix two bugs the capture exposed
Six new sections cover the pages rebuilt this round — the dashboard as it ships,
its edit mode, the widget library, the widget editor, the databases explorer and
the engine catalog drawer. 31/31 sections render.

The capture is a render smoke test, not just a picture, and it earned that here
by surfacing two real bugs that no test could see:

`SearchField` merges its `className` onto its OWN root, so `.skw-lib__search`
IS the `.sk-searchfield` element rather than a wrapper around one. The rule
styling it as a parent matched nothing, leaving the shared `flex: 1 1 240px`
in force — and inside a column-direction filter bar that 240px basis is a
HEIGHT. The widget library's search box was rendering 240px tall.

`GET /auth/me` answers `{user: {...}}`, but Databases read `.role` straight off
the envelope, so `isAdmin` was always false. Every admin got the read-only
console — and with the DDL builder now shipped, would never have seen the button
that runs a CREATE TABLE.

Also carries `postgresql.yaml`, which I left out of the extensions commit.
Without `engine.extensions: true` stock PostgreSQL is not a host for pgvector at
all; the extension tests passed only because the working tree had the change the
commit did not.

Fixture note for whoever edits these next: `/metrics/history` points are NESTED
(`cpu.percent`, `load['1m']`) and carry no network data, `/servers/available` is
a bare array, and `DeployPill` queries `/deployment-jobs` twice and sums — a
single unfiltered payload makes it claim eight running deploys.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 09:59:06 -04:00
Juan DenisandClaude Opus 5 6edeba2ff5 fix(databases): scope the extension listing to apps the caller can see
`GET /databases/engines/<app_id>/extensions` carried only `@jwt_required()`, so
any signed-in user could enumerate any instance on the panel — the image it
runs, its databases, and what it can load. All facts about someone else's
application.

It gates on `can_access_app` now, mirroring the per-app docker DB read next to
it: owner, admin, or an explicit grant. Read access rather than admin-only is
deliberate — a developer who can already see an app should be able to see what
that app can load. The POST was already admin-gated and is unchanged.

The file also declares its `# Bucket: PER-APP` header (plan 29 #9), which is
what the structural sweeps read to know a per-app route file has been thought
about at all.

Found by the repo's own authz sweeps in a full-suite run — four failures, one
cause. Worth noting they fired on a route that has no feature test capable of
catching this: every functional test authenticates as someone entitled to the
data. That is precisely what a structural sweep is for.

Four tests pin the behaviour: denied without access, still visible to the owner,
404 rather than 403 for an app that does not exist (conflating them would make
the endpoint useless to its own caller), and 401 anonymous.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 00:19:46 -04:00
Juan DenisandClaude Opus 5 679082bf16 feat(databases): engine extensions, without the silent failure
The third deferred item. An extension installs INTO a running engine rather than
getting its own container, so it lives in its own namespace:
`backend/templates/extensions/<id>.yaml` with a top-level `extension:` block,
discovered by the same operator-wins two-directory scan the app templates use.
Dropping a YAML in makes it available with no code change, and a test pins that.

Not nested inside the engine template, because pgvector applies to any
PostgreSQL-speaking engine rather than to `postgresql.yaml` — nesting would force
duplication per host. Not a third `kind:` in the app catalog either: an
extension has no compose, port or volume and produces no Application, so every
consumer of `list_all_templates` would have had to learn to skip it.

The hard part is not the SQL. `CREATE EXTENSION vector` is valid and fails on
stock `postgres:16` with "could not open extension control file", because that
image does not ship the library. Two layers, and the statement is never sent
speculatively: an extension declares which images provide it, so a listing marks
an instance unsupported BEFORE anything is clicked; then at install
`pg_available_extensions` is queried first and a negative answer is a 409 with a
remedy, not an attempt. The live probe overrules the image name in both
directions — a private `registry.internal/ops/pg-with-vector` is allowed, and a
`pgvector/pgvector` tag that does not actually carry the library is still
refused.

ServerKit does not recreate a container behind the operator's back. The remedy
instead names an engine template that IS capable, found by matching the
extension's images against each engine template's compose — hence
`postgresql-pgvector.yaml`. TimescaleDB deliberately ships no bundled host and
says so rather than implying one exists.

Compatibility also needed correcting mid-build: matching on wire protocol alone
claimed CockroachDB could host pgvector. It answers on pgwire and cannot load a
single extension. Engines opt in with `engine.extensions: true`, which excludes
it without any Python naming it.

Two fixes fell out. `DatabaseService` had no psql twin of `docker_mysql_execute`
— `managed_db_user_service` carried a private copy with a comment saying as
much; there is now one implementation and it delegates. And that twin sets
`ON_ERROR_STOP=1`, without which psql reports SUCCESS for a statement the server
rejected.

57 tests, covering the zero-code-change property, compatibility, the image
problem in both probe directions, admin gating, and refusals that must not
execute anything.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 23:44:34 -04:00
github-actions[bot] 371df4d205 chore: bump version to 1.7.63 [skip ci] 2026-07-26 03:23:37 +00:00
Juan DenisandClaude Opus 5 42db5b8053 feat(databases): build the table builder and dump import that were deferred
Both were listed as out of scope for this round. That was wrong about the
importer in particular — the restore routes already existed and only the UI was
missing, so "deserves its own round" overstated a wiring job.

The builder is family-aware like the design mock, but split by what ServerKit
can actually reach rather than pretending uniformity. MySQL, PostgreSQL, SQLite
and containerised MySQL get a column grid and a live preview that is
DIALECT-CORRECT — an integer primary key is `BIGINT AUTO_INCREMENT`,
`BIGSERIAL`, or `INTEGER PRIMARY KEY AUTOINCREMENT` depending on the target, not
the mock's MySQL-everywhere. Engines with no query endpoint (Mongo, Redis,
Meilisearch, Influx, Neo4j) generate their own client's syntax to copy, and say
why they cannot run it. Generating a statement we cannot execute and pretending
otherwise would be worse than the honest limit.

This runs real DDL, which the mock never did, so: identifiers are validated
against a strict pattern AND quoted through `quoteIdent`; defaults never become
SQL — numbers stay numbers, NULL/TRUE/FALSE/CURRENT_TIMESTAMP/now() are an
allow-list, anything else is emitted as a quoted literal; and the statement is
always shown before it runs. Verified against injection probes
(`x'; DROP TABLE t; --` renders as a literal).

The importer targets the real contract, not the mock's. `POST
/databases/{mysql,postgresql}/<name>/restore` takes a SERVER-SIDE path — there
is no upload endpoint — so the UI offers dumps already in the backup directory
or a typed path, and says so rather than showing a file picker that could not
work. Neither restore route snapshots first, which the mock claimed it did, so
"back up first" is an explicit checkbox that runs the real backup route and
aborts the import if it fails. Overwriting a database needs the database's name
typed to confirm.

The opt-in for writes turned out to already exist: `readonly: false` on the
existing query routes, gated on admin inside each one. No new endpoint.

Three limits found and surfaced rather than papered over: non-relational engines
have no query endpoint at all; a containerised PostgreSQL cannot execute
anything, because the docker query path resolves a mysql/mariadb client; and
neither SQLite nor containerised databases have a restore route.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 23:18:57 -04:00
Juan DenisandClaude Opus 5 5927e44487 fix(databases): run all twelve engines before claiming they work
The engine templates were written from upstream docs and reviewed carefully, but
only ever read. Every one has now been brought up for real, authenticated
against, and torn down.

CouchDB was broken. A fresh node never created its system databases: `/_all_dbs`
answered `[]`, `_users` did not exist so no non-admin account could ever be
created, and it error-looped on "Missing system database _users". The image
reads only COUCHDB_USER/PASSWORD/SECRET/ERLANG_COOKIE — both
COUCHDB_COUCHDB__SINGLE_NODE and ERL_FLAGS are rejected as unknown_config_value
— so `single_node = true` ships as a local.d config file instead.

Meilisearch and Valkey advertised versions they do not install (1.0 vs the 1.12
that boots; 8.0 vs 8.1.9), caught by comparing the catalog header against what
actually reported for itself.

The method mattered more than the count. Probing auth from INSIDE the container
is not a test and would have passed two engines that were never checked:
postgres initdb writes `host all all 127.0.0.1/32 trust`, so a deliberately
wrong password succeeds on loopback, and the ClickHouse client reads
CLICKHOUSE_PASSWORD out of the container's own environment. Re-run from a remote
source address, both correctly refuse. Every auth assertion here was made from
off-host.

Nine tests cover what can be checked without Docker, led by the one that matters
most: the generated admin password must actually REACH the image. Mongo,
ClickHouse and CouchDB all start happily with authentication disabled when the
password is generated but never passed — silent, and indistinguishable from
working. Also: every ${VAR} resolves, no port binds all interfaces, the catalog
port matches the published one, the declared version is the tag installed, and a
config file can never be rendered over the data volume's mount point (which
would delete persistence without a word).

Verified by reverting each fix and confirming the tests fail with the right
message, then restoring. 67 passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 22:51:04 -04:00
Juan Denis b5160e90bf chore: clean root and docs — drop recovery artifacts, relocate stale audits
- Remove patch_localhost.py (one-off patch script, served its purpose)
- Remove RECOVERY.md + docs/RECOVERED_COMMITS.txt (July-6 incident artifacts)
- Move completed audits and Jan-era strategy docs to docs/audits/
- Update .gitignore for relocated UI_CONSISTENCY_AUDIT.md
2026-07-25 22:22:33 -04:00
Juan DenisandClaude Opus 5 86e44e1077 fix(deploy): stop the deploy pill shouting at the page it links to
Installing an engine navigates straight to `/deployments/:jobId`. The global
"1 deploy running" pill then appeared in the corner of that very console —
a shortcut back to the page you were already reading.

The pill itself is right; its timing was wrong. It is route-aware now: on
`/deployments` and `/deployments/*` it renders nothing and stops polling.
Deliberately NOT filtered by job kind — hiding it only for database installs
would be the parallel-surface mistake, and an engine install is an ordinary
template install on the ordinary pipeline.

On success the console now offers the way back: "Open <db> in Databases",
labelled with the initial database when the install seeded one, landing on
`/databases?engine=<app_id>` with that engine selected and expanded. The
auto-redirect is guarded three ways — a visible countdown, a "Stay here" button
that kills it for good, and it only arms when this page WATCHED the run finish.
Opening a completed install from the activity feed never navigates under you;
that is reading history, not waiting on a deploy.

Also fixes an affordance that had never once rendered. `DbBlankStates` read
`instance.deploy_job_id`, `initial_database` and `error_message` — none of which
`_describe_instance()` returns — so "View install log" was dead markup from the
day it was written. Both are now derived from data the API does return: the job
from the deployments feed, the database through the engine's own `database_var`,
so it holds for any engine template rather than ones that happen to spell it
DB_NAME.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 22:19:12 -04:00
Juan DenisandClaude Opus 5 bccc951d5b feat(databases): make "Not installed" a door instead of a sign
The tree announced `MySQL / MariaDB — Not installed` and then offered nothing to
do about it. We ship engine templates for exactly those now, so the row is
actionable: an Install pill on the row itself, an entry in its context menu, and
the empty-children line reading "Not installed — install MySQL / MariaDB". One
matching template opens the install drawer pre-selected; several open the
catalog pre-filtered.

Candidates are matched on each template's own `engine.protocol`, so MySQL and
MariaDB both satisfy a `mysql` host root without either being named in the UI —
adding a compatible engine template makes it an option here with no code change,
which is the whole point of the template-backed catalog.

A host engine with no installable counterpart now says WHICH kind of nothing it
is — `catalog unavailable` when the backend didn't answer, `no engine template`
when it did and had none — rather than leaving a dead label.

The tree also carries the information the design mock does: child counts per
node, row counts per table, size per database, the engine count on the install
row, and a version badge on installed template engines. Counts come only from
data already loaded, so an unopened node shows nothing rather than a `0`, and
nothing is pre-fetched to fill a badge. The host MySQL/PostgreSQL roots
deliberately keep `Running`/`Stopped`: `/databases/status` returns only
`{installed, running}`, with no version anywhere in that path, so the mock's
`v10.11` is not portable and inventing one would be worse than omitting it.

And an app with no database is no longer an error. `/databases/docker/app/<id>`
answers 400 for an app with no compose to read — a fact about the app, not a
failure to read it — and that surfaced as red "Couldn't load. Right-click to
retry." under a perfectly healthy service. It returns an empty list now, and
each empty state says what is empty: "No databases", "No tables yet",
"No Docker apps".

Also styles `.dbx-tree-node`, which had no rule at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 22:07:35 -04:00
Juan DenisandClaude Opus 5 aa30c859ed fix(databases): the panel could not read the MariaDB it installs
MariaDB 11 dropped the `mysql` symlink in favour of `mariadb`. Every place that
execs a client inside a database container hardcoded `mysql`, so introspecting
any modern MariaDB image failed outright:

    OCI runtime exec failed: exec: "mysql": executable file not found in $PATH

Which is not a hypothetical image. `backend/templates/mariadb.yaml` — the engine
template shipped earlier in this same plan — pins `mariadb:11.4`. Installing
MariaDB through ServerKit produced a database ServerKit could not open, and the
failure surfaced in the tree as a generic "couldn't load".

The client is now resolved per container by probing `--version`, preferring
`mariadb` and falling back to `mysql`, memoised because the answer cannot change
while a container runs. When neither answers it keeps returning `mysql`, so an
unrelated failure still reports the error operators already recognise.

Found by running a real mariadb:11.4 container against the service rather than
trusting the code path; verified end to end (SHOW DATABASES, list databases,
list tables with row counts) before writing the fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 22:01:15 -04:00
Juan DenisandClaude Opus 5 c3e0c9ebf6 feat(dashboard): bring back hovering a chart to read it
The old Recharts graph let you point at a moment and see what every series was
doing there. The ported chart drew lines and nothing else, so a spike was
something you could see but not identify.

Hovering a time series now tracks the nearest sample: a crosshair at that
moment, a dot on each line, and a readout listing every series with its value,
in its own colour, above the time it was sampled. The tooltip flips to the other
side of the crosshair once it would run past the right edge.

The overlay is HTML rather than SVG on purpose. The chart is drawn with
`preserveAspectRatio="none"` so its viewBox stretches to the container — which
would turn every dot into an ellipse. Percentage-positioned elements land on the
same coordinates without inheriting the distortion.

Series of different lengths clamp to their last sample rather than reading past
the end, and the hovered position is stored as a sample index rather than a
pixel so it survives a resize.

`useMetricSeriesSet` now carries timestamps alongside each series; it was
dropping them, which is why a hovered point could not say when it happened.

Left off the stat sparkline deliberately: a tooltip over a 3×2 tile whose value
is already printed above it would cover the number it was explaining.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 21:44:12 -04:00
Juan DenisandClaude Opus 5 fc7bb3a0a1 fix(dashboard): one colour dot per series, not the whole palette every time
Rendering all seven swatches inline on every series row meant a four-series
chart put twenty-eight coloured squares in the settings column, competing with
the controls that actually matter — and pushing the delete button onto its own
line, so each series occupied three rows instead of one.

A colour picker only needs to show the colour. Each series now has a single dot
in its row and the palette opens behind it, so the row is one line again:
colour, resource, metric, delete.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 21:38:49 -04:00
Juan DenisandClaude Opus 5 5180ad4360 fix(dashboard): line up the control row, and let series pick a colour
The bar was a ragged line of mismatched pills: the segmented control, the
refresh select, the icon buttons and Edit each came from a different place with
its own padding and height. Every control in the row is now pinned to 32px and
centred, asserted in one rule rather than trusted from four sources.

Edit, Discard, Add widget and Done were bare `.btn`s, so they floated as loose
labels beside bordered pills. They share the icon buttons shell now.

The refresh select was rendering as an unstyled native control — it kept the
class name of the header that was deleted, and even with the right name it was
missing `appearance: none`, so the browser painted its own light box and system
chevron over everything. Styled properly, with its own chevron and named option
colours so a dark theme cannot produce white-on-white.

Load average shared --accent-bright with CPU, so a chart of both drew two
identical lines. The six metrics now map 1:1 onto the six themeable semantic
tokens; load takes red, which is also what load spiking usually means.

Beyond that, series colour is now yours to set: a swatch row per series in the
editor, offering the semantic tokens rather than hex, so a chart follows
whatever theme is running instead of pinning a colour a skin cannot repaint.

And charts can be stepped again. A sampled metric IS a series of discrete
readings, so holding each value until the next one is arguably the more honest
shape — the squared-off look from before, back as an option next to Legend and
Area fill rather than as the only choice.

Also styles five classes the markup emitted with no rules behind them (TV-mode
bar, empty-board actions, the error hint), caught by sweeping emitted class
names against the compiled CSS.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 21:35:32 -04:00
Juan DenisandClaude Opus 5 62449d7f1f fix(dashboard): give percentages a real 0-100 axis
Charts scaled to their own data, which reads badly in both directions. One
spike to 90% compressed the rest of a CPU series into a flat line along the
bottom, and a disk idling between 0.70% and 0.71% got stretched into dramatic
mountains that meant nothing at all.

A percentage already has an absolute frame of reference, so use it: when every
series on a chart is a percentage the y-axis is pinned to [0, 100]. A spike now
reads as a spike, and a flat metric looks flat because it is.

Only when EVERY series is a percentage. A chart mixing CPU% with load average
keeps scaling to its data — clamping that would bury the load in the floor.
Values outside the domain are clamped into the box rather than painting past it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 21:26:04 -04:00