Commit Graph
351 Commits
Author SHA1 Message Date
Juan Denis c242d4cab5 fix(ssl): a failed certificate check no longer reports fine
Plan 75 §A5. The frontend could always render unknown (a real 'unknown'
state, an n/a chip) — the backend starved it of the signal:

- Doctor's cert check read Domain.ssl_expires_at, a column nothing in
  production writes — structurally incapable of anything but a green
  'No certificates tracked.' Empty now warns: expiry is not monitored.
- openssl -checkend exits non-zero for 'expires soon' AND for
  'unreadable' — an unreadable cert reported needs_renewal=True, and
  the error path vanished the cert from the expiring list entirely.
  Unreadable now returns an error result, and the status endpoint
  surfaces check_errors instead of dropping them.
- A failed monitor cert probe bumped cert_checked_at while keeping the
  old cert_expires_at — a green 'valid / N days' chip from weeks-old
  data next to a fresh checked timestamp. The stale value is cleared;
  the UI falls back to n/a until a probe succeeds.
- get_expiry_alerts dropped unprobeable certs by omission; they now
  appear with severity 'unknown'. get_cert_health reported grade 'F'
  for a DNS timeout; probe failure now yields grade None + error.
- A failed certbot inventory reported '0 certificates, nothing
  expiring' with HTTP 200; list_certificates_report carries the
  errors and the status endpoint exposes them as list_errors.
2026-08-17 22:48:27 -04:00
Juan Denis bb911311ad fix(metrics): unknown is not zero, and zero is not unknown
Two dishonesty directions, one rule (plan 75 §A3/A4): a probe answers a
value or None, never 0 when undetermined — and a real 0 must survive.

Live paths normalized every missing reading to 0 while the history path
stored NULL for the same probe: agent counters an LXC agent can't
determine rendered as a perfectly idle box; load average on unsupported
platforms read 0.00 (and silenced the load alert); CPU freq read 0 MHz;
an empty history window summarized as '0.0 avg'. The mobile summary
called a method that does not exist and rendered 0/0/0 permanently.
One unavailable network counter raised uncaught and froze the entire
live broadcast.

Reverse direction: falsy traps (x if x else None) collapsed a genuinely
idle 0.0 into 'no data' in rollups, averages, and the history model.

Fleet: KPI tiles read '0.0 ms latency' / '100% success' with no data;
never-probed servers exported serverkit_server_up 0 to external
alertmanagers — the up-series now emits only for determined
connectivity. get_all_metrics is per-section fault-tolerant so one
failed probe can't blank the other six.
2026-08-17 22:40:12 -04:00
Juan DenisandClaude Opus 5 fe986c8f35 feat(firewall): guard self-block and default-zone lockouts
The enable() and remove_rule() guards reason about ports. These two paths never
touch the SSH rule and lock the operator out anyway:

  block_ip(<your own address>)          the rule is untouched; your source
                                        address is rejected before reaching it
  set_default_zone(<zone without ssh>)  firewalld re-homes every interface with
                                        no explicit zone; the rules did not
                                        change, the zone around them did

check_ip_block() refuses when the target covers a live SSH peer, the caller's
own HTTP client address, or loopback — nginx forwards to 127.0.0.1, so blocking
it takes the panel down with the session. CIDRs are matched with `ipaddress`,
because 73.244.0.0/16 is the same lockout wearing a wider mask.

Peers come from `ss -Htn state established`, filtered to the sshd ports; `who`
is empty for sessions with no utmp entry. Verified against the live host, whose
only established SSH connection was 146.190.213.37:22 <- 73.244.95.52:59085:

    active_ssh_peers()          -> {'73.244.95.52'}
    block 73.244.95.52          -> refused
    block 73.244.0.0/16         -> refused (covers the same peer)
    block 127.0.0.1             -> refused (loopback)
    block 198.51.100.7          -> allowed

caller_ip is read in the API layer from request.remote_addr and passed in, so
the service stays free of request context.

check_default_zone() asks whether the target zone permits ssh via
--list-services / --list-ports before switching. A failed `ss` probe yields no
peers rather than fabricating conflicts, so a broken probe cannot block every
request. Consistent with the existing guards: undeterminable SSH port refuses,
an unenforcing firewall is a no-op, force=true is the operator accepting it.

/block-ip and /zones/default answer 409 with the conflicting addresses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 19:28:24 -04:00
Juan DenisandClaude Opus 5 7f439e5796 feat(firewall): refuse to remove the last rule admitting SSH
enable() guards the moment the firewall comes up; this guards every moment
after it. With ufw active and denying by default, deleting the rule that admits
SSH severs the very session issuing the delete, and there is no way back in.

Deleting by NUMBER is the sharp edge: the caller passes an opaque index, so the
only way to know what is about to go is to resolve it against
`ufw status numbered` first. Removal by port spec is checked too.

Coverage-aware rather than a blanket block — a duplicate SSH rule may still be
deleted, because another allow rule keeps the port open. Application profiles
(`[1] OpenSSH`) are resolved through `ufw app info`, verified against a real
host: _ports_in('OpenSSH') -> {22}, and '22/tcp (v6)' -> {22} rather than
parsing the address-family suffix as port 6.

deny_port() is implemented as remove_rule(), so it inherits the guard —
denying the SSH port and deleting its allow rule are the same outcome.

Consistent with the enable() guard: an undeterminable SSH port refuses instead
of assuming 22, an inactive firewall is a no-op (nothing is being enforced and
enable() preflights separately), DEFAULT_INPUT_POLICY=ACCEPT short-circuits,
and force=true is the operator accepting the risk.

firewalld: removing service=ssh or the SSH port is checked against
--list-services / --list-ports so other coverage still permits the removal.

DELETE /rules and POST /deny-port answer 409 with the offending rule;
POST /rules/removal-preflight lets the UI warn before the click.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:40:03 -04:00
Juan DenisandClaude Opus 5 454ee552c2 feat(firewall): refuse to enable ufw when it would cut off SSH
`ufw enable` normally prompts "Command may disrupt existing ssh connections.
Proceed?". The service must use `--force enable` because nothing here runs on a
terminal — and --force suppresses exactly that prompt. So the panel had removed
ufw's only safeguard and offered a one-click lockout of a remote box.

Found on a live host: ufw installed and inactive with one staged rule,
`ufw allow 25/tcp`, and sshd on 22. ufw defaults to deny-incoming, so enabling
would have severed the only way back in. Verified against that host:

    ssh_ports()       = [22]
    ufw allowed ports = [25]
    enable('ufw')    -> blocked_by ssh_lockout

check_ssh_lockout() reads the live port from `sshd -T` (authoritative: resolves
Includes, Match blocks and compiled-in defaults) and compares it against
`ufw show added`, the rules that will exist the moment it comes up. Named
application profiles are resolved via `ufw app info` rather than assumed.

An undeterminable SSH port REFUSES rather than falling back to 22 — guessing in
the permissive direction is the lockout this exists to prevent. A DEFAULT_INPUT
_POLICY of ACCEPT short-circuits it, since nothing can be locked out then.

enable(force=True) is the operator accepting the risk; never set by default.
POST /firewall/enable answers 409 (well-formed request, conflicting state) with
the offending port and remediation. GET /firewall/ssh-preflight lets the UI warn
before the click instead of reporting the lockout after it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 10:19:45 -04:00
Juan DenisandClaude Opus 5 33be47b605 feat(monitoring): detect host spec changes and inventory filesystems
A VPS resized from 1 vCPU/1 GB to 4 vCPU/8 GB with a 60 GB volume attached,
and the panel never mentioned any of it. Not a stale cache: a resize forces a
power-off, so the process and its in-memory spec cache die and the next boot
reads the truth. The gap is that nothing ever persisted the PREVIOUS numbers,
so there was nothing to compare against — and the reboot that causes the
change is what destroys the only copy of the baseline.

Adds host_snapshots: capture specs + per-filesystem inventory at boot, diff
against the last row, and emit host.specs_changed / host.storage_advisory on a
real delta. Advisories fire on state transitions only, so an unfixed fstab
entry does not nag on every restart.

Repairs a divergence this exposed: resource_tier_service measured the first of
/var/lib/serverkit,/var/lib/docker,/ while capacity_service hardcoded /. Both
now use host_inventory_service.data_path(), and local headroom reports which
mountpoint it measured.

Notifications render in the existing bell UI, so no frontend change is needed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-17 09:31:18 -04:00
Juan Denis df126515f3 feat(errors): record exceptions via service + admin API + client ingestion
- error_log_service with never-raising dedup record, list/stats/resolve/delete
- /api/v1/error-logs admin blueprint and rate-limited /client endpoint
- 500 errorhandler now records unhandled exceptions without changing the response
- 12 tests covering dedup, filters, resolution, client validation, 500 hook
2026-08-16 18:35:14 -04:00
Juan DenisandClaude Opus 5 3909c34706 fix(api): honest errors for the installer routes, and the right agent repo
Three changes to the endpoints that serve the fleet-agent installers.

A missing installer answered 404. That is what sent issue #101's reporter
looking for a wrong URL -- they filed it as "incorrect paths in setup" -- when
the file had simply never been shipped into the image. The status code produced
the misdiagnosis. It is now 503, the response names the path that was searched,
and the same path is written to the panel log, because the response body does
not survive the `curl -f` the enrollment one-liner uses: curl discards it and
prints only "The requested URL returned error: 404". For an endpoint consumed by
curl, the status code and the server log are the only diagnostics that reach
anyone.

The panel's release lookup pointed at jhd3197/ServerKit and filtered for
`agent-v*` tags, which exist nowhere since the agent moved to its own repo, so
/api/v1/servers/agent/version returned 503 to every agent polling for an update.
It now uses AGENT_GITHUB_REPO (env SERVERKIT_AGENT_GITHUB_REPO) and a ^v(\d…)
tag filter that skips the agent repo's malformed "v" release.

_load_installer() is shared by the .sh and .ps1 routes so they cannot drift, and
SERVERKIT_SCRIPTS_DIR overrides the resolved directory for unusual layouts.

New tests cover the endpoints, the release lookup, and -- the one that would
have caught the original bug -- a packaging invariant that parses the
Dockerfile's COPY sources and asserts every installer the panel serves is
actually copied into the image. No endpoint test could have caught it: pytest
runs against the source tree, where the file always exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 05:27:08 -04:00
Juan DenisandClaude Opus 5 f7be3e0e6e feat(extensions): answer which Python dependencies an install skipped
Plan 55 task 6, backend half. Installing a plugin's requirements runs pip with
the backend's privileges, and a setup.py hook is arbitrary code, so it is
opt-in and off by default -- correct, and the part that was missing is that
nothing ever said so. The skip was a logger.warning: the requirements.txt was
written next to the extension and the panel showed nothing, so an extension
could sit there running with imports it does not have and the operator had no
way to find out short of reading the install log.

pending_requirements() answers it: whether a file is pending, its contents, its
path, the parsed package lines, and the exact env var that changes the
behaviour. GET /plugins/<id>/requirements exposes it, admin-only.

Strictly read-only. Nothing here installs anything -- the plan is explicit that
this surface introduces no auto-install behaviour change -- and a test pins
that by failing if _install_requirements is reached.

The env check the installer inlined is now plugin_pip_enabled(), shared by both
sides. If the surface and the installer ever disagreed about what "enabled"
means, the page would describe a behaviour the installer does not have.

Two honest edges: the content is capped at 64KB before being echoed into a
browser, and pip_enabled is reported alongside pending, because a file written
while the opt-in was off survives turning it on -- the surface reports both
facts rather than guessing which applies.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 17:48:36 -04:00
Juan DenisandClaude Opus 5 bb20db70db feat(extensions): observe permission use, and say which permissions we can't see
Plan 55 task 7, backend half. A manifest's `permissions` array was
declaration-only: the install dialog showed it and nothing ever compared it
against what the extension went on to do.

The capability gate now records every call through it -- including the
refusals, which are the interesting ones -- so `usage_report(slug)` can answer
declared-versus-observed per extension. Observations are in-memory and
per-process by design: the question is "what has this extension done since the
panel started", and persisting it would outlive the evidence and start reading
like an audit trail, which this is not. `has()` deliberately does not record; it
is a question, not a use, and counting it would fabricate activity for callers
that only checked.

The larger half of this change is admitting what the panel cannot see. Task 7
was specified as hooks at "the shell/docker/db helper entry points", and those
do not exist: the SDK exposes no shell, docker, filesystem or network helper at
all, and hands out `db` as raw SQLAlchemy. An extension wanting any of them
imports the host module directly, where no in-process gate is involved. The one
genuinely gated capability is `agent.command:<action>`, because the SDK is the
only way to reach an agent.

So observability is data, not prose: OBSERVABLE_PREFIXES marks what the gate
truly mediates, and the report marks every other declared permission
`observable: false` rather than showing it as "never used". Absence of evidence
is not evidence of absence here, and a UI claiming otherwise would be inventing
enforcement the panel does not have -- exactly the honesty the plan's D2
decision asks for.

GET /plugins/<id>/permissions returns the report, admin-only: it describes what
third-party code has been reaching for.

The UI surface and the docs follow.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 17:44:11 -04:00
Juan DenisandClaude Opus 5 7088298ed7 feat(doctor): connect the fleet doctor that was already written
Plan 72 item A.2. FleetDoctorService, FleetRepairService, the FleetDoctorResult
model, migration 063 and their tests were all complete and reachable from
nothing: no routes, register_jobs() never called, no seeded schedule. The
frontend was further along than the backend -- getFleetDoctorReport() and
runFleetSweep() have been calling /doctor/fleet and /doctor/fleet/run, two
endpoints that did not exist, so the multi-server health story the agent
platform advertises 404'd on contact.

Nothing in those services was redesigned. The one addition is a read-side
aggregator: they could run a sweep and persist rows but had no way to hand
back a per-server report, so fleet_report() composes one. It lists every
registered server rather than only connected ones, which is what makes an
unreachable box a reported state -- last known rows plus connected: false --
instead of a hole in the report. Persisted rows key on check_key while the
UI's field-driven renderer keys on key, so the report emits both.

Three admin-only routes, matching api/doctor.py's gating exactly. The sweep
returns 202 with a job id and never runs on the request thread: it fans out
across agents over the network, and the agent gateway registry is
single-worker and in-memory. Repair is a thin passthrough to FleetRepairService,
which already owns the allowlist, capability gate and audit trail; refusals
come back as per-item results inside a 200, so one offline server cannot fail
a batch. The batch is bounded at 25 so a single request cannot fan an
unbounded number of remote mutations at that single worker.

The schedule is daily like the host doctor but starts 30 minutes behind it,
so the fleet fan-out contends with neither the local sweep nor the boot-time
agent reconnect storm.

The frontend gets a Fleet Doctor section rather than losing its dead methods.
The per-check article was extracted into one component now shared by the host
and fleet lists, so fleet findings render through the same
repairable/repair_ref path instead of a parallel implementation; the confirm
modal says plainly that a fleet repair acts on a remote server through its
agent and is audited.

test_jobs' builtin schedule count moves 21 -> 22 for the new schedule. The
builtin.* kind count is untouched -- this kind is doctor.fleet.run.

Note for anyone reading the suite: tests/test_fleet_doctor_dns_rotation.py is
skipped at module level and proves nothing here. Its rotating-DNS-window API
did not survive the recovery rebuild; that is pre-existing and unrelated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 17:27:59 -04:00
Juan DenisandClaude Opus 5 fc000a3b97 feat(ssl): remember one Let's Encrypt contact instead of asking every time
Plan 73 item 5. The Domains SSL modal asked for a contact email on every
single certificate, and the backend hard-required one per call. The app-detail
Settings tab did remember it -- in localStorage, so the memory was per browser
and invisible to the other surface. The two never agreed, and neither could
help the paths with no form at all: domain attach passed `admin@<host>` when
it had nothing else, an address that usually does not exist, which is where
every expiry notice for those certificates went to die.

There is now a single panel-wide ACME contact in system settings.
get_acme_contact() resolves an explicit address first and falls back to the
stored one; remember_acme_contact() writes it back whenever a request carries
an explicit address. Both are best-effort on the settings side by design: a
settings failure must not break issuance, and failing to remember an address
must not turn a successful certificate into an error. An empty input never
clears a stored contact, since the automatic paths pass whatever they were
given and that is frequently nothing.

Wired into every issuance entry point -- the domain SSL enable route, the
generic certificate route, and domain attach, which now consults the stored
contact before falling back to the admin@<host> guess. The address is only
remembered after issuance actually succeeds; remembering one Let's Encrypt
rejected would be worse than remembering none.

Prefill needed its own endpoint. The settings routes are all admin-gated, but
issuing a certificate for an app you can edit is not an admin action, so
GET /ssl/acme-contact is jwt_required only -- it returns an address the
operator chose for notices, not a credential. The Domains modal and the
Settings tab both prefill from it, and the Settings tab keeps its localStorage
seed so a browser that already has the value shows it without a flash.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 16:22:33 -04:00
Juan DenisandClaude Opus 5 fcc3babe68 fix(dns): catch the stray AAAA record that silently breaks SSL issuance
Plan 73 item 7. Let's Encrypt prefers IPv6 whenever a AAAA record exists, so a
domain whose A record points at this server perfectly still fails HTTP-01
issuance when its AAAA points somewhere else. That shape looked healthy
everywhere in the panel, which is what made it expensive: the operator saw a
verified domain, a green doctor report, and a certificate error naming nothing
either of them had mentioned.

Both DNS surfaces were blind to it for different reasons. The doctor's sweep
returned ok as soon as the server IP was *among* the resolved addresses, so
the extra IPv6 answer never registered -- and that branch is reached precisely
when the A record is correct, i.e. exactly the case worth reporting.
verify_domain used socket.gethostbyname, which is IPv4-only and cannot see a
AAAA record at all.

Both now compare every resolved address against this server: the configured
public IP plus the host's own interface addresses, so a correct dual-stack
setup stays quiet and only a genuine stray is named. Comparison is by parsed
value rather than by spelling, since an expanded 2001:0db8:... form and its
compressed equivalent are the same address, and psutil's zone ids (fe80::1%eth0)
are stripped first. Interface enumeration happens once per doctor sweep
instead of once per domain.

The doctor reports this as a warn, not a fail: the site itself is reachable
and it is a stray record rather than a broken one. Both messages say what to
do -- remove the AAAA record or point it at this server's IPv6 address --
rather than only naming the symptom.

verify_domain keeps its existing response shape. ip_address still carries a
single IPv4 address for the callers that predate this, with the full set added
alongside as ip_addresses, plus aaaa_conflict and a warning string. The
Domains page needed the last part to matter: it printed a green "Domain
verified!" toast for anything that resolved, so a conflict the API now
reports would have gone straight back to being invisible.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 16:17:23 -04:00
Juan DenisandClaude Opus 5 f927b1405e fix(files): make the File Manager's "Web config" shortcut resolve, read-only
Plan 73 item 3. The quick-access rail links the local target at /etc/nginx,
but /etc was never in ALLOWED_ROOTS, so the shortcut returned 403 for every
user including admins -- a dead button, and our own regression: it shipped
after the review that flagged the surrounding SSL story.

/etc/nginx is now an allowed root, but browse-only. The panel generates those
vhosts itself, so a hand edit is silently overwritten on the next domain
change, and a writable /etc/nginx would hand anyone holding files.write a
root-owned config that the panel itself reloads -- a much larger grant than
"let me look at the effective config", which is all the shortcut is for.

READ_ONLY_ROOTS is a second tier under ALLOWED_ROOTS rather than a special
case for this one path: is_path_writable() = allowed AND not read-only, and
every mutating entry point now gates on it instead of on is_path_allowed, so
adding a future root to the list is enough to make the whole write surface
refuse it. Both comparisons resolve with realpath first, so a symlink planted
in a writable root cannot launder a write into the subtree.

copy is deliberately asymmetric -- copying a vhost out to keep only reads the
source, so only the destination is gated -- while move removes its source and
therefore checks both ends. The upload endpoint repeats the check because it
writes to disk directly instead of going through FileService's methods.

Only this subtree is opened, not /etc: /etc/passwd and /etc/shadow stay
denied, and the panel-internal PROTECTED_ROOTS exclusion is untouched
(GHSA-rm3m-9mvw-68fh).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 16:03:25 -04:00
Marcos Paulo Diniz 7a091d440d fix(notifications): reject null updates 2026-08-13 09:28:15 -04:00
Marcos Paulo Diniz e3e399c59d fix(notifications): hide connection test errors
Transport exceptions may contain webhook URLs or bot tokens. Return a stable public error and reject non-object update payloads.
2026-08-13 09:28:15 -04:00
Marcos Paulo Diniz bcccfa139d feat(notifications): select default chat connection 2026-08-13 09:28:15 -04:00
Marcos Paulo Diniz 017795ba08 feat(notifications): test chat connections 2026-08-13 09:28:15 -04:00
Marcos Paulo Diniz c11b302911 feat(notifications): update chat connections 2026-08-13 09:28:15 -04:00
Juan DenisandClaude Opus 5 5f8a3ef204 feat(recycle-bin): applications are soft-deletable, and a delete stops keeping data hostage
Plan 70. Domain went through this and TEN read paths were wrong, four with
effects outside the panel. Application has 211 call sites across 68 files and
drives more -- container lifecycles, nginx, backups, registry pulls, the OS
crontab -- so it was audited in full first (docs/plans/70_...md). 180 sites are
now query_active(); the 35 left unfiltered are each commented in place.

**The delete / purge split is what makes the tombstone honest.** delete_app used
to run compose_down(volumes=True) and rmtree(). Keeping a tombstone AND doing
that would put a Restore button next to a row whose data the delete had already
destroyed -- the one promise a recycle bin cannot break. So:

  DELETE   containers down (VOLUMES KEPT), vhost removed, firewall closed,
           cron suspended, row tombstoned. All reversible.
  RESTORE  vhost rewritten, firewall reopened, cron resumed. Containers left
           STOPPED -- "put it back" must not silently start processes or open
           ports, the same reasoning that stops domain restore re-issuing certs.
  PURGE    volumes removed, uploaded source deleted, cron detached.

register() gained on_purge for that, wired into BOTH purge paths. And
purge_expired had NO scheduler -- fine when a tombstone was just a row, not fine
when it holds data volumes and a source tree, so a daily builtin job now runs
it. Without that, "kept until purge" would have meant kept forever and delete
would have quietly stopped reclaiming disk.

A side effect worth naming: the old _remove_data_flag protection for database
engines is now the DEFAULT for every app, and the flag is carried into
?purge=true so an explicit skip-the-bin delete still honours it.

**Cron was the blocker with no mechanical answer.** before_delete never fires on
a soft delete, and clear_application is LOSSY -- it nulls the association
without recording it, so nothing could put it back. suspend_for_application
instead disables only the jobs that were CURRENTLY ENABLED and stamps each
suspended_by_app; resume re-enables exactly those. That flag is the whole point:
without it a restore switches back on jobs the user deliberately turned off.
clear_application survives as purge-only, where lossiness is correct.

**The indirect reach (audit 3.1) is the class that grep cannot find.**
ContainerSleepPolicy, BackupPolicy, DeploymentJob and WafPolicy are enumerated
on a SCHEDULE and resolve application_id afterwards, so they never appear in a
search for Application.query. Left alone, a tombstoned app's sleep policy is
swept forever and wake_app restarts the containers the delete tore down. Those
loops test Application.deleted_ids().

Deliberately NOT scoped: reservation checks (port, name, directory) -- a
tombstone must keep holding those or a new app takes them and the restore is
refused; api/git.py's webhook VISIBILITY gate, which falls through to
"panel-wide visible" so hiding a tombstone would WIDEN access; and cron
attribution, which only labels a row.

Verified: 15 new tests, the migration replayed on a fresh DB (a tombstone frees
its private_slug, two live rows still collide), and a full delete -> restore
round trip. The 20 remaining full-suite failures are the known
serverkit-wordpress-not-installed class and are identical on a clean tree.

Two traps caught by testing, both recorded in the plan: a bad import made the
ENTIRE purge cleanup silently no-op while purge() still reported success
(_run_purge_hook swallows by design, so assert on the warning, not just ok), and
private_slug had to lose unique=True at the MODEL level or a models-built
database burns the slug regardless of the migration.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 20:26:04 -04:00
Juan DenisandClaude Opus 5 139d5884fe perf(api): stop /status/apps spawning a subprocess per app, and roll up the activity summary
The two worst endpoints in the backend, both measured in plan 69 phase 6.

**/status/apps** ran one `docker compose ps` per app plus a `docker inspect`
per container -- roughly 30 apps x 3 containers = ~120 process spawns per
request -- and it is refilled on a timer, so the cost never amortised. It is
now ONE host-wide `docker ps` whose output is indexed by the compose
project/service/working-dir labels Docker already stamps on every container,
shared between callers through a 3-second in-memory snapshot.

The snapshot is deliberately module memory rather than CacheService: it must be
droppable synchronously, because a status that survives the action that changed
it reads as a broken panel. So `invalidate(app_id)` is called from start, image
apply, deploy and rollback. 3s is short enough that a missed hook self-corrects
within one poll.

**/admin/activity/summary** ran two 90-day loops issuing one COUNT per day =
180 sequential queries. Now one `GROUP BY func.date(...)` per series with the
gaps zero-filled in Python, so the response shape is byte-identical. `func.date`
is portable across the SQLite and PostgreSQL this app supports. A third N+1 fell
out on the way: the top-users block did one SELECT per user, now a single IN.

New tests assert the zero-fill specifically -- a day with no rows must still
appear and the series length must match the window, which is exactly what a
GROUP BY drops if you forget.

1140 backend tests pass. The 3 failures in that run (test_fail2ban_jail_service,
test_url_swap, test_site_routing) all reproduce on a clean tree -- the last is
the known "serverkit-wordpress extension not installed" class.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 18:15:53 -04:00
Juan DenisandClaude Opus 5 2eeb58daa0 perf(api): opt /servers into the query layer
Second endpoint after /apps, same pattern. Server.to_dict grows a `fields`
parameter so $select skips the derived work rather than only trimming the
payload: `onboarding_progress` is a JSON walk that is ~29% of a row and is read
by exactly one screen, and `group_name` is a relationship load. A caller that
needs {id, name} -- Domains and Backups both do -- now pays for neither.

`group_name` is eager-loaded when it IS wanted, so the list stops issuing one
SELECT per server for it.

Two details this route needed that /apps did not: it answers with a bare JSON
array, so no paging envelope is emitted (that would break every caller), and an
explicit $orderby now wins over the hardcoded by-name sort while the default is
preserved when absent.

3 more tests; the 293 existing server/fleet/agent tests still pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 13:16:58 -04:00
Juan DenisandClaude Opus 5 cf7c20864f perf(api): $select/$filter/$orderby/$skip/$top, and opt /apps in
Field selection did not exist anywhere in the backend (`fields=`/`select=`: 0
hits across 90+ API files) and neither did eager loading (joinedload/
selectinload/subqueryload: 0 matches). /apps sends ~2,319 B per row where the
list UI reads ~411, over an unbounded `query.all()` -- for an admin with no
workspace header that is literally SELECT * FROM applications.

Ports the author's conuco_v2 query layer as app/api/_query.py, with two
deliberate differences:

- ORM, not Core. conuco resolves fields through `table.c`; here they resolve
  against the mapper's column set, and that set is the ALLOWLIST -- an unknown
  field is a 400, never a getattr into arbitrary model attributes.
- $select is not only a column narrowing. That is where the win actually is:
  Application.to_dict() costs ~3 extra queries per row, and two of them
  (image_scans, image_update_checks) are lazy='dynamic', so the attribute IS a
  Query and NO loader option can hoist them. Declining a field is the only way
  not to pay for it. to_dict(fields=...) now skips the loads it was not asked
  for, so $select=id,name costs zero extra queries per row instead of three.

/apps also eager-loads server/project/environment/linked_app -- but only when
the request actually asked for the names they resolve, since a joinedload for
a field nobody selected is a wider row for nothing.

Every parameter is optional and to_dict(fields=None) is the full legacy
payload, so adopting this changed no existing caller's response. Paging meta
appears only once a caller pages.

$view is deliberately not implemented: ServerKit's saved views are per-user
rows already resolved in the browser, so a second server-side copy of the same
concept would only be somewhere for the two to disagree.

19 new tests. The 2 failures in the wider run (test_fail2ban_jail_service,
test_url_swap) reproduce on a clean tree -- pre-existing, not from this change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 12:11:39 -04:00
Juan DenisandClaude Opus 5 eee14e6a79 fix(domains): stop soft-delete tombstones leaking into every read path
Making Domain soft-deletable silently changed the meaning of every existing
`Domain.query`: before, it could only return live rows. An audit of all 16 call
sites found ten that were now wrong, several with side effects outside the
panel.

The ones that reached the outside world:

  setup_reconcile_service.py:267  the only is_primary row after deleting the
      primary IS a tombstone -- and it drives a WP-CLI search-replace across
      the whole WordPress database. It would have repointed a live site at a
      host nginx no longer serves.
  setup_reconcile_service.py:83   DNS backfill enumerated deleted domains and
      called ensure_a_record() -- creating a real record at Cloudflare/Route53
      for a domain the user removed.
  api/domains.py:367 (enable_ssl)  a real ACME order for a host that is in no
      server_name: the HTTP-01 challenge cannot be served, burning Let's
      Encrypt failed-validation limits for the whole registered domain, and on
      success installing a wrong-CN cert on the LIVE vhost.
  doctor_service.py:267           permanent red "does not resolve" rows that no
      action could clear, and the same set is the allowlist for one-click DNS
      repair -- so Repair created a provider record for a deleted domain.

And the ones that broke the feature itself:

  domain_attach_service.py:79     a tombstone counted as a name clash, so you
      could not move a hostname to another app after removing it. Migration 083
      made the unique index partial precisely to free the name; this re-imposed
      the burn in application code.
  domain_attach_service.py:71     attach found the tombstone, reported success,
      created no live row, published nothing -- and then mutated a recycle-bin
      record.
  site_domain_service.py:600      same for give_subdomain: success, a URL, a DNS
      record, and a site that was never published.
  api/domains.py:532              a second vhost renderer (admin Regenerate)
      writing tombstones back into server_name.

Application.domains still includes tombstones ON PURPOSE -- filtering it with a
primaryjoin would, with delete-orphan, make SQLAlchemy HARD-delete the row the
recycle bin exists to keep. `Application.live_domains` is the accessor for
"what this app actually serves", and to_dict() uses it so the /apps payload no
longer lists deleted domains.

Per-domain routes now 404 a tombstone instead of editing it.

Seven tests cover the leaks that had real consequences.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 09:27:05 -04:00
Juan DenisandClaude Opus 5 cb1580214a fix(domains): adding or removing one domain no longer drops the site to HTTP
create_domain and delete_domain each called NginxService.create_site directly,
passing no ssl_cert / ssl_key / micro_cache. create_site takes those as
arguments, so the regenerated vhost came out HTTP-only and cache-less: on a
site covered by a base wildcard cert, touching ANY domain silently knocked the
whole site — every other domain on it — back to plain HTTP, and dropped its
micro-cache directives. Invisible until someone loaded the site.

Both now publish through SiteDomainService.write_app_vhost via one local seam,
so add and remove cannot disagree about what the served config should be. That
also fixes two lesser asymmetries: the rewrite was gated on app_type == docker,
so php/static/flask/wordpress vhosts kept advertising a domain after it was
removed; and it never called enable_site.

ALSO FIXES A REGRESSION I INTRODUCED WITH SOFT DELETE. app_vhost_kwargs read
`Domain.query`, which before soft delete could only return live rows and now
returns tombstones too. The vhost renderer would have written a deleted domain
straight back into server_name — the domain would keep being served after
deletion, and drift detection (which renders through this same function) would
report correct config as drifted. Now query_active().

Three tests, each failing against the previous code: the wildcard cert and
micro-cache survive in the resolved kwargs, a soft-deleted domain is absent
from them, and an end-to-end delete leaves the surviving domain on HTTPS.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 09:05:56 -04:00
Juan DenisandClaude Opus 5 f49a3e6cfc feat(recycle-bin): soft delete for records, with restore and purge
Deleting is the only action in a control panel that repeating cannot undo, so
records a person destroys by hand now leave a tombstone and the Recycle Bin
hands them back.

SoftDeleteMixin adds deleted_at/deleted_by_id plus soft_delete/restore and
query_active()/query_deleted(). recycle_bin_service is a REGISTRY: a model
registers with a label function and an optional on_restore hook, so the API
lists and restores across types without knowing what a domain is. Adding the
next soft-deletable model is one register() call.

Applied to Domain (a real, destructive case -- deleting one used to lose the
record outright) and SavedView. DELETE /domains/<id> now tombstones and still
tears the vhost down; restoring a row does NOT re-do side effects unless the
type supplies on_restore, which is the honest default.

API: GET /recycle-bin (all kinds, newest first), POST
/recycle-bin/<kind>/<id>/restore, DELETE /recycle-bin/<kind>/<id> (purge,
admin-only -- the one irreversible action), POST /recycle-bin/purge-expired
with a 30-day default retention.

THE PART THAT NEEDED CARE: both tables had a plain UNIQUE covering what is now
a tombstone, so deleting a domain would have permanently burned its name and
deleting a view would have blocked re-creating one by the same name. Migration
083 replaces each with a PARTIAL unique index predicated on deleted_at IS NULL
-- supported by SQLite 3.8+ and Postgres. Tests pin exactly that: a deleted
name can be reused, and the live row stays unique.

Also gives SavedView a `slug` (backfilled), the handle the upcoming
?view=<slug> links resolve against. A tombstoned view does not reserve its
slug, so deleting a view frees the link name too.

Migration is idempotent on both columns and indexes: `deleted_at` is
index=True on the mixin, so a database built from the models already has that
index, and a bare create_index would abort the upgrade on exactly the installs
we most want to keep working.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 07:14:55 -04:00
Juan DenisandClaude Opus 5 3d9bf09bea fix(api): GET /servers never sent the metrics the servers list renders
The CPU/Memory/Disk gauges read `server.metrics`, the same key the
per-server status endpoint uses (servers.py:757). The list endpoint never
populated it -- to_dict() defaults to include_metrics=False, and even when
asked it emits `latest_metrics` -- so on a live panel every gauge cell fell
back to the "no data" dash.

It survived because the screenshot fixtures DO supply `metrics`
(sections.mjs:294): the UI looked correct in every capture and was empty in
production. The fixture had encoded the intended contract; the backend never
implemented it.

Batched deliberately: Server.metrics is lazy='dynamic', so the obvious
per-row lookup costs one extra SELECT per server. Newest row per server =
highest id (autoincrement on insert), which also avoids the tie a
max(timestamp) join hits for same-second rows -- two queries total,
regardless of server count.

Tests pin all three properties, and each fails without the fix: the key and
field names the table reads, absence (not {}) for a server that never
reported, and that the query count stays flat as servers are added.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 05:52:37 -04:00
Juan DenisandTyler Boykin 9fb0a6cab5 fix(security): close git argument injection in repo_url/app_path/branch (GHSA-8vx6-432p-h62q)
An authenticated user of any role could inject arguments into git
subprocesses running as the backend user (root under systemd):

- repo_url was passed unvalidated to git ls-remote/clone, enabling the
  ext:: transport (host command execution on git < 2.52 defaults) and
  file:// local-repository disclosure (all git versions).
- app_path on POST /deploy/clone was passed without a '--' terminator,
  enabling --upload-pack=<cmd> option injection independent of git
  version and protocol.*.allow policy.
- branch flowed into 'git fetch origin <branch>' as an unterminated
  refspec.

Fix (new app/utils/git_security.py applied at every affected sink):

- repo_url scheme allowlist (https/http/git/ssh + scp-like SSH);
  rejects ext::/file://, local paths, leading '-', whitespace.
- '-c protocol.ext.allow=never -c protocol.file.allow=never' pinned on
  every remote-touching git call, plus a restrictive GIT_ALLOW_PROTOCOL
  in the subprocess environment.
- '--' positional terminator before URL/path/refspec arguments —
  co-load-bearing with protocol pinning, which alone does not stop
  option injection such as --upload-pack=<cmd>.
- branch/ref validation (leading '-', ref-format metacharacters) for
  clone --branch, fetch refspecs and stored deploy configs.
- POST /deploy/clone now confines app_path to the managed apps root,
  mirroring how /apps/from-repository derives it server-side.

Adds 31 regression tests; bumps version to 1.7.86.

Reported-by: CaptBoykin
Co-authored-by: Tyler Boykin <25166954+CaptBoykin@users.noreply.github.com>
2026-08-11 19:43:46 -04:00
Juan Denis ede0f0fdbd feat(activity): audit feed filters by target_id
AuditService.get_logs gains a target_id filter (paired with target_type) and
/admin/activity/feed passes both through — the per-resource activity
timeline can now ask 'what happened to THIS app'.
2026-08-09 13:53:59 -04:00
Juan Denis b6c257bfc7 feat(views): per-user saved table views API
SavedView model (user/page/name/state JSON/is_default, unique per
user+page+name), idempotent 082 migration, saved_view_service with
one-default-per-page enforcement, and a JWT-scoped /api/v1/views CRUD
blueprint. Covered by 9 endpoint tests.
2026-08-09 02:32:55 -04:00
Juan Denis 7c6b0f3958 security: hook seam lifecycle + registrant binding
Audit remediation (extension seams):
- disable/uninstall tear down the extension's backup kinds, event
  types, and template providers (unregister_capabilities by slug);
  enable re-registers (idempotent)
- wordpress_bridge fails closed unless the extension row is ACTIVE;
  orphaned-site db snapshot route returns clean 503
- provider registrations bound to their registrant: cross-slug
  replace/hijack rejected for backup kinds, event types, template
  providers
2026-08-05 13:36:33 -04:00
Juan Denis 670e9d8af7 security: signature lifecycle, update-path consent gates, download hardening
Audit remediation (install/signing surface):
- sig verdict reset on every install path (no stale verified badge)
- registry update route mirrors install consent gates (unreviewed 404,
  unsigned-first-party/untrusted 409 until acknowledged, bad sig 400);
  update offers carry installed_source_type; frontend risk dialog gains
  update branch
- first_party entries without verified pinned-key signature require
  acknowledge (community reviewed exempt, review-stamp-bound)
- downloads: https-only with per-hop redirect revalidation, private/
  link-local IP rejection, 100MB zip + 64KB sig caps, github token
  https-only
- extraction: declared+actual 250MB decompression cap with cleanup
- install/update audit logs record the signature verdict
2026-08-05 13:36:20 -04:00
Juan Denis 72eaba34f2 refactor(wordpress): decouple core from the in-tree extension
Bridge, blueprint registration, app/attach flows and builtin job
handlers no longer assume the WP builtin is present; WP install paths
gitignored like other extracted extensions. WP absent = feature absent
gracefully (plan 52 phase 5, D4).
2026-08-05 02:37:48 -04:00
Juan Denis 8d106aa5a0 feat(backups): dynamic target-provider registry
VALID_TARGET_TYPES is now provider-driven; the wordpress_site branch
moves behind a provider the WP extension registers. ProtectionPanel /
RestoreDrawer read available types from the API; policies whose provider
is absent surface a clear missing-provider state instead of crashing
(plan 52 phase 3 task 13).
2026-08-05 01:18:11 -04:00
Juan Denis 31697cda9e feat(extensions): enforce signature policy in install flow
Invalid signature hard-blocks (400); untrusted key requires explicit
acknowledge (409 backstop); registry index carries signature/publisher
fields; install pinned to previewed sha256 with re-verified signature.
2026-08-04 23:34:15 -04:00
Juan Denis d961bd23f0 fix(api): close IDOR/authz gaps across deployment jobs, vaults, queues, docker, databases
Proactive hardening from an internal audit of the same broken-authorization
class as GHSA-6w78-q5vm-rfmh (authentication checked, app/workspace-level
authorization skipped):

- deployment_jobs: job detail/logs/retry resolve the job and gate on
  can_access_app / can_operate_app (app-less jobs: requester or admin);
  global job listing is admin-only
- secrets_webhooks: every vault/secret/webhook-endpoint/delivery by-id
  route resolves to its workspace and 404-seals non-members; secret
  reveal and signing-secret regeneration require workspace owner/admin;
  dead local admin_required placeholder removed
- shared_resources: variable-group CRUD and attach/detach gate on the
  resolved scope (workspace role / can_edit_app / can_operate_app),
  closing deploy-time env injection into foreign apps
- queue_bus: message receive/complete/fail/list on system-owned groups
  are admin-only; user-owned groups restricted to their owner
- docker: raw container inspect (exposes env vars) and container logs
  are admin-only, matching Decision 7; per-app logs stay app-gated
- databases: raw SQL query endpoints, host-level metadata reads, and
  process lists are admin-only; per-app managed database flows unchanged

Adds 4 persona-matrix regression suites (74 tests) and bumps the
test-count ratchet.
2026-08-04 08:53:27 -04:00
Juan DenisandTyler Boykin 783e195304 fix(git): enforce app-level authorization on deployment read endpoints
GHSA-6w78-q5vm-rfmh: four git deployment/webhook read endpoints checked
authentication but skipped app-level authorization, letting any active
user (even a foreign viewer) read cross-tenant deployment history and
logs that can contain secrets from pre/post-deploy scripts.

- git.deployments/<id>: resolve deployment -> app, 404/403 via
  app_access_tier before returning any log bytes
- git.deployments/webhook/<id>: gate on _webhook_visible_to (app-linked
  webhooks follow app visibility; unlinked stay panel-wide, plan 29 #11)
- deploy.history & deploy.webhook-logs: app_id queries require
  app_access_tier; global (no app_id) queries are panel-admin only

Adds backend/tests/test_git_deployment_authz.py (9 persona-matrix
regression tests) and bumps the test-count ratchet.

Co-authored-by: Tyler Boykin <25166954+CaptBoykin@users.noreply.github.com>
2026-08-03 23:44:56 -04:00
Juan Denis 937cbb2b81 feat(setup): capacity-aware install profiles and resource advisory
- install_profile_service: record what install.sh put on the box
  (minimal/standard/full), recommended from detected hardware and
  overridable; thresholds mirrored by recommend_profile in install.sh
- setup wizard: replace the hard ResourceGate/SetupStepTier with
  SetupStepCapacity + SetupStepSecurity
- ResourceAdvisory: inline capacity warning at the point of action
  (WordPress/Docker/Services/Deployments), backed by headroom.fits
- RequiresDocker component for docker-dependent surfaces
- install.sh/update.sh: profile detection + tests
  (test_install.sh, test_update.sh, test_server_capacity.py)
2026-07-31 14:56:37 -04:00
Juan DenisandClaude Opus 5 cceb444d27 fix(monitoring): give the TLS certificate read its own clock
_maybe_attach_certificate throttled on last_check_at, which is the probe clock.
An active monitor is checked every 30s, so that condition was never older than
the 6h refresh window: the certificate would be read once and then never
refreshed, leaving a stale expiry date on the detail page forever.

Adds cert_checked_at (migration 081) and throttles on that instead. The stamp
is written even when the read fails, so an unreachable TLS endpoint is retried
on the same cadence rather than on every single probe.

Also serves the Monitors list sparkline from one bulk query per page instead of
one per row, bounded by a time window so a busy monitor's history cannot crowd
out a quiet one's the way a single global LIMIT would.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 00:22:56 -04:00
Juan DenisandClaude Opus 5 9512975ebc feat(monitoring): promote status components to first-class monitors
A StatusComponent was already a full synthetic check (http/tcp/dns/smtp/ping,
target, interval, timeout, HealthCheck history, uptime windows, auto-incidents)
— it was just framed as "a component of a status page" and its engine lived
inside the serverkit-status extension, so watching a site required publishing a
status page. Promote it:

- migration 081 makes status_components.page_id and status_incidents.page_id
  nullable, and adds the columns backing the monitor UI (is_paused,
  check_method, expected_status, keyword, follow_redirects, verify_tls, retries,
  consecutive_failures, cert_issuer, cert_expires_at)
- new core app/services/monitor_service.py owns the check engine; the extension
  keeps page publishing and delegates
- new /api/v1/monitors (CRUD, check, pause, history, uptime buckets, incidents)
- new builtin.monitor_check sweep on a 30s tick — until now checks only ran when
  someone pressed "Check now", so a configured monitor never noticed an outage

Check-engine defects fixed while moving it:

- ping ignored the process return code and could only ever report 'up'
- ping used Linux-only flags with no guard, breaking the Windows dev server
- http ignored its own method/expected_status/follow_redirects/verify_tls
- smtp was documented on the column but unimplemented; keyword did not exist
- run_check never recomputed uptime and never opened or resolved an incident —
  only the WordPress health-sync path did, so network monitors never paged
- TLS issuer/expiry are now captured (throttled to one read per 6h)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 23:29:02 -04:00
Juan Denis ddf2e948d1 fix(security): enforce files.read on reads; de-shell syslog unit filter
Follow-ups from the GHSA-4wqh-7f4f-5qmx / GHSA-mc93-rc3x-fpgq review:

- File Manager read endpoints (browse/info/read/search/disk-*/analyze/
  type-breakdown/download + s3/browse, s3/read, s3/download-url) only
  required a valid JWT, so a user whose files.read permission is revoked
  via custom permissions could still read and download files. They now
  require permission_required('files', 'read'), matching the write side.

- LogService._read_syslog() interpolated the unit name into a bash -c
  string using subprocess.list2cmdline(), which is cmd.exe-style quoting
  and does not stop $(...)/backtick expansion under bash. grep now runs
  as a plain argv list (with --) and the tail happens in Python.
  Reachable only on non-systemd hosts by admins, but the same bug class
  as the Postfix advisory.

Regression tests: revoked files.read 403s every read endpoint; the
syslog filter never invokes a shell and passes payloads verbatim.
2026-07-26 15:40:14 -04:00
Juan DenisandAruvasgaChithan 696226002b fix(security): enforce files.write permission on File Manager mutations (GHSA-4wqh-7f4f-5qmx)
Mutating File Manager endpoints (write/create/mkdir/delete/rename/copy/
move/chmod/upload and the S3 equivalents) only required a valid JWT, so a
viewer account (files.write=false) could write and delete files. They now
require the files.write permission via permission_required(). Read
endpoints are unchanged. Adds regression tests covering both advisories.

Co-authored-by: AruvasgaChithan <AruvasgaChithan@users.noreply.github.com>
2026-07-26 15:19:22 -04:00
Juan DenisandClaude Opus 5 6edeba2ff5 fix(databases): scope the extension listing to apps the caller can see
`GET /databases/engines/<app_id>/extensions` carried only `@jwt_required()`, so
any signed-in user could enumerate any instance on the panel — the image it
runs, its databases, and what it can load. All facts about someone else's
application.

It gates on `can_access_app` now, mirroring the per-app docker DB read next to
it: owner, admin, or an explicit grant. Read access rather than admin-only is
deliberate — a developer who can already see an app should be able to see what
that app can load. The POST was already admin-gated and is unchanged.

The file also declares its `# Bucket: PER-APP` header (plan 29 #9), which is
what the structural sweeps read to know a per-app route file has been thought
about at all.

Found by the repo's own authz sweeps in a full-suite run — four failures, one
cause. Worth noting they fired on a route that has no feature test capable of
catching this: every functional test authenticates as someone entitled to the
data. That is precisely what a structural sweep is for.

Four tests pin the behaviour: denied without access, still visible to the owner,
404 rather than 403 for an app that does not exist (conflating them would make
the endpoint useless to its own caller), and 401 anonymous.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 00:19:46 -04:00
Juan DenisandClaude Opus 5 679082bf16 feat(databases): engine extensions, without the silent failure
The third deferred item. An extension installs INTO a running engine rather than
getting its own container, so it lives in its own namespace:
`backend/templates/extensions/<id>.yaml` with a top-level `extension:` block,
discovered by the same operator-wins two-directory scan the app templates use.
Dropping a YAML in makes it available with no code change, and a test pins that.

Not nested inside the engine template, because pgvector applies to any
PostgreSQL-speaking engine rather than to `postgresql.yaml` — nesting would force
duplication per host. Not a third `kind:` in the app catalog either: an
extension has no compose, port or volume and produces no Application, so every
consumer of `list_all_templates` would have had to learn to skip it.

The hard part is not the SQL. `CREATE EXTENSION vector` is valid and fails on
stock `postgres:16` with "could not open extension control file", because that
image does not ship the library. Two layers, and the statement is never sent
speculatively: an extension declares which images provide it, so a listing marks
an instance unsupported BEFORE anything is clicked; then at install
`pg_available_extensions` is queried first and a negative answer is a 409 with a
remedy, not an attempt. The live probe overrules the image name in both
directions — a private `registry.internal/ops/pg-with-vector` is allowed, and a
`pgvector/pgvector` tag that does not actually carry the library is still
refused.

ServerKit does not recreate a container behind the operator's back. The remedy
instead names an engine template that IS capable, found by matching the
extension's images against each engine template's compose — hence
`postgresql-pgvector.yaml`. TimescaleDB deliberately ships no bundled host and
says so rather than implying one exists.

Compatibility also needed correcting mid-build: matching on wire protocol alone
claimed CockroachDB could host pgvector. It answers on pgwire and cannot load a
single extension. Engines opt in with `engine.extensions: true`, which excludes
it without any Python naming it.

Two fixes fell out. `DatabaseService` had no psql twin of `docker_mysql_execute`
— `managed_db_user_service` carried a private copy with a comment saying as
much; there is now one implementation and it delegates. And that twin sets
`ON_ERROR_STOP=1`, without which psql reports SUCCESS for a statement the server
rejected.

57 tests, covering the zero-code-change property, compatibility, the image
problem in both probe directions, admin gating, and refusals that must not
execute anything.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 23:44:34 -04:00
Juan DenisandClaude Opus 5 3070353631 feat(databases): install any engine, because an engine is just a template
You could not install a database from ServerKit. The tree's four roots were
literals, only MySQL and PostgreSQL reported real state, and no install path
existed anywhere in the backend.

Rather than add one, this makes a database engine an ordinary app template that
declares a top-level `engine:` block — family, wire protocol, admin user, which
variable holds the password, what it calls a "table". Presence of that block is
the only thing that lists a template in the Databases catalog, and installing it
is the template pipeline that already existed: same deployment job, same
streaming console.

The consequence is the point. Adding CockroachDB is adding a YAML file. So is
adding something we have never heard of — author it locally or sync a template
repo — and the install form still works, because it is generated from the
template's own `variables[]` rather than hand-written per engine. There is no
engine model, no engine table, no engine install mechanism to keep in step.
`test_dropping_a_new_engine_yaml_lists_it` is the design; if it ever needs code
to pass, this went wrong.

Twelve engines ship: postgresql, mysql, mariadb, mongodb, redis, valkey,
cockroachdb, clickhouse, influxdb, neo4j, meilisearch, couchdb. Each pins its
image, binds to loopback, names a data volume, and was checked against the
upstream image's actual environment contract — a wrong variable name is a
container that boots with authentication silently disabled.

Two safety notes worth reading. Engines bind 127.0.0.1 unless explicitly
exposed, and the flag is applied after caller variables so a handcrafted payload
cannot quietly publish one. And CockroachDB single-node runs `--insecure`
because password auth needs a certificate-backed cluster: for engines with no
credentials at all the loopback bind is the only control, so public exposure is
refused outright rather than warned about. A test pins the passwordless list to
exactly cockroachdb, so a future template that drops its password variable
fails loudly instead of opening a hole.

Also surfaced at last: `ManagedDatabase.owner_application_id` has been accepted
by the create endpoints for ages with no UI to set it. Creating a database now
offers "Attach to application".

Deferred deliberately: new-table builder, SQL dump import, engine extensions
(pgvector, TimescaleDB) — each a data-mutation surface deserving its own round.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 20:09:05 -04:00
Juan DenisandClaude Opus 5 010b632530 feat(dashboard): boards that survive the browser they were built in
The dashboard's layout lived in localStorage as `{id, visible}` — a visibility
toggle wearing a layout's clothes. It carried no geometry, so widget width was
baked into SCSS (`.dash-kpi { grid-column: span 3 }`) and "make the chart wider"
was a code change. And it stayed on one machine.

`DashboardBoard` gives a user real boards: named, ordered, each holding a JSON
array of placed widgets with x/y/w/h and per-instance config. Three defaults
ship — overview, infrastructure, apps & deploys — seeded on a user's first GET
and restorable per board.

Validation is deliberate about the boring cases: geometry must fit the 12-column
grid, ids must be unique within a board, bools are rejected as coordinates
(`bool` subclasses `int`), and a rejected PUT leaves the stored board untouched
rather than half-applied. A board belonging to another user is a 404, not a 403
— there is no reason to confirm it exists.

The shipped defaults are checked by rebuilding their occupancy map cell by cell,
so editing DEFAULT_BOARDS into an overlap fails the suite instead of shipping a
broken first-run layout.

31 tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 19:30:51 -04:00
Juan DenisandClaude Opus 5 92e0750262 feat(sdk): a second widget surface — types users place, not just slots
`contributions.widgets` mounts a component at a host-chosen slot. That is the
wrong shape for the new dashboard, where the user decides what goes on a board
and where. So this adds `dashboard_widgets` alongside it: a registry of
placeable widget TYPES that surface in the widget library, each instance
positioned, sized and configured by the person looking at it.

The two are easy to confuse, so the docstring, the manifest spec and
EXTENSIONS.md all now say plainly which is which, and a test asserts a
dashboard widget never leaks into the fixed-slot list.

SDK_VERSION 1.0.0 -> 1.1.0 (added surface, older extensions unaffected),
bumped in lock-step across the JS export and the backend mirror.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 19:24:41 -04:00
Juan DenisandClaude Opus 5 ef3712f97a fix(services): real numbers in Resource Usage, and a Related card that finishes
Two more of the same fault as the logs: the frontend reading fields the backend
does not send at that level.

Resource Usage showed 0.0% CPU, 0.0% memory and N/A for everything else on a
healthy container. The stats route answers {stats: {...}} carrying docker's own
CLI keys, and the caller read the top level of the envelope — so every field was
undefined and fell through to its zero. It unwraps `stats` now, which is where
CPUPerc, MemUsage, NetIO and PIDs have been all along.

Related Resources sat on "Loading…" permanently because GET /apps/<id>/related
was never implemented. The client has always called it, the 404 was swallowed by
an empty catch, and the card reads null as "still loading". The endpoint now
exists — domains, managed databases, backup policy and deploy count, scoped by
the same access check as the rest of the app routes, each section gathered
independently so a missing table degrades that row instead of failing the card.
The catch also settles to an empty object now, so a future failure shows "no
related resources" rather than spinning forever.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 17:13:36 -04:00
Juan DenisandClaude Opus 5 da2b77f3d2 feat(templates): tell people whether a template actually fits the server
Deploying checked three things — the name is free, the server row exists, the
directory isn't taken. Nothing asked whether the box could carry the thing, so
a 2 GB app landed on a 1 GB VPS at full speed and the first sign of trouble was
the OOM killer.

Every piece was already here and unconnected: templates describe what they cost
(`requirements: memory / storage`), servers report what they have
(Server.total_memory / total_disk) and what they are using (the newest
ServerMetrics sample). This is the join, answering in the shape a person asks
it — does it fit, will it be tight, or can't we tell?

Advisory on purpose. It reports; it never refuses. An estimate is an estimate,
real usage depends on configuration and traffic, so the operator keeps the
final say and "tight" is a sentence to read rather than a wall to argue with.
A full server still returns valid:true, and a test asserts that so this cannot
quietly become a gate.

Three judgements worth stating:

- The reserve is a floor-and-fraction pair (10% of the box, at least 256 MB RAM
  / 2 GB disk). A bare percentage is wrong at both ends: 10% of 1 GB is
  nothing, 10% of 64 GB is more than most apps need.
- `unknown` outranks `ok`. No recent sample, or a template that declares
  nothing, says so — a confident wrong answer is worse than no answer.
- Free memory comes from psutil's `available`, not `free`: cache and buffers
  are reclaimable, and counting them as used calls every healthy Linux box
  full.

Catalog coverage goes from 52/106 to 106/106. The 54 estimates are each app's
typical idle-to-light-use footprint, taken from the project's own documented
minimum where it publishes one and rounded to the values the existing 52
already use — sonarqube 4 GB because that is its documented heap floor, immich
4 GB for the ML models, prometheus 20 GB because retention dominates, traefik
256 MB.

Surfaced two ways: a capacity block alongside (never inside) `errors` on
validate-install, and a read-only GET the deploy drawer can poll while the
operator is still choosing a target, so the answer updates as they switch
servers instead of waiting for a submit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 15:48:36 -04:00
Juan DenisandClaude Opus 5 e8e94a126f feat(plugins): let extensions contribute backup kinds
An extension with something worth backing up had to hand-roll it — no policy,
no cron mirror, no run history, no retention, no offsite copy, no verification,
no restore, and no place in the Protection panel. A registered kind inherits
every one of those, because everything downstream of the target_type switches
is already generic: the run lifecycle, retention (count, age, incremental-chain
ancestry), the API blueprint, and the panel itself, which threads target_type
through without ever switching on its value.

No migration: target_type is an unconstrained String, and the allowed set lived
purely in a Python tuple. A kind supplies resolve() and execute(), plus an
optional restore().

Four switches had to learn about the registry, and the order matters in three
of them:

- validate_target_type is the keystone — every policy route funnels through it,
  so accepting a registered kind opens the whole blueprint at once.
- _resolve_target and _execute_backup check the registry BEFORE their core
  branches, because both end in an application fallthrough that swallows any
  unrecognised type.
- run_restore_job likewise. This is the sharp one: without it a plugin's
  archive would be unpacked over an app directory by core's else-branch.

A kind that didn't implement restore is refused at request time with a reason,
rather than queueing a job that takes the policy lock and runs a safety backup
before failing.

The registry normalises what plugins hand back — target_type is forced rather
than defaulted, so a kind cannot describe itself as an 'application' and be
restored as one; execute() must return (path, size, meta) with a real path; and
the metadata retention and the panel read is filled in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 13:59:51 -04:00