This is the change behind the headline symptom. A DevTools capture of /domains
showed ~70s of network activity and a 404 taking 28.77 SECONDS. The backend
measured that same request at 1.0ms in its own api_usage_logs table. A routing
404 runs no handler and touches no database, so it cannot spend 28s working —
the entire gap was Stalled/Queueing in the browser.
The cause: in dev the API was same-origin with the Vite dev server, which serves
each of the ~760 source files as its own request. A browser allows six
concurrent HTTP/1.1 connections per origin, so twenty API calls queued behind
hundreds of module requests. Splitting the origin doubles the usable sockets
and, more to the point, stops a slow API call from blocking a JS module.
Same-origin was not an arbitrary choice — the comment it replaces warned about
"CORS/preflight redirects". Both halves of that are handled:
- CORS: config.py DEFAULT_CORS_ORIGINS already lists the Vite dev origin on
both localhost and 127.0.0.1, so nothing new is being allowed.
- Preflight: every panel request carries an Authorization header, which makes
it non-simple. Without Access-Control-Max-Age the browser re-runs the
preflight every few seconds (Chrome caches 5s by default), so a cross-origin
panel would pay two round trips per call forever — worse than what it
replaces. CORS now sends max_age=3600, verified against the real create_app()
rather than assumed: a preflight from http://localhost:41921 returns 200 with
Access-Control-Max-Age: 3600 and the matching allow-origin. That header is
worth having regardless, since any deployment serving the panel from a
different origin than the API was paying the same tax.
Escape hatch: VITE_API_PROXY=true restores same-origin for a setup with an
origin CORS does not expect. Socket.IO is deliberately untouched — it stays on
window.location.origin in dev, because a WebSocket upgrades out of the HTTP
connection pool and gains nothing from the split.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Incidents was showing the same idea twice: a converged "All incidents" table, and
under it a second table headed FLEET ALERTS with its own Active/Acknowledged/
Resolved segment row. A host alert IS an incident whose source is a host alert —
the SOURCE column already said so. FleetAlertsPanel is deleted (188 lines, and
Incidents was its only importer); alerts are folded into the existing row merge
with server -> subject, metric/value/threshold -> what happened, severity ->
impact, status -> state. Ack and Resolve move into the row-actions cell, shown
only for rows that have an alert id and are not resolved. The segment became an
"Acknowledged" builtin view alongside the Active/Resolved ones that already
covered its other two buckets.
Config Templates: the Compliant/Drifted/Unknown KPI band and its progress bar are
gone, replaced by a per-row Compliance column plus Drifted/Compliant builtin
views. The band could say three assignments had drifted but never which template
caused it; the column answers that per row. This needed the template payload to
carry its own tallies, which is the same single query the bare assignment count
already cost.
Recycle bin: the All / domains / saved-views segment row is gone — those buckets
are a filter on the Type column, and the Type column's own menu does it with live
per-value counts and can be saved into a view. Settings tab panes now sit on the
same card surface every other table in the app does.
Users: Add User and Invite User move onto the chrome line next to the filter and
"..." instead of each occupying a row of their own above its table.
SSL certificates: was a card list with a KPI band, one tab away from /domains
being a full table. Now a table with saved views (Expiring soon / Valid / Failed
or missing), search, column filters and export. The leading tile takes the row's
state too — a fixed green shield beside an expired certificate contradicted the
Status cell next to it.
Service detail: the Environment and Events tabs become tables with the inline
chrome an inner tab gets.
Fixes found by verification, in code this round touched:
- ServerTemplates had four columns with a `value` accessor and no `render`, so
DataTable fell back to row[key]: two rendered blank, one printed a raw array,
and `services` printed a list of OBJECTS, which React refuses to render — that
one took the whole tab down for any template declaring a service. Pre-existing,
and invisible because the screenshot fixture sent `services: ['sshd']` as
strings. The fixture now matches to_dict() so it cannot hide it again.
- useTopbarChrome returns a ready `actions` value. All eight hoisting pages
destructured only `portal` and dropped the fallback, so outside a tab group the
filter button and "..." would have gone nowhere rather than inline.
- sameViewState compares only the `page` keys a page still captures. Retiring a
key (as the recycle bin just did) otherwise stranded every view a user had
already saved with it, pinning "Unsaved" on with no way to clear it.
- The environment tab's "Applies to" column was gated on the app's CURRENT
compose services, so a variable targeting a since-renamed service showed
nowhere and read as applying to everything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 70. Domain went through this and TEN read paths were wrong, four with
effects outside the panel. Application has 211 call sites across 68 files and
drives more -- container lifecycles, nginx, backups, registry pulls, the OS
crontab -- so it was audited in full first (docs/plans/70_...md). 180 sites are
now query_active(); the 35 left unfiltered are each commented in place.
**The delete / purge split is what makes the tombstone honest.** delete_app used
to run compose_down(volumes=True) and rmtree(). Keeping a tombstone AND doing
that would put a Restore button next to a row whose data the delete had already
destroyed -- the one promise a recycle bin cannot break. So:
DELETE containers down (VOLUMES KEPT), vhost removed, firewall closed,
cron suspended, row tombstoned. All reversible.
RESTORE vhost rewritten, firewall reopened, cron resumed. Containers left
STOPPED -- "put it back" must not silently start processes or open
ports, the same reasoning that stops domain restore re-issuing certs.
PURGE volumes removed, uploaded source deleted, cron detached.
register() gained on_purge for that, wired into BOTH purge paths. And
purge_expired had NO scheduler -- fine when a tombstone was just a row, not fine
when it holds data volumes and a source tree, so a daily builtin job now runs
it. Without that, "kept until purge" would have meant kept forever and delete
would have quietly stopped reclaiming disk.
A side effect worth naming: the old _remove_data_flag protection for database
engines is now the DEFAULT for every app, and the flag is carried into
?purge=true so an explicit skip-the-bin delete still honours it.
**Cron was the blocker with no mechanical answer.** before_delete never fires on
a soft delete, and clear_application is LOSSY -- it nulls the association
without recording it, so nothing could put it back. suspend_for_application
instead disables only the jobs that were CURRENTLY ENABLED and stamps each
suspended_by_app; resume re-enables exactly those. That flag is the whole point:
without it a restore switches back on jobs the user deliberately turned off.
clear_application survives as purge-only, where lossiness is correct.
**The indirect reach (audit 3.1) is the class that grep cannot find.**
ContainerSleepPolicy, BackupPolicy, DeploymentJob and WafPolicy are enumerated
on a SCHEDULE and resolve application_id afterwards, so they never appear in a
search for Application.query. Left alone, a tombstoned app's sleep policy is
swept forever and wake_app restarts the containers the delete tore down. Those
loops test Application.deleted_ids().
Deliberately NOT scoped: reservation checks (port, name, directory) -- a
tombstone must keep holding those or a new app takes them and the restore is
refused; api/git.py's webhook VISIBILITY gate, which falls through to
"panel-wide visible" so hiding a tombstone would WIDEN access; and cron
attribution, which only labels a row.
Verified: 15 new tests, the migration replayed on a fresh DB (a tombstone frees
its private_slug, two live rows still collide), and a full delete -> restore
round trip. The 20 remaining full-suite failures are the known
serverkit-wordpress-not-installed class and are identical on a clean tree.
Two traps caught by testing, both recorded in the plan: a bad import made the
ENTIRE purge cleanup silently no-op while purge() still reported success
(_run_purge_hook swallows by design, so assert on the warning, not just ok), and
private_slug had to lose unique=True at the MODEL level or a models-built
database burns the slug regardless of the migration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two worst endpoints in the backend, both measured in plan 69 phase 6.
**/status/apps** ran one `docker compose ps` per app plus a `docker inspect`
per container -- roughly 30 apps x 3 containers = ~120 process spawns per
request -- and it is refilled on a timer, so the cost never amortised. It is
now ONE host-wide `docker ps` whose output is indexed by the compose
project/service/working-dir labels Docker already stamps on every container,
shared between callers through a 3-second in-memory snapshot.
The snapshot is deliberately module memory rather than CacheService: it must be
droppable synchronously, because a status that survives the action that changed
it reads as a broken panel. So `invalidate(app_id)` is called from start, image
apply, deploy and rollback. 3s is short enough that a missed hook self-corrects
within one poll.
**/admin/activity/summary** ran two 90-day loops issuing one COUNT per day =
180 sequential queries. Now one `GROUP BY func.date(...)` per series with the
gaps zero-filled in Python, so the response shape is byte-identical. `func.date`
is portable across the SQLite and PostgreSQL this app supports. A third N+1 fell
out on the way: the top-users block did one SELECT per user, now a single IN.
New tests assert the zero-fill specifically -- a day with no rows must still
appear and the series length must match the window, which is exactly what a
GROUP BY drops if you forget.
1140 backend tests pass. The 3 failures in that run (test_fail2ban_jail_service,
test_url_swap, test_site_routing) all reproduce on a clean tree -- the last is
the known "serverkit-wordpress extension not installed" class.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Second endpoint after /apps, same pattern. Server.to_dict grows a `fields`
parameter so $select skips the derived work rather than only trimming the
payload: `onboarding_progress` is a JSON walk that is ~29% of a row and is read
by exactly one screen, and `group_name` is a relationship load. A caller that
needs {id, name} -- Domains and Backups both do -- now pays for neither.
`group_name` is eager-loaded when it IS wanted, so the list stops issuing one
SELECT per server for it.
Two details this route needed that /apps did not: it answers with a bare JSON
array, so no paging envelope is emitted (that would break every caller), and an
explicit $orderby now wins over the hardcoded by-name sort while the default is
preserved when absent.
3 more tests; the 293 existing server/fleet/agent tests still pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
on_restore_domain deliberately does not re-issue a certificate -- ssl_enabled
can still be True over cert files that expired while the row sat in the bin,
and quietly talking to Let's Encrypt is not something a "put it back" button
should do. That decision stays.
What was missing is telling anyone. The domain returns on plain HTTP while its
row still reads ssl_enabled, so the first sign is a browser warning.
A restore now has three outcomes instead of two: the side effect FAILED
(warning, error toast), it succeeded but left something actionable (notice,
warning toast), or it just worked. An on_restore hook returns a string to raise
a notice; it rides on the item so it cannot be confused with the error slot.
The notice only fires when nothing is actually serving HTTPS for that name --
app_vhost_kwargs re-attaches a covering wildcard, and warning about a domain
that came back on HTTPS anyway would just be noise.
3 tests: notice present, notice suppressed under a covering wildcard, and no
notice for a domain that was never HTTPS.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Field selection did not exist anywhere in the backend (`fields=`/`select=`: 0
hits across 90+ API files) and neither did eager loading (joinedload/
selectinload/subqueryload: 0 matches). /apps sends ~2,319 B per row where the
list UI reads ~411, over an unbounded `query.all()` -- for an admin with no
workspace header that is literally SELECT * FROM applications.
Ports the author's conuco_v2 query layer as app/api/_query.py, with two
deliberate differences:
- ORM, not Core. conuco resolves fields through `table.c`; here they resolve
against the mapper's column set, and that set is the ALLOWLIST -- an unknown
field is a 400, never a getattr into arbitrary model attributes.
- $select is not only a column narrowing. That is where the win actually is:
Application.to_dict() costs ~3 extra queries per row, and two of them
(image_scans, image_update_checks) are lazy='dynamic', so the attribute IS a
Query and NO loader option can hoist them. Declining a field is the only way
not to pay for it. to_dict(fields=...) now skips the loads it was not asked
for, so $select=id,name costs zero extra queries per row instead of three.
/apps also eager-loads server/project/environment/linked_app -- but only when
the request actually asked for the names they resolve, since a joinedload for
a field nobody selected is a wider row for nothing.
Every parameter is optional and to_dict(fields=None) is the full legacy
payload, so adopting this changed no existing caller's response. Paging meta
appears only once a caller pages.
$view is deliberately not implemented: ServerKit's saved views are per-user
rows already resolved in the browser, so a second server-side copy of the same
concept would only be somewhere for the two to disagree.
19 new tests. The 2 failures in the wider run (test_fail2ban_jail_service,
test_url_swap) reproduce on a clean tree -- pre-existing, not from this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two gaps found by re-reading the seams rather than the pages.
useViewLink was wired into useTableChrome but NOT into useGridConfig, so
/domains -- the page the whole grid was built for -- was the one list page that
could not produce or accept a view link. Both hooks share it now, so a DataGrid
page and a DataTable page emit the same URLs.
SavedView had no pre_restore. Deleting a view frees its name AND its slug (the
unique indexes are partial), so either can be taken by the time you press
Restore; clearing the tombstone would then violate the index and 500 at commit.
Same guard Domain already had.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Making Domain soft-deletable silently changed the meaning of every existing
`Domain.query`: before, it could only return live rows. An audit of all 16 call
sites found ten that were now wrong, several with side effects outside the
panel.
The ones that reached the outside world:
setup_reconcile_service.py:267 the only is_primary row after deleting the
primary IS a tombstone -- and it drives a WP-CLI search-replace across
the whole WordPress database. It would have repointed a live site at a
host nginx no longer serves.
setup_reconcile_service.py:83 DNS backfill enumerated deleted domains and
called ensure_a_record() -- creating a real record at Cloudflare/Route53
for a domain the user removed.
api/domains.py:367 (enable_ssl) a real ACME order for a host that is in no
server_name: the HTTP-01 challenge cannot be served, burning Let's
Encrypt failed-validation limits for the whole registered domain, and on
success installing a wrong-CN cert on the LIVE vhost.
doctor_service.py:267 permanent red "does not resolve" rows that no
action could clear, and the same set is the allowlist for one-click DNS
repair -- so Repair created a provider record for a deleted domain.
And the ones that broke the feature itself:
domain_attach_service.py:79 a tombstone counted as a name clash, so you
could not move a hostname to another app after removing it. Migration 083
made the unique index partial precisely to free the name; this re-imposed
the burn in application code.
domain_attach_service.py:71 attach found the tombstone, reported success,
created no live row, published nothing -- and then mutated a recycle-bin
record.
site_domain_service.py:600 same for give_subdomain: success, a URL, a DNS
record, and a site that was never published.
api/domains.py:532 a second vhost renderer (admin Regenerate)
writing tombstones back into server_name.
Application.domains still includes tombstones ON PURPOSE -- filtering it with a
primaryjoin would, with delete-orphan, make SQLAlchemy HARD-delete the row the
recycle bin exists to keep. `Application.live_domains` is the accessor for
"what this app actually serves", and to_dict() uses it so the /apps payload no
longer lists deleted domains.
Per-domain routes now 404 a tombstone instead of editing it.
Seven tests cover the leaks that had real consequences.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
create_domain and delete_domain each called NginxService.create_site directly,
passing no ssl_cert / ssl_key / micro_cache. create_site takes those as
arguments, so the regenerated vhost came out HTTP-only and cache-less: on a
site covered by a base wildcard cert, touching ANY domain silently knocked the
whole site — every other domain on it — back to plain HTTP, and dropped its
micro-cache directives. Invisible until someone loaded the site.
Both now publish through SiteDomainService.write_app_vhost via one local seam,
so add and remove cannot disagree about what the served config should be. That
also fixes two lesser asymmetries: the rewrite was gated on app_type == docker,
so php/static/flask/wordpress vhosts kept advertising a domain after it was
removed; and it never called enable_site.
ALSO FIXES A REGRESSION I INTRODUCED WITH SOFT DELETE. app_vhost_kwargs read
`Domain.query`, which before soft delete could only return live rows and now
returns tombstones too. The vhost renderer would have written a deleted domain
straight back into server_name — the domain would keep being served after
deletion, and drift detection (which renders through this same function) would
report correct config as drifted. Now query_active().
Three tests, each failing against the previous code: the wildcard cert and
micro-cache survive in the resolved kwargs, a soft-deleted domain is absent
from them, and an end-to-end delete leaves the surviving domain on HTTPS.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The teardown map for delete_domain landed after the hook shipped and found a
real hole in it.
When the deleted domain was the app's LAST one, delete_domain runs
disable_site() + delete_site() -- the sites-enabled SYMLINK is gone, not just
the server_name. The hook called NginxService.create_site and reloaded, which
rewrites the file and leaves it UNSERVED. Restoring the last domain of an app
looked like it worked and served nothing.
SiteDomainService.write_app_vhost is the correct seam and fixes three things at
once:
1. it calls enable_site, so the symlink comes back;
2. app_vhost_kwargs re-attaches a covering wildcard cert and the micro-cache
flag -- a bare create_site drops both, so a restored site would silently
fall back to plain HTTP;
3. it handles every host-nginx app type. delete_domain only touches nginx for
docker apps, so php/static/flask/wordpress vhosts still list the deleted
domain; the rewrite is idempotent, so running it there repairs the
asymmetry rather than preserving it.
New test pins the case that was broken -- restoring an app's only domain must
call BOTH create_site and enable_site -- and it fails against the previous
implementation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deleting a domain did two things: tombstone the row and rewrite the app's
nginx vhost so traffic stopped. Restoring only undid the first, so a domain
could be "restored" and still not served. on_restore_domain rewrites the vhost
from the app's live domains -- the mirror of what delete_domain does -- and
reloads.
Adds a `pre_restore` hook to the registry, because "restorable" is a moving
target: deleting a domain FREES its name (the unique index covers live rows
only), so something else may hold it by the time you press Restore. Clearing
the tombstone then violates the index and raises IntegrityError at commit -- a
500 for what is really an explainable conflict. Now it refuses with the name of
the app that took it. Same guard for a parent Application that no longer
exists.
A restored domain that was primary does NOT become a second primary: whatever
stayed live keeps the role and this row returns as an alias.
The certificate is deliberately not re-issued. ssl_enabled may still be true
with cert files that expired while the domain sat in the bin, and quietly
talking to Let's Encrypt is not something a "put it back" button should do --
the domain returns on HTTP and SSL is re-enabled explicitly.
Five tests, including the two that would otherwise be 500s: name taken again,
and app gone. A failed vhost write warns and KEEPS the restore rather than
rolling back a row the user asked for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deleting is the only action in a control panel that repeating cannot undo, so
records a person destroys by hand now leave a tombstone and the Recycle Bin
hands them back.
SoftDeleteMixin adds deleted_at/deleted_by_id plus soft_delete/restore and
query_active()/query_deleted(). recycle_bin_service is a REGISTRY: a model
registers with a label function and an optional on_restore hook, so the API
lists and restores across types without knowing what a domain is. Adding the
next soft-deletable model is one register() call.
Applied to Domain (a real, destructive case -- deleting one used to lose the
record outright) and SavedView. DELETE /domains/<id> now tombstones and still
tears the vhost down; restoring a row does NOT re-do side effects unless the
type supplies on_restore, which is the honest default.
API: GET /recycle-bin (all kinds, newest first), POST
/recycle-bin/<kind>/<id>/restore, DELETE /recycle-bin/<kind>/<id> (purge,
admin-only -- the one irreversible action), POST /recycle-bin/purge-expired
with a 30-day default retention.
THE PART THAT NEEDED CARE: both tables had a plain UNIQUE covering what is now
a tombstone, so deleting a domain would have permanently burned its name and
deleting a view would have blocked re-creating one by the same name. Migration
083 replaces each with a PARTIAL unique index predicated on deleted_at IS NULL
-- supported by SQLite 3.8+ and Postgres. Tests pin exactly that: a deleted
name can be reused, and the live row stays unique.
Also gives SavedView a `slug` (backfilled), the handle the upcoming
?view=<slug> links resolve against. A tombstoned view does not reserve its
slug, so deleting a view frees the link name too.
Migration is idempotent on both columns and indexes: `deleted_at` is
index=True on the mixin, so a database built from the models already has that
index, and a bare create_index would abort the upgrade on exactly the installs
we most want to keep working.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CPU/Memory/Disk gauges read `server.metrics`, the same key the
per-server status endpoint uses (servers.py:757). The list endpoint never
populated it -- to_dict() defaults to include_metrics=False, and even when
asked it emits `latest_metrics` -- so on a live panel every gauge cell fell
back to the "no data" dash.
It survived because the screenshot fixtures DO supply `metrics`
(sections.mjs:294): the UI looked correct in every capture and was empty in
production. The fixture had encoded the intended contract; the backend never
implemented it.
Batched deliberately: Server.metrics is lazy='dynamic', so the obvious
per-row lookup costs one extra SELECT per server. Newest row per server =
highest id (autoincrement on insert), which also avoids the tie a
max(timestamp) join hits for same-second rows -- two queries total,
regardless of server count.
Tests pin all three properties, and each fails without the fix: the key and
field names the table reads, absence (not {}) for a server that never
reported, and that the query count stays flat as servers are added.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The queue-bus consumers, job system and metrics collector share one SQLite
file with request handling, and the default journal mode lets any writer
lock the whole file — the sporadic 'database is locked' errors in the
notification/job consumers. Connections now open with journal_mode=WAL
(readers and a writer coexist), busy_timeout=30s and synchronous=NORMAL.
Skipped under TESTING, where conftest owns connection pragmas.
An authenticated user of any role could inject arguments into git
subprocesses running as the backend user (root under systemd):
- repo_url was passed unvalidated to git ls-remote/clone, enabling the
ext:: transport (host command execution on git < 2.52 defaults) and
file:// local-repository disclosure (all git versions).
- app_path on POST /deploy/clone was passed without a '--' terminator,
enabling --upload-pack=<cmd> option injection independent of git
version and protocol.*.allow policy.
- branch flowed into 'git fetch origin <branch>' as an unterminated
refspec.
Fix (new app/utils/git_security.py applied at every affected sink):
- repo_url scheme allowlist (https/http/git/ssh + scp-like SSH);
rejects ext::/file://, local paths, leading '-', whitespace.
- '-c protocol.ext.allow=never -c protocol.file.allow=never' pinned on
every remote-touching git call, plus a restrictive GIT_ALLOW_PROTOCOL
in the subprocess environment.
- '--' positional terminator before URL/path/refspec arguments —
co-load-bearing with protocol pinning, which alone does not stop
option injection such as --upload-pack=<cmd>.
- branch/ref validation (leading '-', ref-format metacharacters) for
clone --branch, fetch refspecs and stored deploy configs.
- POST /deploy/clone now confines app_path to the managed apps root,
mirroring how /apps/from-repository derives it server-side.
Adds 31 regression tests; bumps version to 1.7.86.
Reported-by: CaptBoykin
Co-authored-by: Tyler Boykin <25166954+CaptBoykin@users.noreply.github.com>
AuditService.get_logs gains a target_id filter (paired with target_type) and
/admin/activity/feed passes both through — the per-resource activity
timeline can now ask 'what happened to THIS app'.
SavedView model (user/page/name/state JSON/is_default, unique per
user+page+name), idempotent 082 migration, saved_view_service with
one-default-per-page enforcement, and a JWT-scoped /api/v1/views CRUD
blueprint. Covered by 9 endpoint tests.
With the remote index memoized, re-parsing every bundled YAML on each call
was the entire remaining cost of a Templates page load. Entries are now
memoized per file and keyed on (st_mtime_ns, st_size), so a sync or an edit
invalidates exactly the file that changed: measured on the real catalog,
list_local_templates() goes from 304ms to 0.89ms (118 templates), and the
full catalog listing from 1707ms to 1.26ms.
The writers need no invalidation call - mtime keying covers create, overwrite
and delete, the last because the listing only returns what the directory scan
finds. Cache entries are pruned to what is still on disk so a long-running
panel that churns templates cannot grow it without bound.
Failed parses are cached too, keyed on the same stat tuple, so one malformed
YAML no longer costs a parse on every render; it is retried the moment the
file changes. A file that fails to parse still does not claim its id, so a
valid file of the same name in the fallback directory can supply it - pinned
by a test, since that is easy to lose in a refactor.
Entries are deep-copied out: a shallow copy would share the nested
categories/repo/engine containers, so a caller appending to categories would
corrupt every later listing.
Also swaps os.listdir for os.scandir (DirEntry.stat() reuses data the scan
already returned) and sorts directory entries, which makes a same-id
collision (foo.yaml vs foo.yml) resolve identically on every platform instead
of depending on filesystem order.
Both bugs were latent only while DEFAULT_REPOS pointed at a 404. Now that
serverkit.ai proxies a live registry, they bite on every Templates page load.
list_all_templates() concatenated local + remote with no id merge, so every
bundled template was listed twice the moment a real registry answered.
Measured against the live registry: 224 entries for 116 unique ids. It now
merges by id with local winning, matching get_template(), which already
resolves local directories before any repo; among repos, the first enabled
one to claim an id keeps it.
fetch_remote_templates() ran inline on every listing with a 30s timeout and
no memoization, so one unreachable repo stalled the whole catalog on every
render. It is now memoized with a TTL (5m success, 60s failure), a 10s
timeout, and a last-good fallback so a transient blip cannot blank a catalog
the panel has already shown. The last-good payload is carried forward across
repeated failures rather than being overwritten by the first one. The cache
is dropped in save_config(), the single choke point for repo add/remove/
enable (sync_templates routes through it too), and force=True bypasses it.
11 proving tests, including the repeated-failure case and a regression test
that a cached entry cannot be corrupted by a caller mutating it.
DEFAULT_REPOS pointed at raw.githubusercontent.com/serverkit/templates --
the org `serverkit` does not exist (the registry is jhd3197/serverkit-templates,
and its default branch is master, not main). So it has 404'd for its entire
existence: no panel has ever fetched a template remotely, and every /templates
page has been showing only the 118 bundled backend/templates/*.yaml.
Now points at https://serverkit.ai/templates, which proxies the registry and
was built for this consumer -- it serves <repo_url>/index.json and the
<repo_url>/templates/<id>.yaml path this class derives, behind a TTL cache
with last-good fallback. Using the product domain also means a branch rename
upstream cannot silently empty every panel's catalog, which is the exact
failure this is fixing.
Fixing the default alone would only help fresh installs, because get_config
returns whatever a saved templates.json holds. Known-dead URLs are therefore
healed on read. Nothing is lost by rewriting a URL that never once resolved.
The checksum half is here rather than in a follow-up because this commit
ACTIVATES a download path that has never run: sync_templates wrote whatever
came back straight to disk, ignoring the sha256 the index pins for every one
of the 106 official entries. Turning on an unverified fetch days after the
panel learned to verify ed25519 signatures on extensions would be shipping a
known downgrade. Mismatch is now a hard refusal that never reaches disk;
absent hash is allowed and counted, mirroring unsigned-vs-invalid for
extensions. Content is written as bytes so the file on disk is exactly what
was hashed.
Note get_template's in-memory remote read (it parses without saving) is not
covered -- it has no index entry on hand and would need an extra fetch.
DEPLOY ORDER: serverkit.ai must ship the master-branch fix before this helps.
Until then the proxy 502s and sync finds nothing, same as today.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Runtime-ESM extensions (starting with serverkit-wordpress) need slot
mounting and error containment from the host; additive, versioned in
lockstep frontend/backend.
Create/update validation fails loudly naming the missing extension
instead of orphaning monitors; health sweep no-ops with zero WP sites
(plan 52 phase 3 task 17). StatusComponent.wordpress_site_id stays core
per D1.
The wordpress template kind + wordpress-external-db preflight move behind
a template-provider hook the extension supplies; the backend filters the
catalog so the WP card simply does not appear when WP is absent — no
frontend change needed (plan 52 phase 3 task 16).
Extensions register event types at load; wordpress.* types now come from
the WP extension instead of a hardcoded core catalog (plan 52 phase 3
task 15). emit_wp stays core as a dumb emitter helper.
VALID_TARGET_TYPES is now provider-driven; the wordpress_site branch
moves behind a provider the WP extension registers. ProtectionPanel /
RestoreDrawer read available types from the API; policies whose provider
is absent surface a clear missing-provider state instead of crashing
(plan 52 phase 3 task 13).
Extensions can register providers into core engines at load (backup
targets, event types, template kinds) via a core_hooks module; idempotent,
process-lifetime like blueprint registration. WP extension supplies its
entries through builtin-extensions/serverkit-wordpress/backend/core_hooks.py
(plan 52 phase 3, D4).
Proactive hardening from an internal audit of the same broken-authorization
class as GHSA-6w78-q5vm-rfmh (authentication checked, app/workspace-level
authorization skipped):
- deployment_jobs: job detail/logs/retry resolve the job and gate on
can_access_app / can_operate_app (app-less jobs: requester or admin);
global job listing is admin-only
- secrets_webhooks: every vault/secret/webhook-endpoint/delivery by-id
route resolves to its workspace and 404-seals non-members; secret
reveal and signing-secret regeneration require workspace owner/admin;
dead local admin_required placeholder removed
- shared_resources: variable-group CRUD and attach/detach gate on the
resolved scope (workspace role / can_edit_app / can_operate_app),
closing deploy-time env injection into foreign apps
- queue_bus: message receive/complete/fail/list on system-owned groups
are admin-only; user-owned groups restricted to their owner
- docker: raw container inspect (exposes env vars) and container logs
are admin-only, matching Decision 7; per-app logs stay app-gated
- databases: raw SQL query endpoints, host-level metadata reads, and
process lists are admin-only; per-app managed database flows unchanged
Adds 4 persona-matrix regression suites (74 tests) and bumps the
test-count ratchet.
GHSA-6w78-q5vm-rfmh: four git deployment/webhook read endpoints checked
authentication but skipped app-level authorization, letting any active
user (even a foreign viewer) read cross-tenant deployment history and
logs that can contain secrets from pre/post-deploy scripts.
- git.deployments/<id>: resolve deployment -> app, 404/403 via
app_access_tier before returning any log bytes
- git.deployments/webhook/<id>: gate on _webhook_visible_to (app-linked
webhooks follow app visibility; unlinked stay panel-wide, plan 29 #11)
- deploy.history & deploy.webhook-logs: app_id queries require
app_access_tier; global (no app_id) queries are panel-admin only
Adds backend/tests/test_git_deployment_authz.py (9 persona-matrix
regression tests) and bumps the test-count ratchet.
Co-authored-by: Tyler Boykin <25166954+CaptBoykin@users.noreply.github.com>
Copilot review on the GHSA-rm3m-9mvw-68fh patch caught two issues:
- _BACKEND_DIR/_INSTALL_DIR were one level too shallow (backend/app and
backend/ instead of backend/ and the install root), which left
<install>/.env and frontend/dist reachable on the deployed layout —
the exact PoC path from the advisory. _BACKEND_DIR stays in
PROTECTED_ROOTS so flat layouts (code at /app) are covered too.
- The ALLOWED_ROOTS check used a bare startswith, so e.g. /optfoo
matched the /opt root. Both checks now require an exact match or a
path-boundary prefix.
New tests pin the level math (the existing cases computed the expected
layout independently and passed on Windows for the wrong reason) and
simulate the deployed layout with the install dir inside an allowed root.
New builtin job security-feed (daily) pulls the normalized advisory feed
from serverkit.ai/api/security-feed.json (a cached proxy of the repo's
published GitHub Security Advisories) and:
- notifies admins (security.alert, critical) when the running panel
version falls inside an advisory's affected range, once per advisory;
- fires a one-time post-fix reminder after an upgrade crosses a fix
boundary when the advisory declares a "## Post-upgrade actions" section
(e.g. rotate JWT_SECRET_KEY after GHSA-rm3m-9mvw-68fh).
Dedupe keys, the last-good feed copy, and the previously-seen version live
in SystemSettings, so restarts never re-notify and outbound failures serve
the cached feed. Opt out via the security_feed.enabled setting. Privacy
page on serverkit.ai updated to disclose the daily call.
The 1.7.68 RBAC fix enforced files.read on read endpoints, but the default
viewer role legitimately has files.read=true, and on the documented
install.sh layout the panel install dir (/opt/serverkit) sits under the
allowed root /opt. A viewer could read the backend .env, steal
JWT_SECRET_KEY / SERVERKIT_ENCRYPTION_KEY, and forge admin sessions.
FileService now carries PROTECTED_ROOTS (the panel install dir and
SERVERKIT_CONFIG_DIR), checked inside is_path_allowed before the
ALLOWED_ROOTS test, so every file-manager operation (read, write, browse,
download, upload, ...) refuses panel-internal paths for every role.
Regression tests: protected paths rejected by is_path_allowed; viewer
read/download of the backend .env returns 403; allowed roots keep working.
Co-authored-by: CaptBoykin <CaptBoykin@users.noreply.github.com>
- install_profile_service: record what install.sh put on the box
(minimal/standard/full), recommended from detected hardware and
overridable; thresholds mirrored by recommend_profile in install.sh
- setup wizard: replace the hard ResourceGate/SetupStepTier with
SetupStepCapacity + SetupStepSecurity
- ResourceAdvisory: inline capacity warning at the point of action
(WordPress/Docker/Services/Deployments), backed by headroom.fits
- RequiresDocker component for docker-dependent surfaces
- install.sh/update.sh: profile detection + tests
(test_install.sh, test_update.sh, test_server_capacity.py)
_maybe_attach_certificate throttled on last_check_at, which is the probe clock.
An active monitor is checked every 30s, so that condition was never older than
the 6h refresh window: the certificate would be read once and then never
refreshed, leaving a stale expiry date on the detail page forever.
Adds cert_checked_at (migration 081) and throttles on that instead. The stamp
is written even when the read fails, so an unreachable TLS endpoint is retried
on the same cadence rather than on every single probe.
Also serves the Monitors list sparkline from one bulk query per page instead of
one per row, bounded by a time window so a busy monitor's history cannot crowd
out a quiet one's the way a single global LIMIT would.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A StatusComponent was already a full synthetic check (http/tcp/dns/smtp/ping,
target, interval, timeout, HealthCheck history, uptime windows, auto-incidents)
— it was just framed as "a component of a status page" and its engine lived
inside the serverkit-status extension, so watching a site required publishing a
status page. Promote it:
- migration 081 makes status_components.page_id and status_incidents.page_id
nullable, and adds the columns backing the monitor UI (is_paused,
check_method, expected_status, keyword, follow_redirects, verify_tls, retries,
consecutive_failures, cert_issuer, cert_expires_at)
- new core app/services/monitor_service.py owns the check engine; the extension
keeps page publishing and delegates
- new /api/v1/monitors (CRUD, check, pause, history, uptime buckets, incidents)
- new builtin.monitor_check sweep on a 30s tick — until now checks only ran when
someone pressed "Check now", so a configured monitor never noticed an outage
Check-engine defects fixed while moving it:
- ping ignored the process return code and could only ever report 'up'
- ping used Linux-only flags with no guard, breaking the Windows dev server
- http ignored its own method/expected_status/follow_redirects/verify_tls
- smtp was documented on the column but unimplemented; keyword did not exist
- run_check never recomputed uptime and never opened or resolved an incident —
only the WordPress health-sync path did, so network monitors never paged
- TLS issuer/expiry are now captured (throttled to one read per 6h)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The page kept a shape the rest of the app moved off: a four-card KPI
strip, its own in-body search input, a hand-rolled table, four icon
buttons per row, and an edit modal on row click.
It now matches /domains: the segmented filter carries the counts (no
KPI strip), SearchField lives in the top bar, the list is the shared
DataTable, and a row opens the DS Drawer. The top bar keeps its title
— /cron has no tab group, and the reference mock shows a titled bar
there; the body was what was stale.
Last run / Next run / Status columns need data list_jobs() never
returned: it skipped the schedule_human / next_run / run-tracking joins
that jobs_for_application() has done since plan 34, so the admin list
could only ever show enabled-vs-disabled. Apply the same enrichment,
with tests covering the schedule join, the last-run join, and the
no-history case.
The drawer's run history uses GET /cron/jobs/<id>/runs, which already
existed and only ScheduledTasksCard consumed. It's admin-only and
empty until a job carries the tracking shim, so both cases are stated
in the drawer (with an enable-tracking action) rather than rendering a
blank table.
Also fixes styles that never applied: .run-output-*, .spinner-inline
and the modal field overrides were nested under .cron-page, but Radix
Dialog and Sheet portal to <body>. Hoisted to top level.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- log_service: grep the service/unit filter with -F so names like
'nginx.service' are matched literally instead of as a regex.
- log_service: clamp the in-Python tail like 'tail -n' — lines=0 sliced
as [-0:] == all lines, negatives kept nearly everything.
- test_files_rbac: add files/upload and files/s3/upload to WRITE_CASES,
and disk-usage/disk-mounts/analyze/type-breakdown to READ_CASES, so the
permission gates on the full /api/v1/files surface are covered.
- create_app(): skip background daemons (queue consumers, metrics, job
system, analytics flush, linked-panel client) when loaded by a Flask CLI
one-shot such as 'flask db upgrade', or when SERVERKIT_SKIP_BACKGROUND=1.
They query the DB before migrations run, so any error there (corrupt DB,
pre-migration schema) aborted the migration and sank the whole update.
- update.sh: set SERVERKIT_SKIP_BACKGROUND=1 for both flask db upgrade
invocations as an explicit contract.
- update.sh: fail fast with an actionable message when the SQLite slot copy
fails PRAGMA integrity_check, instead of dying mid-boot with 'database
disk image is malformed'.
- update.sh: rebuild the release tarball's prebuilt venv when its baked
VIRTUAL_ENV path resolves to the old slot (it is baked as
/opt/serverkit/venv), which previously ran new code on the old slot's
dependencies during migration. Resolved-path comparison keeps the fast
prebuilt path for fresh installs.
Follow-ups from the GHSA-4wqh-7f4f-5qmx / GHSA-mc93-rc3x-fpgq review:
- File Manager read endpoints (browse/info/read/search/disk-*/analyze/
type-breakdown/download + s3/browse, s3/read, s3/download-url) only
required a valid JWT, so a user whose files.read permission is revoked
via custom permissions could still read and download files. They now
require permission_required('files', 'read'), matching the write side.
- LogService._read_syslog() interpolated the unit name into a bash -c
string using subprocess.list2cmdline(), which is cmd.exe-style quoting
and does not stop $(...)/backtick expansion under bash. grep now runs
as a plain argv list (with --) and the tail happens in Python.
Reachable only on non-systemd hosts by admins, but the same bug class
as the Postfix advisory.
Regression tests: revoked files.read 403s every read endpoint; the
syslog filter never invokes a shell and passes payloads verbatim.
Mutating File Manager endpoints (write/create/mkdir/delete/rename/copy/
move/chmod/upload and the S3 equivalents) only required a valid JWT, so a
viewer account (files.write=false) could write and delete files. They now
require the files.write permission via permission_required(). Read
endpoints are unchanged. Adds regression tests covering both advisories.
Co-authored-by: AruvasgaChithan <AruvasgaChithan@users.noreply.github.com>
The hostname parameter was concatenated into a bash -c string executed as
root via run_privileged(). Validate the hostname against a strict charset
and pipe debconf selections via stdin so no user input ever reaches a
shell command line.
Co-authored-by: tonghuaroot <tonghuaroot@gmail.com>
`GET /databases/engines/<app_id>/extensions` carried only `@jwt_required()`, so
any signed-in user could enumerate any instance on the panel — the image it
runs, its databases, and what it can load. All facts about someone else's
application.
It gates on `can_access_app` now, mirroring the per-app docker DB read next to
it: owner, admin, or an explicit grant. Read access rather than admin-only is
deliberate — a developer who can already see an app should be able to see what
that app can load. The POST was already admin-gated and is unchanged.
The file also declares its `# Bucket: PER-APP` header (plan 29 #9), which is
what the structural sweeps read to know a per-app route file has been thought
about at all.
Found by the repo's own authz sweeps in a full-suite run — four failures, one
cause. Worth noting they fired on a route that has no feature test capable of
catching this: every functional test authenticates as someone entitled to the
data. That is precisely what a structural sweep is for.
Four tests pin the behaviour: denied without access, still visible to the owner,
404 rather than 403 for an app that does not exist (conflating them would make
the endpoint useless to its own caller), and 401 anonymous.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The third deferred item. An extension installs INTO a running engine rather than
getting its own container, so it lives in its own namespace:
`backend/templates/extensions/<id>.yaml` with a top-level `extension:` block,
discovered by the same operator-wins two-directory scan the app templates use.
Dropping a YAML in makes it available with no code change, and a test pins that.
Not nested inside the engine template, because pgvector applies to any
PostgreSQL-speaking engine rather than to `postgresql.yaml` — nesting would force
duplication per host. Not a third `kind:` in the app catalog either: an
extension has no compose, port or volume and produces no Application, so every
consumer of `list_all_templates` would have had to learn to skip it.
The hard part is not the SQL. `CREATE EXTENSION vector` is valid and fails on
stock `postgres:16` with "could not open extension control file", because that
image does not ship the library. Two layers, and the statement is never sent
speculatively: an extension declares which images provide it, so a listing marks
an instance unsupported BEFORE anything is clicked; then at install
`pg_available_extensions` is queried first and a negative answer is a 409 with a
remedy, not an attempt. The live probe overrules the image name in both
directions — a private `registry.internal/ops/pg-with-vector` is allowed, and a
`pgvector/pgvector` tag that does not actually carry the library is still
refused.
ServerKit does not recreate a container behind the operator's back. The remedy
instead names an engine template that IS capable, found by matching the
extension's images against each engine template's compose — hence
`postgresql-pgvector.yaml`. TimescaleDB deliberately ships no bundled host and
says so rather than implying one exists.
Compatibility also needed correcting mid-build: matching on wire protocol alone
claimed CockroachDB could host pgvector. It answers on pgwire and cannot load a
single extension. Engines opt in with `engine.extensions: true`, which excludes
it without any Python naming it.
Two fixes fell out. `DatabaseService` had no psql twin of `docker_mysql_execute`
— `managed_db_user_service` carried a private copy with a comment saying as
much; there is now one implementation and it delegates. And that twin sets
`ON_ERROR_STOP=1`, without which psql reports SUCCESS for a statement the server
rejected.
57 tests, covering the zero-code-change property, compatibility, the image
problem in both probe directions, admin gating, and refusals that must not
execute anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MariaDB 11 dropped the `mysql` symlink in favour of `mariadb`. Every place that
execs a client inside a database container hardcoded `mysql`, so introspecting
any modern MariaDB image failed outright:
OCI runtime exec failed: exec: "mysql": executable file not found in $PATH
Which is not a hypothetical image. `backend/templates/mariadb.yaml` — the engine
template shipped earlier in this same plan — pins `mariadb:11.4`. Installing
MariaDB through ServerKit produced a database ServerKit could not open, and the
failure surfaced in the tree as a generic "couldn't load".
The client is now resolved per container by probing `--version`, preferring
`mariadb` and falling back to `mysql`, memoised because the answer cannot change
while a container runs. When neither answers it keeps returning `mysql`, so an
unrelated failure still reports the error operators already recognise.
Found by running a real mariadb:11.4 container against the service rather than
trusting the code path; verified end to end (SHOW DATABASES, list databases,
list tables with row counts) before writing the fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
