The first pass left routes that hand the body straight to a service
(create_zone(data), MonitorService.create(data), ...). Those services
read it as a dict, so a body of null, [] or a bare string still
crashed them into a 500. Agent long-poll, external webhook receivers
and the routes that already type-check their body are left alone.
328 more routes used a JSON body as a dict straight from
request.get_json(), so a body of null, [] or a bare string crashed them
into a 500. Bare reads become json_object(); the
'get_json(silent=True) or {}' reads of action routes become
json_object(optional=True), which still treats a missing body as {}.
Routes that take arrays or pass the body through untouched are left
alone. The full backend suite shows no new failures.
Also carries three files' bare section citations (apps, backups,
notifications) and the extended citation guard, which needed them.
33 routes wrapped their body in try/except Exception only to return
unexpected_response(e), which records the crash and answers the same
body the global 500 handler already does. Every site was directly in a
route function, so the handler sees the same exceptions. SAML metadata
relied on its wrapper for an unconfigured IdP; that is now a typed 404
instead of a logged crash on a public, probed route.
Comments, docstrings and a few strings cited plan numbers ("plan 29
#9", "plan 86 §B2") from a git-ignored folder, so they pointed at
nothing; one reached users in a bottleneck hint. Code and frontend
strings are unchanged apart from those citations (checked by AST and
token comparison). A vhost still carrying the old asset comment reads
as undrifted, and a guard test keeps new citations out.
Toasts showed backend prose verbatim, with no code to translate
against and "application" where the UI says "service". Not-found,
missing-field, permission, conflict and agent-offline errors now come
from factories in app/exceptions.py that pair one fixed message with
a dotted code (not_found.<resource>, validation.required,
permission.denied, conflict.exists, agent.offline), so the client can
translate them.
The shared app-loading helpers and the apps, domains, servers,
databases and backups routes raise those errors instead of hand
shaping bodies, which drops the hand-shaped body count from 1132 to
850. Five routes picked an HTTP status by comparing error text; they
now key off a result code or a sentinel so rewording can't change a
status. Five broad excepts that echoed str(exc) now log the crash and
return the request id instead.
A deleted app deliberately keeps reserving its name (Plan 70: a restore
must never collide), but every clash site said only "already exists" —
pointing the operator at an app that is in no list (GH report: delete an
app, then a same-name deploy fails with no way forward).
app_name_clash_error() now distinguishes a tombstone from a live row and
names the release path (restore or purge from Settings -> Recycle Bin);
all five clash sites go through it. Live-row wording is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The drawer showed '<name>.<base> — Published automatically once the deploy
finishes' whenever a managed base domain existed, but publishing was gated
on a per-template auto_domain flag the operator can't see. For most of the
catalog the install landed on host:port instead — a link a default firewall
doesn't even expose — and the domain had to be redone by hand in the
service's Domain settings.
The promise on screen is now the contract: when the drawer displays a
domain it sends auto_domain: true, which rides the install plan to the
finalizer and overrides the template's silence (the forced flag also feeds
the SERVICE_FQDN magic context, so tokens and the published host agree).
Absent flag = the template YAML still decides, as before.
The drawer also stops promising what can't be delivered: the domain field
only renders for compose templates targeting the local server — the panel's
nginx can't proxy a remote container, and the repo pipeline has no
auto-publish step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GHSA-r4q7-x795-vr4j was one forgotten check, but the audit behind it found
the same shape across the surface: `@jwt_required()` authenticates, it does
not authorize, and ServerKit has no default-deny to catch the difference.
This closes the routes that audit left open — fleet-monitor alert
acknowledge/resolve, recycle-bin restore, nginx test/diff, registrar
sync/test, server-template drift checks, uptime tracking on/off, server ping
and remote compose ps/logs, refresh-capabilities, template
test-db-connection, deploy branches, buildpack detect — plus three inline
gaps: a viewer could widen their own API key's scopes, the demo-deploy flag
was doing double duty as an authorization one, and tagging a shared resource
had no write check at all while attaching one did.
`_ensure_group_mutable` only ever blocked *system* queue groups, so every
queue_bus mutation now also calls `_ensure_group_accessible` (owner-or-admin)
— renaming, draining or deleting another user's group was open to any
authenticated caller.
The rbac policy decorators authenticate through `auth_required()`, so they
replace `@jwt_required()` rather than stack under it; stacking rejects
API-key callers before the policy can authorize them
(tests/test_auth_decorator_boundaries.py enforces this). Where a route then
needed the caller it reads `get_current_user()` — `get_jwt_identity()` raises
for an API-key request. The decorators also carry a `_sk_authz` marker so a
static test can see the decision from the source.
`migrations.get_migration_history` moves to `require_admin_user()` for the
same reason its siblings did: the inline `User.ROLE_ADMIN` check it kept was
already reading two names this module no longer imports, so the route raised
NameError for every caller, admin included.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 76 milestone B, drift/bug row 4. 43 API handlers answered HTTP 500 from
their own `except Exception`, so the global 500 handler never ran — and that
handler was the ONLY caller of error_log_service.record_error(source='backend').
Every one of those crashes was absent from /monitoring/errors, and the page read
empty precisely because someone had "handled" them.
35 of the 43 also returned str(exc) to the caller. Two returned the full
traceback (templates.py, which also print()ed it), and several logged with a
locally-minted uuid4 'ref' that correlated with nothing but a log line.
The recording side of the 500 handler moves to app/error_reporting.py, so it is
importable rather than reachable only by not-catching:
record_unexpected(exc) -> logs, rolls the session back, records, returns
the request id
unexpected_response(exc) -> the above plus the documented crash body
The 500 handler now calls the same pair, so there is one recorder rather than
one recorder and 43 routes that quietly opted out of it.
Caller-visible change on those endpoints: the body is now the documented
{'error': 'Internal server error', 'status': 500, 'code': 'internal_error',
'request_id': ...} instead of the exception text. That is the point — an
unexpected exception's text is written for whoever reads the log and routinely
carries connection strings, absolute paths, and SQL. The request id is the part
a caller can act on, and it matches the row in the error log. No frontend code
read the 'ref' or 'trace' keys this drops (checked).
Two sites keep their own wording and call record_unexpected instead: sso.py's
login failures should not read as a generic crash on the login page.
docker.py's bulk endpoint keeps its per-item `results` — the caller's half-done
work survives, only the exception text is dropped.
Ratchet: tests/unreported_crash_census.py + its test, held at **0**, not a
legacy baseline to shrink. It counts blanket handlers that answer 500 without
calling either reporter, and is scoped to app/api — a service catching an
exception and returning a value is making a domain decision, not declaring an
HTTP crash. Verified non-vacuous: an injected handler fails it, and it passes
again on revert.
Measured and deliberately still open: 56 API handlers swallow a crash and
answer 200. A failure reported as success is worse, but converging it is
milestone C's response-envelope work, not this row.
Gate: tests/test_unexpected_error_reporting.py (11, including the two that pin
why the old shape lost the crash), test_unreported_crash_ratchet.py (9), and
the 66 test files covering every touched module: 1105 passed, 7 skipped, 1
pre-existing Windows-only failure (test_fleet_doctor's socket.gaierror).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 70. Domain went through this and TEN read paths were wrong, four with
effects outside the panel. Application has 211 call sites across 68 files and
drives more -- container lifecycles, nginx, backups, registry pulls, the OS
crontab -- so it was audited in full first (docs/plans/70_...md). 180 sites are
now query_active(); the 35 left unfiltered are each commented in place.
**The delete / purge split is what makes the tombstone honest.** delete_app used
to run compose_down(volumes=True) and rmtree(). Keeping a tombstone AND doing
that would put a Restore button next to a row whose data the delete had already
destroyed -- the one promise a recycle bin cannot break. So:
DELETE containers down (VOLUMES KEPT), vhost removed, firewall closed,
cron suspended, row tombstoned. All reversible.
RESTORE vhost rewritten, firewall reopened, cron resumed. Containers left
STOPPED -- "put it back" must not silently start processes or open
ports, the same reasoning that stops domain restore re-issuing certs.
PURGE volumes removed, uploaded source deleted, cron detached.
register() gained on_purge for that, wired into BOTH purge paths. And
purge_expired had NO scheduler -- fine when a tombstone was just a row, not fine
when it holds data volumes and a source tree, so a daily builtin job now runs
it. Without that, "kept until purge" would have meant kept forever and delete
would have quietly stopped reclaiming disk.
A side effect worth naming: the old _remove_data_flag protection for database
engines is now the DEFAULT for every app, and the flag is carried into
?purge=true so an explicit skip-the-bin delete still honours it.
**Cron was the blocker with no mechanical answer.** before_delete never fires on
a soft delete, and clear_application is LOSSY -- it nulls the association
without recording it, so nothing could put it back. suspend_for_application
instead disables only the jobs that were CURRENTLY ENABLED and stamps each
suspended_by_app; resume re-enables exactly those. That flag is the whole point:
without it a restore switches back on jobs the user deliberately turned off.
clear_application survives as purge-only, where lossiness is correct.
**The indirect reach (audit 3.1) is the class that grep cannot find.**
ContainerSleepPolicy, BackupPolicy, DeploymentJob and WafPolicy are enumerated
on a SCHEDULE and resolve application_id afterwards, so they never appear in a
search for Application.query. Left alone, a tombstoned app's sleep policy is
swept forever and wake_app restarts the containers the delete tore down. Those
loops test Application.deleted_ids().
Deliberately NOT scoped: reservation checks (port, name, directory) -- a
tombstone must keep holding those or a new app takes them and the restore is
refused; api/git.py's webhook VISIBILITY gate, which falls through to
"panel-wide visible" so hiding a tombstone would WIDEN access; and cron
attribution, which only labels a row.
Verified: 15 new tests, the migration replayed on a fresh DB (a tombstone frees
its private_slug, two live rows still collide), and a full delete -> restore
round trip. The 20 remaining full-suite failures are the known
serverkit-wordpress-not-installed class and are identical on a clean tree.
Two traps caught by testing, both recorded in the plan: a bad import made the
ENTIRE purge cleanup silently no-op while purge() still reported success
(_run_purge_hook swallows by design, so assert on the warning, not just ok), and
private_slug had to lose unique=True at the MODEL level or a models-built
database burns the slug regardless of the migration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The third deferred item. An extension installs INTO a running engine rather than
getting its own container, so it lives in its own namespace:
`backend/templates/extensions/<id>.yaml` with a top-level `extension:` block,
discovered by the same operator-wins two-directory scan the app templates use.
Dropping a YAML in makes it available with no code change, and a test pins that.
Not nested inside the engine template, because pgvector applies to any
PostgreSQL-speaking engine rather than to `postgresql.yaml` — nesting would force
duplication per host. Not a third `kind:` in the app catalog either: an
extension has no compose, port or volume and produces no Application, so every
consumer of `list_all_templates` would have had to learn to skip it.
The hard part is not the SQL. `CREATE EXTENSION vector` is valid and fails on
stock `postgres:16` with "could not open extension control file", because that
image does not ship the library. Two layers, and the statement is never sent
speculatively: an extension declares which images provide it, so a listing marks
an instance unsupported BEFORE anything is clicked; then at install
`pg_available_extensions` is queried first and a negative answer is a 409 with a
remedy, not an attempt. The live probe overrules the image name in both
directions — a private `registry.internal/ops/pg-with-vector` is allowed, and a
`pgvector/pgvector` tag that does not actually carry the library is still
refused.
ServerKit does not recreate a container behind the operator's back. The remedy
instead names an engine template that IS capable, found by matching the
extension's images against each engine template's compose — hence
`postgresql-pgvector.yaml`. TimescaleDB deliberately ships no bundled host and
says so rather than implying one exists.
Compatibility also needed correcting mid-build: matching on wire protocol alone
claimed CockroachDB could host pgvector. It answers on pgwire and cannot load a
single extension. Engines opt in with `engine.extensions: true`, which excludes
it without any Python naming it.
Two fixes fell out. `DatabaseService` had no psql twin of `docker_mysql_execute`
— `managed_db_user_service` carried a private copy with a comment saying as
much; there is now one implementation and it delegates. And that twin sets
`ON_ERROR_STOP=1`, without which psql reports SUCCESS for a statement the server
rejected.
57 tests, covering the zero-code-change property, compatibility, the image
problem in both probe directions, admin gating, and refusals that must not
execute anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
You could not install a database from ServerKit. The tree's four roots were
literals, only MySQL and PostgreSQL reported real state, and no install path
existed anywhere in the backend.
Rather than add one, this makes a database engine an ordinary app template that
declares a top-level `engine:` block — family, wire protocol, admin user, which
variable holds the password, what it calls a "table". Presence of that block is
the only thing that lists a template in the Databases catalog, and installing it
is the template pipeline that already existed: same deployment job, same
streaming console.
The consequence is the point. Adding CockroachDB is adding a YAML file. So is
adding something we have never heard of — author it locally or sync a template
repo — and the install form still works, because it is generated from the
template's own `variables[]` rather than hand-written per engine. There is no
engine model, no engine table, no engine install mechanism to keep in step.
`test_dropping_a_new_engine_yaml_lists_it` is the design; if it ever needs code
to pass, this went wrong.
Twelve engines ship: postgresql, mysql, mariadb, mongodb, redis, valkey,
cockroachdb, clickhouse, influxdb, neo4j, meilisearch, couchdb. Each pins its
image, binds to loopback, names a data volume, and was checked against the
upstream image's actual environment contract — a wrong variable name is a
container that boots with authentication silently disabled.
Two safety notes worth reading. Engines bind 127.0.0.1 unless explicitly
exposed, and the flag is applied after caller variables so a handcrafted payload
cannot quietly publish one. And CockroachDB single-node runs `--insecure`
because password auth needs a certificate-backed cluster: for engines with no
credentials at all the loopback bind is the only control, so public exposure is
refused outright rather than warned about. A test pins the passwordless list to
exactly cockroachdb, so a future template that drops its password variable
fails loudly instead of opening a hole.
Also surfaced at last: `ManagedDatabase.owner_application_id` has been accepted
by the create endpoints for ages with no UI to set it. Creating a database now
offers "Attach to application".
Deferred deliberately: new-table builder, SQL dump import, engine extensions
(pgvector, TimescaleDB) — each a data-mutation surface deserving its own round.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deploying checked three things — the name is free, the server row exists, the
directory isn't taken. Nothing asked whether the box could carry the thing, so
a 2 GB app landed on a 1 GB VPS at full speed and the first sign of trouble was
the OOM killer.
Every piece was already here and unconnected: templates describe what they cost
(`requirements: memory / storage`), servers report what they have
(Server.total_memory / total_disk) and what they are using (the newest
ServerMetrics sample). This is the join, answering in the shape a person asks
it — does it fit, will it be tight, or can't we tell?
Advisory on purpose. It reports; it never refuses. An estimate is an estimate,
real usage depends on configuration and traffic, so the operator keeps the
final say and "tight" is a sentence to read rather than a wall to argue with.
A full server still returns valid:true, and a test asserts that so this cannot
quietly become a gate.
Three judgements worth stating:
- The reserve is a floor-and-fraction pair (10% of the box, at least 256 MB RAM
/ 2 GB disk). A bare percentage is wrong at both ends: 10% of 1 GB is
nothing, 10% of 64 GB is more than most apps need.
- `unknown` outranks `ok`. No recent sample, or a template that declares
nothing, says so — a confident wrong answer is worse than no answer.
- Free memory comes from psutil's `available`, not `free`: cache and buffers
are reclaimable, and counting them as used calls every healthy Linux box
full.
Catalog coverage goes from 52/106 to 106/106. The 54 estimates are each app's
typical idle-to-light-use footprint, taken from the project's own documented
minimum where it publishes one and rounded to the values the existing 52
already use — sonarqube 4 GB because that is its documented heap floor, immich
4 GB for the ML models, prometheus 20 GB because retention dominates, traefik
256 MB.
Surfaced two ways: a capacity block alongside (never inside) `errors` on
validate-install, and a read-only GET the deploy drawer can poll while the
operator is still choosing a target, so the answer updates as they switch
servers instead of waiting for a submit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deploy templates were a hardcoded frontend constant (AgentSite, with a
hand-fabricated manifest and an EXAMPLE badge). They now live in the backend
catalog as `kind: repo` YAML entries that blend with the 105 compose templates.
- template schema gains `kind: compose | repo` (absent = compose, so every
existing template is untouched); repo templates carry a `repo` block with a
`url` instead of a compose stack
- backend/templates/agentsite.yaml (kind: repo) replaces the JS constant
- list + detail payloads serve `kind`/`repo`; category derivation picks up
repo templates automatically
- GET /api/v1/templates/<id>/manifest inspects a PUBLIC repo via
RepositoryManifestService.analyze_public_repo (raw host, no clone, no OAuth)
and falls back to the template's declared hints, labeled `template-hints`
- index.json + catalog-cert test recognize repo-kind; ratchet floor 2475->2498
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Four foundational enhancements toward the deploy-code developer experience:
- Container status aggregator (C7): new ContainerStatusService applies a
deterministic priority hierarchy (degraded > restarting > running:unhealthy >
starting > running:healthy > exited > unknown) over an app's containers,
cached with a short TTL. Exposed at /api/v1/status and pushed over a new
container_status Socket.IO channel; Applications page now renders an
aggregated ContainerStatusPill with sub-status tooltip.
- API token scopes + sensitive data masking (C10): require_scope() decorator
enforces fine-grained scopes for API-key requests (pass-through for JWT),
a canonical SCOPES catalog at /api/v1/api-keys/scopes, and a reusable
sensitive_data_filter (mask_sensitive) honoring secrets:read. Frontend gains
a grouped ApiKeyScopesModal picker and per-key scope pills.
- Server onboarding state machine (C8): new onboarding lifecycle
(pending -> validating -> installing_prerequisites -> installing_docker ->
pairing_agent -> ready/failed) with ServerOnboardingService, a
server_onboarding_logs table, servers.onboarding_* columns, REST endpoints,
and an OnboardingWizard surfaced on the server detail overview.
- Declarative template catalog (C5): documented catalog schema and
magic-variable resolution (SERVICE_PASSWORD_/USER_/FQDN_/URL_/BASE64_*)
layered onto the existing YAML template loader without changing behavior for
templates that don't use them; GET /api/v1/templates/catalog/schema.
Migration 029 adds the onboarding table + columns. 60 new tests pass; frontend
builds clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Makes the expanded catalog usable end-to-end:
- #24 Auto-domain: templates can opt in with 'auto_domain: true'; on install the app is published at <slug>.<base_domain> via the existing SiteDomainService (HTTPS only if the wildcard cert is already set up — never forces SSL; remote installs skipped).
- #25 Category consolidation: getCategoryIcon now covers the new tags (ai/llm/search/finance/documents/rss/vpn/dns/identity/observability/...).
- #26 Icon strategy: each template ships an inline lucide-style base64 icon with a keyed lucide fallback map; documented in Templates.jsx.
- #27 Update flow: update_app now normalizes list-format variables and re-renders the files: section + bind mounts, so converted/file-based templates (litellm, signoz, posthog) update correctly.
- #28 Community repo: TemplateService.build_repo_index()/export_repo_index() generate the index.json the sync mechanism already expects, exposed via GET /repos/index and POST /repos/index/export.
Roadmap: SERVICES_ROADMAP #24-#28.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add app/utils/formatting.py with format_bytes() and delegate 9 duplicated
human-readable byte formatters across services.
- Import admin_required from app.middleware.rbac in 15 API modules instead of
redefining it locally.
- Extend app/utils/slug.py with validate_app_name() and use it for template
install/validation and apps slug normalization.
A grant now reaches every app-related blueprint, not just apps.py + WordPress, so
a shared application is fully manageable by its grantees.
- Promoted the access helpers to a shared seam: ResourceGrantService.can_access_app
(owner/admin/any grant -> reads) and can_edit_app (owner/admin/editor grant ->
mutations); apps.py now delegates to them.
- Swept 33 ownership checks across 6 blueprints (categorized read vs mutate via a
parallel mapping workflow): builds.py (12), domains.py (7, incl. SSL + domain
CRUD), deploy.py (6), private_urls.py (5), templates.py (2), databases.py (1).
Reads -> any grant; mutates -> editor grant.
- Fixed the same latent int-vs-str ownership bug in all 33 (now user.id).
Deferred: python.py's single get_app_or_404 helper gates read+mutate+delete
together and needs per-method splitting before it's safe to make grant-aware.
Verified: cross-blueprint integration test (grantee reaches builds deployments;
stranger 403), full suite 131 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce a deployment jobs system to support cross-server template installations and remote execution. Adds DeploymentJob and DeploymentJobLog models, DeploymentJobService for job lifecycle, and DeploymentPlanRunner to execute plans locally or via agents. TemplateService gains build_install_plan and related helpers to produce executable plans; Template installs can run async (202) or synchronously. Backend API/CLI updated: new deployment-jobs endpoints, apps API uses RemoteDockerService for remote compose actions, and CLI commands (deploy-template, deployment-status, list-servers) were added. Agent and config changes: file access validation with allowed_paths, file write create_dirs option, default allowed paths, and install script/systemd read-write paths updated to permit /var/lib/serverkit and /var/serverkit. Migration file for deployment_jobs schema included.
Introduces clickable filter buttons to toggle CPU, memory, and disk metrics in MetricsGraph, replacing the static legend. Refines app name validation in both backend and frontend for template installation, allowing 2+ character names and improving error handling. WorkflowBuilder now handles missing workflow data gracefully and supports a fullscreen layout. Updates related styles for new filter legend and fullscreen workflow view.
POST /templates/test-db-connection - validates MySQL connection
- Required: host, user, password, database
- Optional: port (default 3306)
- Returns success/error with detailed message
- Includes warning if pymysql not available
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implements endpoints for managing and querying MySQL/PostgreSQL databases running in Docker containers, including listing containers, databases, tables, and executing queries. Adds endpoints for retrieving app logs and real-time status for Docker apps. Updates template variable processing to flag auto-generated and hidden variables. Extends DatabaseService with Docker MySQL/PostgreSQL discovery and query utilities.
Same bug as before - the validate_installation function was using
.items() on variables expecting a dict, but templates now use
list format.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add APP_NAME as automatic variable for all template installs
- Handle both list and dict formats for template variables
- Update uptime-kuma, portainer, node-app templates with proper format
- Use ${APP_NAME} for container names instead of hardcoded values
- Standardize port variable names to HTTP_PORT
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Introduces backend API endpoints and service logic for managing application templates, including listing, installing, updating, and repository management. Adds several example template YAML files and an index for templates. Updates frontend to support template browsing and installation, including new page, sidebar link, API integration, and styles. Removes the obsolete ROADMAP.md.
Juan Denis