An audit found 102 FK columns with no index — every join, per-parent
listing, and the delete cascades from _delete_cascade_policy scanned
whole tables on them (deployments.app_id, audit_logs.user_id,
dns_records.zone_id, webhook_logs.webhook_id, ...). All FK columns now
declare index=True; migration 096 creates the same indexes on existing
installs, deriving the list from the live schema at upgrade time (so
it is idempotent and cannot drift from what an install actually has).
Ratchet: test_every_fk_column_is_indexed pins the rule with no
allowlist — a new FK column is born indexed or the build fails.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every NOT NULL FK from the KNOWN_DANGLING_ON_DELETE audit now declares
its parent-side relationship, so the delete-cascade policy door covers
it: AI conversations/pending actions, API keys, dashboards, event
subscriptions, login links, sent invitations, resource grants (users);
env vars + history, previews + settings, WAF policy (applications);
surveys, doctor results, proxy stack, tunnels (servers); manifests
(projects); DDNS hosts (zones); exposed services (tunnels); drills
(backup policies); rollouts (agent versions); site plugin installs
(WordPress sites); queue messages (groups). The ratchet set is now
EMPTY and stays that way.
One new deliberate exception: Workspace.projects — projects carry
environments and app groupings, so delete_workspace now refuses (409)
while any remain instead of silently destroying them (previously it
crashed with IntegrityError).
Migration 095 sweeps rows already orphaned on existing installs by
pre-fix hard-deletes — including an application's leftover env-var
secrets, which a reused SQLite row id could have attached to a future
app.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the 29 hand-written cascade='all, delete-orphan' declarations
from c1d4c0af/1704d60b/79a78c3c with a single mapper hook
(_delete_cascade_policy): at configure time, every one-to-many whose
child FK is NOT NULL gets the delete cascade automatically — with a
NOT NULL FK the children cannot outlive the parent, so the only choice
is cascade or crash. New models (extensions included — registering one
re-fires the hook) are covered with zero ceremony; nothing to remember.
Relationships where the parent delete should be REFUSED instead opt
out in the module's DELIBERATELY_UNCASCADED registry with their
justification (User.applications guarded by delete_user's 409,
CloudProvider.servers with no delete path). The sweep test now proves
the door works rather than policing model authors, and pins the
registry so it can only shrink.
Model files revert to their pre-fix text: delete semantics live in one
place instead of twenty.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up to 1704d60b, prompted by 'can this happen elsewhere?' — a
mapper-wide audit found 22 more one-to-many relationships whose child
FK is NOT NULL but which lacked a delete cascade, so hard-deleting the
parent (user/server/workspace/template/plugin delete, deployment
retention pruning, WP site teardown) raised the same IntegrityError as
the Recycle Bin purge.
20 get the cascade: plugin installs, deployment diffs, metric alerts,
template assignments (both sides), WP activities/sync/promotion jobs
(both sides), workspace members/keys, and the per-user detail rows
(passkeys, OAuth identities, notification prefs, sanitization
profiles, source connections, workflows, workspace memberships).
2 stay uncascaded on purpose, now enforced: User.applications (delete_
user refuses with 409 while the user owns apps — apps must go through
app delete + Recycle Bin purge so containers/volumes are cleaned) and
CloudProvider.servers (no delete path exists; dropping VM inventory
must stay a deliberate decision). The invariant sweep is now global
across every mapper with exactly that allowlist, which may only shrink.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Field follow-up to c1d4c0af: the live box then crashed on
deployments.app_id — dynamic relationships get the same FK null-out
on parent delete as the one-to-ones. Cascade every one-to-many from
Application whose child FK is NOT NULL (deployments, config
snapshots, image scans, SBOMs, image update checks, git deployments),
and pin the invariant with a mapper-inspection sweep so a future
NOT NULL child cannot reintroduce the purge crash.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Purging an app from the Recycle Bin crashed with IntegrityError
(NOT NULL constraint failed: container_scale_policies.application_id)
when the app had an auto-scale policy: with no delete cascade,
SQLAlchemy nulls the child FK on parent delete, and these one-to-one
rows declare it NOT NULL. Scale policy, sleep policy and wp_site all
share the shape; all three now ride the purge.
Reported on Discord (vaultwarden/hedgedoc delete-reinstall flow).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Migrations 065/068 have always created servers.management_mode and
servers.allow_agent_update_observed, but the model fields,
MANAGEMENT_MODES, is_managed, and agent_registry.observed_blocked_count
were lost in the 2026-07 data loss — POST /management-mode and
GET /observed-status raised AttributeError, and a mode flip that got
past validation would land on a transient Python attribute and
evaporate. test_survey_api carried skip-stubs documenting the hole.
Restored: the two columns (specs mirror the migrations), the mode
constant and is_managed property, to_dict emission, and a minimal
observed-block counter on the registry. The four skip-stubs are now
real tests, including one proving the flip persists across sessions.
The dispatch-level Observe guard remains hollow and is tracked in
plan 82 §H. events_json on notification_preferences is likewise
re-mapped (column carried by migration 059; its resolution surface is
still lost) so pre-loss override data survives.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RestorePoint hand-rolled json.loads for payload/coverage — now reads
through JsonColumnMixin._json_read like every other model, keeping the
models-json ratchet green.
test_restoring_the_last_domain_re_enables_the_site mocked create_site
and counted enable_site calls, but since write_vhost became the one
enabling door (enable=True re-links sites-enabled and create_site goes
through it), write_app_vhost no longer calls enable_site separately —
test_site_domain_writer_does_not_enable_twice pins that. The restore
test now asserts through write_vhost's enable flag instead, still
failing if a restore path ever opts out of enabling.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 79 A1/A2/A4 + B1/B2.
Delivery deliberately avoids the usual http-backend fetch. `en` is bundled;
other locales are dynamic imports Vite emits as chunks from the app's own
static origin. The sign-in and setup screens render before any API call
succeeds, so a fetched locale either flashes English or -- when the backend
is the thing that is broken -- never arrives, on the one screen whose job is
to say so. It also keeps air-gapped and subpath installs working with no
second path to 404.
Precedence (user row -> localStorage -> panel default -> navigator -> en)
mirrors ThemeContext's, including the split provider: LocaleProvider sits
outermost so a toast raised during theme init is already translatable, and
LocaleSync bridges auth state down, exactly as ThemeSync does.
users.language is nullable on purpose -- "follow the panel default" is a
different state from an explicit choice of English, so existing rows are not
backfilled. The backend restates the supported list to validate writes; a
parity test fails the build if it drifts from the frontend manifest, which
stays canonical.
Two bugs the tests caught, both real rather than test artifacts:
- normalize_language('ES') returned None. The shape gate was case-sensitive,
so two of the three forms browsers actually send were rejected.
- PUT /me is throttled 3/minute for credential edits, and language rides the
same route. Switching language three times returned 429 -- and because the
client applies the change locally first, the preference silently failed to
persist and reverted on the next sign-in elsewhere, defeating the column.
Cosmetic-only payloads are now exempt from that limit; credential edits
stay throttled (proved both ways) and the app-wide 100/minute still holds.
eslint moves to ecmaVersion 'latest' because import attributes are ES2025 and
Node's test runner requires them for the manifest imports.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First of the regret-ranked adoptions: a plain delete now tombstones the
row (migration 088; no unique constraints, so no partial-index rewrite).
The database content and its managed BackupPolicy survive the bin window
— restore brings everything back intact; PURGE removes the policy but
NEVER drops database content; the explicit ?drop=true path stays
immediately destructive. Active read paths (list/get/adopt/manifest/env
references/app detail) switch to query_active(); the backup executor
deliberately stays tombstone-blind so bin-parked data keeps backing up.
Server (98 query sites, fleet enrollment + HMAC secrets) is the next
ranked adoption — deliberately its own pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
env_variable's class-level SECRET_KEY-derived Fernet (reached
cross-model by shared_resource) and sso_service's private stack now
WRITE through the one key path (SERVERKIT_ENCRYPTION_KEY) and dual-READ:
one path first, legacy key second — permanent window, so rows never
rewritten keep decrypting and no bulk decrypt-all/re-encrypt-all pass
ever runs (PR #94 postmortem). The '[DECRYPTION_ERROR]' sentinel and
None-on-failure token semantics are preserved.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Opt-in to_dict over __table__.columns: auto-isoformat datetimes,
*_encrypted always dropped, __serialize_exclude__ for redacted columns,
serialize_extra() for computed fields (named to dodge ServerMetrics'
literal 'extra' column). 20 purely-mechanical to_dicts across 15 files
convert; genuinely custom shapes keep their hand-written methods (plan
75 §F's ruling stands). Field selection stays api/_query.py's job.
Gate: key-for-key parity against the pre-conversion key sets frozen
from git HEAD (only the Timestamp pair may be additive).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RunLogStream accepts any RunLike (__run_kind__ + id + to_dict, optional
result/step members): one room grammar run_<kind>_<id>, one batched
persisted run_log/run_status event pair, and a GET
/api/v1/runs/<kind>/<id>/logs?after_id polling twin over the new
run_log_entries table (migration 087). Deploys keep DeploymentJobLog and
dual-emit the legacy deploy_log/deploy_status during the migration
window. The unified Job runtime — which had NO live channel — adopts it
in the consumer (start/succeeded/failed frames; handlers can log via
stream_for(job)). The dead pipeline_* scaffolding was retired in E2.
Gate: emit -> persisted row -> after_id resync contract test; consumer
adoption test. MIGRATION_INVENTORY.md regenerated (census moved).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 11 hand-written status+completed_at(+error_message) transition
blocks in deployment_job_service collapse into mark_running/
mark_succeeded/mark_failed. The model keeps its stored 'succeeded'
spelling via the mixin override until this domain's data migration;
error text routes through __error_attr__ = 'error_message'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
~60 call sites across 24 model files now ride _json_read with each
site's exact default preserved; 34 classes newly adopt the mixin
(registry contract test expanded to 36). Reverses the old 'bare sites
stay raising' ruling: a corrupt row degrades to its default instead of
500ing every serializing endpoint (docstring updated). Ratchet:
json.loads( appears only in json_column_mixin.py.
Known pre-existing: test_host_inventory enumerate_filesystems failures
on this machine (verified via clean-tree stash rerun).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
models/status.py defines the six lifecycle constants, TERMINAL/ACTIVE
sets, and normalize() mapping every measured legacy spelling (succeeded/
completed/done/failure/error/canceled/scheduled) home at the read edge —
stored strings migrate per-domain later, never wholesale.
RunLifecycleMixin centralizes mark_running/succeeded/failed/cancelled
transitions (status + started_at/completed_at/duration), with per-model
spelling overrides so adoption never silently rewrites a domain's stored
vocabulary. Parity gate: every measured spelling normalizes canonical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TimestampMixin (byte-identical to the hand-written pair — zero schema
migrations) adopted by 52 classes across 44 model files; the utcnow
tz-aware migration is now a one-file edit. uuid_pk() replaces the 13
pasted String(36) uuid4 lambdas across 7 files (ai.py's hex variant
stays grandfathered, documented). Deviant shapes (indexed created_at,
sanitization_profile's default-less updated_at, created-only models)
deliberately keep their own columns.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six more models own their *_encrypted columns through the descriptor
(linked_panel, source_connection, container_registry, managed_database,
registrar_connection, pending_agent); eight service/api files stop
importing utils.crypto for them. legacy_plaintext mode mirrors
decrypt_secret_safe so pre-encryption rows keep reading until rewritten
— this also fixes registrar/source-connection reads that used the
RAISING decrypt variant on columns that may hold legacy plaintext.
Ratchet test freezes the remaining service-level crypto importers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One crypto path for model-owned *_encrypted columns: assignment encrypts
and RAISES on failure (server.py's accessors print()-ed the error and
silently stored nothing — an agent could 'pair' yet never verify a
signature); read decrypts tolerantly (None + logged warning). is_set()
standardizes the has_secret masking flag. Server adopts it for
api_secret / api_secret_pending; legacy method names kept as delegates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two dishonesty directions, one rule (plan 75 §A3/A4): a probe answers a
value or None, never 0 when undetermined — and a real 0 must survive.
Live paths normalized every missing reading to 0 while the history path
stored NULL for the same probe: agent counters an LXC agent can't
determine rendered as a perfectly idle box; load average on unsupported
platforms read 0.00 (and silenced the load alert); CPU freq read 0 MHz;
an empty history window summarized as '0.0 avg'. The mobile summary
called a method that does not exist and rendered 0/0/0 permanently.
One unavailable network counter raised uncaught and froze the entire
live broadcast.
Reverse direction: falsy traps (x if x else None) collapsed a genuinely
idle 0.0 into 'no data' in rollups, averages, and the history model.
Fleet: KPI tiles read '0.0 ms latency' / '100% success' with no data;
never-probed servers exported serverkit_server_up 0 to external
alertmanagers — the up-series now emits only for determined
connectivity. get_all_metrics is per-section fault-tolerant so one
failed probe can't blank the other six.
Every model with a JSON-in-Text column hand-rolled the same try/except
json.loads pair — get_details/get_context/get_headers/get_payload and a
dozen more, each a place the next fix would have to land N times. The
mixin owns the mechanics (empty-is-default, corrupt-is-default so a
hand-edited row never 500s an endpoint, optional shape check,
falsy-on-write); the per-model parts — accessor name, column, default —
stay per-model.
Conversion is behaviour-preserving: identical fallbacks, identical
on-write empty forms (NULL vs '[]' vs '{}'). Deliberately NOT converted:
accessors that let json.loads raise on corrupt data (collapsing those
would be a behaviour change, not a consolidation), and inline parses in
to_dict (per-entity serialization choice). A handful of 'except
Exception' reads narrow to the dominant (JSONDecodeError, TypeError) —
the only exceptions json.loads raises for str input.
A VPS resized from 1 vCPU/1 GB to 4 vCPU/8 GB with a 60 GB volume attached,
and the panel never mentioned any of it. Not a stale cache: a resize forces a
power-off, so the process and its in-memory spec cache die and the next boot
reads the truth. The gap is that nothing ever persisted the PREVIOUS numbers,
so there was nothing to compare against — and the reboot that causes the
change is what destroys the only copy of the baseline.
Adds host_snapshots: capture specs + per-filesystem inventory at boot, diff
against the last row, and emit host.specs_changed / host.storage_advisory on a
real delta. Advisories fire on state transitions only, so an unfixed fstab
entry does not nag on every restart.
Repairs a divergence this exposed: resource_tier_service measured the first of
/var/lib/serverkit,/var/lib/docker,/ while capacity_service hardcoded /. Both
now use host_inventory_service.data_path(), and local headroom reports which
mountpoint it measured.
Notifications render in the existing bell UI, so no frontend change is needed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review catch on #97. The chat connection test failure log interpolated
DeliveryResult.error verbatim. _deliver_webhook builds that from str(exc) on a
network error, and requests embeds the full URL in its exception text -- so the
line could carry a Discord/Slack/Teams webhook token or a Telegram bot token,
all of which live in the URL *path*. to_dict() masks `destination` even from an
admin, and /api/logs serves the log files back to the panel, so the log line
leaked past the API's own masking.
Dropping the reason would have restored the earlier problem of a failed test
button leaving no diagnostics anywhere, so it is redacted instead: URLs keep
scheme and host and lose the path, and bare bot tokens are substituted. The
host is what makes the line worth keeping and is not a secret.
Also normalise ChatWebhookConnection.raw_credentials() to {} when the stored
JSON is well-formed but not an object. A hand-edited or corrupt row could hold
`[]`, which every caller then treated as a mapping -- credentials() called
.items() on it, so to_dict() and therefore the list endpoint raised before the
write path was even reachable. Guarding at the source covers all callers and
matches the existing categories() pattern.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Incidents was showing the same idea twice: a converged "All incidents" table, and
under it a second table headed FLEET ALERTS with its own Active/Acknowledged/
Resolved segment row. A host alert IS an incident whose source is a host alert —
the SOURCE column already said so. FleetAlertsPanel is deleted (188 lines, and
Incidents was its only importer); alerts are folded into the existing row merge
with server -> subject, metric/value/threshold -> what happened, severity ->
impact, status -> state. Ack and Resolve move into the row-actions cell, shown
only for rows that have an alert id and are not resolved. The segment became an
"Acknowledged" builtin view alongside the Active/Resolved ones that already
covered its other two buckets.
Config Templates: the Compliant/Drifted/Unknown KPI band and its progress bar are
gone, replaced by a per-row Compliance column plus Drifted/Compliant builtin
views. The band could say three assignments had drifted but never which template
caused it; the column answers that per row. This needed the template payload to
carry its own tallies, which is the same single query the bare assignment count
already cost.
Recycle bin: the All / domains / saved-views segment row is gone — those buckets
are a filter on the Type column, and the Type column's own menu does it with live
per-value counts and can be saved into a view. Settings tab panes now sit on the
same card surface every other table in the app does.
Users: Add User and Invite User move onto the chrome line next to the filter and
"..." instead of each occupying a row of their own above its table.
SSL certificates: was a card list with a KPI band, one tab away from /domains
being a full table. Now a table with saved views (Expiring soon / Valid / Failed
or missing), search, column filters and export. The leading tile takes the row's
state too — a fixed green shield beside an expired certificate contradicted the
Status cell next to it.
Service detail: the Environment and Events tabs become tables with the inline
chrome an inner tab gets.
Fixes found by verification, in code this round touched:
- ServerTemplates had four columns with a `value` accessor and no `render`, so
DataTable fell back to row[key]: two rendered blank, one printed a raw array,
and `services` printed a list of OBJECTS, which React refuses to render — that
one took the whole tab down for any template declaring a service. Pre-existing,
and invisible because the screenshot fixture sent `services: ['sshd']` as
strings. The fixture now matches to_dict() so it cannot hide it again.
- useTopbarChrome returns a ready `actions` value. All eight hoisting pages
destructured only `portal` and dropped the fallback, so outside a tab group the
filter button and "..." would have gone nowhere rather than inline.
- sameViewState compares only the `page` keys a page still captures. Retiring a
key (as the recycle bin just did) otherwise stranded every view a user had
already saved with it, pinning "Unsaved" on with no way to clear it.
- The environment tab's "Applies to" column was gated on the app's CURRENT
compose services, so a variable targeting a since-renamed service showed
nowhere and read as applying to everything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 70. Domain went through this and TEN read paths were wrong, four with
effects outside the panel. Application has 211 call sites across 68 files and
drives more -- container lifecycles, nginx, backups, registry pulls, the OS
crontab -- so it was audited in full first (docs/plans/70_...md). 180 sites are
now query_active(); the 35 left unfiltered are each commented in place.
**The delete / purge split is what makes the tombstone honest.** delete_app used
to run compose_down(volumes=True) and rmtree(). Keeping a tombstone AND doing
that would put a Restore button next to a row whose data the delete had already
destroyed -- the one promise a recycle bin cannot break. So:
DELETE containers down (VOLUMES KEPT), vhost removed, firewall closed,
cron suspended, row tombstoned. All reversible.
RESTORE vhost rewritten, firewall reopened, cron resumed. Containers left
STOPPED -- "put it back" must not silently start processes or open
ports, the same reasoning that stops domain restore re-issuing certs.
PURGE volumes removed, uploaded source deleted, cron detached.
register() gained on_purge for that, wired into BOTH purge paths. And
purge_expired had NO scheduler -- fine when a tombstone was just a row, not fine
when it holds data volumes and a source tree, so a daily builtin job now runs
it. Without that, "kept until purge" would have meant kept forever and delete
would have quietly stopped reclaiming disk.
A side effect worth naming: the old _remove_data_flag protection for database
engines is now the DEFAULT for every app, and the flag is carried into
?purge=true so an explicit skip-the-bin delete still honours it.
**Cron was the blocker with no mechanical answer.** before_delete never fires on
a soft delete, and clear_application is LOSSY -- it nulls the association
without recording it, so nothing could put it back. suspend_for_application
instead disables only the jobs that were CURRENTLY ENABLED and stamps each
suspended_by_app; resume re-enables exactly those. That flag is the whole point:
without it a restore switches back on jobs the user deliberately turned off.
clear_application survives as purge-only, where lossiness is correct.
**The indirect reach (audit 3.1) is the class that grep cannot find.**
ContainerSleepPolicy, BackupPolicy, DeploymentJob and WafPolicy are enumerated
on a SCHEDULE and resolve application_id afterwards, so they never appear in a
search for Application.query. Left alone, a tombstoned app's sleep policy is
swept forever and wake_app restarts the containers the delete tore down. Those
loops test Application.deleted_ids().
Deliberately NOT scoped: reservation checks (port, name, directory) -- a
tombstone must keep holding those or a new app takes them and the restore is
refused; api/git.py's webhook VISIBILITY gate, which falls through to
"panel-wide visible" so hiding a tombstone would WIDEN access; and cron
attribution, which only labels a row.
Verified: 15 new tests, the migration replayed on a fresh DB (a tombstone frees
its private_slug, two live rows still collide), and a full delete -> restore
round trip. The 20 remaining full-suite failures are the known
serverkit-wordpress-not-installed class and are identical on a clean tree.
Two traps caught by testing, both recorded in the plan: a bad import made the
ENTIRE purge cleanup silently no-op while purge() still reported success
(_run_purge_hook swallows by design, so assert on the warning, not just ok), and
private_slug had to lose unique=True at the MODEL level or a models-built
database burns the slug regardless of the migration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Second endpoint after /apps, same pattern. Server.to_dict grows a `fields`
parameter so $select skips the derived work rather than only trimming the
payload: `onboarding_progress` is a JSON walk that is ~29% of a row and is read
by exactly one screen, and `group_name` is a relationship load. A caller that
needs {id, name} -- Domains and Backups both do -- now pays for neither.
`group_name` is eager-loaded when it IS wanted, so the list stops issuing one
SELECT per server for it.
Two details this route needed that /apps did not: it answers with a bare JSON
array, so no paging envelope is emitted (that would break every caller), and an
explicit $orderby now wins over the hardcoded by-name sort while the default is
preserved when absent.
3 more tests; the 293 existing server/fleet/agent tests still pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Field selection did not exist anywhere in the backend (`fields=`/`select=`: 0
hits across 90+ API files) and neither did eager loading (joinedload/
selectinload/subqueryload: 0 matches). /apps sends ~2,319 B per row where the
list UI reads ~411, over an unbounded `query.all()` -- for an admin with no
workspace header that is literally SELECT * FROM applications.
Ports the author's conuco_v2 query layer as app/api/_query.py, with two
deliberate differences:
- ORM, not Core. conuco resolves fields through `table.c`; here they resolve
against the mapper's column set, and that set is the ALLOWLIST -- an unknown
field is a 400, never a getattr into arbitrary model attributes.
- $select is not only a column narrowing. That is where the win actually is:
Application.to_dict() costs ~3 extra queries per row, and two of them
(image_scans, image_update_checks) are lazy='dynamic', so the attribute IS a
Query and NO loader option can hoist them. Declining a field is the only way
not to pay for it. to_dict(fields=...) now skips the loads it was not asked
for, so $select=id,name costs zero extra queries per row instead of three.
/apps also eager-loads server/project/environment/linked_app -- but only when
the request actually asked for the names they resolve, since a joinedload for
a field nobody selected is a wider row for nothing.
Every parameter is optional and to_dict(fields=None) is the full legacy
payload, so adopting this changed no existing caller's response. Paging meta
appears only once a caller pages.
$view is deliberately not implemented: ServerKit's saved views are per-user
rows already resolved in the browser, so a second server-side copy of the same
concept would only be somewhere for the two to disagree.
19 new tests. The 2 failures in the wider run (test_fail2ban_jail_service,
test_url_swap) reproduce on a clean tree -- pre-existing, not from this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Making Domain soft-deletable silently changed the meaning of every existing
`Domain.query`: before, it could only return live rows. An audit of all 16 call
sites found ten that were now wrong, several with side effects outside the
panel.
The ones that reached the outside world:
setup_reconcile_service.py:267 the only is_primary row after deleting the
primary IS a tombstone -- and it drives a WP-CLI search-replace across
the whole WordPress database. It would have repointed a live site at a
host nginx no longer serves.
setup_reconcile_service.py:83 DNS backfill enumerated deleted domains and
called ensure_a_record() -- creating a real record at Cloudflare/Route53
for a domain the user removed.
api/domains.py:367 (enable_ssl) a real ACME order for a host that is in no
server_name: the HTTP-01 challenge cannot be served, burning Let's
Encrypt failed-validation limits for the whole registered domain, and on
success installing a wrong-CN cert on the LIVE vhost.
doctor_service.py:267 permanent red "does not resolve" rows that no
action could clear, and the same set is the allowlist for one-click DNS
repair -- so Repair created a provider record for a deleted domain.
And the ones that broke the feature itself:
domain_attach_service.py:79 a tombstone counted as a name clash, so you
could not move a hostname to another app after removing it. Migration 083
made the unique index partial precisely to free the name; this re-imposed
the burn in application code.
domain_attach_service.py:71 attach found the tombstone, reported success,
created no live row, published nothing -- and then mutated a recycle-bin
record.
site_domain_service.py:600 same for give_subdomain: success, a URL, a DNS
record, and a site that was never published.
api/domains.py:532 a second vhost renderer (admin Regenerate)
writing tombstones back into server_name.
Application.domains still includes tombstones ON PURPOSE -- filtering it with a
primaryjoin would, with delete-orphan, make SQLAlchemy HARD-delete the row the
recycle bin exists to keep. `Application.live_domains` is the accessor for
"what this app actually serves", and to_dict() uses it so the /apps payload no
longer lists deleted domains.
Per-domain routes now 404 a tombstone instead of editing it.
Seven tests cover the leaks that had real consequences.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deleting is the only action in a control panel that repeating cannot undo, so
records a person destroys by hand now leave a tombstone and the Recycle Bin
hands them back.
SoftDeleteMixin adds deleted_at/deleted_by_id plus soft_delete/restore and
query_active()/query_deleted(). recycle_bin_service is a REGISTRY: a model
registers with a label function and an optional on_restore hook, so the API
lists and restores across types without knowing what a domain is. Adding the
next soft-deletable model is one register() call.
Applied to Domain (a real, destructive case -- deleting one used to lose the
record outright) and SavedView. DELETE /domains/<id> now tombstones and still
tears the vhost down; restoring a row does NOT re-do side effects unless the
type supplies on_restore, which is the honest default.
API: GET /recycle-bin (all kinds, newest first), POST
/recycle-bin/<kind>/<id>/restore, DELETE /recycle-bin/<kind>/<id> (purge,
admin-only -- the one irreversible action), POST /recycle-bin/purge-expired
with a 30-day default retention.
THE PART THAT NEEDED CARE: both tables had a plain UNIQUE covering what is now
a tombstone, so deleting a domain would have permanently burned its name and
deleting a view would have blocked re-creating one by the same name. Migration
083 replaces each with a PARTIAL unique index predicated on deleted_at IS NULL
-- supported by SQLite 3.8+ and Postgres. Tests pin exactly that: a deleted
name can be reused, and the live row stays unique.
Also gives SavedView a `slug` (backfilled), the handle the upcoming
?view=<slug> links resolve against. A tombstoned view does not reserve its
slug, so deleting a view frees the link name too.
Migration is idempotent on both columns and indexes: `deleted_at` is
index=True on the mixin, so a database built from the models already has that
index, and a bare create_index would abort the upgrade on exactly the installs
we most want to keep working.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SavedView model (user/page/name/state JSON/is_default, unique per
user+page+name), idempotent 082 migration, saved_view_service with
one-default-per-page enforcement, and a JWT-scoped /api/v1/views CRUD
blueprint. Covered by 9 endpoint tests.
VALID_TARGET_TYPES is now provider-driven; the wordpress_site branch
moves behind a provider the WP extension registers. ProtectionPanel /
RestoreDrawer read available types from the API; policies whose provider
is absent surface a clear missing-provider state instead of crashing
(plan 52 phase 3 task 13).
_maybe_attach_certificate throttled on last_check_at, which is the probe clock.
An active monitor is checked every 30s, so that condition was never older than
the 6h refresh window: the certificate would be read once and then never
refreshed, leaving a stale expiry date on the detail page forever.
Adds cert_checked_at (migration 081) and throttles on that instead. The stamp
is written even when the read fails, so an unreachable TLS endpoint is retried
on the same cadence rather than on every single probe.
Also serves the Monitors list sparkline from one bulk query per page instead of
one per row, bounded by a time window so a busy monitor's history cannot crowd
out a quiet one's the way a single global LIMIT would.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A StatusComponent was already a full synthetic check (http/tcp/dns/smtp/ping,
target, interval, timeout, HealthCheck history, uptime windows, auto-incidents)
— it was just framed as "a component of a status page" and its engine lived
inside the serverkit-status extension, so watching a site required publishing a
status page. Promote it:
- migration 081 makes status_components.page_id and status_incidents.page_id
nullable, and adds the columns backing the monitor UI (is_paused,
check_method, expected_status, keyword, follow_redirects, verify_tls, retries,
consecutive_failures, cert_issuer, cert_expires_at)
- new core app/services/monitor_service.py owns the check engine; the extension
keeps page publishing and delegates
- new /api/v1/monitors (CRUD, check, pause, history, uptime buckets, incidents)
- new builtin.monitor_check sweep on a 30s tick — until now checks only ran when
someone pressed "Check now", so a configured monitor never noticed an outage
Check-engine defects fixed while moving it:
- ping ignored the process return code and could only ever report 'up'
- ping used Linux-only flags with no guard, breaking the Windows dev server
- http ignored its own method/expected_status/follow_redirects/verify_tls
- smtp was documented on the column but unimplemented; keyword did not exist
- run_check never recomputed uptime and never opened or resolved an incident —
only the WordPress health-sync path did, so network monitors never paged
- TLS issuer/expiry are now captured (throttled to one read per 6h)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dashboard's layout lived in localStorage as `{id, visible}` — a visibility
toggle wearing a layout's clothes. It carried no geometry, so widget width was
baked into SCSS (`.dash-kpi { grid-column: span 3 }`) and "make the chart wider"
was a code change. And it stayed on one machine.
`DashboardBoard` gives a user real boards: named, ordered, each holding a JSON
array of placed widgets with x/y/w/h and per-instance config. Three defaults
ship — overview, infrastructure, apps & deploys — seeded on a user's first GET
and restorable per board.
Validation is deliberate about the boring cases: geometry must fit the 12-column
grid, ids must be unique within a board, bools are rejected as coordinates
(`bool` subclasses `int`), and a rejected PUT leaves the stored board untouched
rather than half-applied. A board belonging to another user is a 404, not a 403
— there is no reason to confirm it exists.
The shipped defaults are checked by rebuilding their occupancy map cell by cell,
so editing DEFAULT_BOARDS into an overlap fails the suite instead of shipping a
broken first-run layout.
31 tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An extension that needed to remember anything had to add a model to core AND an
Alembic migration for it — which is why extension tables like
ext_serverkit_minecraft_* sit in the panel's own migration history. That is a
steep tax on a small extension and it puts plugin schema in the panel's upgrade
path. A plugin now writes to a key/value store instead:
mine = store.for_plugin('serverkit-minecraft')
mine.set('world:1:seed', 8675309)
Values are JSON, so dicts and lists round-trip without hand-serialising, and
rows are namespaced by slug so two plugins cannot collide on a key. Reach for a
real table when you need to query BY the data — this is a key/value store, not
a database, and the docstring says so.
Because the table is shared by every installed plugin, three things are refused
rather than trusted: a value that isn't JSON-serializable, a value over 256 KB,
and an empty or overlong key. Validation happens at set() rather than at commit
— by commit time the error is opaque and the session is already poisoned, so
the next unrelated write fails too.
has() exists because get() cannot distinguish a stored None from a missing key,
and a plugin storing None deliberately would otherwise have no way to tell.
Concurrent writers race on insert; the unique constraint makes that safe and
the loser applies its write to the winner's row instead of failing the caller.
Store rows follow the same rule as extension-declared tables: an ordinary
uninstall leaves them, so a reinstall finds its state where it left it, and
"uninstall and delete data" removes them. The purge runs after the uninstall
lifecycle hook, since a hook may legitimately read its own store on the way out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 1 housekeeping for the full-isolation round:
- Move DatabaseSnapshot + SyncJob out of models/wordpress_site.py into a neutral
models/db_snapshot.py (pure module move, tables unchanged); wordpress_site.py
re-exports both for import compatibility.
- Untrack the stale committed live copies of serverkit-gui (sibling repo, no
in-tree source) and serverkit-email (source of truth: builtin-extensions/
serverkit-email/), matching how every other extracted builtin is already
gitignored. Files stay on disk; only removed from the index.
- .gitignore: add the email + gui frontend/backend artifact paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lets a panel link itself to a master ServerKit panel as a worker WITHOUT
installing the standalone Go agent: the panel registers via the master's
existing token flow, then runs an embedded agent client (HMAC auth + HTTP
long-poll transport) that dispatches agent commands (file:write,
docker:compose:up/ps/logs, system:info/metrics) onto its own local
services. The master needs zero changes — a linked panel is just another
agent-backed Server row with scoped, revocable credentials.
- linked_panel_config table (migration 077) + Fernet-encrypted secret
- embedded agent client: connect/poll/result loop, heartbeat metrics,
per-command threads, allowed-roots guard on file:write
- API: GET/POST/DELETE /api/v1/linked-panel (+ startup resume hook)
- Servers page: LinkedPanelCard (link form, live status, unlink)
- deployment_job_logs.id: BigInteger -> with_variant(Integer, 'sqlite');
BIGINT PK never autoincrements on SQLite, so the first log insert crashed
every fresh install (jobs stuck 'running' at step 0 with no logs)
- migration 076 rebuilds the table on existing SQLite databases
- runner: first status/log/telemetry inside try, non-raising log(),
rollback-safe failure marking — any failure now fails the job visibly
- service: guard enqueue, crash-proof run_job, fail jobs left 'running'
across restarts at boot
- repo-based services (POST /apps/from-repository) now enqueue a real
'app_deploy' DeploymentJob (build + start via DeploymentService.deploy)
and return deploy_job_id
- compose override injects restart: unless-stopped for services without a
policy, so apps come back after reboot
- GET /deployment-jobs gains an app_id filter
- frontend: shared DeploymentJobProgress component (timeout + visible poll
errors) used by the Templates InstallModal and the service Logs tab;
repo wizard navigates to the Logs tab to watch the deploy live; install
success lands on the Logs tab; failure toasts surface real server errors
DeploymentRunner.run() logged self.job.target_server_name, but the model
only exposed that value as a to_dict() key, so every deploy job crashed
with AttributeError at its first step and was marked failed before any
work ran. Add a target_server_name property and reuse it in to_dict().
Introduces a full-stack Test Sandbox feature to run distro matrix tests in Docker from the UI: new backend API/service, sandbox_runs model + migration, run/log endpoints, and a new frontend page with mode selection, run history, live polling, cancel, and per-distro logs.
Also adds ARM64 scripts CI coverage (ubuntu-24.04-arm) and strengthens install/test automation for minimal images and overlay rebuilds: early curl/bootstrap index refresh, stricter venv validation, release-venv glibc fallback, Node reprovisioning on source fallback, rsync --delete overlay behavior, npm bootstrap in VM install, Windows tar pinning in PowerShell harness, plus updated test docs and distro coverage notes.
