POST /servers/register needs no login and ran a slow PBKDF2 check
against every pending server per request, so a few IPs under the
5/min limit could keep the single worker's CPU busy. Tokens carry 256
random bits, so store a SHA-256 digest and find the row directly;
hashes issued before this still verify until they expire.
Comments cited plan sections as "(§G3)" or "(§8 unification)", which
point into the same git-ignored plans the plan-number citations did.
References next to a real document (docs/REDESIGN_MAP.md §3) stay. Code
and runtime strings are unchanged (AST and token comparison).
Comments, docstrings and a few strings cited plan numbers ("plan 29
#9", "plan 86 §B2") from a git-ignored folder, so they pointed at
nothing; one reached users in a bottleneck hint. Code and frontend
strings are unchanged apart from those citations (checked by AST and
token comparison). A vhost still carrying the old asset comment reads
as undrifted, and a guard test keeps new citations out.
Uptime columns defaulted to 100.0 and the pages formatted a missing
value as 100.00%, so a new monitor showed a perfect month. The average
also used 'uptime || 100', turning a fully down component (0%) into
100%. Missing data now reads as a dash and 0 stays 0; both status pages
share one helper.
The server access helper and the app-name clash checks queried the DB
from the API layer, which the controller-boundary scan rejects. They
now go through WorkspaceService.get_accessible_server and
Application.name_clash_error; the baseline drops by three.
Plan 87 phase C. A compose app on slots runs each slot as its own
project, <name>-a / <name>-b, rendered from the merged compose config:
container_name stripped, only the web service published, on the
slot's loopback port, and every named volume pinned to the name it has
today. Without the pin compose prefixes volumes with the project name
and slot B would start on empty ones; the real-Docker leg checks this.
Stateful services (a database, cache or queue with a named volume)
cannot run twice. They move once, on an explicit previewed split, into
a shared <name>-data project that both slots reach by the same names.
The webhook, template-update and rollback paths deploy through the
engine instead of stopping the stack. The --scale web=2 rolling
restart and the Webhook.zero_downtime flag are deleted: it assumed a
service named web, ignored its own health result, and no UI set it.
Compose slot images get immutable per-deployment tags and are pruned
with the rest.
Plan 87 phase B. An app that opts in keeps two slots, like the
panel updater's two installs. A deploy boots the new release in the
idle slot on its own loopback port while the live one serves, holds
it to the health gate, then switches nginx by moving app.port to the
new slot and rewriting the vhost, so every app.port reader follows
without edits. A watch window switches back on its own if the
release falls over; the old slot stays warm for an instant switch
back, then is stopped (not removed) by a standby sweep.
Opting in adopts the running container as slot a with no restart.
Eligibility is shown with reasons and re-checked on every deploy;
opting out folds the slots back on the next in-place deploy. Slot
images are never pruned and slot ports stay reserved.
Plan 87 phase A. Fixes the deploy bugs that make a rollback or a
health check lie:
- builds get an immutable tag per deployment, so rolling back runs
the image that actually ran instead of whatever :latest became;
old tags are pruned so this is not a new disk leak
- the deployment records the commit after the pull, not before
- the health wait is a real gate: a 4xx fails, it takes three passes
in a row, an exited container fails at once, and it raises
- a per-app deploy lock makes a second deploy wait for the first
- a compose rollback rebuilds and recreates its containers, and a
rollback with no version finds the previous release again
- a failed template update says when restoring the old stack failed
Health path, timeout and the 4xx rule are editable in Settings.
Replica counts could be changed, but nginx only proxied to one port
and the auto-scale sweep was never scheduled, so neither the
Auto-scale section nor its five routes could work; the nginx
upstream builder had no UI at all. With one live copy per app they
are gone, and the scale-policy table is dropped. Running containers
are not touched. The scaling spec now says replicas are not
pursued.
The Docker vhost set no caching at all. An opt-in rule serves files
whose name carries a content hash with a one-year immutable
Cache-Control, proxying exactly like location /. A name only counts
as hashed with 8+ characters and a digit, so app-settings.js is never
pinned. Off by default: the panel cannot know an app's asset layout.
An installed PostgreSQL can switch on a transaction-mode PgBouncer
sidecar through its managed override, with no host port, on the
shared network. fromService gains pooledUrl; apps keep the direct
connectionString unless they switch, because transaction pooling
breaks prepared statements in some drivers, LISTEN and advisory
locks, and the card says so.
A new collector template receives OTLP; Attach → Tracing gives an
app the standard OTEL_EXPORTER_OTLP_* variables, and a Jaeger
attached to the collector as its trace store receives the spans.
ServerKit does not instrument apps; it only wires the path. A docker
leg sends a span from an app and finds it in Jaeger.
The Prometheus, Grafana and Loki templates installed but connected
to nothing. A Grafana install now takes Metrics and Logs
attachments: the panel creates the data source through Grafana's
API with the recorded admin login, and a metrics attach also imports
a dashboard of the panel's fleet metrics. An attached app joins the
shared network, so Grafana reaches the source by name.
Installed Redis and a running app were unrelated until someone wrote
the env by hand. The Overview's Attached services card reuses an
installed service or installs one, writes the fromService references
(REDIS_URL, AMQP_URL/BROKER_URL, the S3_* set), shows that kind's
failure mode, and offers a redeploy. Detach removes exactly what the
attach wrote.
MinIO's images no longer pull, so object storage is now the Garage
template. Attach storage creates a bucket named for the app and a key
that can read and write only that bucket, keeps the secret in an
internal vault, and wires S3_* and AWS_* env through fromService and
fromSecret. Detach revokes the key and removes the env but keeps the
bucket and its objects. A docker leg proves the scope against a
real Garage.
passive_deletes left removal to ON DELETE CASCADE, which SQLite
does not enforce here (foreign_keys is never switched on), so a
hard-deleted app left its rollup rows behind.
Each site now sets its own TTL (10s default, 300s cap), and serves
the last good page while the app errors or the entry refreshes, with
one upstream request per missing entry. The cache key carries the
host, so purge clears only this site's files instead of every site's
cache. The settings panel shows the 24h hit ratio.
The route's access check moved into the service, where the per-app
authz sweeps can't see it; it is back in the route. The rollup table
now has a parent-side cascade so a purged app takes its rows, and
reads its histogram through the JSON column mixin.
A one-minute builtin tick tails each app's timed access log from a
stored byte offset and rolls it into minute and hour rows: status
classes, cache HIT/MISS/BYPASS/STALE and a latency histogram, so
p50/p95/p99 merge exactly across ticks and into hours. Minute rows
go after 48 h and hour rows after 30 days on the same tick. The
Overview tab gets a Traffic card.
Keep connection persistence behind the service boundary, use typed errors and shared JSON parsing, and refresh reviewed inventories. The new GET connection catalog accepts API keys so API clients managing connections can discover IDs without exposing credentials; existing JWT-only routes stay unchanged. Add regression coverage and release notes.
An audit found 102 FK columns with no index — every join, per-parent
listing, and the delete cascades from _delete_cascade_policy scanned
whole tables on them (deployments.app_id, audit_logs.user_id,
dns_records.zone_id, webhook_logs.webhook_id, ...). All FK columns now
declare index=True; migration 096 creates the same indexes on existing
installs, deriving the list from the live schema at upgrade time (so
it is idempotent and cannot drift from what an install actually has).
Ratchet: test_every_fk_column_is_indexed pins the rule with no
allowlist — a new FK column is born indexed or the build fails.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every NOT NULL FK from the KNOWN_DANGLING_ON_DELETE audit now declares
its parent-side relationship, so the delete-cascade policy door covers
it: AI conversations/pending actions, API keys, dashboards, event
subscriptions, login links, sent invitations, resource grants (users);
env vars + history, previews + settings, WAF policy (applications);
surveys, doctor results, proxy stack, tunnels (servers); manifests
(projects); DDNS hosts (zones); exposed services (tunnels); drills
(backup policies); rollouts (agent versions); site plugin installs
(WordPress sites); queue messages (groups). The ratchet set is now
EMPTY and stays that way.
One new deliberate exception: Workspace.projects — projects carry
environments and app groupings, so delete_workspace now refuses (409)
while any remain instead of silently destroying them (previously it
crashed with IntegrityError).
Migration 095 sweeps rows already orphaned on existing installs by
pre-fix hard-deletes — including an application's leftover env-var
secrets, which a reused SQLite row id could have attached to a future
app.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the 29 hand-written cascade='all, delete-orphan' declarations
from c1d4c0af/1704d60b/79a78c3c with a single mapper hook
(_delete_cascade_policy): at configure time, every one-to-many whose
child FK is NOT NULL gets the delete cascade automatically — with a
NOT NULL FK the children cannot outlive the parent, so the only choice
is cascade or crash. New models (extensions included — registering one
re-fires the hook) are covered with zero ceremony; nothing to remember.
Relationships where the parent delete should be REFUSED instead opt
out in the module's DELIBERATELY_UNCASCADED registry with their
justification (User.applications guarded by delete_user's 409,
CloudProvider.servers with no delete path). The sweep test now proves
the door works rather than policing model authors, and pins the
registry so it can only shrink.
Model files revert to their pre-fix text: delete semantics live in one
place instead of twenty.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up to 1704d60b, prompted by 'can this happen elsewhere?' — a
mapper-wide audit found 22 more one-to-many relationships whose child
FK is NOT NULL but which lacked a delete cascade, so hard-deleting the
parent (user/server/workspace/template/plugin delete, deployment
retention pruning, WP site teardown) raised the same IntegrityError as
the Recycle Bin purge.
20 get the cascade: plugin installs, deployment diffs, metric alerts,
template assignments (both sides), WP activities/sync/promotion jobs
(both sides), workspace members/keys, and the per-user detail rows
(passkeys, OAuth identities, notification prefs, sanitization
profiles, source connections, workflows, workspace memberships).
2 stay uncascaded on purpose, now enforced: User.applications (delete_
user refuses with 409 while the user owns apps — apps must go through
app delete + Recycle Bin purge so containers/volumes are cleaned) and
CloudProvider.servers (no delete path exists; dropping VM inventory
must stay a deliberate decision). The invariant sweep is now global
across every mapper with exactly that allowlist, which may only shrink.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Field follow-up to c1d4c0af: the live box then crashed on
deployments.app_id — dynamic relationships get the same FK null-out
on parent delete as the one-to-ones. Cascade every one-to-many from
Application whose child FK is NOT NULL (deployments, config
snapshots, image scans, SBOMs, image update checks, git deployments),
and pin the invariant with a mapper-inspection sweep so a future
NOT NULL child cannot reintroduce the purge crash.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Purging an app from the Recycle Bin crashed with IntegrityError
(NOT NULL constraint failed: container_scale_policies.application_id)
when the app had an auto-scale policy: with no delete cascade,
SQLAlchemy nulls the child FK on parent delete, and these one-to-one
rows declare it NOT NULL. Scale policy, sleep policy and wp_site all
share the shape; all three now ride the purge.
Reported on Discord (vaultwarden/hedgedoc delete-reinstall flow).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Migrations 065/068 have always created servers.management_mode and
servers.allow_agent_update_observed, but the model fields,
MANAGEMENT_MODES, is_managed, and agent_registry.observed_blocked_count
were lost in the 2026-07 data loss — POST /management-mode and
GET /observed-status raised AttributeError, and a mode flip that got
past validation would land on a transient Python attribute and
evaporate. test_survey_api carried skip-stubs documenting the hole.
Restored: the two columns (specs mirror the migrations), the mode
constant and is_managed property, to_dict emission, and a minimal
observed-block counter on the registry. The four skip-stubs are now
real tests, including one proving the flip persists across sessions.
The dispatch-level Observe guard remains hollow and is tracked in
plan 82 §H. events_json on notification_preferences is likewise
re-mapped (column carried by migration 059; its resolution surface is
still lost) so pre-loss override data survives.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RestorePoint hand-rolled json.loads for payload/coverage — now reads
through JsonColumnMixin._json_read like every other model, keeping the
models-json ratchet green.
test_restoring_the_last_domain_re_enables_the_site mocked create_site
and counted enable_site calls, but since write_vhost became the one
enabling door (enable=True re-links sites-enabled and create_site goes
through it), write_app_vhost no longer calls enable_site separately —
test_site_domain_writer_does_not_enable_twice pins that. The restore
test now asserts through write_vhost's enable flag instead, still
failing if a restore path ever opts out of enabling.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 79 A1/A2/A4 + B1/B2.
Delivery deliberately avoids the usual http-backend fetch. `en` is bundled;
other locales are dynamic imports Vite emits as chunks from the app's own
static origin. The sign-in and setup screens render before any API call
succeeds, so a fetched locale either flashes English or -- when the backend
is the thing that is broken -- never arrives, on the one screen whose job is
to say so. It also keeps air-gapped and subpath installs working with no
second path to 404.
Precedence (user row -> localStorage -> panel default -> navigator -> en)
mirrors ThemeContext's, including the split provider: LocaleProvider sits
outermost so a toast raised during theme init is already translatable, and
LocaleSync bridges auth state down, exactly as ThemeSync does.
users.language is nullable on purpose -- "follow the panel default" is a
different state from an explicit choice of English, so existing rows are not
backfilled. The backend restates the supported list to validate writes; a
parity test fails the build if it drifts from the frontend manifest, which
stays canonical.
Two bugs the tests caught, both real rather than test artifacts:
- normalize_language('ES') returned None. The shape gate was case-sensitive,
so two of the three forms browsers actually send were rejected.
- PUT /me is throttled 3/minute for credential edits, and language rides the
same route. Switching language three times returned 429 -- and because the
client applies the change locally first, the preference silently failed to
persist and reverted on the next sign-in elsewhere, defeating the column.
Cosmetic-only payloads are now exempt from that limit; credential edits
stay throttled (proved both ways) and the app-wide 100/minute still holds.
eslint moves to ecmaVersion 'latest' because import attributes are ES2025 and
Node's test runner requires them for the manifest imports.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First of the regret-ranked adoptions: a plain delete now tombstones the
row (migration 088; no unique constraints, so no partial-index rewrite).
The database content and its managed BackupPolicy survive the bin window
— restore brings everything back intact; PURGE removes the policy but
NEVER drops database content; the explicit ?drop=true path stays
immediately destructive. Active read paths (list/get/adopt/manifest/env
references/app detail) switch to query_active(); the backup executor
deliberately stays tombstone-blind so bin-parked data keeps backing up.
Server (98 query sites, fleet enrollment + HMAC secrets) is the next
ranked adoption — deliberately its own pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
env_variable's class-level SECRET_KEY-derived Fernet (reached
cross-model by shared_resource) and sso_service's private stack now
WRITE through the one key path (SERVERKIT_ENCRYPTION_KEY) and dual-READ:
one path first, legacy key second — permanent window, so rows never
rewritten keep decrypting and no bulk decrypt-all/re-encrypt-all pass
ever runs (PR #94 postmortem). The '[DECRYPTION_ERROR]' sentinel and
None-on-failure token semantics are preserved.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Opt-in to_dict over __table__.columns: auto-isoformat datetimes,
*_encrypted always dropped, __serialize_exclude__ for redacted columns,
serialize_extra() for computed fields (named to dodge ServerMetrics'
literal 'extra' column). 20 purely-mechanical to_dicts across 15 files
convert; genuinely custom shapes keep their hand-written methods (plan
75 §F's ruling stands). Field selection stays api/_query.py's job.
Gate: key-for-key parity against the pre-conversion key sets frozen
from git HEAD (only the Timestamp pair may be additive).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RunLogStream accepts any RunLike (__run_kind__ + id + to_dict, optional
result/step members): one room grammar run_<kind>_<id>, one batched
persisted run_log/run_status event pair, and a GET
/api/v1/runs/<kind>/<id>/logs?after_id polling twin over the new
run_log_entries table (migration 087). Deploys keep DeploymentJobLog and
dual-emit the legacy deploy_log/deploy_status during the migration
window. The unified Job runtime — which had NO live channel — adopts it
in the consumer (start/succeeded/failed frames; handlers can log via
stream_for(job)). The dead pipeline_* scaffolding was retired in E2.
Gate: emit -> persisted row -> after_id resync contract test; consumer
adoption test. MIGRATION_INVENTORY.md regenerated (census moved).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 11 hand-written status+completed_at(+error_message) transition
blocks in deployment_job_service collapse into mark_running/
mark_succeeded/mark_failed. The model keeps its stored 'succeeded'
spelling via the mixin override until this domain's data migration;
error text routes through __error_attr__ = 'error_message'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
~60 call sites across 24 model files now ride _json_read with each
site's exact default preserved; 34 classes newly adopt the mixin
(registry contract test expanded to 36). Reverses the old 'bare sites
stay raising' ruling: a corrupt row degrades to its default instead of
500ing every serializing endpoint (docstring updated). Ratchet:
json.loads( appears only in json_column_mixin.py.
Known pre-existing: test_host_inventory enumerate_filesystems failures
on this machine (verified via clean-tree stash rerun).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
models/status.py defines the six lifecycle constants, TERMINAL/ACTIVE
sets, and normalize() mapping every measured legacy spelling (succeeded/
completed/done/failure/error/canceled/scheduled) home at the read edge —
stored strings migrate per-domain later, never wholesale.
RunLifecycleMixin centralizes mark_running/succeeded/failed/cancelled
transitions (status + started_at/completed_at/duration), with per-model
spelling overrides so adoption never silently rewrites a domain's stored
vocabulary. Parity gate: every measured spelling normalizes canonical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TimestampMixin (byte-identical to the hand-written pair — zero schema
migrations) adopted by 52 classes across 44 model files; the utcnow
tz-aware migration is now a one-file edit. uuid_pk() replaces the 13
pasted String(36) uuid4 lambdas across 7 files (ai.py's hex variant
stays grandfathered, documented). Deviant shapes (indexed created_at,
sanitization_profile's default-less updated_at, created-only models)
deliberately keep their own columns.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six more models own their *_encrypted columns through the descriptor
(linked_panel, source_connection, container_registry, managed_database,
registrar_connection, pending_agent); eight service/api files stop
importing utils.crypto for them. legacy_plaintext mode mirrors
decrypt_secret_safe so pre-encryption rows keep reading until rewritten
— this also fixes registrar/source-connection reads that used the
RAISING decrypt variant on columns that may hold legacy plaintext.
Ratchet test freezes the remaining service-level crypto importers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One crypto path for model-owned *_encrypted columns: assignment encrypts
and RAISES on failure (server.py's accessors print()-ed the error and
silently stored nothing — an agent could 'pair' yet never verify a
signature); read decrypts tolerantly (None + logged warning). is_set()
standardizes the has_secret masking flag. Server adopts it for
api_secret / api_secret_pending; legacy method names kept as delegates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Juan Denis