Faro access grants install keys through the agent's ssh:keys
actions. A locked-down server had no canned profile for that short
of Full Access; ssh_access grants ssh:keys:* plus the health reads.
The SSL-enable and server-overview changes put session and query
calls in the routes, tripping the controller-boundary ratchet. Move
them into SiteDomainService and WorkspaceService, raise a typed error
for bad allowed_ips, point the ACME-contact test at the new vhost
door, and regenerate the baseline and inventory (476 -> 453).
POST /servers/register needs no login and ran a slow PBKDF2 check
against every pending server per request, so a few IPs under the
5/min limit could keep the single worker's CPU busy. Tokens carry 256
random bits, so store a SHA-256 digest and find the row directly;
hashes issued before this still verify until they expire.
Enabling SSL on a custom domain edited the vhost in place, but every
later rewrite (deploy, slot switch, domain attach, drift repair,
restore) rebuilt it from Domain rows that only knew wildcard certs,
so HTTPS silently disappeared while the row still said enabled.
Rebuilds now resolve each domain's own cert and render one server set
per certificate when domains differ. Enable/attach publish through
write_app_vhost and only claim SSL when nginx accepted it, the key
path is stored under its real column (legacy rows fall back to
certbot's privkey.pem), disable drops the HTTPS block, and a repeat
add_ssl_to_site swaps the cert instead of breaking nginx -t.
pg_create_database/user and pg_grant/revoke built superuser SQL from
raw names and passwords, unlike the MySQL helpers beside them. A quote
in a password broke CREATE USER, and the create routes ignored that
and returned credentials for a user that did not exist. Validate
identifiers and privilege lists, escape the password, and surface a
failed user as user_error instead of fake credentials.
Only three routes checked can_access_server; the rest loaded the server
by id, so a developer outside a workspace could run cron, write files,
install packages or widen agent permissions on its servers, and the
list/overview/docker picker showed every server. Every <server_id>
route now resolves through the workspace check, lists filter in SQL,
and PUT /servers/<id> validates allowed_ips like PUT /allowed-ips.
A sweep test walks the URL map so a new unguarded route fails CI.
Both used psutil without importing it, so every call has been a 500
since they were written. Nothing calls them: the process list and
service status come from /processes and /processes/services.
The first pass left routes that hand the body straight to a service
(create_zone(data), MonitorService.create(data), ...). Those services
read it as a dict, so a body of null, [] or a bare string still
crashed them into a 500. Agent long-poll, external webhook receivers
and the routes that already type-check their body are left alone.
The value went straight to a privileged journalctl -p. As its own argv
element it could not inject options, but anything other than levels
0-7 now gets a 400 instead of a journalctl error.
328 more routes used a JSON body as a dict straight from
request.get_json(), so a body of null, [] or a bare string crashed them
into a 500. Bare reads become json_object(); the
'get_json(silent=True) or {}' reads of action routes become
json_object(optional=True), which still treats a missing body as {}.
Routes that take arrays or pass the body through untouched are left
alone. The full backend suite shows no new failures.
Also carries three files' bare section citations (apps, backups,
notifications) and the extended citation guard, which needed them.
Comments cited plan sections as "(§G3)" or "(§8 unification)", which
point into the same git-ignored plans the plan-number citations did.
References next to a real document (docs/REDESIGN_MAP.md §3) stay. Code
and runtime strings are unchanged (AST and token comparison).
The 3,000-line file mixed core server routes with Docker, agent
install/update/fleet and remote host operations. Those sections move
verbatim into servers_docker, servers_agents and servers_remote on the
same blueprint; the registered route map (127 rules, methods and
endpoint names) is identical before and after.
The controller-boundary scan only treated files that create a
Blueprint as controllers, so an area module would have escaped it; it
now also counts files that register routes.
/system/health always said healthy, even with the database gone, so
container healthchecks and the installer trusted a panel that could not
serve; it now answers 503 when SELECT 1 fails. It also gave anonymous
callers the canonical domain and whether secrets encryption is set up;
those details now need a valid token (the Settings page sends one).
33 routes wrapped their body in try/except Exception only to return
unexpected_response(e), which records the crash and answers the same
body the global 500 handler already does. Every site was directly in a
route function, so the handler sees the same exceptions. SAML metadata
relied on its wrapper for an unconfigured IdP; that is now a typed 404
instead of a logged crash on a public, probed route.
The cron proxy answered an offline agent with 503 and a refused command
with 403, while 22 of the 24 Docker proxy routes answered 500 for both,
so the UI could not tell an offline agent from a crash. Every remote
handler now uses the same rule.
Eleven routes passed request.get_json() straight to data.get(...), so
a body of null, [] or a bare string crashed them into a 500. They now
read the body through json_object(), which answers a typed 400.
QueueBusError is an expected, caller-actionable failure, so it now
subclasses ApplicationError and the global handler answers it with its
own status. The 21 queue-bus routes each wrapped their whole body in
try/except only to do that by hand. Also records the two workspace
queries the previous commit moved out of the API layer.
The same "active workspace, else my workspaces, else all for an admin"
rule was written out in projects, environments and search. It now lives
in WorkspaceService, with a request-aware wrapper for routes.
Nine route files each loaded the live app and checked access, with
four return shapes, docstrings promising error tuples the code never
returned, and 41 dead 'if err: return err' branches. require_app()
backed by ResourceGrantService.get_app_for does it once; conceal=True
keeps the 404-for-denied answer the previews, WAF and slot routes
already gave, so no route's status changes. Nine query sites leave the
API layer.
Comments, docstrings and a few strings cited plan numbers ("plan 29
#9", "plan 86 §B2") from a git-ignored folder, so they pointed at
nothing; one reached users in a bottleneck hint. Code and frontend
strings are unchanged apart from those citations (checked by AST and
token comparison). A vhost still carrying the old asset comment reads
as undrifted, and a guard test keeps new citations out.
The server access helper and the app-name clash checks queried the DB
from the API layer, which the controller-boundary scan rejects. They
now go through WorkspaceService.get_accessible_server and
Application.name_clash_error; the baseline drops by three.
The checks added to apps, servers and 2FA verify returned hand-shaped
{'error'} bodies, pushing that census past its ceiling. They now raise
the typed errors (a RateLimitedError joins them for 429). create_app
loads the user once through get_current_user(), which drops the
identity census to 68. Regenerates the migration inventory, stale since
the raw-subprocess ceiling fell to 23.
Failed codes counted toward the account lockout, but verify never
checked it, so guessing could continue past the lock from fresh IPs.
A TOTP code also stayed reusable for its whole window, and a pending-
login token could complete more than one login. Spent codes and tokens
are now recorded in-process (the panel runs one worker) until they
would have expired anyway.
Masking was opt-in from the client (?mask=true), so a viewer grant on
an app read every secret value through the list, the single-variable
read and the .env export. Secret values now go only to users who can
edit the app.
POST/PUT /apps let any developer set root_path, compose_file,
systemd_unit and managed_by, and store a raw name. Root-run code later
runs docker compose from that root, renders the name into an nginx
vhost and joins it onto sites-available for rm -f. The host-binding
fields are now admin-only (as /manual already was), create slugs the
name and checks for clashes, and a rename must already be a slug.
The nginx site methods all share write_vhost's plain-filename check, so
enable/disable/delete can no longer be pointed outside the site dirs.
Creating a terminal, regenerating the pairing token and moving a server
checked the role but not workspace membership, so a developer outside a
workspace could get a shell on its servers or re-pair them to their own
agent. REST now uses the rule the Socket.IO rooms already applied, in
one helper. The Default workspace counts as shared: servers created
without a context land there and nobody is a member of it.
Toasts showed backend prose verbatim, with no code to translate
against and "application" where the UI says "service". Not-found,
missing-field, permission, conflict and agent-offline errors now come
from factories in app/exceptions.py that pair one fixed message with
a dotted code (not_found.<resource>, validation.required,
permission.denied, conflict.exists, agent.offline), so the client can
translate them.
The shared app-loading helpers and the apps, domains, servers,
databases and backups routes raise those errors instead of hand
shaping bodies, which drops the hand-shaped body count from 1132 to
850. Five routes picked an HTTP status by comparing error text; they
now key off a result code or a sentinel so rewording can't change a
status. Five broad excepts that echoed str(exc) now log the crash and
return the request id instead.
The deploy paths already know which host ports are taken (app rows,
slot rows, Docker mappings, a bind test) but only used that to pick a
port silently. Exposing it lets the shared port input say "8001 is
used by shop-api" and offer the next free port, instead of failing
at deploy time.
Vela holds one ServerKit API key as a host-side secret and polls the
fleet list, so this read route moves from bare @jwt_required() to
auth_required() + require_scope('servers:read') — a deliberate,
per-route jwt_only_census exception, not a sweep. JWT callers are
unchanged; a restricted key without servers:read gets 403 from the
scope middleware, no credentials get 401.
Scope choice: the existing servers:read catalog entry already covers
'list servers, agents, and metrics', and the serverkit-gui read routes
(capabilities/frame/surface, sibling repo) take the same scope, so one
key scope serves the whole Vela polling surface.
The census ceiling ratchets down to 600 and the migration inventory is
regenerated, as both ratchets require. tests/test_servers_api_key_scope.py
pins the four cases: scoped key 200, wrong-scope key 403, JWT 200,
anonymous 401.
Until now the connect client refused every http and sio stream, so
Open panel in ServerKit Cloud showed a blank page. The client now
bridges them to the loopback listener, the panel serves its page
under the relay's /t/<device>/ prefix, and the SPA prefixes its
router, API and Socket.IO paths from it. The Cloud grant is checked
against Cloud's JWKS before signing in the panel user with the same
email; panel two-factor still applies.
Plan 87 phase E, compose part. For a compose app on slots, the live
release is a slot project, not the project its own compose file names.
Status maps to the live project only (a stopped standby never reads as
a degraded app), start/stop/restart and sleep/wake act on it, container
listings show the live and standby projects, and an image update runs
as a queued slot deploy instead of recreating the stack in place.
Plan 87 phase C. A compose app on slots runs each slot as its own
project, <name>-a / <name>-b, rendered from the merged compose config:
container_name stripped, only the web service published, on the
slot's loopback port, and every named volume pinned to the name it has
today. Without the pin compose prefixes volumes with the project name
and slot B would start on empty ones; the real-Docker leg checks this.
Stateful services (a database, cache or queue with a named volume)
cannot run twice. They move once, on an explicit previewed split, into
a shared <name>-data project that both slots reach by the same names.
The webhook, template-update and rollback paths deploy through the
engine instead of stopping the stack. The --scale web=2 rolling
restart and the Webhook.zero_downtime flag are deleted: it assumed a
service named web, ignored its own health result, and no UI set it.
Compose slot images get immutable per-deployment tags and are pruned
with the rest.
Plan 87 phase D. Both slots share the database, so what changes it
runs once, before the switch, while the old release still serves:
the app's managed databases are dumped (a warning, never a blocker),
then the release command runs in a one-off container of the new
image. A failed release aborts the deploy with the live slot
untouched. The command comes from the app's setting, the Procfile
release: line, or serverkit.yaml; a rollback never re-runs it.
Rolling code back never rolls the database back: after a switch
back the slot card offers the pre-deploy snapshot as an explicit
second step with a data-loss warning. stop_old_before_release is
the announced-downtime fallback for schema changes that cannot stay
backward-compatible for one deploy.
Plan 87 phase B. An app that opts in keeps two slots, like the
panel updater's two installs. A deploy boots the new release in the
idle slot on its own loopback port while the live one serves, holds
it to the health gate, then switches nginx by moving app.port to the
new slot and rewriting the vhost, so every app.port reader follows
without edits. A watch window switches back on its own if the
release falls over; the old slot stays warm for an instant switch
back, then is stopped (not removed) by a standby sweep.
Opting in adopts the running container as slot a with no restart.
Eligibility is shown with reasons and re-checked on every deploy;
opting out folds the slots back on the next in-place deploy. Slot
images are never pruned and slot ports stay reserved.
Plan 87 phase A. Fixes the deploy bugs that make a rollback or a
health check lie:
- builds get an immutable tag per deployment, so rolling back runs
the image that actually ran instead of whatever :latest became;
old tags are pruned so this is not a new disk leak
- the deployment records the commit after the pull, not before
- the health wait is a real gate: a 4xx fails, it takes three passes
in a row, an exited container fails at once, and it raises
- a per-app deploy lock makes a second deploy wait for the first
- a compose rollback rebuilds and recreates its containers, and a
rollback with no version finds the previous release again
- a failed template update says when restoring the old stack failed
Health path, timeout and the 4xx rule are editable in Settings.
Replica counts could be changed, but nginx only proxied to one port
and the auto-scale sweep was never scheduled, so neither the
Auto-scale section nor its five routes could work; the nginx
upstream builder had no UI at all. With one live copy per app they
are gone, and the scale-policy table is dropped. Running containers
are not touched. The scaling spec now says replicas are not
pursued.
The Docker vhost set no caching at all. An opt-in rule serves files
whose name carries a content hash with a one-year immutable
Cache-Control, proxying exactly like location /. A name only counts
as hashed with 8+ characters and a digit, so app-settings.js is never
pinned. Off by default: the panel cannot know an app's asset layout.
A rule-based check reads the app's request metrics, its container
CPU and the CPU and top queries of the databases it references, and
names what fits: page cache for a busy app over an idle database,
a cache or index when one query dominates a busy database, slot
deploys for errors during deploys, and TTL for a low hit ratio. Each
hint links to the fix and states its failure mode; nothing is added
automatically.
An installed PostgreSQL can switch on a transaction-mode PgBouncer
sidecar through its managed override, with no host port, on the
shared network. fromService gains pooledUrl; apps keep the direct
connectionString unless they switch, because transaction pooling
breaks prepared statements in some drivers, LISTEN and advisory
locks, and the card says so.
The Prometheus, Grafana and Loki templates installed but connected
to nothing. A Grafana install now takes Metrics and Logs
attachments: the panel creates the data source through Grafana's
API with the recorded admin login, and a metrics attach also imports
a dashboard of the panel's fleet metrics. An attached app joins the
shared network, so Grafana reaches the source by name.
The metrics endpoint only accepted a PROMETHEUS_TOKEN env var that
almost no install sets, and the Prometheus template's config never
reached its container. The panel now keeps its own scrape token in
an internal vault, hands it to templates as SERVERKIT_METRICS_TOKEN,
and the template ships a prometheus.yml with a serverkit job.
The explorer showed live sessions but nothing about which queries
cost the most or whether the cache was working. A container's new
Insights tab lists top queries by total time (pg_stat_statements,
performance_schema digests) with connections in use and the buffer
cache hit ratio, and says why a list is empty. PostgreSQL's
pg_stat_statements is switched on through the config tuner, keeping
existing preloads, so its rollback undoes it.
Juan Denis