Faro access grants install keys through the agent's ssh:keys
actions. A locked-down server had no canned profile for that short
of Full Access; ssh_access grants ssh:keys:* plus the health reads.
The SSL-enable and server-overview changes put session and query
calls in the routes, tripping the controller-boundary ratchet. Move
them into SiteDomainService and WorkspaceService, raise a typed error
for bad allowed_ips, point the ACME-contact test at the new vhost
door, and regenerate the baseline and inventory (476 -> 453).
POST /servers/register needs no login and ran a slow PBKDF2 check
against every pending server per request, so a few IPs under the
5/min limit could keep the single worker's CPU busy. Tokens carry 256
random bits, so store a SHA-256 digest and find the row directly;
hashes issued before this still verify until they expire.
Only three routes checked can_access_server; the rest loaded the server
by id, so a developer outside a workspace could run cron, write files,
install packages or widen agent permissions on its servers, and the
list/overview/docker picker showed every server. Every <server_id>
route now resolves through the workspace check, lists filter in SQL,
and PUT /servers/<id> validates allowed_ips like PUT /allowed-ips.
A sweep test walks the URL map so a new unguarded route fails CI.
The first pass left routes that hand the body straight to a service
(create_zone(data), MonitorService.create(data), ...). Those services
read it as a dict, so a body of null, [] or a bare string still
crashed them into a 500. Agent long-poll, external webhook receivers
and the routes that already type-check their body are left alone.
328 more routes used a JSON body as a dict straight from
request.get_json(), so a body of null, [] or a bare string crashed them
into a 500. Bare reads become json_object(); the
'get_json(silent=True) or {}' reads of action routes become
json_object(optional=True), which still treats a missing body as {}.
Routes that take arrays or pass the body through untouched are left
alone. The full backend suite shows no new failures.
Also carries three files' bare section citations (apps, backups,
notifications) and the extended citation guard, which needed them.
The 3,000-line file mixed core server routes with Docker, agent
install/update/fleet and remote host operations. Those sections move
verbatim into servers_docker, servers_agents and servers_remote on the
same blueprint; the registered route map (127 rules, methods and
endpoint names) is identical before and after.
The controller-boundary scan only treated files that create a
Blueprint as controllers, so an area module would have escaped it; it
now also counts files that register routes.
The cron proxy answered an offline agent with 503 and a refused command
with 403, while 22 of the 24 Docker proxy routes answered 500 for both,
so the UI could not tell an offline agent from a crash. Every remote
handler now uses the same rule.
Eleven routes passed request.get_json() straight to data.get(...), so
a body of null, [] or a bare string crashed them into a 500. They now
read the body through json_object(), which answers a typed 400.
Comments, docstrings and a few strings cited plan numbers ("plan 29
#9", "plan 86 §B2") from a git-ignored folder, so they pointed at
nothing; one reached users in a bottleneck hint. Code and frontend
strings are unchanged apart from those citations (checked by AST and
token comparison). A vhost still carrying the old asset comment reads
as undrifted, and a guard test keeps new citations out.
The server access helper and the app-name clash checks queried the DB
from the API layer, which the controller-boundary scan rejects. They
now go through WorkspaceService.get_accessible_server and
Application.name_clash_error; the baseline drops by three.
The checks added to apps, servers and 2FA verify returned hand-shaped
{'error'} bodies, pushing that census past its ceiling. They now raise
the typed errors (a RateLimitedError joins them for 429). create_app
loads the user once through get_current_user(), which drops the
identity census to 68. Regenerates the migration inventory, stale since
the raw-subprocess ceiling fell to 23.
Creating a terminal, regenerating the pairing token and moving a server
checked the role but not workspace membership, so a developer outside a
workspace could get a shell on its servers or re-pair them to their own
agent. REST now uses the rule the Socket.IO rooms already applied, in
one helper. The Default workspace counts as shared: servers created
without a context land there and nobody is a member of it.
Toasts showed backend prose verbatim, with no code to translate
against and "application" where the UI says "service". Not-found,
missing-field, permission, conflict and agent-offline errors now come
from factories in app/exceptions.py that pair one fixed message with
a dotted code (not_found.<resource>, validation.required,
permission.denied, conflict.exists, agent.offline), so the client can
translate them.
The shared app-loading helpers and the apps, domains, servers,
databases and backups routes raise those errors instead of hand
shaping bodies, which drops the hand-shaped body count from 1132 to
850. Five routes picked an HTTP status by comparing error text; they
now key off a result code or a sentinel so rewording can't change a
status. Five broad excepts that echoed str(exc) now log the crash and
return the request id instead.
Vela holds one ServerKit API key as a host-side secret and polls the
fleet list, so this read route moves from bare @jwt_required() to
auth_required() + require_scope('servers:read') — a deliberate,
per-route jwt_only_census exception, not a sweep. JWT callers are
unchanged; a restricted key without servers:read gets 403 from the
scope middleware, no credentials get 401.
Scope choice: the existing servers:read catalog entry already covers
'list servers, agents, and metrics', and the serverkit-gui read routes
(capabilities/frame/surface, sibling repo) take the same scope, so one
key scope serves the whole Vela polling surface.
The census ceiling ratchets down to 600 and the migration inventory is
regenerated, as both ratchets require. tests/test_servers_api_key_scope.py
pins the four cases: scoped key 200, wrong-scope key 403, JWT 200,
anonymous 401.
GHSA-r4q7-x795-vr4j was one forgotten check, but the audit behind it found
the same shape across the surface: `@jwt_required()` authenticates, it does
not authorize, and ServerKit has no default-deny to catch the difference.
This closes the routes that audit left open — fleet-monitor alert
acknowledge/resolve, recycle-bin restore, nginx test/diff, registrar
sync/test, server-template drift checks, uptime tracking on/off, server ping
and remote compose ps/logs, refresh-capabilities, template
test-db-connection, deploy branches, buildpack detect — plus three inline
gaps: a viewer could widen their own API key's scopes, the demo-deploy flag
was doing double duty as an authorization one, and tagging a shared resource
had no write check at all while attaching one did.
`_ensure_group_mutable` only ever blocked *system* queue groups, so every
queue_bus mutation now also calls `_ensure_group_accessible` (owner-or-admin)
— renaming, draining or deleting another user's group was open to any
authenticated caller.
The rbac policy decorators authenticate through `auth_required()`, so they
replace `@jwt_required()` rather than stack under it; stacking rejects
API-key callers before the policy can authorize them
(tests/test_auth_decorator_boundaries.py enforces this). Where a route then
needed the caller it reads `get_current_user()` — `get_jwt_identity()` raises
for an API-key request. The decorators also carry a `_sk_authz` marker so a
static test can see the decision from the source.
`migrations.get_migration_history` moves to `require_admin_user()` for the
same reason its siblings did: the inline `User.ROLE_ADMIN` check it kept was
already reading two names this module no longer imports, so the route raised
NameError for every caller, admin included.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Migrate all 20 direct agent_registry.send_command call sites (api/servers,
terminal, fleet doctor/repair/fleet incl. the upgrade Thread target,
deployment_runner, survey, tunnel broker/publish, plugins SDK) onto the
shared dispatcher, and widen the ratchet from remote_*_service.py to a
tree-wide AST scan covering builtin-extensions/. The survey test's
transport stub gets the transport's real keyword signature.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 76 milestone A, drift/bug row 2 — scheduled as a fix, not a refactor,
because it is a behaviour difference shipping today.
rbac.get_current_user() is the only lookup that knows about both auth
mechanisms: it returns g.api_key_user for an X-API-Key request and falls back
to the JWT. 73 call sites answered "who is calling" with
`User.query.get(get_jwt_identity())` instead, behind 19 near-identical private
helpers (12 named get_current_user(), shadowing the real door).
The audit that scheduled this row predicted those return None for API-key
callers. Measured, it is worse: auth_required() deliberately skips
verify_jwt_in_request() once the key middleware has authenticated, so there is
no JWT context and get_jwt_identity() *raises*. Nine handlers reached through
an API-key-capable policy decorator therefore returned 500 to every API-key
caller:
bandwidth.get_app_bandwidth @viewer_required
servers.set_server_workspace @developer_required
servers.create_server @developer_required
sso.get_sso_config / update_provider_config / update_general_settings
source_connections.update_{github,gitlab,bitbucket}_admin_config
source_connections.complete_github_app_manifest
databases.execute_{mysql,pg,sqlite,docker}_query (see below)
Three service helpers had the same lookup inside a blanket `except Exception`,
which converted the raise into a silent user_id=None — the write landed and
the audit trail lost the actor (shared_resource_service, cf_ops_change_service,
secrets_webhooks).
The four databases.execute_*_query sites were a second admin check *inside* an
already-@admin_required route: dead code that always passed for JWT callers and
crashed for API-key ones. They now call require_admin_user() rather than
re-deriving the answer.
Also folds in a sixth _require_admin() found in the cloudflare-ops extension
source (29 gates, the same inverted contract as jobs.py), and fixes
serverkit-gui's blueprint at its sibling-repo source rather than in this repo's
live copy.
Ratchet (second-wave rule 1 — a migration without one regrows):
tests/identity_door_census.py counts both bypass shapes by AST, direct and the
two-line `uid = get_jwt_identity(); User.query.get(uid)` variant. The ceiling is
87, all of them on @jwt_required()-only routes where reading the JWT is correct
today; the ratchet's third assertion holds the API-key-capable population at
exactly zero, so migrating one of those routes to a policy decorator without
fixing its lookup now fails. Verified non-vacuous: an injected bypass fails both
assertions and passes again on revert.
The first wave's controller-boundary baseline is regenerated — this removes 74
persistence crossings (593 -> 519), a debt reduction with no new entries.
Gate: tests/test_api_key_identity.py (7, each red before the fix),
test_identity_door_ratchet.py (10), test_api_controller_boundaries.py (4), plus
the 65 test files covering every touched module: 1024 passed, 10 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three changes to the endpoints that serve the fleet-agent installers.
A missing installer answered 404. That is what sent issue #101's reporter
looking for a wrong URL -- they filed it as "incorrect paths in setup" -- when
the file had simply never been shipped into the image. The status code produced
the misdiagnosis. It is now 503, the response names the path that was searched,
and the same path is written to the panel log, because the response body does
not survive the `curl -f` the enrollment one-liner uses: curl discards it and
prints only "The requested URL returned error: 404". For an endpoint consumed by
curl, the status code and the server log are the only diagnostics that reach
anyone.
The panel's release lookup pointed at jhd3197/ServerKit and filtered for
`agent-v*` tags, which exist nowhere since the agent moved to its own repo, so
/api/v1/servers/agent/version returned 503 to every agent polling for an update.
It now uses AGENT_GITHUB_REPO (env SERVERKIT_AGENT_GITHUB_REPO) and a ^v(\d…)
tag filter that skips the agent repo's malformed "v" release.
_load_installer() is shared by the .sh and .ps1 routes so they cannot drift, and
SERVERKIT_SCRIPTS_DIR overrides the resolved directory for unusual layouts.
New tests cover the endpoints, the release lookup, and -- the one that would
have caught the original bug -- a packaging invariant that parses the
Dockerfile's COPY sources and asserts every installer the panel serves is
actually copied into the image. No endpoint test could have caught it: pytest
runs against the source tree, where the file always exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Second endpoint after /apps, same pattern. Server.to_dict grows a `fields`
parameter so $select skips the derived work rather than only trimming the
payload: `onboarding_progress` is a JSON walk that is ~29% of a row and is read
by exactly one screen, and `group_name` is a relationship load. A caller that
needs {id, name} -- Domains and Backups both do -- now pays for neither.
`group_name` is eager-loaded when it IS wanted, so the list stops issuing one
SELECT per server for it.
Two details this route needed that /apps did not: it answers with a bare JSON
array, so no paging envelope is emitted (that would break every caller), and an
explicit $orderby now wins over the hardcoded by-name sort while the default is
preserved when absent.
3 more tests; the 293 existing server/fleet/agent tests still pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CPU/Memory/Disk gauges read `server.metrics`, the same key the
per-server status endpoint uses (servers.py:757). The list endpoint never
populated it -- to_dict() defaults to include_metrics=False, and even when
asked it emits `latest_metrics` -- so on a live panel every gauge cell fell
back to the "no data" dash.
It survived because the screenshot fixtures DO supply `metrics`
(sections.mjs:294): the UI looked correct in every capture and was empty in
production. The fixture had encoded the intended contract; the backend never
implemented it.
Batched deliberately: Server.metrics is lazy='dynamic', so the obvious
per-row lookup costs one extra SELECT per server. Newest row per server =
highest id (autoincrement on insert), which also avoids the tie a
max(timestamp) join hits for same-second rows -- two queries total,
regardless of server count.
Tests pin all three properties, and each fails without the fix: the key and
field names the table reads, absence (not {}) for a server that never
reported, and that the query count stays flat as servers are added.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- version discovery pages agent-v tags (?per_page=100, 3 pages) so
panel releases can no longer push agent tags off page 1; panel
injects its resolved agent version into the served script and its
own releases fetch is paginated too (A1)
- downloaded agent binary is sha256-verified against the release's
checksums.txt: halt on mismatch, warn-and-continue when absent (A2)
- value-taking options error loudly when the value is missing; busybox
adduser fallback; no GNU-only grep -oP; pipe-safe source guard (A3)
- test-routing.sh fixed (PASS++ suicide under set -e, guarded probes,
curl 000000 false-pass) rather than deleted - still live tooling
- new scripts/test/test_agent_install.sh (18 passed); backend server
tests 110 passed
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four foundational enhancements toward the deploy-code developer experience:
- Container status aggregator (C7): new ContainerStatusService applies a
deterministic priority hierarchy (degraded > restarting > running:unhealthy >
starting > running:healthy > exited > unknown) over an app's containers,
cached with a short TTL. Exposed at /api/v1/status and pushed over a new
container_status Socket.IO channel; Applications page now renders an
aggregated ContainerStatusPill with sub-status tooltip.
- API token scopes + sensitive data masking (C10): require_scope() decorator
enforces fine-grained scopes for API-key requests (pass-through for JWT),
a canonical SCOPES catalog at /api/v1/api-keys/scopes, and a reusable
sensitive_data_filter (mask_sensitive) honoring secrets:read. Frontend gains
a grouped ApiKeyScopesModal picker and per-key scope pills.
- Server onboarding state machine (C8): new onboarding lifecycle
(pending -> validating -> installing_prerequisites -> installing_docker ->
pairing_agent -> ready/failed) with ServerOnboardingService, a
server_onboarding_logs table, servers.onboarding_* columns, REST endpoints,
and an OnboardingWizard surfaced on the server detail overview.
- Declarative template catalog (C5): documented catalog schema and
magic-variable resolution (SERVICE_PASSWORD_/USER_/FQDN_/URL_/BASE64_*)
layered onto the existing YAML template loader without changing behavior for
templates that don't use them; GET /api/v1/templates/catalog/schema.
Migration 029 adds the onboarding table + columns. 60 new tests pass; frontend
builds clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Enforce max_servers quota on server creation and workspace moves
- Add my_role/my_effective_role to single workspace GET response
- Add Navigation Permissions editor to Workspace Settings
- Store full active workspace object for nav/brand settings
Finish #33's two named workspace-scoping slices, backend-only (no schema
migration, no frontend changes):
- Domains: get_domains scopes through the parent Application via scope_query
(own + granted, workspace-narrowed); also surfaces granted apps' domains,
closing a gap vs the per-domain routes. Admin-no-context path unchanged.
- Databases: the app-linked /docker/databases enumeration scopes via
scope_query; host-level MySQL/PG/SQLite stay global system resources
(documented inline).
- Role reconciliation: WorkspaceService.effective_role + can_write_in_workspace
intersect global User.role with per-workspace WorkspaceMember.role,
narrow-only (a workspace role can only reduce capability; system admin never
capped). Enforced at apps/servers create + reassign seams (viewer -> 403,
only when a workspace context is active). Reconciled role surfaced on
GET /workspaces (my_role/my_effective_role).
6 new tests; full backend suite 193 passed / 12 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
POST /<id>/agent/update was @jwt_required() only, so a read-only viewer could
trigger agent binary replacement + service restart across the whole fleet —
inconsistent with every other state-changing server route, which is
@developer_required. Add @developer_required and a test proving a viewer gets
403 while a developer clears the gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes the organizing loop — existing resources were stuck wherever the
backfill put them (Default); now they can be moved into client workspaces.
- PUT /apps/<id>/workspace (owner-or-admin on the app) and
PUT /servers/<id>/workspace (developer+), both requiring the TARGET be a
workspace the caller can access (member or admin). A null target moves the
resource back to Default.
- UI: an 'Apps' management modal on each Workspaces card (mirrors the Members
modal) with move-in / move-out. It fetches with allWorkspaces (an empty
X-Workspace-Id header) so it can see apps in other workspaces to move them in,
sidestepping the ambient scope.
- Fixed a real bug caught by tests: the app ownership check compared app.user_id
to the stringified get_jwt_identity() (int != str -> spurious 403); now uses
user.id.
Verified: 2 new endpoint tests (ownership + target-membership + null->Default +
role gates), full suite 122 passed, frontend lint-clean + production build.
Servers reassignment is API-only for now; the modal is apps-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A stale or forbidden X-Workspace-Id (deleted workspace, removed member) now degrades to no-scope instead of 403/404, so an ambient workspace-context header can never break the apps/servers lists. Safe by construction: falling back to no-scope only ever shows the user their own resources and stamps the default workspace on create — it never grants access to, or creates in, a foreign workspace. Also drops the error-tuple from the four call sites.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cross-cutting backbone the rest of the #33 agency-scale milestone needs,
landed as a contained, narrow-only foundation rather than a 36-route rewrite.
- Nullable workspace_id FK on applications + servers (Alembic 015 off 014) with
an idempotent raw-SQL backfill: create a Default workspace, attach every
workspace-less row, and make every existing user a member so nobody loses
visibility once scoping is activated. Also creates the workspace_id indexes
that upgraded installs would otherwise lack.
- Centralized scope helper in WorkspaceService: ensure_default_workspace,
resolve_workspace_id (validates a requested workspace; membership-checked with
admin bypass; rejects deactivated accounts on the workspace path), and
scope_query.
- Opt-in + narrow-only: a request filters by workspace only when it carries an
X-Workspace-Id header / ?workspace_id=. scope_query INTERSECTS the workspace
filter with the existing per-user ownership filter, so it can only narrow
visibility, never broaden it — with no context the behavior is byte-identical
to before (admin->all, owner->own, servers global). This closes the footgun
where the backfill would otherwise let a non-admin enumerate all resources via
the Default workspace.
- Wired into apps.py + servers.py (list + create): replaces the duplicated
ownership branch and stamps new resources with the active-or-default workspace.
Deferred to follow-on slices: broad route adoption, per-site ACL, the
workspace<->User role reconciliation, server-side branding, and a frontend
workspace selector.
Verified: 10 new pytest tests (scope-narrowing, a can't-broaden-access
regression, deactivated-user gate, raw-SQL backfill against real SQLite +
idempotency), full suite 120 passed, a real end-to-end alembic upgrade to 015 on
a fresh DB (columns + indexes), and a 2-lens adversarial review (3 findings, all
fixed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce connstring package and tests to decode panel "connection strings" and add UI/CLI hooks to accept them. Move file access handlers into a dedicated file_handlers.go with strict allowlist checks that resolve symlinks and validate parent dirs to prevent escapes; add unit tests covering symlink and allowlist behaviour. Add credential-rotation HMAC verification, command blocking and configurable exec timeout resolution, gate system:exec registration by feature flag, and fix Docker logs reading to use io.ReadAll with a size cap. Misc: buffer restart channel to avoid IPC hangs, improve UDP discovery error handling, expose IPC token via a local endpoint and add IPCTokenPath, and wire the pairer to handle connection-string pairing and persist credentials. Overall changes focus on security hardening, UX for single-string pairing, and reliability fixes.
Four real root causes that produced the symptoms we've been chasing:
1. agent.log was 0 bytes because io.MultiWriter aborts the entire
write chain on the first writer error. On Windows services where
os.Stdout writes can transiently fail, the lumberjack file writer
never got the slog records — events.json (direct os.File) worked
fine while agent.log silently stayed empty for the entire process
lifetime. Replace io.MultiWriter with resilientWriter that
independently writes to each underlying writer and swallows
per-writer errors. Also surface logger init failures (MkdirAll,
OpenFile) instead of silently falling back to stdout-only, and
emit an explicit "logger initialized" startup line so an empty
file is now an obvious bug rather than ambiguous.
2. WS connections appeared to "drop every 2-3 minutes" but actually
the read loop had no ReadDeadline at all — ReadMessage() blocked
forever on a network-killed connection until the OS-level TCP
timeout fired, which is exactly what produced the
ws_connected → ws_disconnected pattern in events.json. Add
readWait = pingInterval*2.5 + 5s, reset it on every successful
read AND in SetPongHandler. Now CF/ngrok killing the conn at the
network layer (without sending a WS close frame) is detected
within seconds, the read errors out cleanly, the Manager's flap
detector counts it, and we fall back to polling far faster.
3. Newly-registered servers defaulted to permissions=[] which made
every action 403. The user hit "Permission denied" on
docker:image:list, system:info, agent:recapabilities, and probably
half the rest. Default to ['*'] when no profile or list is
specified — single-tenant deployments shouldn't need to manually
open a profile editor before anything works.
4. Migration 006 retroactively fixes already-stuck servers: any row
with permissions=[] gets ['*']. Non-empty admin-defined ACLs are
left alone. Solves "I registered before this fix and now I can't
do anything" without bulldozing legitimate configurations.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the panel restarts or the agent process crashes without a clean
disconnect, Server.status stays 'online' in the DB even though
agent_registry has no in-memory record. The UI then shows a
contradiction: "online" pill in the header, "Agent not connected"
on the Docker tab.
Reconcile in get_server: if there's no in-memory agent and the row
says online, flip to offline and persist. The capabilities cache +
last_seen still surface so the user sees the last known state, but
the UI now correctly reflects that the agent isn't reachable right
now.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Users had to SSH in to run `cloudflared tunnel login` before any of
the Tunnels tab worked. Wire up the OAuth flow so the panel can
trigger it and surface the auth URL for the user to open from their
own browser.
Agent:
- New cloudflared.LoginEvent + Manager.Login() that spawns the
process, regex-matches the dash.cloudflare.com/argotunnel URL on
stdout/stderr, and watches cert.pem in parallel as a reliable
completion signal. 15-minute hard ceiling.
- New cloudflared:login action returns {job_id, channel} immediately;
the goroutine emits start → status (with auth_url) → done events
on the job channel.
- Non-linux stub returns "unsupported".
- jobs.HasTerminated() helper so handler shutdown paths can avoid
double-emitting the terminator.
Panel:
- POST /servers/<id>/cloudflared/login + RemoteCloudflaredService.login
- ApiService.startRemoteCloudflaredLogin
Frontend:
- CloudflaredTab gains a "Login to Cloudflare" CTA when not authed.
Click → starts the agent flow, joins the job's Socket.IO room,
listens for the auth_url event, and renders an Open-in-Browser
button. Once the agent emits done, refreshes capabilities + status
+ tunnel list automatically — the rest of the tab unlocks itself.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The agent reports a rich capability payload (sudo mode, runtime
managers, detected runtimes, allowed paths, systemd JSON support)
that the panel was discarding once the WebSocket disconnected. The
Overview tab also gave no visibility into what the agent had
discovered, which left users guessing why a feature tab was missing
or a Docker call returned empty.
Backend:
- Server model gains seven nullable columns: cached_capabilities,
cached_runtimes, cached_runtime_managers, cached_allowed_paths,
cached_sudo, cached_systemd_json, capabilities_at. Migration 005
is idempotent (checks existing columns before adding) so it's safe
on already-bootstrapped databases.
- agent_registry.update_capabilities() now persists a snapshot to the
Server row whenever the agent reports state. Wrapped in try/except
so a DB hiccup doesn't drop the in-memory update.
- /servers/<id> falls back to the cached snapshot when the agent is
offline and exposes capabilities_stale + capabilities_at so the UI
can badge stale data.
Frontend:
- New SystemStatusCard renders sudo mode, package manager, runtime
managers, detected runtime versions, capability map (as colored
chips), and file-access roots. "Stale" badge + relative timestamp
+ Refresh button (calls agent:recapabilities).
- DockerTab now distinguishes "agent online but no docker daemon"
from "no containers found", with concrete remediation hints. Load
errors surface as a banner instead of being swallowed to console.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 3 (frontend), Phase 4 (template library), Phase 5 (pyenv runtimes).
Frontend:
- New Packages tab on ServerDetail: search, install/remove, upgrade-all,
update-cache, quick-install presets (nginx, redis, mariadb, postgres,
docker, fail2ban, certbot, htop). Capability-gated on packages.
- New Services tab: list systemd units with state filter, start/stop/
restart/logs per row, daemon-reload at top. Capability-gated on systemd.
- JobProgressModal subscribes to a server-stream Socket.IO room and
renders live install/build progress until the agent emits phase=done.
- ApiService: 13 packages/services methods + 9 runtime methods +
refresh-capabilities helper.
- LESS partials for both new tabs imported in main.scss.
Template library: 9 new one-click stacks added to ServerTemplate
TEMPLATE_LIBRARY: lemp, lamp, docker-host, node-host, python-host, redis,
postgres, cloudflared, fail2ban. Reuses existing packages/services/
firewall_rules JSON columns; the workflow engine's agent_command +
capability_gate nodes drive execution against the new agent surface.
Phase 5 — pyenv / pyenv-win runtime management:
- Agent: 9 new runtimes:* actions (list, pyenv:bootstrap, python:installed,
python:available, python:install, python:uninstall, python:set_global,
python:set_local, python:current). Bootstrap and install stream via
the job-channel pattern. pyenvLock mutex serializes mutations.
Capability probe surfaces RuntimeManagers["python"] = pyenv | pyenv-win.
- Panel: remote_runtimes_service.py + 9 routes under /servers/<id>/runtimes.
- Frontend: ApiService methods to drive the new endpoints.
(Dedicated RuntimesTab component left as a follow-up; the API
surface is the foundation.)
Build, vet, and tests pass across all three layers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Foundation for the remote server setup experience: agent gains
auto-sudo, package and systemd handler additions, missing docker
verbs, and a job-channel registry for streaming long-running
operations. Panel exposes the new agent surface as /servers/<id>/
packages, /services, and /refresh-capabilities routes.
Agent:
- New protocol actions: agent:recapabilities, packages:update_cache/
install_async/upgrade/search/info, systemd:daemon_reload/list_units/
logs/logs_follow, docker:container:create/exec, docker:image:build,
docker:volume:create, docker:network:create.
- Auto-sudo helper (sudo -n) probed at boot, surfaced as a capability;
apt invocations get DEBIAN_FRONTEND=noninteractive + dpkg force-confold.
- jobs registry with ring-buffer replay so panels that subscribe to a
job:<id> channel mid-flight still see prior progress events.
- Capability re-probe is mutex-guarded and additive (transient probe
failures don't drop features the panel already saw).
- Existing systemd handlers now sudo and use 2-min timeouts for cold
service starts (postgresql etc.).
Panel:
- remote_packages_service.py and remote_systemd_service.py mirror the
cron service shape (thin send_command dispatcher).
- 13 new routes on servers.py covering packages CRUD + search/info,
systemd unit control + logs + daemon-reload, and capability refresh.
- Long-running install/upgrade return {job_id, channel} for the
frontend to subscribe to via Socket.IO.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add Phase 4 agent primitives (package management, systemd control, bounded exec) and advertise file-access roots so the panel can target agents safely. Introduces agent/internal/agent/workflow_handlers.go with handlers for packages, systemd and system:exec, registers those actions, and extends capabilities.Probe and capability messages to include runtimes and allowed_paths.
Backend: add RemoteFileService and new /servers/<id>/files/* endpoints, store advertised allowed_paths in agent_registry and expose a getter, normalize agent system metrics shape in remote_docker_service, and wire capabilities/allowed_paths into the servers list. Extend workflow_engine with agent_command, capability_gate and runtime_gate nodes and basic runtime-version comparison.
Frontend: add TargetPicker component and update FileManager to support remote targets (browse/read/write via agent), guard unsupported remote ops, and normalize agent file responses for the UI. Overall this enables remote file browsing and Phase 4 workflows against connected agents.
Introduce Cloudflare Tunnel management end-to-end.
Highlights:
- Add agent/internal/cloudflared package (Manager interface, Linux impl, non-Linux stub) to shell out to cloudflared and detect auth via cert.pem.
- Wire cloudflared Manager into the agent and register cloudflared:* handlers for status/list/create/route/delete.
- Extend protocol.Capabilities and agent payloads with runtimes detection and a cloudflared capability; implement runtime version probes.
- Add backend API endpoints and RemoteCloudflaredService that forward cloudflared actions to agents; update agent registry to store runtimes.
- Frontend: new CloudflaredTab UI, API client methods for remote cloudflared operations, styles, and UX tweaks (plus minor component/class refinements in LogViewer, ProcessTable, FileManager, StyleGuide, terminal styles).
Purpose: allow the panel to manage Cloudflare named tunnels by delegating commands to the agent's cloudflared CLI, while surfacing binary/auth state (install vs authenticated) and detected language runtimes.
Introduce remote cron management across agent, backend and frontend.
- Agent: add a cron.Manager to agent, register handlers (cron:status,list,add,remove,toggle) and implement thin JSON handlers. Add agent/pkg/protocol message constants.
- Cron package: new platform-agnostic interface with Linux implementation (crontab-driven) and non-Linux stub. Includes content-based stable IDs, validation, list/add/remove/toggle and human-friendly schedule descriptions.
- Backend: expose /servers/<id>/cron/* endpoints and add RemoteCronService to proxy requests to agents via agent_registry, mapping agent responses to HTTP codes.
- Frontend: enhance Tabs component with overflow "More" popover; add API calls for remote cron ops; wire a Cron tab in ServerDetail (guarded by server.capabilities.cron) with UI for listing, adding, toggling and removing jobs; add Cron UI in Git page by switching to the new Tabs component.
Notes: cron functionality is Linux-only; non-Linux agents return an explicit unsupported status. Validation prevents unsafe shell operators and requires absolute command paths.
Probe and report agent feature capabilities (cron, docker, systemd, nginx, etc.) so the panel can gate target pickers without extra round-trips.
Key changes:
- Add agent/internal/capabilities: probes host for features and platform/distro, returns protocol.CapabilitiesMessage cached at agent startup.
- Agent: store probed capabilities, send them on every transport reconnect (sendCapabilities) and include them when using poll transport.
- Protocol: introduce TypeCapabilities and CapabilitiesMessage (map[string]bool + platform/distro fields).
- Poll transport: accept CapabilitiesMessage in Send(), serialize to raw payload and queue for next /poll; support one-shot capabilities in poll body.
- Backend: handle capabilities from socket (on_capabilities) and poll endpoint; persist capabilities in in-memory ConnectedAgent (agent_registry) with helpers get_capabilities/has_capability; include capabilities/platform info in servers API.
- New fleet API (GET /api/v1/fleet/targets): list runnable targets (local + connected agents) and optionally filter agents by a requested feature using the advertised capability map.
Rationale: make agent feature availability explicit and discoverable to the panel so UI pickers can show only applicable agents without asking agents on every UI interaction.
Introduce a fallback polling transport and manager: add internal/transport (Manager + poll client) so the agent can fall back from WebSocket to REST long-poll when tunnels mangle WS frames. Wire the Manager into the Agent and expose transport mode in IPC status.
Windows/service fixes: add service-scoped mutex to prevent concurrent service starts, probe IPC port on start to fail fast if occupied, and mirror main.Version into agent package at init. Fix spawned wizard visibility by setting HideWindow=false and add focus helpers that force foreground on Windows.
UI/asset changes: enable DevTools by default (env to disable), inject a small JS shim to forward console errors/navigation readiness to Go logs, hide WebView host until bundle ready (with timeout), and wrap static assets with no-store headers to avoid stale cached bundles. Add local /local/status endpoint so the UI can detect registration when IPC is unreachable.
Pairing/service handling: ensure service stop/start errors are surfaced, wait for service to reach RUNNING after pairing and surface helpful error messages if it doesn't. Implement platform-specific waitForServiceRunning and service control stubs.
Auth/backend alignment: expand API key prefix from 8 to 12 chars to match backend panel expectations and avoid authentication mismatches.
Misc: add focus_windows.go (Windows-only), various small refactors and logging improvements to make blank-page and pairing failures debuggable.
Introduce a deployment jobs system to support cross-server template installations and remote execution. Adds DeploymentJob and DeploymentJobLog models, DeploymentJobService for job lifecycle, and DeploymentPlanRunner to execute plans locally or via agents. TemplateService gains build_install_plan and related helpers to produce executable plans; Template installs can run async (202) or synchronously. Backend API/CLI updated: new deployment-jobs endpoints, apps API uses RemoteDockerService for remote compose actions, and CLI commands (deploy-template, deployment-status, list-servers) were added. Agent and config changes: file access validation with allowed_paths, file write create_dirs option, default allowed paths, and install script/systemd read-write paths updated to permit /var/lib/serverkit and /var/serverkit. Migration file for deployment_jobs schema included.
Add end-to-end support for removing remote Docker networks and volumes and normalize remote API responses. Server: new Flask endpoints to DELETE remote networks/volumes and helpers to compute external base/ws URLs and v1 API paths. Agent: register new handler docker:network:remove and Client.RemoveNetwork implementation. Backend service: RemoteDockerService.remove_network uses agent registry for remote servers and falls back to local DockerService. Models: expand Server permission logic to map agent actions to profile/legacy scopes (DOCKER_READ_ACTIONS, scope expansion and wildcard matching) and add unit tests covering permission mappings. Frontend: handle agent-backed response shapes (unwrapRemoteData, normalizeListResponse), call new removeRemoteNetwork/removeRemoteVolume API methods, update download/install URLs to /api/v1, and tweak several UI texts. Install scripts: change config/log directories, pass --config when registering/starting agent, and update systemd/service/unit and Windows service install paths. README: small wording updates about fleet/remote deployment and monitoring.
Adds cross-server Fleet Monitor endpoints and models, plus broad security and robustness improvements across agent, backend, and tooling.
Key changes:
- New fleet monitor API (endpoints for heatmap, comparison, alerts, thresholds, anomalies, forecast, search, exports, and Prometheus) and service skeleton.
- New DB models for ServerAlertThreshold and MetricAlert to support metric alerts.
- Agent discovery/registration hardening: HMAC-signed discovery requests, signature/timestamp validation, minimal discovery responses, HMAC-based unregister headers, and enforced command timeout cap in agent command handling.
- WebSocket and agent client TLS handling gated by SERVERKIT_INSECURE_TLS env var; added auth rate limiting on agent gateway.
- Docker-compose: parameterize agent image via AGENT_VERSION and avoid running as root by adding docker group access.
- Backend security & policy: tightened CSP script/style sources, request size limit, 2FA pending-token rejection on non-2FA endpoints, registration flow/guards, admin-only SSO admin routes via middleware, rate limits on sensitive endpoints, and logging for auth/2FA events.
- Database service hardening: identifier validation, avoid passing passwords on CLI (use env vars), safer subprocess usage, and docker exec handling improvements.
- CronService: command validation to prevent shell injection and run commands without shell=True; build_service and other subprocess usages updated similarly.
- Misc: .env.example notes clarified for encryption key generation, invitation token generation improved, agent registry timestamp window reduced, various logging and input validations added, frontend FleetMonitor page and styles stubbed in, and Vite/config/style updates.
Overall this commit implements fleet monitoring scaffolding and a set of security, validation, and operational improvements to make agent discovery, registration, command execution, and backend services safer and more robust.
Introduce full agent fleet management and discovery features across backend, agent, and frontend. Backend: add AgentVersion and AgentRollout models, extend Server/ServerGroup/ServerCommand/AgentSession with fleet, queue and latency fields; implement AgentFleetService (bulk upgrades, staged rollouts, queued commands, retries, diagnostics) and DiscoveryService (UDP broadcast scan); add new servers API endpoints for fleet operations. Agent: add UDP discovery responder, agent:update handler to trigger updater, and wire updater.UpdateTo to install specific versions; extend protocol messages with discovery and agent:update types. Registry: deliver queued commands on agent connect and track heartbeat latency. Frontend: add AgentFleet page and route, plus API client updates. Also add SECURITY_AUDIT.md to .gitignore. These changes enable discovery, staged rollouts, offline command queuing with retries, and fleet health/diagnostics.
Replace hardcoded repository URL from serverkit/serverkit to jhd3197/ServerKit across agent Dockerfile, README, packaging scripts (deb/rpm) and install scripts so downloads and metadata point to the correct repo. Backend install endpoints now replace the repo placeholder with the GITHUB_REPO constant, and the frontend Windows install command was adjusted to use the -Server parameter instead of -ServerUrl. These changes ensure install scripts, packages, and generated docs reference the intended repository and that generated installer commands match the agent's expected flags.
Juan Denis