Twenty-one imports whose last caller left during the plan-76/77 sweeps:
mostly 'import subprocess' in services now routed through the shared
runners, plus time/requests in the CachedRemoteIndex adopters, threading
in agent_gateway, os in api/system, User in api/bandwidth, run_privileged
in nginx_advanced_service, and the dead SENSITIVE_SETTING_PARTS alias in
audit_service (zero importers). Each removal AST-verified unused before
deletion; all modules compile and the app factory boots.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The full-suite run tripped four ratchets on plan-77's own additions —
exactly their job: /api/v1/runs moves onto auth_required (the JWT-only
ceiling holds at 605) and behind run_log_service.list_run_logs (no
model access in the controller); the core-blueprint snapshot records
runs_bp deliberately (108 -> 109 + sha); sso_service joins the crypto
allowlist (C2 dual-read lives there by design); the controller-boundary
baseline re-records the two pre-existing ManagedDatabase crossings under
their B5 query_active spelling (count unchanged).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First of the regret-ranked adoptions: a plain delete now tombstones the
row (migration 088; no unique constraints, so no partial-index rewrite).
The database content and its managed BackupPolicy survive the bin window
— restore brings everything back intact; PURGE removes the policy but
NEVER drops database content; the explicit ?drop=true path stays
immediately destructive. Active read paths (list/get/adopt/manifest/env
references/app detail) switch to query_active(); the backup executor
deliberately stays tombstone-blind so bin-parked data keeps backing up.
Server (98 query sites, fleet enrollment + HMAC secrets) is the next
ranked adoption — deliberately its own pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RunLogStream accepts any RunLike (__run_kind__ + id + to_dict, optional
result/step members): one room grammar run_<kind>_<id>, one batched
persisted run_log/run_status event pair, and a GET
/api/v1/runs/<kind>/<id>/logs?after_id polling twin over the new
run_log_entries table (migration 087). Deploys keep DeploymentJobLog and
dual-emit the legacy deploy_log/deploy_status during the migration
window. The unified Job runtime — which had NO live channel — adopts it
in the consumer (start/succeeded/failed frames; handlers can log via
stream_for(job)). The dead pipeline_* scaffolding was retired in E2.
Gate: emit -> persisted row -> after_id resync contract test; consumer
adoption test. MIGRATION_INVENTORY.md regenerated (census moved).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
config.py's parsers move to app/utils/env.py and are imported back; the
three incompatible truthiness variants collapse to one TRUTHY tuple.
Fixes the measured bug: SERVERKIT_ALLOW_PRIVATE_DOWNLOADS=on silently
read false in plugin_service (its tuple lacked 'on'); same for the pip
gate, SERVERKIT_STAGING, and SERVERKIT_DEMO_MODE. Ratchet: no new
os.environ truthiness tuples outside utils/env.py.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
agent_registry.authenticate_and_register owns the full handshake
(rate-limit -> HMAC -> allowed-IPs/anomaly -> new-IP -> version ->
register) and ingest_agent_state owns heartbeat/sysinfo/capability
side-effects; the WS namespace and long-poll endpoints become thin
adapters translating error codes to emit/disconnect vs jsonify/status.
The per-IP auth limiter moves to the registry (re-exported from the
gateway for existing callers/tests). Gate: parametrized ws/poll matrix
(bad HMAC, blocked IP, rate limit, new-IP anomaly) + a word-anchored
ratchet that the adapters never re-grow the sequence.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six more models own their *_encrypted columns through the descriptor
(linked_panel, source_connection, container_registry, managed_database,
registrar_connection, pending_agent); eight service/api files stop
importing utils.crypto for them. legacy_plaintext mode mirrors
decrypt_secret_safe so pre-encryption rows keep reading until rewritten
— this also fixes registrar/source-connection reads that used the
RAISING decrypt variant on columns that may hold legacy plaintext.
Ratchet test freezes the remaining service-level crypto importers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
QueryParseError is now a ValidationError subclass — the global handler
answers it as a 400 and per-route translations become unnecessary (the
existing ones keep working). PageQuery parses, defaults, and bounds
page/per_page in one place; admin audit-logs, activity feed, error logs,
and telemetry events adopt it (telemetry keeps its 200 cap). git.py and
source_connections.py keep hand-parsed params deliberately — they proxy
pagination to Gitea/GitHub/GitLab/Bitbucket, not to DB queries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The inbound PR webhook must keep answering 200 (providers disable failing
hooks), but its preview-settings crash only reached logger.debug — the
last route swallowing an exception without putting it on the record. It
now goes through record_unexpected before the in-band answer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DbConfigTunerService raises ValidationError (preconditions/settings),
DependencyUnavailableError (engine/container operations, with the
restore-then-report semantics preserved in the message), and NotFoundError
(nothing to roll back to); db_tuner.py drops its three error-key
ternaries. The census docstring now records the completed migration and
corrects its own stale 'eight callers' blocker. Census 3 -> 0 — held as an
invariant from here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SecretVaultService, SecretService, and WebhookGatewayService raise typed
errors and return bare data; secrets_webhooks.py drops all twelve
{'success': bool} translations including its two prose sniffs. Bulk
create keeps per-item error collection (catching ApplicationError).
Replay now answers 200 with success:false when the forward target refuses
— previously a failed forward was a 404 with error:null — and the
WebhooksTab toast distinguishes the two. The public /webhooks/receive
wire protocol is intentionally unchanged (external senders depend on it).
Sniffing census 6 -> 4.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
theme_registry_service.install returns the theme dict and raises
NotFound/Validation/DependencyUnavailable; the route's not-found prose
sniff is gone. import_theme keeps its tuple contract for the import
route. Sniffing census 7 -> 6.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
update_ip raises AuthenticationError for a bad/disabled token (401 via the
global handler, as before) and ValidationError for a bad IP; the route's
except-ValueError message sniff is gone. Sniffing census 8 -> 7.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
enqueue_scan/resolve_app_path raise NotFoundError/ValidationError; the
scan routes drop their except-ValueError prose sniffs and let the global
handler map. Sniffing census 9 -> 8.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
check_application raises NotFoundError/ValidationError and returns the
check dict bare; the route stops re-deriving status codes from message
text. Sniffing census 10 -> 9.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
saved_view_service raises ValidationError/NotFoundError/ConflictError and
returns bare dicts; views.py drops the (value, err) tuple translations and
its not-found prose sniff. Duplicate names are now 409. Sniffing census
11 -> 10.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The C1/C2 reference conversion: FileService raises typed errors
(PermissionDenied/NotFound/Conflict/Validation) and returns bare data;
files.py stops translating {'success': bool} envelopes and stops choosing
status codes by sniffing error prose. Denials are now 403 regardless of
wording, missing paths 404, already-exists 409. Success bodies keep the
compatibility 'success': True key so consumers are unchanged. The plan's
'eight callers' blocker was a substring-grep artifact — FileService has
exactly one consumer. Sniffing census 23 -> 11.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Migrate all 20 direct agent_registry.send_command call sites (api/servers,
terminal, fleet doctor/repair/fleet incl. the upgrade Thread target,
deployment_runner, survey, tunnel broker/publish, plugins SDK) onto the
shared dispatcher, and widen the ratchet from remote_*_service.py to a
tree-wide AST scan covering builtin-extensions/. The survey test's
transport stub gets the transport's real keyword signature.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-up to a7d5a0b9, and a hole in its ratchet.
queue_bus.py answered 21 crashes through its own `_handle_error(e)`, which
returned 500 with str(e) for anything that was not a QueueBusError. Those are
exactly the unreported crashes milestone B set out to remove, and the census
missed every one of them: it looked for a literal 500 in the except handler, and
here the 500 lived one call away in the helper.
A ratchet that a one-line helper defeats is not a ratchet, so the census now
follows one level of indirection — a return of a same-module function that
itself answers 500 without reporting counts as answering 500. Verified
non-vacuous with a probe module using exactly that shape.
`_handle_error` now keeps a QueueBusError's own message and status (it is an
expected, caller-actionable failure) and routes anything else through
unexpected_response, fixing all 21 sites at once.
Worth recording how this was found: measuring the "handlers that swallow and
answer 200" population, which the plan had at 56. The real number is 1 — the
earlier figure counted any handler whose return the AST could not resolve to a
status, which included every `return unexpected_response(exc)` the migration had
just introduced and every `return _handle_error(e)`. The plan is corrected.
Gate: -k queue_bus (66 passed), crash ratchet + reporting tests (20 passed).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 76 milestone A, the third row. 23 handlers decided admin access inline,
all in two files, in two shapes.
18 were the mid-route gate this milestone already built a door for:
current_user_id = get_jwt_identity()
user = User.query.get(current_user_id)
if user.role != 'admin':
return jsonify({'error': 'Admin access required'}), 403
which is four lines to say require_admin_user(), and carried the two-line
identity lookup with it. Migrating them closes both rows at once: the identity
census drops 87 -> 69 and the controller-boundary baseline loses 18 persistence
crossings (528 -> 510), with no new entries.
It also fixes a latent 500: where the check omitted the `not user or` guard, a
token naming a deleted user hit `None.role` instead of answering 403.
The other 5 are `not user.is_admin and app.user_id != user.id` — admin OR
owner, which is a different question. Those keep their shape and only converge
the truth source from `role != 'admin'` to `is_admin`; routing them through
can_access_app would silently widen access, since that also honours resource
grants.
The migration initially left `current_user_id` referenced but unbound in four
places. Two were caught by tests; the other two (create_manual_app,
upload_app_archive) have no coverage and were caught by an AST scan for names
used-but-never-assigned. Worth remembering that a mechanical removal needs the
scan, not just the suite.
Gate: -k "system or apps or app_ or host_snapshot or manual_app or upload",
304 passed / 13 skipped; all seven ratchets green (52 passed).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 76 milestone B, drift/bug row 4. 43 API handlers answered HTTP 500 from
their own `except Exception`, so the global 500 handler never ran — and that
handler was the ONLY caller of error_log_service.record_error(source='backend').
Every one of those crashes was absent from /monitoring/errors, and the page read
empty precisely because someone had "handled" them.
35 of the 43 also returned str(exc) to the caller. Two returned the full
traceback (templates.py, which also print()ed it), and several logged with a
locally-minted uuid4 'ref' that correlated with nothing but a log line.
The recording side of the 500 handler moves to app/error_reporting.py, so it is
importable rather than reachable only by not-catching:
record_unexpected(exc) -> logs, rolls the session back, records, returns
the request id
unexpected_response(exc) -> the above plus the documented crash body
The 500 handler now calls the same pair, so there is one recorder rather than
one recorder and 43 routes that quietly opted out of it.
Caller-visible change on those endpoints: the body is now the documented
{'error': 'Internal server error', 'status': 500, 'code': 'internal_error',
'request_id': ...} instead of the exception text. That is the point — an
unexpected exception's text is written for whoever reads the log and routinely
carries connection strings, absolute paths, and SQL. The request id is the part
a caller can act on, and it matches the row in the error log. No frontend code
read the 'ref' or 'trace' keys this drops (checked).
Two sites keep their own wording and call record_unexpected instead: sso.py's
login failures should not read as a generic crash on the login page.
docker.py's bulk endpoint keeps its per-item `results` — the caller's half-done
work survives, only the exception text is dropped.
Ratchet: tests/unreported_crash_census.py + its test, held at **0**, not a
legacy baseline to shrink. It counts blanket handlers that answer 500 without
calling either reporter, and is scoped to app/api — a service catching an
exception and returning a value is making a domain decision, not declaring an
HTTP crash. Verified non-vacuous: an injected handler fails it, and it passes
again on revert.
Measured and deliberately still open: 56 API handlers swallow a crash and
answer 200. A failure reported as success is worse, but converging it is
milestone C's response-envelope work, not this row.
Gate: tests/test_unexpected_error_reporting.py (11, including the two that pin
why the old shape lost the crash), test_unreported_crash_ratchet.py (9), and
the 66 test files covering every touched module: 1105 passed, 7 skipped, 1
pre-existing Windows-only failure (test_fleet_doctor's socket.gaierror).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fe02ce8e removed queue_bus.py's private _current_user(), which was the only
code using the model. The automatic dead-import sweep kept it because the word
"User" also appears in a docstring two lines into _ensure_group_accessible.
Gate: -k queue_bus, 26 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 76 milestone A, drift/bug row 2 — scheduled as a fix, not a refactor,
because it is a behaviour difference shipping today.
rbac.get_current_user() is the only lookup that knows about both auth
mechanisms: it returns g.api_key_user for an X-API-Key request and falls back
to the JWT. 73 call sites answered "who is calling" with
`User.query.get(get_jwt_identity())` instead, behind 19 near-identical private
helpers (12 named get_current_user(), shadowing the real door).
The audit that scheduled this row predicted those return None for API-key
callers. Measured, it is worse: auth_required() deliberately skips
verify_jwt_in_request() once the key middleware has authenticated, so there is
no JWT context and get_jwt_identity() *raises*. Nine handlers reached through
an API-key-capable policy decorator therefore returned 500 to every API-key
caller:
bandwidth.get_app_bandwidth @viewer_required
servers.set_server_workspace @developer_required
servers.create_server @developer_required
sso.get_sso_config / update_provider_config / update_general_settings
source_connections.update_{github,gitlab,bitbucket}_admin_config
source_connections.complete_github_app_manifest
databases.execute_{mysql,pg,sqlite,docker}_query (see below)
Three service helpers had the same lookup inside a blanket `except Exception`,
which converted the raise into a silent user_id=None — the write landed and
the audit trail lost the actor (shared_resource_service, cf_ops_change_service,
secrets_webhooks).
The four databases.execute_*_query sites were a second admin check *inside* an
already-@admin_required route: dead code that always passed for JWT callers and
crashed for API-key ones. They now call require_admin_user() rather than
re-deriving the answer.
Also folds in a sixth _require_admin() found in the cloudflare-ops extension
source (29 gates, the same inverted contract as jobs.py), and fixes
serverkit-gui's blueprint at its sibling-repo source rather than in this repo's
live copy.
Ratchet (second-wave rule 1 — a migration without one regrows):
tests/identity_door_census.py counts both bypass shapes by AST, direct and the
two-line `uid = get_jwt_identity(); User.query.get(uid)` variant. The ceiling is
87, all of them on @jwt_required()-only routes where reading the JWT is correct
today; the ratchet's third assertion holds the API-key-capable population at
exactly zero, so migrating one of those routes to a policy decorator without
fixing its lookup now fails. Verified non-vacuous: an injected bypass fails both
assertions and passes again on revert.
The first wave's controller-boundary baseline is regenerated — this removes 74
persistence crossings (593 -> 519), a debt reduction with no new entries.
Gate: tests/test_api_key_identity.py (7, each red before the fix),
test_identity_door_ratchet.py (10), test_api_controller_boundaries.py (4), plus
the 65 test files covering every touched module: 1024 passed, 10 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 76 milestone A, drift/bug row 3. Five API modules had grown a private
_require_admin() with *three mutually incompatible* return contracts:
jobs.py -> returns the user; None means DENIED
dns_cutover.py -> returns None on success; a response tuple means denied
waf.py -> same as dns_cutover
manifests.py -> (user, None) on success, (None, response) on denial
themes.py -> same as manifests
jobs.py's is the exact inverse of dns_cutover.py's, so a guard copied between
the two reads identically at the call site and fails open. Nothing in the code
told you which contract you were looking at.
Replace all five with rbac.require_admin_user(), which raises a typed
PermissionDeniedError the global ApplicationError handler already renders. A
gate that raises has no return contract to get backwards.
Two convergences ride along, both toward the stricter shared behaviour:
- manifests.py gated on `user.role != 'admin'` where the others used
`is_admin`. These are the same predicate today (is_admin is defined as
role == ROLE_ADMIN), so no route changes status, but there is now one
truth source rather than two.
- none of the five checked is_active, so a deactivated admin still passed
them; @admin_required has always checked it. The shared door checks it, so
these 25 call sites now reject deactivated admins like every other
admin route.
The unknown-user case keeps its 403 'Admin access required' rather than
becoming a 401/404: a non-admin caller should not learn from the status code
whether the account behind the token still exists.
Gate: tests/test_jobs.py, test_themes.py, test_waf.py, test_dns_cutover_api.py
(82 passed) and -k manifest (152 passed).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 75 §G1, the remaining 33 files. Every raw subprocess call site in
backend/app that can go through a door now does. What is left is what
should be:
8 app/utils/system.py the doors themselves
8 database_service mysqldump|gzip, gunzip|mysql, pg_dump|gzip,
gunzip|psql — a pipe is two processes
7 build/docker/db_sync/ftp streaming output, gzip pipes
1 log_service tail -f
2 serverkit-analytics sibling repo; see below
Extensions were migrated in BOTH trees. builtin-extensions/ is the source
and backend/app/plugins/ the live copy, compared by normalised hash;
test_builtin_extension_drift stays green. serverkit-email has both sides
tracked, serverkit-ftp and serverkit-cloud-provision only the source.
serverkit-analytics (2 calls) is deliberately untouched: it has no source
in builtin-extensions/ at all, and backend/app/plugins/serverkit-analytics
is a gitignored materialised copy. Editing it would be a change that
cannot be committed and is overwritten on the next sync — that fix belongs
in the serverkit-analytics repo.
Exit-code semantics are preserved wherever they carry meaning, with a
comment at each site, because they are the thing a mechanical migration
destroys: clamscan 1 = INFECTED, grype 1 = vulnerabilities FOUND, yara
1 = matches, cron_runner 127 = never launched. Those read
result['returncode'], never result['success'].
run_checked gained two things real call sites needed: it yields when the
caller directs the streams itself (writing straight to a log file), rather
than adding capture_output on top and raising ValueError; and text/errors
pass through, so site_importers/ssh keeps decoding with errors='replace'
instead of gaining a UnicodeDecodeError on a bad byte.
A regression this migration introduced, caught by the suite and fixed:
install_profile_service._docker_usable() is deliberately TRI-STATE —
None means the probe could not run, because inside LXC the docker client
is usually present while the daemon never started, and callers "must not
treat a broken probe as proof of absence". Writing it as
run_checked(...)['success'] turned an exec failure into False, i.e. "docker
is not usable" — a fact nobody established. That is the §A violation this
plan exists to remove, reintroduced by the migration meant to prevent it.
test_probe_honesty_doctor caught it. Every other run_checked(...)['success']
caller was then audited; the rest are genuinely two-state, and their
originals also returned False on a missing binary.
Seven more slips of the same shape were caught before running by an AST
check for `x = run_checked(...)` followed by `x.stdout/.stderr/.returncode`
— a dict read as an object, which raises only on the failure path, so
happy-path tests stay green.
api_controller_boundary_baseline.json: pure deletion. Routing api/docker.py
through DockerService removed the last 6 subprocess crossings, so the
`subprocess` category is gone from the API layer entirely.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 75 §A5. The frontend could always render unknown (a real 'unknown'
state, an n/a chip) — the backend starved it of the signal:
- Doctor's cert check read Domain.ssl_expires_at, a column nothing in
production writes — structurally incapable of anything but a green
'No certificates tracked.' Empty now warns: expiry is not monitored.
- openssl -checkend exits non-zero for 'expires soon' AND for
'unreadable' — an unreadable cert reported needs_renewal=True, and
the error path vanished the cert from the expiring list entirely.
Unreadable now returns an error result, and the status endpoint
surfaces check_errors instead of dropping them.
- A failed monitor cert probe bumped cert_checked_at while keeping the
old cert_expires_at — a green 'valid / N days' chip from weeks-old
data next to a fresh checked timestamp. The stale value is cleared;
the UI falls back to n/a until a probe succeeds.
- get_expiry_alerts dropped unprobeable certs by omission; they now
appear with severity 'unknown'. get_cert_health reported grade 'F'
for a DNS timeout; probe failure now yields grade None + error.
- A failed certbot inventory reported '0 certificates, nothing
expiring' with HTTP 200; list_certificates_report carries the
errors and the status endpoint exposes them as list_errors.
Two dishonesty directions, one rule (plan 75 §A3/A4): a probe answers a
value or None, never 0 when undetermined — and a real 0 must survive.
Live paths normalized every missing reading to 0 while the history path
stored NULL for the same probe: agent counters an LXC agent can't
determine rendered as a perfectly idle box; load average on unsupported
platforms read 0.00 (and silenced the load alert); CPU freq read 0 MHz;
an empty history window summarized as '0.0 avg'. The mobile summary
called a method that does not exist and rendered 0/0/0 permanently.
One unavailable network counter raised uncaught and froze the entire
live broadcast.
Reverse direction: falsy traps (x if x else None) collapsed a genuinely
idle 0.0 into 'no data' in rollups, averages, and the history model.
Fleet: KPI tiles read '0.0 ms latency' / '100% success' with no data;
never-probed servers exported serverkit_server_up 0 to external
alertmanagers — the up-series now emits only for determined
connectivity. get_all_metrics is per-section fault-tolerant so one
failed probe can't blank the other six.
The enable() and remove_rule() guards reason about ports. These two paths never
touch the SSH rule and lock the operator out anyway:
block_ip(<your own address>) the rule is untouched; your source
address is rejected before reaching it
set_default_zone(<zone without ssh>) firewalld re-homes every interface with
no explicit zone; the rules did not
change, the zone around them did
check_ip_block() refuses when the target covers a live SSH peer, the caller's
own HTTP client address, or loopback — nginx forwards to 127.0.0.1, so blocking
it takes the panel down with the session. CIDRs are matched with `ipaddress`,
because 73.244.0.0/16 is the same lockout wearing a wider mask.
Peers come from `ss -Htn state established`, filtered to the sshd ports; `who`
is empty for sessions with no utmp entry. Verified against the live host, whose
only established SSH connection was 146.190.213.37:22 <- 73.244.95.52:59085:
active_ssh_peers() -> {'73.244.95.52'}
block 73.244.95.52 -> refused
block 73.244.0.0/16 -> refused (covers the same peer)
block 127.0.0.1 -> refused (loopback)
block 198.51.100.7 -> allowed
caller_ip is read in the API layer from request.remote_addr and passed in, so
the service stays free of request context.
check_default_zone() asks whether the target zone permits ssh via
--list-services / --list-ports before switching. A failed `ss` probe yields no
peers rather than fabricating conflicts, so a broken probe cannot block every
request. Consistent with the existing guards: undeterminable SSH port refuses,
an unenforcing firewall is a no-op, force=true is the operator accepting it.
/block-ip and /zones/default answer 409 with the conflicting addresses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
enable() guards the moment the firewall comes up; this guards every moment
after it. With ufw active and denying by default, deleting the rule that admits
SSH severs the very session issuing the delete, and there is no way back in.
Deleting by NUMBER is the sharp edge: the caller passes an opaque index, so the
only way to know what is about to go is to resolve it against
`ufw status numbered` first. Removal by port spec is checked too.
Coverage-aware rather than a blanket block — a duplicate SSH rule may still be
deleted, because another allow rule keeps the port open. Application profiles
(`[1] OpenSSH`) are resolved through `ufw app info`, verified against a real
host: _ports_in('OpenSSH') -> {22}, and '22/tcp (v6)' -> {22} rather than
parsing the address-family suffix as port 6.
deny_port() is implemented as remove_rule(), so it inherits the guard —
denying the SSH port and deleting its allow rule are the same outcome.
Consistent with the enable() guard: an undeterminable SSH port refuses instead
of assuming 22, an inactive firewall is a no-op (nothing is being enforced and
enable() preflights separately), DEFAULT_INPUT_POLICY=ACCEPT short-circuits,
and force=true is the operator accepting the risk.
firewalld: removing service=ssh or the SSH port is checked against
--list-services / --list-ports so other coverage still permits the removal.
DELETE /rules and POST /deny-port answer 409 with the offending rule;
POST /rules/removal-preflight lets the UI warn before the click.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ufw enable` normally prompts "Command may disrupt existing ssh connections.
Proceed?". The service must use `--force enable` because nothing here runs on a
terminal — and --force suppresses exactly that prompt. So the panel had removed
ufw's only safeguard and offered a one-click lockout of a remote box.
Found on a live host: ufw installed and inactive with one staged rule,
`ufw allow 25/tcp`, and sshd on 22. ufw defaults to deny-incoming, so enabling
would have severed the only way back in. Verified against that host:
ssh_ports() = [22]
ufw allowed ports = [25]
enable('ufw') -> blocked_by ssh_lockout
check_ssh_lockout() reads the live port from `sshd -T` (authoritative: resolves
Includes, Match blocks and compiled-in defaults) and compares it against
`ufw show added`, the rules that will exist the moment it comes up. Named
application profiles are resolved via `ufw app info` rather than assumed.
An undeterminable SSH port REFUSES rather than falling back to 22 — guessing in
the permissive direction is the lockout this exists to prevent. A DEFAULT_INPUT
_POLICY of ACCEPT short-circuits it, since nothing can be locked out then.
enable(force=True) is the operator accepting the risk; never set by default.
POST /firewall/enable answers 409 (well-formed request, conflicting state) with
the offending port and remediation. GET /firewall/ssh-preflight lets the UI warn
before the click instead of reporting the lockout after it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A VPS resized from 1 vCPU/1 GB to 4 vCPU/8 GB with a 60 GB volume attached,
and the panel never mentioned any of it. Not a stale cache: a resize forces a
power-off, so the process and its in-memory spec cache die and the next boot
reads the truth. The gap is that nothing ever persisted the PREVIOUS numbers,
so there was nothing to compare against — and the reboot that causes the
change is what destroys the only copy of the baseline.
Adds host_snapshots: capture specs + per-filesystem inventory at boot, diff
against the last row, and emit host.specs_changed / host.storage_advisory on a
real delta. Advisories fire on state transitions only, so an unfixed fstab
entry does not nag on every restart.
Repairs a divergence this exposed: resource_tier_service measured the first of
/var/lib/serverkit,/var/lib/docker,/ while capacity_service hardcoded /. Both
now use host_inventory_service.data_path(), and local headroom reports which
mountpoint it measured.
Notifications render in the existing bell UI, so no frontend change is needed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three changes to the endpoints that serve the fleet-agent installers.
A missing installer answered 404. That is what sent issue #101's reporter
looking for a wrong URL -- they filed it as "incorrect paths in setup" -- when
the file had simply never been shipped into the image. The status code produced
the misdiagnosis. It is now 503, the response names the path that was searched,
and the same path is written to the panel log, because the response body does
not survive the `curl -f` the enrollment one-liner uses: curl discards it and
prints only "The requested URL returned error: 404". For an endpoint consumed by
curl, the status code and the server log are the only diagnostics that reach
anyone.
The panel's release lookup pointed at jhd3197/ServerKit and filtered for
`agent-v*` tags, which exist nowhere since the agent moved to its own repo, so
/api/v1/servers/agent/version returned 503 to every agent polling for an update.
It now uses AGENT_GITHUB_REPO (env SERVERKIT_AGENT_GITHUB_REPO) and a ^v(\d…)
tag filter that skips the agent repo's malformed "v" release.
_load_installer() is shared by the .sh and .ps1 routes so they cannot drift, and
SERVERKIT_SCRIPTS_DIR overrides the resolved directory for unusual layouts.
New tests cover the endpoints, the release lookup, and -- the one that would
have caught the original bug -- a packaging invariant that parses the
Dockerfile's COPY sources and asserts every installer the panel serves is
actually copied into the image. No endpoint test could have caught it: pytest
runs against the source tree, where the file always exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 55 task 6, backend half. Installing a plugin's requirements runs pip with
the backend's privileges, and a setup.py hook is arbitrary code, so it is
opt-in and off by default -- correct, and the part that was missing is that
nothing ever said so. The skip was a logger.warning: the requirements.txt was
written next to the extension and the panel showed nothing, so an extension
could sit there running with imports it does not have and the operator had no
way to find out short of reading the install log.
pending_requirements() answers it: whether a file is pending, its contents, its
path, the parsed package lines, and the exact env var that changes the
behaviour. GET /plugins/<id>/requirements exposes it, admin-only.
Strictly read-only. Nothing here installs anything -- the plan is explicit that
this surface introduces no auto-install behaviour change -- and a test pins
that by failing if _install_requirements is reached.
The env check the installer inlined is now plugin_pip_enabled(), shared by both
sides. If the surface and the installer ever disagreed about what "enabled"
means, the page would describe a behaviour the installer does not have.
Two honest edges: the content is capped at 64KB before being echoed into a
browser, and pip_enabled is reported alongside pending, because a file written
while the opt-in was off survives turning it on -- the surface reports both
facts rather than guessing which applies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 55 task 7, backend half. A manifest's `permissions` array was
declaration-only: the install dialog showed it and nothing ever compared it
against what the extension went on to do.
The capability gate now records every call through it -- including the
refusals, which are the interesting ones -- so `usage_report(slug)` can answer
declared-versus-observed per extension. Observations are in-memory and
per-process by design: the question is "what has this extension done since the
panel started", and persisting it would outlive the evidence and start reading
like an audit trail, which this is not. `has()` deliberately does not record; it
is a question, not a use, and counting it would fabricate activity for callers
that only checked.
The larger half of this change is admitting what the panel cannot see. Task 7
was specified as hooks at "the shell/docker/db helper entry points", and those
do not exist: the SDK exposes no shell, docker, filesystem or network helper at
all, and hands out `db` as raw SQLAlchemy. An extension wanting any of them
imports the host module directly, where no in-process gate is involved. The one
genuinely gated capability is `agent.command:<action>`, because the SDK is the
only way to reach an agent.
So observability is data, not prose: OBSERVABLE_PREFIXES marks what the gate
truly mediates, and the report marks every other declared permission
`observable: false` rather than showing it as "never used". Absence of evidence
is not evidence of absence here, and a UI claiming otherwise would be inventing
enforcement the panel does not have -- exactly the honesty the plan's D2
decision asks for.
GET /plugins/<id>/permissions returns the report, admin-only: it describes what
third-party code has been reaching for.
The UI surface and the docs follow.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 72 item A.2. FleetDoctorService, FleetRepairService, the FleetDoctorResult
model, migration 063 and their tests were all complete and reachable from
nothing: no routes, register_jobs() never called, no seeded schedule. The
frontend was further along than the backend -- getFleetDoctorReport() and
runFleetSweep() have been calling /doctor/fleet and /doctor/fleet/run, two
endpoints that did not exist, so the multi-server health story the agent
platform advertises 404'd on contact.
Nothing in those services was redesigned. The one addition is a read-side
aggregator: they could run a sweep and persist rows but had no way to hand
back a per-server report, so fleet_report() composes one. It lists every
registered server rather than only connected ones, which is what makes an
unreachable box a reported state -- last known rows plus connected: false --
instead of a hole in the report. Persisted rows key on check_key while the
UI's field-driven renderer keys on key, so the report emits both.
Three admin-only routes, matching api/doctor.py's gating exactly. The sweep
returns 202 with a job id and never runs on the request thread: it fans out
across agents over the network, and the agent gateway registry is
single-worker and in-memory. Repair is a thin passthrough to FleetRepairService,
which already owns the allowlist, capability gate and audit trail; refusals
come back as per-item results inside a 200, so one offline server cannot fail
a batch. The batch is bounded at 25 so a single request cannot fan an
unbounded number of remote mutations at that single worker.
The schedule is daily like the host doctor but starts 30 minutes behind it,
so the fleet fan-out contends with neither the local sweep nor the boot-time
agent reconnect storm.
The frontend gets a Fleet Doctor section rather than losing its dead methods.
The per-check article was extracted into one component now shared by the host
and fleet lists, so fleet findings render through the same
repairable/repair_ref path instead of a parallel implementation; the confirm
modal says plainly that a fleet repair acts on a remote server through its
agent and is audited.
test_jobs' builtin schedule count moves 21 -> 22 for the new schedule. The
builtin.* kind count is untouched -- this kind is doctor.fleet.run.
Note for anyone reading the suite: tests/test_fleet_doctor_dns_rotation.py is
skipped at module level and proves nothing here. Its rotating-DNS-window API
did not survive the recovery rebuild; that is pre-existing and unrelated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 73 item 5. The Domains SSL modal asked for a contact email on every
single certificate, and the backend hard-required one per call. The app-detail
Settings tab did remember it -- in localStorage, so the memory was per browser
and invisible to the other surface. The two never agreed, and neither could
help the paths with no form at all: domain attach passed `admin@<host>` when
it had nothing else, an address that usually does not exist, which is where
every expiry notice for those certificates went to die.
There is now a single panel-wide ACME contact in system settings.
get_acme_contact() resolves an explicit address first and falls back to the
stored one; remember_acme_contact() writes it back whenever a request carries
an explicit address. Both are best-effort on the settings side by design: a
settings failure must not break issuance, and failing to remember an address
must not turn a successful certificate into an error. An empty input never
clears a stored contact, since the automatic paths pass whatever they were
given and that is frequently nothing.
Wired into every issuance entry point -- the domain SSL enable route, the
generic certificate route, and domain attach, which now consults the stored
contact before falling back to the admin@<host> guess. The address is only
remembered after issuance actually succeeds; remembering one Let's Encrypt
rejected would be worse than remembering none.
Prefill needed its own endpoint. The settings routes are all admin-gated, but
issuing a certificate for an app you can edit is not an admin action, so
GET /ssl/acme-contact is jwt_required only -- it returns an address the
operator chose for notices, not a credential. The Domains modal and the
Settings tab both prefill from it, and the Settings tab keeps its localStorage
seed so a browser that already has the value shows it without a flash.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 73 item 7. Let's Encrypt prefers IPv6 whenever a AAAA record exists, so a
domain whose A record points at this server perfectly still fails HTTP-01
issuance when its AAAA points somewhere else. That shape looked healthy
everywhere in the panel, which is what made it expensive: the operator saw a
verified domain, a green doctor report, and a certificate error naming nothing
either of them had mentioned.
Both DNS surfaces were blind to it for different reasons. The doctor's sweep
returned ok as soon as the server IP was *among* the resolved addresses, so
the extra IPv6 answer never registered -- and that branch is reached precisely
when the A record is correct, i.e. exactly the case worth reporting.
verify_domain used socket.gethostbyname, which is IPv4-only and cannot see a
AAAA record at all.
Both now compare every resolved address against this server: the configured
public IP plus the host's own interface addresses, so a correct dual-stack
setup stays quiet and only a genuine stray is named. Comparison is by parsed
value rather than by spelling, since an expanded 2001:0db8:... form and its
compressed equivalent are the same address, and psutil's zone ids (fe80::1%eth0)
are stripped first. Interface enumeration happens once per doctor sweep
instead of once per domain.
The doctor reports this as a warn, not a fail: the site itself is reachable
and it is a stray record rather than a broken one. Both messages say what to
do -- remove the AAAA record or point it at this server's IPv6 address --
rather than only naming the symptom.
verify_domain keeps its existing response shape. ip_address still carries a
single IPv4 address for the callers that predate this, with the full set added
alongside as ip_addresses, plus aaaa_conflict and a warning string. The
Domains page needed the last part to matter: it printed a green "Domain
verified!" toast for anything that resolved, so a conflict the API now
reports would have gone straight back to being invisible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 73 item 3. The quick-access rail links the local target at /etc/nginx,
but /etc was never in ALLOWED_ROOTS, so the shortcut returned 403 for every
user including admins -- a dead button, and our own regression: it shipped
after the review that flagged the surrounding SSL story.
/etc/nginx is now an allowed root, but browse-only. The panel generates those
vhosts itself, so a hand edit is silently overwritten on the next domain
change, and a writable /etc/nginx would hand anyone holding files.write a
root-owned config that the panel itself reloads -- a much larger grant than
"let me look at the effective config", which is all the shortcut is for.
READ_ONLY_ROOTS is a second tier under ALLOWED_ROOTS rather than a special
case for this one path: is_path_writable() = allowed AND not read-only, and
every mutating entry point now gates on it instead of on is_path_allowed, so
adding a future root to the list is enough to make the whole write surface
refuse it. Both comparisons resolve with realpath first, so a symlink planted
in a writable root cannot launder a write into the subtree.
copy is deliberately asymmetric -- copying a vhost out to keep only reads the
source, so only the destination is gated -- while move removes its source and
therefore checks both ends. The upload endpoint repeats the check because it
writes to disk directly instead of going through FileService's methods.
Only this subtree is opened, not /etc: /etc/passwd and /etc/shadow stay
denied, and the panel-internal PROTECTED_ROOTS exclusion is untouched
(GHSA-rm3m-9mvw-68fh).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
