Resuming uptime tracking at boot read the history through a helper
that first created /var/lib/serverkit. The image runs unprivileged, so
that makedirs raised and the app never started. Reading now creates
nothing, and a failed resume is logged instead of aborting boot.
Comments cited plan sections as "(§G3)" or "(§8 unification)", which
point into the same git-ignored plans the plan-number citations did.
References next to a real document (docs/REDESIGN_MAP.md §3) stay. Code
and runtime strings are unchanged (AST and token comparison).
The worst cases the audit named: docstrings and header comments that
told how three old copies drifted, or quoted subprocess counts that
were stale the day they were written, instead of stating the contract.
Each now says what the function does and keeps one line of why. Also
drops the always-zero n_cloud counter that existed only to keep a boot
log line's or-chain intact.
Comments, docstrings and a few strings cited plan numbers ("plan 29
#9", "plan 86 §B2") from a git-ignored folder, so they pointed at
nothing; one reached users in a bottleneck hint. Code and frontend
strings are unchanged apart from those citations (checked by AST and
token comparison). A vhost still carrying the old asset comment reads
as undrifted, and a guard test keeps new citations out.
The broker only delivers pending messages, so anything in flight when
the process died was never delivered again and its Job row read
running forever. At boot, before any consumer polls, interrupted jobs
are failed visibly (a half-run deploy or restore is not safe to repeat
blind) and every other stranded message is redelivered or, when out of
attempts, dead-lettered.
Both loops only started from an API call, so every restart silently
stopped threshold alerting while the UI still showed it enabled. Uptime
tracking now persists its on/off flag, the monitor loop runs inside an
app context so its telemetry writes stop failing, and SMTP gets a
timeout so a dead mail host cannot wedge the loop.
Until now the connect client refused every http and sio stream, so
Open panel in ServerKit Cloud showed a blank page. The client now
bridges them to the loopback listener, the panel serves its page
under the relay's /t/<device>/ prefix, and the SPA prefixes its
router, API and Socket.IO paths from it. The Cloud grant is checked
against Cloud's JWKS before signing in the panel user with the same
email; panel two-factor still applies.
- Panel start writes a logrotate file for the panel's logs and a
default json-file size cap into Docker's daemon.json (merged, never
restarting Docker); the retention tick keeps the newest 10 update
logs and 10 build logs per app.
- The previous blue/green slot is removed once the current one has
been live storage.previous_slot_hours (24), never mid-update.
- Disks under 50 GB get shorter retention defaults once per install;
values the operator changed are left alone.
A fresh panel no longer lets whoever reaches it first become admin:
registering the first account needs a one-time code the installer
prints (`serverkit setup-code` repeats it). SERVERKIT_ADMIN_EMAIL and
SERVERKIT_ADMIN_PASSWORD create the admin unattended instead.
Git apps now deploy what users expect: a repository with a compose
file deploys every service it declares, "Deploy latest" builds the
branch's newest commit, and push webhooks rebuild the containers.
Also fixes branch switching on single-branch clones, uploads of a
lone file, compose v1 port preflight, and CLI commands that ignored
their flags.
Centralize backup schedule calculations and shared request/form flows, encode query values, and move repository finalization into its service. Add browser CI coverage, repair stale test fixtures, and provide opt-in request and build measurement tools.
- app/services/connect_client.py: RelayClient background thread (WS hello
signed device_id|ts|nonce, 20s pings, jittered 1s->60s backoff with reset
after 2 stable minutes, >6 reconnects/10min flap log, close-code ->
state_reason mapping, terminal revoked, long-poll fallback with 15min WS
re-probe); connect-state.json runtime state merged into status();
disconnect() stops the client and removes the state file; run_doctor()
implements the connect check table
- app/services/connect_keys.py: strict load_keypair() for the transport
- app/__init__.py: start_client_if_paired() next to the linked-panel hook
- cli.py: connect status shows transport/last-seen/relay-instance; new
connect doctor command (exit 1 on hard-check failure)
- requirements.txt: websockets==15.0.1 (WS client; the SocketIO stack is
server-side only)
- tests/test_connect_relay.py: 21 unit tests for backoff, flap detection,
close-code mapping, hello signature, state file, loop transitions
The 1791-line security omnibus splits along tool lines and the install-gated
surfaces leave core for five standalone extensions - serverkit-clamav
(ClamAV+YARA scanning, quarantine, scan jobs), serverkit-fail2ban
(management UI; probes stay core on Fail2banJailService), serverkit-lynis,
serverkit-auto-updates, and serverkit-image-scan (headless CVE/SBOM
scanning; models stay core per the plan 52 D1 data seam). Each mounts its
routes back at the same /api/v1/security prefix when installed, so route
shapes never change; absent extensions mean plain 404s, no dead UI.
Core keeps the zero-host-package baseline: status/config, legacy integrity,
suspicious activity/events, SSH keys, IP lists, the audit (fail2ban check
now probes via the jail service; updates check was already self-contained),
scoped FIM, and the scan/alert logs + notification helpers extensions write
through. Job handlers for security.malware_scan/lynis_scan/image_scan now
come exclusively from the extensions' manifest jobs key.
Security page becomes core-tabs-only; extension tabs contribute into the
security tab group (3a groupId). Overview posture drops the hardcoded
ClamAV rows and offers the Marketplace when no security extension is
installed. Upgrade parity: run_registry_auto_install() installs each
extension from the registry once, gated on the host tool actually being
present, fail-soft until the registry entry is published.
Wizard round 2 (Ph5): a security-posture chooser (minimal/recommended/
hardened) on the Summary step, catalog-resolved so unpublished slugs never
render dead installs; persisted as onboarding_security_posture.
Moved test suites re-home to the extension repos; panel keeps
test_security_suite_extraction.py (18 proving tests: absent-shape 404s,
lean summary, jail-service probes, migration gates, posture wiring).
docs/API_SURFACE.md regenerated (34 operations moved out).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
flask-limiter picks RATELIMIT_STORAGE_URI up from app.config at init_app,
so surfacing the env var in config (only when non-empty) lets production
deployments share limit state across processes and silences the in-memory
storage warning. Unset keeps in-memory storage, matching the deliberate
single-worker gunicorn design. Documented in .env.example with probe and
mechanism tests.
Expose disk_reclaim_service behind /api/v1/system/disk/reclaim: a
read-only report and a confirm-gated POST whose keys are re-validated
against a fresh scan server-side (safe candidates only — telemetry
stays CLI-only). ?wait=false runs it as a background job; both paths
are audit-logged.
The curated fix shows up where the problem is reported: a failing
disk.headroom doctor check and an alerting disk tile each offer
'Reclaim space', which previews measured candidates before handing
the run to Operations.
The last hand-rolled truthiness read in the app factory: it accepted only
the literal '1', invisible to the env-door ratchet's pattern. env_bool()
keeps '1' working (the updater's contract) and adds the door's documented
true/yes/on spellings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 76 milestone B, drift/bug row 4. 43 API handlers answered HTTP 500 from
their own `except Exception`, so the global 500 handler never ran — and that
handler was the ONLY caller of error_log_service.record_error(source='backend').
Every one of those crashes was absent from /monitoring/errors, and the page read
empty precisely because someone had "handled" them.
35 of the 43 also returned str(exc) to the caller. Two returned the full
traceback (templates.py, which also print()ed it), and several logged with a
locally-minted uuid4 'ref' that correlated with nothing but a log line.
The recording side of the 500 handler moves to app/error_reporting.py, so it is
importable rather than reachable only by not-catching:
record_unexpected(exc) -> logs, rolls the session back, records, returns
the request id
unexpected_response(exc) -> the above plus the documented crash body
The 500 handler now calls the same pair, so there is one recorder rather than
one recorder and 43 routes that quietly opted out of it.
Caller-visible change on those endpoints: the body is now the documented
{'error': 'Internal server error', 'status': 500, 'code': 'internal_error',
'request_id': ...} instead of the exception text. That is the point — an
unexpected exception's text is written for whoever reads the log and routinely
carries connection strings, absolute paths, and SQL. The request id is the part
a caller can act on, and it matches the row in the error log. No frontend code
read the 'ref' or 'trace' keys this drops (checked).
Two sites keep their own wording and call record_unexpected instead: sso.py's
login failures should not read as a generic crash on the login page.
docker.py's bulk endpoint keeps its per-item `results` — the caller's half-done
work survives, only the exception text is dropped.
Ratchet: tests/unreported_crash_census.py + its test, held at **0**, not a
legacy baseline to shrink. It counts blanket handlers that answer 500 without
calling either reporter, and is scoped to app/api — a service catching an
exception and returning a value is making a domain decision, not declaring an
HTTP crash. Verified non-vacuous: an injected handler fails it, and it passes
again on revert.
Measured and deliberately still open: 56 API handlers swallow a crash and
answer 200. A failure reported as success is worse, but converging it is
milestone C's response-envelope work, not this row.
Gate: tests/test_unexpected_error_reporting.py (11, including the two that pin
why the old shape lost the crash), test_unreported_crash_ratchet.py (9), and
the 66 test files covering every touched module: 1105 passed, 7 skipped, 1
pre-existing Windows-only failure (test_fleet_doctor's socket.gaierror).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 500 handler shares the app-context-scoped db.session with the view that
just failed, which breaks two ways.
The one that matters: when the crash IS a database error, that session is
already in a failed transaction, so record_error's very first query raises
PendingRollbackError -- swallowed by its own never-raise contract. Database-
caused 500s were therefore the one class of crash the error tracker could never
show, and they are the class most worth seeing. 72 API views commit with no
try/except anywhere in the function, so this is not a narrow path.
The second: record_error's commit() flushes whatever ORM state the crashed view
left pending, persisting a half-applied write on the way out.
Rolling back first fixes both. Flask-SQLAlchemy's teardown rolls this session
back moments later anyway, so a clean request is unaffected.
Reviewed as an alternative to a partial unique index on the dedup fingerprint,
which was rejected: record_error swallows everything and returns None, so an
IntegrityError would DROP the error report rather than dedupe it; set_resolved
commits unguarded, so un-resolving an old row sharing a fingerprint would 500 a
legitimate admin action; and CREATE UNIQUE INDEX would abort the migration on
exactly the installs that already accumulated duplicates. The race is real but
its blast radius is duplicate rows in an admin list -- nothing reads
fingerprint for uniqueness, and the lookup is .first(), not .one().
Both new tests fail with the rollback removed (mutation-checked by line
number). The first draft of the DB-error test did NOT: a hand-built
IntegrityError never poisons the session, so it has to be a real failed flush.
Race reported as High by Copilot on the error-tracker PR.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A VPS resized from 1 vCPU/1 GB to 4 vCPU/8 GB with a 60 GB volume attached,
and the panel never mentioned any of it. Not a stale cache: a resize forces a
power-off, so the process and its in-memory spec cache die and the next boot
reads the truth. The gap is that nothing ever persisted the PREVIOUS numbers,
so there was nothing to compare against — and the reboot that causes the
change is what destroys the only copy of the baseline.
Adds host_snapshots: capture specs + per-filesystem inventory at boot, diff
against the last row, and emit host.specs_changed / host.storage_advisory on a
real delta. Advisories fire on state transitions only, so an unfixed fstab
entry does not nag on every restart.
Repairs a divergence this exposed: resource_tier_service measured the first of
/var/lib/serverkit,/var/lib/docker,/ while capacity_service hardcoded /. Both
now use host_inventory_service.data_path(), and local headroom reports which
mountpoint it measured.
Notifications render in the existing bell UI, so no frontend change is needed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tracing issue #101 turned up that a panel failing to serve its own agent
installer left no diagnostic trail at all. The backend had exactly one error
handler -- the 404/SPA one -- no 500 or HTTPException handler, and nothing
anywhere configured logging, while only 2 of 100 app/api modules ever call
logger.error.
Adds:
- an errorhandler(500) that logs the unhandled exception with its method and
path and returns JSON. Flask routes here only when it is not propagating, so
pytest and the dev server still re-raise with a full traceback.
- an errorhandler(HTTPException) so framework-generated errors under /api/
(405 from the router, 413, abort()) return JSON instead of Werkzeug's HTML
page to clients that are parsing JSON. Non-API paths keep the HTML page, and
the more specific 404/500 handlers still win for those codes.
- _configure_logging(): a stderr handler with a timestamp/level/logger-name
formatter, level from LOG_LEVEL. Services across the codebase log through
logging.getLogger(__name__); with no root handler those records fell through
to logging's last-resort handler -- WARNING and above only, unformatted, INFO
dropped on the floor.
Two things the container run caught while writing this: Flask's default handler
has to come off app.logger or every record prints twice (once as %(module)s via
Flask's handler, once as %(name)s via ours, both reaching stderr through
propagation), and the level must not key off DEBUG alone -- TestingConfig sets
DEBUG=True, which would have moved the whole test suite to DEBUG logging as a
side effect. Both are pinned by tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 72 item A.2. FleetDoctorService, FleetRepairService, the FleetDoctorResult
model, migration 063 and their tests were all complete and reachable from
nothing: no routes, register_jobs() never called, no seeded schedule. The
frontend was further along than the backend -- getFleetDoctorReport() and
runFleetSweep() have been calling /doctor/fleet and /doctor/fleet/run, two
endpoints that did not exist, so the multi-server health story the agent
platform advertises 404'd on contact.
Nothing in those services was redesigned. The one addition is a read-side
aggregator: they could run a sweep and persist rows but had no way to hand
back a per-server report, so fleet_report() composes one. It lists every
registered server rather than only connected ones, which is what makes an
unreachable box a reported state -- last known rows plus connected: false --
instead of a hole in the report. Persisted rows key on check_key while the
UI's field-driven renderer keys on key, so the report emits both.
Three admin-only routes, matching api/doctor.py's gating exactly. The sweep
returns 202 with a job id and never runs on the request thread: it fans out
across agents over the network, and the agent gateway registry is
single-worker and in-memory. Repair is a thin passthrough to FleetRepairService,
which already owns the allowlist, capability gate and audit trail; refusals
come back as per-item results inside a 200, so one offline server cannot fail
a batch. The batch is bounded at 25 so a single request cannot fan an
unbounded number of remote mutations at that single worker.
The schedule is daily like the host doctor but starts 30 minutes behind it,
so the fleet fan-out contends with neither the local sweep nor the boot-time
agent reconnect storm.
The frontend gets a Fleet Doctor section rather than losing its dead methods.
The per-check article was extracted into one component now shared by the host
and fleet lists, so fleet findings render through the same
repairable/repair_ref path instead of a parallel implementation; the confirm
modal says plainly that a fleet repair acts on a remote server through its
agent and is audited.
test_jobs' builtin schedule count moves 21 -> 22 for the new schedule. The
builtin.* kind count is untouched -- this kind is doctor.fleet.run.
Note for anyone reading the suite: tests/test_fleet_doctor_dns_rotation.py is
skipped at module level and proves nothing here. Its rotating-DNS-window API
did not survive the recovery rebuild; that is pre-existing and unrelated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This is the change behind the headline symptom. A DevTools capture of /domains
showed ~70s of network activity and a 404 taking 28.77 SECONDS. The backend
measured that same request at 1.0ms in its own api_usage_logs table. A routing
404 runs no handler and touches no database, so it cannot spend 28s working —
the entire gap was Stalled/Queueing in the browser.
The cause: in dev the API was same-origin with the Vite dev server, which serves
each of the ~760 source files as its own request. A browser allows six
concurrent HTTP/1.1 connections per origin, so twenty API calls queued behind
hundreds of module requests. Splitting the origin doubles the usable sockets
and, more to the point, stops a slow API call from blocking a JS module.
Same-origin was not an arbitrary choice — the comment it replaces warned about
"CORS/preflight redirects". Both halves of that are handled:
- CORS: config.py DEFAULT_CORS_ORIGINS already lists the Vite dev origin on
both localhost and 127.0.0.1, so nothing new is being allowed.
- Preflight: every panel request carries an Authorization header, which makes
it non-simple. Without Access-Control-Max-Age the browser re-runs the
preflight every few seconds (Chrome caches 5s by default), so a cross-origin
panel would pay two round trips per call forever — worse than what it
replaces. CORS now sends max_age=3600, verified against the real create_app()
rather than assumed: a preflight from http://localhost:41921 returns 200 with
Access-Control-Max-Age: 3600 and the matching allow-origin. That header is
worth having regardless, since any deployment serving the panel from a
different origin than the API was paying the same tax.
Escape hatch: VITE_API_PROXY=true restores same-origin for a setup with an
origin CORS does not expect. Socket.IO is deliberately untouched — it stays on
window.location.origin in dev, because a WebSocket upgrades out of the HTTP
connection pool and gains nothing from the split.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deleting is the only action in a control panel that repeating cannot undo, so
records a person destroys by hand now leave a tombstone and the Recycle Bin
hands them back.
SoftDeleteMixin adds deleted_at/deleted_by_id plus soft_delete/restore and
query_active()/query_deleted(). recycle_bin_service is a REGISTRY: a model
registers with a label function and an optional on_restore hook, so the API
lists and restores across types without knowing what a domain is. Adding the
next soft-deletable model is one register() call.
Applied to Domain (a real, destructive case -- deleting one used to lose the
record outright) and SavedView. DELETE /domains/<id> now tombstones and still
tears the vhost down; restoring a row does NOT re-do side effects unless the
type supplies on_restore, which is the honest default.
API: GET /recycle-bin (all kinds, newest first), POST
/recycle-bin/<kind>/<id>/restore, DELETE /recycle-bin/<kind>/<id> (purge,
admin-only -- the one irreversible action), POST /recycle-bin/purge-expired
with a 30-day default retention.
THE PART THAT NEEDED CARE: both tables had a plain UNIQUE covering what is now
a tombstone, so deleting a domain would have permanently burned its name and
deleting a view would have blocked re-creating one by the same name. Migration
083 replaces each with a PARTIAL unique index predicated on deleted_at IS NULL
-- supported by SQLite 3.8+ and Postgres. Tests pin exactly that: a deleted
name can be reused, and the live row stays unique.
Also gives SavedView a `slug` (backfilled), the handle the upcoming
?view=<slug> links resolve against. A tombstoned view does not reserve its
slug, so deleting a view frees the link name too.
Migration is idempotent on both columns and indexes: `deleted_at` is
index=True on the mixin, so a database built from the models already has that
index, and a bare create_index would abort the upgrade on exactly the installs
we most want to keep working.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The queue-bus consumers, job system and metrics collector share one SQLite
file with request handling, and the default journal mode lets any writer
lock the whole file — the sporadic 'database is locked' errors in the
notification/job consumers. Connections now open with journal_mode=WAL
(readers and a writer coexist), busy_timeout=30s and synchronous=NORMAL.
Skipped under TESTING, where conftest owns connection pragmas.
SavedView model (user/page/name/state JSON/is_default, unique per
user+page+name), idempotent 082 migration, saved_view_service with
one-default-per-page enforcement, and a JWT-scoped /api/v1/views CRUD
blueprint. Covered by 9 endpoint tests.
A StatusComponent was already a full synthetic check (http/tcp/dns/smtp/ping,
target, interval, timeout, HealthCheck history, uptime windows, auto-incidents)
— it was just framed as "a component of a status page" and its engine lived
inside the serverkit-status extension, so watching a site required publishing a
status page. Promote it:
- migration 081 makes status_components.page_id and status_incidents.page_id
nullable, and adds the columns backing the monitor UI (is_paused,
check_method, expected_status, keyword, follow_redirects, verify_tls, retries,
consecutive_failures, cert_issuer, cert_expires_at)
- new core app/services/monitor_service.py owns the check engine; the extension
keeps page publishing and delegates
- new /api/v1/monitors (CRUD, check, pause, history, uptime buckets, incidents)
- new builtin.monitor_check sweep on a 30s tick — until now checks only ran when
someone pressed "Check now", so a configured monitor never noticed an outage
Check-engine defects fixed while moving it:
- ping ignored the process return code and could only ever report 'up'
- ping used Linux-only flags with no guard, breaking the Windows dev server
- http ignored its own method/expected_status/follow_redirects/verify_tls
- smtp was documented on the column but unimplemented; keyword did not exist
- run_check never recomputed uptime and never opened or resolved an incident —
only the WordPress health-sync path did, so network monitors never paged
- TLS issuer/expiry are now captured (throttled to one read per 6h)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- create_app(): skip background daemons (queue consumers, metrics, job
system, analytics flush, linked-panel client) when loaded by a Flask CLI
one-shot such as 'flask db upgrade', or when SERVERKIT_SKIP_BACKGROUND=1.
They query the DB before migrations run, so any error there (corrupt DB,
pre-migration schema) aborted the migration and sank the whole update.
- update.sh: set SERVERKIT_SKIP_BACKGROUND=1 for both flask db upgrade
invocations as an explicit contract.
- update.sh: fail fast with an actionable message when the SQLite slot copy
fails PRAGMA integrity_check, instead of dying mid-boot with 'database
disk image is malformed'.
- update.sh: rebuild the release tarball's prebuilt venv when its baked
VIRTUAL_ENV path resolves to the old slot (it is baked as
/opt/serverkit/venv), which previously ran new code on the old slot's
dependencies during migration. Resolved-path comparison keeps the fast
prebuilt path for fresh installs.
You could not install a database from ServerKit. The tree's four roots were
literals, only MySQL and PostgreSQL reported real state, and no install path
existed anywhere in the backend.
Rather than add one, this makes a database engine an ordinary app template that
declares a top-level `engine:` block — family, wire protocol, admin user, which
variable holds the password, what it calls a "table". Presence of that block is
the only thing that lists a template in the Databases catalog, and installing it
is the template pipeline that already existed: same deployment job, same
streaming console.
The consequence is the point. Adding CockroachDB is adding a YAML file. So is
adding something we have never heard of — author it locally or sync a template
repo — and the install form still works, because it is generated from the
template's own `variables[]` rather than hand-written per engine. There is no
engine model, no engine table, no engine install mechanism to keep in step.
`test_dropping_a_new_engine_yaml_lists_it` is the design; if it ever needs code
to pass, this went wrong.
Twelve engines ship: postgresql, mysql, mariadb, mongodb, redis, valkey,
cockroachdb, clickhouse, influxdb, neo4j, meilisearch, couchdb. Each pins its
image, binds to loopback, names a data volume, and was checked against the
upstream image's actual environment contract — a wrong variable name is a
container that boots with authentication silently disabled.
Two safety notes worth reading. Engines bind 127.0.0.1 unless explicitly
exposed, and the flag is applied after caller variables so a handcrafted payload
cannot quietly publish one. And CockroachDB single-node runs `--insecure`
because password auth needs a certificate-backed cluster: for engines with no
credentials at all the loopback bind is the only control, so public exposure is
refused outright rather than warned about. A test pins the passwordless list to
exactly cockroachdb, so a future template that drops its password variable
fails loudly instead of opening a hole.
Also surfaced at last: `ManagedDatabase.owner_application_id` has been accepted
by the create endpoints for ages with no UI to set it. Creating a database now
offers "Attach to application".
Deferred deliberately: new-table builder, SQL dump import, engine extensions
(pgvector, TimescaleDB) — each a data-mutation surface deserving its own round.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dashboard's layout lived in localStorage as `{id, visible}` — a visibility
toggle wearing a layout's clothes. It carried no geometry, so widget width was
baked into SCSS (`.dash-kpi { grid-column: span 3 }`) and "make the chart wider"
was a code change. And it stayed on one machine.
`DashboardBoard` gives a user real boards: named, ordered, each holding a JSON
array of placed widgets with x/y/w/h and per-instance config. Three defaults
ship — overview, infrastructure, apps & deploys — seeded on a user's first GET
and restorable per board.
Validation is deliberate about the boring cases: geometry must fit the 12-column
grid, ids must be unique within a board, bools are rejected as coordinates
(`bool` subclasses `int`), and a rejected PUT leaves the stored board untouched
rather than half-applied. A board belonging to another user is a 404, not a 403
— there is no reason to confirm it exists.
The shipped defaults are checked by rebuilding their occupancy map cell by cell,
so editing DEFAULT_BOARDS into an overlap fails the suite instead of shipping a
broken first-run layout.
31 tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Lets a panel link itself to a master ServerKit panel as a worker WITHOUT
installing the standalone Go agent: the panel registers via the master's
existing token flow, then runs an embedded agent client (HMAC auth + HTTP
long-poll transport) that dispatches agent commands (file:write,
docker:compose:up/ps/logs, system:info/metrics) onto its own local
services. The master needs zero changes — a linked panel is just another
agent-backed Server row with scoped, revocable credentials.
- linked_panel_config table (migration 077) + Fernet-encrypted secret
- embedded agent client: connect/poll/result loop, heartbeat metrics,
per-command threads, allowed-roots guard on file:write
- API: GET/POST/DELETE /api/v1/linked-panel (+ startup resume hook)
- Servers page: LinkedPanelCard (link form, live status, unlink)
Introduces a full-stack Test Sandbox feature to run distro matrix tests in Docker from the UI: new backend API/service, sandbox_runs model + migration, run/log endpoints, and a new frontend page with mode selection, run history, live polling, cancel, and per-distro logs.
Also adds ARM64 scripts CI coverage (ubuntu-24.04-arm) and strengthens install/test automation for minimal images and overlay rebuilds: early curl/bootstrap index refresh, stricter venv validation, release-venv glibc fallback, Node reprovisioning on source fallback, rsync --delete overlay behavior, npm bootstrap in VM install, Windows tar pinning in PowerShell harness, plus updated test docs and distro coverage notes.
Analytics, CrowdSec, DNS Server, GPU Monitor, Kubernetes, Mail Server and
Automations (tramo) now live in their own repositories and install from the
registry as versioned, sha256-pinned zips. Their sources leave
builtin-extensions/, their pre-bundled frontends leave frontend/src/plugins/,
and registry_index.json gains a `bundled: false` entry for each so the
Marketplace offers them as normal registry installs.
The panel keeps only the wiring: comments at the former mount points explain
where each surface now comes from, and the gpu auto-install gate in
extension_migration.py is dropped since a host-probe can no longer decide to
re-vendor an extension that is not in this tree.
Also untracks backend/app/plugins/serverkit-{gpu,k8s,tramo}/, which were the
panel's own install artifacts rather than source. Committing those shipped a
second copy of code that now lives in three other repos, guaranteed to drift.
.gitignore now excludes the install path for every extracted slug on both the
backend and frontend side, so a local install can never be committed back.
Four tests from test_tramo_extension.py exercise panel core rather than the
extension (the retired /api/v1/workflows blueprint, the plan-45 event emitters
ported to EventService, and the docker_service double-emit fix); they move to
test_tramo_panel_core.py and stay here. The rest of the extension suites go
with their repos.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rate-limit buckets, login lockout and audit-log source IPs hand-parsed the
leftmost X-Forwarded-For token — a client-controlled value — letting an
attacker rotate it to dodge per-IP limits or forge audit trails.
- config: add TRUST_PROXY_HEADERS / TRUSTED_PROXY_HOPS (+ _env_bool/_env_int)
- create_app: wrap in gated Werkzeug ProxyFix (rightmost trusted hops) before
the limiter/handlers so every remote_addr consumer sees the real client
- new app/utils/client_ip.py::get_client_ip() — the single trusted seam
- replace all six leftmost-XFF parsers (rate_limit, audit middleware,
api_key_auth, api_analytics, audit_service, ddns) with get_client_ip()
- deploy: root .env.example ships TRUST_PROXY_HEADERS=true (behind nginx);
backend/.env.example documents it off for dev
- tests: test_trusted_client_ip.py (7) incl. real flask-limiter bucket isolation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plan 47 Phase 2. The status-pages blueprint (public /status/<slug> + /badge +
authenticated management API) and its service move into
builtin-extensions/serverkit-status/backend/ (entry_point status_pages:
status_pages_bp, url_prefix unchanged). Core no longer registers /api/v1/status
(management) or imports the service; the unused /api/v1/observability/status-pages
alias is dropped. The StatusPage/StatusComponent/etc models stay core (G2). The
public status/badge pages work whenever the extension is installed.
The core WordPress health-check job (run_health_checks) now reaches
StatusPageService.sync_component_from_health via get_installed_extension_attr and
no-ops cleanly on a lean panel.
Tests: test_status_extraction.py (8) — extraction shape, install=routes,
health-job no-op, relocated status-page mutation + component-check authz.
Adjusted test_observability_namespace + test_api_authz_hardening accordingly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plan 47 Phase 2. The tunnels blueprint + broker/publish/netutil services move
into builtin-extensions/serverkit-remote-access/backend/ (entry_point tunnels:
tunnels_bp, url_prefix unchanged). Core no longer registers /api/v1/tunnels or
imports the services; the Tunnel/ExposedService models stay core (G2).
New reusable core->extension seam get_installed_extension_attr(slug, module,
attr): the agent gateway reaches TunnelBrokerService.schedule_reconcile through
it, so tunnel self-heal still fires when the extension is installed and no-ops
cleanly otherwise. Backend re-acquisition on upgrade is covered by the Phase-2
migration.
Tests: test_tunnel_netutil.py + test_tunnel_reconcile.py re-homed into
test_remote_access_extraction.py (17), which installs the extension and runs the
helpers/reconcile against its modules.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plan 47 Phase 2. The cloud-provider blueprint + service move out of core into
builtin-extensions/serverkit-cloud-provision/backend/ (entry_point, url_prefix
unchanged). Core no longer registers /api/v1/cloud or imports the service; the
CloudProvider/CloudServer models stay core (G2). The one-time legacy cloud-secret
encryption is dropped from core boot (already run on any panel reaching this
version; the extension encrypts on write).
Also adds run_backend_acquisition() — a one-shot migration that re-acquires the
now-extracted backend on upgraded panels that installed a converted builtin
frontend-only (whose API used to come from core). Without it, ftp/cloud/etc would
lose their API on upgrade. Wired into create_app after run_auto_install.
Tests: test_cloud_provision_extraction.py (7, incl. relocated secret-at-rest +
backend-acquisition migration); secret-encryption cloud block moved out of core.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plan 47 Phase 2. The FTP blueprint + service move out of core into
builtin-extensions/serverkit-ftp/backend/ (entry_point ftp:ftp_bp, url_prefix
unchanged). Core no longer registers /api/v1/ftp or imports ftp_service — a
fresh panel that never touches FTP loads none of it. Already in
CONVERTED_BUILTIN_SLUGS, so upgrades re-acquire the backend automatically.
Acceptance (tests/test_ftp_extraction.py): uninstalled = no /api/v1/ftp routes +
app.services.ftp_service gone; installed = blueprint hot-loads and /ftp/status
responds; uninstall removes it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backstop for the updater-side skip: remove_retired_extensions() deletes a
retired extension''s copy-installed files and InstalledPlugin row on every
boot, BEFORE the plugin loader, so it is never loaded, never repaired
back, and never present for an older/offline updater to carry forward
into the next frontend build. Also drops serverkit-workflows from
CONVERTED_BUILTIN_SLUGS -- it no longer ships, so the auto-install
candidate was a permanent no-op.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The React-Flow Workflow Builder is replaced by the Automations extension
(tramo). Remove its core Python engine and port its four event emitters to
the panel event bus so tramo workflows can trigger on them via the events
bridge.
- event_service.py: new EVENT_CATALOG entries health.check_failed, git.push,
monitor.high_cpu, monitor.high_memory.
- docker_service.py: drop the app_stopped emit (app.stopped already fires via
the audit trail -- no double-emit).
- environment_health_service/git_service/monitoring_service: emit the new
panel events via EventService instead of WorkflowEventBus.
- Delete app/api/workflows.py, app/services/workflow_engine.py,
app/services/workflow_service.py; drop the blueprint + startup job
registration; neutralize the cron workflow-scheduler handler to a no-op.
- KEEP app/models/workflow.py + its tables (read-only legacy export lives in
the tramo blueprint; table drop is a future cleanup).
- Remove the two workflow-engine test classes from test_jobs.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GET /api/v1/search?q= (JWT, 2-char minimum) fans a term out across the core
entity types — services, servers, domains, databases, WordPress sites, cron
jobs (admin-only), installed extensions, and vaults (names only, never secret
values) — reusing each domain's existing access helpers so a member only sees
what their grants allow. Per-type cap of 5; each entity block is isolated so one
failing type degrades gracefully instead of 500-ing the search.
Wires the frontend api.search() async provider (200ms debounce) into the
palette's entity group, replacing the fetch-everything-on-open prefetch. Adds
tests/test_search.py covering shape, min-length, cap, member-vs-admin filtering,
cron admin-only, and the vaults names-only rule.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
app/api/survey.py defines survey_bp + its /api/v1/servers/<id>/survey routes
(plan 27/28) and its own docstring says "Mounted at /api/v1/servers (see
app/__init__.py)", but the 2026-07-08 recovery rebuild dropped the
register_blueprint call, leaving every survey/management-mode/observed-status
route a 404. The survey_service logic survived intact. Re-register it under
/api/v1/servers (same prefix as servers_bp, no route conflict). Un-blocks the
restored test_survey_api catalog/snapshot route coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Juan Denis