Every subscriber starts its own docker logs process but emitted into
the shared app room, so with two viewers open each saw every line
twice, got the other's tail replayed, and both saw 'ended' when either
stream stopped. Deliver to the owning socket while room membership
still gates it, keeping the per-line access re-check.
Every emit re-validated the socket's saved JWT including exp, and
access tokens last 15 minutes, so metrics, terminals, logs and
notifications were cut a quarter-hour after connecting; clients do
not reconnect after a server-side disconnect. Re-checks now skip exp
only; revocation, auth_version and deactivation still end the socket,
and connecting still needs an unexpired token.
Comments, docstrings and a few strings cited plan numbers ("plan 29
#9", "plan 86 §B2") from a git-ignored folder, so they pointed at
nothing; one reached users in a bottleneck hint. Code and frontend
strings are unchanged apart from those citations (checked by AST and
token comparison). A vhost still carrying the old asset comment reads
as undrifted, and a guard test keeps new citations out.
The stop flag was checked between lines, but readline() blocks until
the log grows, so closing the viewer on a quiet log left a sudo tail -f
running for the life of the panel. A watcher now kills tail on stop.
The socket emitter also gave up after 30s without a line, silently
leaving the client subscribed to nothing; it now runs until the stream
is stopped or replaced.
Creating a terminal, regenerating the pairing token and moving a server
checked the role but not workspace membership, so a developer outside a
workspace could get a shell on its servers or re-pair them to their own
agent. REST now uses the rule the Socket.IO rooms already applied, in
one helper. The Default workspace counts as shared: servers created
without a context land there and nobody is a member of it.
Toasts showed backend prose verbatim, with no code to translate
against and "application" where the UI says "service". Not-found,
missing-field, permission, conflict and agent-offline errors now come
from factories in app/exceptions.py that pair one fixed message with
a dotted code (not_found.<resource>, validation.required,
permission.denied, conflict.exists, agent.offline), so the client can
translate them.
The shared app-loading helpers and the apps, domains, servers,
databases and backups routes raise those errors instead of hand
shaping bodies, which drops the hand-shaped body count from 1132 to
850. Five routes picked an HTTP status by comparing error text; they
now key off a result code or a sentinel so rewording can't change a
status. Five broad excepts that echoed str(exc) now log the crash and
return the request id instead.
Require valid access-token sessions, scope application and server rooms, and enforce run visibility consistently across sockets and polling. Add positive-delivery and cross-user isolation regressions.
RunLogStream accepts any RunLike (__run_kind__ + id + to_dict, optional
result/step members): one room grammar run_<kind>_<id>, one batched
persisted run_log/run_status event pair, and a GET
/api/v1/runs/<kind>/<id>/logs?after_id polling twin over the new
run_log_entries table (migration 087). Deploys keep DeploymentJobLog and
dual-emit the legacy deploy_log/deploy_status during the migration
window. The unified Job runtime — which had NO live channel — adopts it
in the consumer (start/succeeded/failed frames; handlers can log via
stream_for(job)). The dead pipeline_* scaffolding was retired in E2.
Gate: emit -> persisted row -> after_id resync contract test; consumer
adoption test. MIGRATION_INVENTORY.md regenerated (census moved).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
register_channel() generates each subscribe/unsubscribe pair with ONE
auth model (connected-identity base check + optional per-channel gate)
and rooms built on sockets_rooms; metrics, container_status, terminal
and deploy convert. The pipeline_* channel (zero callers anywhere) is
deleted. Gate: per-channel auth tests driving the generated handlers;
ratchet freezes the two remaining raw handlers (file/container logs,
which own per-sid OS resources).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One shape for process polling loops: Event-based stop (immediate, not
sleep-bounded), lock-guarded idempotent start/stop, optional app-context
per tick, real logger error handling (the sockets loops reported errors
via print() — unreadable in prod), and a run_while predicate for
subscriber-scoped loops. The metrics and container-status broadcasters
convert; thread_ownership now carries ONE entry for the helper's spawn
site instead of a row per hand-rolled thread.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
app/sockets_rooms.py owns every Socket.IO room-name builder (the grammar
lived in ~25 scattered f-strings); app/constants.py SOCKET_EVENTS is
mirrored verbatim by frontend/src/constants/events.js and a contract
test parses the JS so a renamed event fails in CI instead of silently
killing a stream. sockets.py / agent_gateway / notification bus and the
two frontend components that hand-derived rooms now use the builders;
ratchet forbids new room f-strings in the realtime layer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plan 70. Domain went through this and TEN read paths were wrong, four with
effects outside the panel. Application has 211 call sites across 68 files and
drives more -- container lifecycles, nginx, backups, registry pulls, the OS
crontab -- so it was audited in full first (docs/plans/70_...md). 180 sites are
now query_active(); the 35 left unfiltered are each commented in place.
**The delete / purge split is what makes the tombstone honest.** delete_app used
to run compose_down(volumes=True) and rmtree(). Keeping a tombstone AND doing
that would put a Restore button next to a row whose data the delete had already
destroyed -- the one promise a recycle bin cannot break. So:
DELETE containers down (VOLUMES KEPT), vhost removed, firewall closed,
cron suspended, row tombstoned. All reversible.
RESTORE vhost rewritten, firewall reopened, cron resumed. Containers left
STOPPED -- "put it back" must not silently start processes or open
ports, the same reasoning that stops domain restore re-issuing certs.
PURGE volumes removed, uploaded source deleted, cron detached.
register() gained on_purge for that, wired into BOTH purge paths. And
purge_expired had NO scheduler -- fine when a tombstone was just a row, not fine
when it holds data volumes and a source tree, so a daily builtin job now runs
it. Without that, "kept until purge" would have meant kept forever and delete
would have quietly stopped reclaiming disk.
A side effect worth naming: the old _remove_data_flag protection for database
engines is now the DEFAULT for every app, and the flag is carried into
?purge=true so an explicit skip-the-bin delete still honours it.
**Cron was the blocker with no mechanical answer.** before_delete never fires on
a soft delete, and clear_application is LOSSY -- it nulls the association
without recording it, so nothing could put it back. suspend_for_application
instead disables only the jobs that were CURRENTLY ENABLED and stamps each
suspended_by_app; resume re-enables exactly those. That flag is the whole point:
without it a restore switches back on jobs the user deliberately turned off.
clear_application survives as purge-only, where lossiness is correct.
**The indirect reach (audit 3.1) is the class that grep cannot find.**
ContainerSleepPolicy, BackupPolicy, DeploymentJob and WafPolicy are enumerated
on a SCHEDULE and resolve application_id afterwards, so they never appear in a
search for Application.query. Left alone, a tombstoned app's sleep policy is
swept forever and wake_app restarts the containers the delete tore down. Those
loops test Application.deleted_ids().
Deliberately NOT scoped: reservation checks (port, name, directory) -- a
tombstone must keep holding those or a new app takes them and the restore is
refused; api/git.py's webhook VISIBILITY gate, which falls through to
"panel-wide visible" so hiding a tombstone would WIDEN access; and cron
attribution, which only labels a row.
Verified: 15 new tests, the migration replayed on a fresh DB (a tombstone frees
its private_slug, two live rows still collide), and a full delete -> restore
round trip. The 20 remaining full-suite failures are the known
serverkit-wordpress-not-installed class and are identical on a clean tree.
Two traps caught by testing, both recorded in the plan: a bad import made the
ENTIRE purge cleanup silently no-op while purge() still reported success
(_run_purge_hook swallows by design, so assert on the warning, not just ok), and
private_slug had to lose unique=True at the MODEL level or a models-built
database burns the slug regardless of the migration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Introduce app/services/run_log_service.py — the single crash-proof write seam
for all deploy-path logging (plan 51 D4/D5/D6/D10). It batches writes (one DB
commit + one socket emit per flush instead of commit-per-line), keeps a truthful
80-line in-memory failure tail, records per-step timings, sanitizes ANSI/\r,
caps persisted rows at 5000, and matches a plain-language failure hint. The
DeploymentPlanRunner now funnels log/step/close through it.
Add DockerService.compose_up_streaming (Popen + --ansi never --progress plain,
line callback) so template installs stream the real pull/build transcript live
and a failure persists a meaningful tail — the biggest reason install errors
were unreadable. The runner's local compose.up step adopts it.
Replace the orphaned build_* socket surface (zero subscribers, grep-verified)
with the Deploy Console channel: emit_deploy_log / emit_deploy_status to room
deploy_{job_id}, plus subscribe_deploy/unsubscribe_deploy (authenticated).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Four foundational enhancements toward the deploy-code developer experience:
- Container status aggregator (C7): new ContainerStatusService applies a
deterministic priority hierarchy (degraded > restarting > running:unhealthy >
starting > running:healthy > exited > unknown) over an app's containers,
cached with a short TTL. Exposed at /api/v1/status and pushed over a new
container_status Socket.IO channel; Applications page now renders an
aggregated ContainerStatusPill with sub-status tooltip.
- API token scopes + sensitive data masking (C10): require_scope() decorator
enforces fine-grained scopes for API-key requests (pass-through for JWT),
a canonical SCOPES catalog at /api/v1/api-keys/scopes, and a reusable
sensitive_data_filter (mask_sensitive) honoring secrets:read. Frontend gains
a grouped ApiKeyScopesModal picker and per-key scope pills.
- Server onboarding state machine (C8): new onboarding lifecycle
(pending -> validating -> installing_prerequisites -> installing_docker ->
pairing_agent -> ready/failed) with ServerOnboardingService, a
server_onboarding_logs table, servers.onboarding_* columns, REST endpoints,
and an OnboardingWizard surfaced on the server detail overview.
- Declarative template catalog (C5): documented catalog schema and
magic-variable resolution (SERVICE_PASSWORD_/USER_/FQDN_/URL_/BASE64_*)
layered onto the existing YAML template loader without changing behavior for
templates that don't use them; GET /api/v1/templates/catalog/schema.
Migration 029 adds the onboarding table + columns. 60 new tests pass; frontend
builds clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reusable, non-blocking notification layer on top of the Queue Bus:
notify.send(event, to, data) records a Notification + per-channel
NotificationDelivery, queues them, and a daemon consumer delivers via
pluggable channel adapters (in-app, email, chat) inheriting retry,
backoff and dead-letter.
- Jinja2 email system: branded, dark-mode, Outlook-safe base layout +
per-event templates (security/backup/monitoring/welcome/invitation).
The legacy inline f-string email was removed and now renders here.
- In-app center API (inbox/unread-count/mark-read) + per-user Socket.IO push.
- Admin delivery log (stats/filter/retry).
- Email providers (SMTP/SendGrid/Postmark/SES/Mailgun) with encrypted
credentials + test; the email adapter prefers the default provider and
falls back to the legacy SMTP config; surfaced in ConnectionRegistry.
- NotificationService.send_all() is now a non-blocking shim that routes
system alerts through the bus. Migrations 023 + 024.
Tests: backend/tests/test_notification_bus.py (20). Full suite: 391 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Socket.IO connect handler only checked that the JWT decoded — it never
confirmed the account still exists or is active, and recorded no identity. The
generic join_room and subscribe_terminal handlers then performed no
authorization, so a read-only viewer (or a deactivated account with an
unexpired token) could attach to a live PTY stream room and watch a privileged
shell.
- connect: resolve the JWT identity to a live, active User and stash
{user_id, role} per sid; reject missing/deactivated accounts.
- subscribe_terminal and join_room (for _terminal: rooms): require
admin/developer, matching the @developer_required on terminal creation.
- disconnect: drop the per-sid identity record.
Job-progress and cloudflared-login streaming (non-terminal server rooms) are
unaffected — that data is already reachable over REST by any authenticated user.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The "Terminal & Logs" page had no terminal. Now a Terminal tab hosts the
xterm console (RemoteTerminal) against paired agent servers, behind a
demo-style target rail (.term-shell: agent list with glowing status
dots, offline rows disabled, first online server auto-selected).
RemoteTerminal was built but never mountable - it imported a
nonexistent SocketContext and listened for a terminal:<sid> event that
nothing emits. Fixed the full output path:
- backend sockets.py: new subscribe_terminal/unsubscribe_terminal
handlers join the browser into the agent gateway stream room
(server_<id>_terminal:<session>) after validating the session via
TerminalService.get_session
- RemoteTerminal: rewired to services/socket.js; consumes server_stream
events filtered by the session channel (agent PTY -> gateway ->
xterm), re-subscribes on socket reconnect, unsubscribes on unmount
VALID_TABS gains terminal (appended - logs stays the default landing
since the shell needs a connected agent; panel-host PTY remains a S5
gap, TerminalService rejects local).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace generic @jwt_required() usage with role-based decorators and switch Socket.IO token handling to the auth payload.
- Replace jwt_required with viewer_required/admin_required across API modules (backend/app/api/cron.py, firewall.py, ftp.py, git.py, metrics.py) and update imports accordingly.
- Require JWT for previously public system/private endpoints (backend/app/api/system.py, private_urls.py).
- Change Socket.IO server connection handler to accept auth payload and read token from auth (backend/app/sockets.py) to avoid token leakage in query strings.
- Update frontend clients to send token via socket auth instead of query (frontend/src/pages/WordPressProject.jsx, frontend/src/services/socket.js).
These changes add finer-grained RBAC control for endpoints and improve socket auth security by removing tokens from URLs/logs.
Introduces a comprehensive REST API for WordPress multi-environment management, including endpoints for project listing, environment CRUD, promotion, sync, comparison, locking, activity logs, container logs, resource limits, basic auth, WP-CLI, health checks, disk usage, bulk operations, and auto-sync scheduling. Adds new models for environment activity, promotion jobs, and sanitization profiles, and integrates auto-sync scheduling into the app startup. Updates model imports and registration to include the new models.
- Add subscribe_container_logs event for real-time log streaming
- Add unsubscribe_container_logs event to stop streaming
- Stream logs via docker logs --follow subprocess
- Parse log lines with timestamp and level detection
- Emit container_log events to room logs_{app_id}
- Handle container_log_ended when container stops
- Clean up processes on disconnect or unsubscribe
- Support service selection for compose apps
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Introduces REST API endpoints for build and deployment management, including build configuration, triggering, logs, deployment history, rollback, and diff functionality. Adds new models for Deployment and DeploymentDiff, and implements BuildService and DeploymentService for orchestrating build and deploy workflows. Updates application initialization to register the new API blueprint and adjusts model exports accordingly. Frontend and style changes are not included in this summary.
Add backend and frontend foundations for ServerKit, including Flask API structure, SQLAlchemy models, service layers, and React app with authentication context and service hooks. Implement core API endpoints for authentication, application management, Docker, databases, and system operations. Update README with project overview, tech stack, and setup instructions. Add development roadmap and example environment configuration.
Juan Denis