Comments cited plan sections as "(§G3)" or "(§8 unification)", which
point into the same git-ignored plans the plan-number citations did.
References next to a real document (docs/REDESIGN_MAP.md §3) stay. Code
and runtime strings are unchanged (AST and token comparison).
Comments, docstrings and a few strings cited plan numbers ("plan 29
#9", "plan 86 §B2") from a git-ignored folder, so they pointed at
nothing; one reached users in a bottleneck hint. Code and frontend
strings are unchanged apart from those citations (checked by AST and
token comparison). A vhost still carrying the old asset comment reads
as undrifted, and a guard test keeps new citations out.
Toasts showed backend prose verbatim, with no code to translate
against and "application" where the UI says "service". Not-found,
missing-field, permission, conflict and agent-offline errors now come
from factories in app/exceptions.py that pair one fixed message with
a dotted code (not_found.<resource>, validation.required,
permission.denied, conflict.exists, agent.offline), so the client can
translate them.
The shared app-loading helpers and the apps, domains, servers,
databases and backups routes raise those errors instead of hand
shaping bodies, which drops the hand-shaped body count from 1132 to
850. Five routes picked an HTTP status by comparing error text; they
now key off a result code or a sentinel so rewording can't change a
status. Five broad excepts that echoed str(exc) now log the crash and
return the request id instead.
Plan 87 phase C. A compose app on slots runs each slot as its own
project, <name>-a / <name>-b, rendered from the merged compose config:
container_name stripped, only the web service published, on the
slot's loopback port, and every named volume pinned to the name it has
today. Without the pin compose prefixes volumes with the project name
and slot B would start on empty ones; the real-Docker leg checks this.
Stateful services (a database, cache or queue with a named volume)
cannot run twice. They move once, on an explicit previewed split, into
a shared <name>-data project that both slots reach by the same names.
The webhook, template-update and rollback paths deploy through the
engine instead of stopping the stack. The --scale web=2 rolling
restart and the Webhook.zero_downtime flag are deleted: it assumed a
service named web, ignored its own health result, and no UI set it.
Compose slot images get immutable per-deployment tags and are pruned
with the rest.
Plan 87 phase F. A slot deploy job lists its real stages (Preflight,
Build, Boot slot X, Health gate, Switch, Watch) and advances through
them as the engine reaches each one, so PipelineStrip renders them
unchanged. The overview gets a slot card: which release is live,
which is the standby and how long it stays warm, with Switch back;
when slots are off it lists why the app does or does not qualify.
Settings gains the opt-in, watch window, standby minutes and the
shared-volume confirmation.
Plan 87 phase B. An app that opts in keeps two slots, like the
panel updater's two installs. A deploy boots the new release in the
idle slot on its own loopback port while the live one serves, holds
it to the health gate, then switches nginx by moving app.port to the
new slot and rewriting the vhost, so every app.port reader follows
without edits. A watch window switches back on its own if the
release falls over; the old slot stays warm for an instant switch
back, then is stopped (not removed) by a standby sweep.
Opting in adopts the running container as slot a with no restart.
Eligibility is shown with reasons and re-checked on every deploy;
opting out folds the slots back on the next in-place deploy. Slot
images are never pruned and slot ports stay reserved.
Plan 87 phase A. Fixes the deploy bugs that make a rollback or a
health check lie:
- builds get an immutable tag per deployment, so rolling back runs
the image that actually ran instead of whatever :latest became;
old tags are pruned so this is not a new disk leak
- the deployment records the commit after the pull, not before
- the health wait is a real gate: a 4xx fails, it takes three passes
in a row, an exited container fails at once, and it raises
- a per-app deploy lock makes a second deploy wait for the first
- a compose rollback rebuilds and recreates its containers, and a
rollback with no version finds the previous release again
- a failed template update says when restoring the old stack failed
Health path, timeout and the 4xx rule are editable in Settings.
Webhooks were the only way to hear a core event, and app.deployed
sat in the catalog without ever being emitted. EventService now
calls registered in-process listeners on every emit (best-effort,
torn down with the extension), and a successful deploy emits
app.deployed with the app's hostnames, which a CDN extension uses to
purge exactly those hosts.
Only the Procfile web: line was read, so worker: and scheduler:
never ran. Each now runs as serverkit-app-<id>-<process> from the
same image with the app's env and volumes, no port, and a restart
policy; start, stop and restart follow the web container, and a line
removed from the Procfile goes on the next deploy. Single-container
apps also join the shared service network, which only compose apps
did, and the app's container list now shows the web and its workers.
A fresh panel no longer lets whoever reaches it first become admin:
registering the first account needs a one-time code the installer
prints (`serverkit setup-code` repeats it). SERVERKIT_ADMIN_EMAIL and
SERVERKIT_ADMIN_PASSWORD create the admin unattended instead.
Git apps now deploy what users expect: a repository with a compose
file deploys every service it declares, "Deploy latest" builds the
branch's newest commit, and push webhooks rebuild the containers.
Also fixes branch switching on single-branch clones, uploads of a
lone file, compose v1 port preflight, and CLI commands that ignored
their flags.
Plan 75 §G1, the remaining 33 files. Every raw subprocess call site in
backend/app that can go through a door now does. What is left is what
should be:
8 app/utils/system.py the doors themselves
8 database_service mysqldump|gzip, gunzip|mysql, pg_dump|gzip,
gunzip|psql — a pipe is two processes
7 build/docker/db_sync/ftp streaming output, gzip pipes
1 log_service tail -f
2 serverkit-analytics sibling repo; see below
Extensions were migrated in BOTH trees. builtin-extensions/ is the source
and backend/app/plugins/ the live copy, compared by normalised hash;
test_builtin_extension_drift stays green. serverkit-email has both sides
tracked, serverkit-ftp and serverkit-cloud-provision only the source.
serverkit-analytics (2 calls) is deliberately untouched: it has no source
in builtin-extensions/ at all, and backend/app/plugins/serverkit-analytics
is a gitignored materialised copy. Editing it would be a change that
cannot be committed and is overwritten on the next sync — that fix belongs
in the serverkit-analytics repo.
Exit-code semantics are preserved wherever they carry meaning, with a
comment at each site, because they are the thing a mechanical migration
destroys: clamscan 1 = INFECTED, grype 1 = vulnerabilities FOUND, yara
1 = matches, cron_runner 127 = never launched. Those read
result['returncode'], never result['success'].
run_checked gained two things real call sites needed: it yields when the
caller directs the streams itself (writing straight to a log file), rather
than adding capture_output on top and raising ValueError; and text/errors
pass through, so site_importers/ssh keeps decoding with errors='replace'
instead of gaining a UnicodeDecodeError on a bad byte.
A regression this migration introduced, caught by the suite and fixed:
install_profile_service._docker_usable() is deliberately TRI-STATE —
None means the probe could not run, because inside LXC the docker client
is usually present while the daemon never started, and callers "must not
treat a broken probe as proof of absence". Writing it as
run_checked(...)['success'] turned an exec failure into False, i.e. "docker
is not usable" — a fact nobody established. That is the §A violation this
plan exists to remove, reintroduced by the migration meant to prevent it.
test_probe_honesty_doctor caught it. Every other run_checked(...)['success']
caller was then audited; the rest are genuinely two-state, and their
originals also returned False on a missing binary.
Seven more slips of the same shape were caught before running by an AST
check for `x = run_checked(...)` followed by `x.stdout/.stderr/.returncode`
— a dict read as an object, which raises only on the failure path, so
happy-path tests stay green.
api_controller_boundary_baseline.json: pure deletion. Routing api/docker.py
through DockerService removed the last 6 subprocess crossings, so the
`subprocess` category is gone from the API layer entirely.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 72 item B.1. Every redeploy path reached its point of no return -- a
compose down, or a stop+remove -- before anything had checked that the new
version could actually run. An invalid compose file, an image that no longer
pulls, or a port somebody else took stopped the site and left it stopped,
nginx 502ing until a human intervened. The fix is not blue/green (that is
B.2/B.3, deliberately not attempted here). It is ordering.
deploy_preflight_service runs everything checkable while the old containers
keep serving: docker compose config --quiet -- wiring the long-existing
validate_compose_file, whose only caller until now was a manual API endpoint
-- then image resolution to completion, then host port conflicts. Nothing in
it stops, removes, recreates or rewrites anything; its only side effects are
pulls and builds, which are additive. Each caller aborts on a blocking
finding, and the log line says the running deployment was left untouched,
because that is the whole user-visible payoff.
Three paths were unsafe and are fixed; each now carries an explicit marker at
the line past which the live deployment is gone.
The single-container path had the sharpest version of the bug: the registry
pull sat BELOW the stop and remove, so a registry hiccup left the app with no
container at all. The build already ran earlier, so the pull is the only
pre-traffic requirement, and it now happens first. The template update
renders the new compose to a sibling dotfile and validates and pulls THAT, so
the live docker-compose.yml is untouched until validation passes; same
directory, so relative bind mounts and .env resolve identically. The git
webhook default builds before it stops, and its later up --build is a cache
hit.
Everything is fail-open, which matters more than the checks themselves: when
compose config cannot be introspected the dependent checks are skipped, and a
pull failure is downgraded to advisory when every referenced image is already
present locally, so a locally-tagged image that was never pushed keeps
deploying exactly as before. A preflight must never abort a deploy the old
code would have run.
The port check needed reframing rather than reuse. The probe itself is
ManifestApplyService._port_bound, called through the class so the existing
test seam serves both surfaces, but a bare probe flags the deploy's own live
container and would abort every ordinary redeploy. It only fires for ports the
new config wants that the running project does not already publish, and skips
entirely when ownership cannot be determined. It is deliberately not applied
to the single-container path, where app.port is held by the app's own
container when one exists and there is nothing live to lose when one does not.
Paths already safe were left alone rather than given ceremony: the
zero_downtime restart builds first and never downs, template install has no
live container, and the upload/rollback/image-update paths already pull or
build before compose up.
Also fixes a pre-existing bug found in update_app: install_info was assigned
only inside a try but read unconditionally later, so a missing or corrupt
.serverkit-template.json raised NameError *after* compose_down and left the app
stopped with a new compose that was never started -- the exact outage this
item targets.
No existing test asserted the old unsafe ordering, so nothing had to be
rewritten. The new tests prove the ordering directly: on a validation failure
compose_down is never called, the template's compose file is byte-for-byte
unchanged, and stop_container/remove_container are never reached on a pull
failure.
Not verified: no real Docker was exercised -- every docker seam is stubbed, so
build/pull exit behaviour on a mixed project is reasoned-about, not observed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Plan 70. Domain went through this and TEN read paths were wrong, four with
effects outside the panel. Application has 211 call sites across 68 files and
drives more -- container lifecycles, nginx, backups, registry pulls, the OS
crontab -- so it was audited in full first (docs/plans/70_...md). 180 sites are
now query_active(); the 35 left unfiltered are each commented in place.
**The delete / purge split is what makes the tombstone honest.** delete_app used
to run compose_down(volumes=True) and rmtree(). Keeping a tombstone AND doing
that would put a Restore button next to a row whose data the delete had already
destroyed -- the one promise a recycle bin cannot break. So:
DELETE containers down (VOLUMES KEPT), vhost removed, firewall closed,
cron suspended, row tombstoned. All reversible.
RESTORE vhost rewritten, firewall reopened, cron resumed. Containers left
STOPPED -- "put it back" must not silently start processes or open
ports, the same reasoning that stops domain restore re-issuing certs.
PURGE volumes removed, uploaded source deleted, cron detached.
register() gained on_purge for that, wired into BOTH purge paths. And
purge_expired had NO scheduler -- fine when a tombstone was just a row, not fine
when it holds data volumes and a source tree, so a daily builtin job now runs
it. Without that, "kept until purge" would have meant kept forever and delete
would have quietly stopped reclaiming disk.
A side effect worth naming: the old _remove_data_flag protection for database
engines is now the DEFAULT for every app, and the flag is carried into
?purge=true so an explicit skip-the-bin delete still honours it.
**Cron was the blocker with no mechanical answer.** before_delete never fires on
a soft delete, and clear_application is LOSSY -- it nulls the association
without recording it, so nothing could put it back. suspend_for_application
instead disables only the jobs that were CURRENTLY ENABLED and stamps each
suspended_by_app; resume re-enables exactly those. That flag is the whole point:
without it a restore switches back on jobs the user deliberately turned off.
clear_application survives as purge-only, where lossiness is correct.
**The indirect reach (audit 3.1) is the class that grep cannot find.**
ContainerSleepPolicy, BackupPolicy, DeploymentJob and WafPolicy are enumerated
on a SCHEDULE and resolve application_id afterwards, so they never appear in a
search for Application.query. Left alone, a tombstoned app's sleep policy is
swept forever and wake_app restarts the containers the delete tore down. Those
loops test Application.deleted_ids().
Deliberately NOT scoped: reservation checks (port, name, directory) -- a
tombstone must keep holding those or a new app takes them and the restore is
refused; api/git.py's webhook VISIBILITY gate, which falls through to
"panel-wide visible" so hiding a tombstone would WIDEN access; and cron
attribution, which only labels a row.
Verified: 15 new tests, the migration replayed on a fresh DB (a tombstone frees
its private_slug, two live rows still collide), and a full delete -> restore
round trip. The 20 remaining full-suite failures are the known
serverkit-wordpress-not-installed class and are identical on a clean tree.
Two traps caught by testing, both recorded in the plan: a bad import made the
ENTIRE purge cleanup silently no-op while purge() still reported success
(_run_purge_hook swallows by design, so assert on the warning, not just ok), and
private_slug had to lose unique=True at the MODEL level or a models-built
database burns the slug regardless of the migration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two worst endpoints in the backend, both measured in plan 69 phase 6.
**/status/apps** ran one `docker compose ps` per app plus a `docker inspect`
per container -- roughly 30 apps x 3 containers = ~120 process spawns per
request -- and it is refilled on a timer, so the cost never amortised. It is
now ONE host-wide `docker ps` whose output is indexed by the compose
project/service/working-dir labels Docker already stamps on every container,
shared between callers through a 3-second in-memory snapshot.
The snapshot is deliberately module memory rather than CacheService: it must be
droppable synchronously, because a status that survives the action that changed
it reads as a broken panel. So `invalidate(app_id)` is called from start, image
apply, deploy and rollback. 3s is short enough that a missed hook self-corrects
within one poll.
**/admin/activity/summary** ran two 90-day loops issuing one COUNT per day =
180 sequential queries. Now one `GROUP BY func.date(...)` per series with the
gaps zero-filled in Python, so the response shape is byte-identical. `func.date`
is portable across the SQLite and PostgreSQL this app supports. A third N+1 fell
out on the way: the top-users block did one SELECT per user, now a single IN.
New tests assert the zero-fill specifically -- a day with no rows must still
appear and the series length must match the window, which is exactly what a
GROUP BY drops if you forget.
1140 backend tests pass. The 3 failures in that run (test_fail2ban_jail_service,
test_url_swap, test_site_routing) all reproduce on a clean tree -- the last is
the known "serverkit-wordpress extension not installed" class.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Give a service tracked, named persistent storage that survives redeploys,
replacing fragile relative bind mounts (./mysql-data:/var/lib/mysql).
- AppVolume model (namespaced serverkit-app-{id}-{slug}, unique per mount path)
+ migration 043.
- volume_service: create (docker volume + row, dup-path refusal), list_for_app
(rows + live inspect), delete (row-only by default; wipe only when the app is
stopped and no running container uses it), measure, convert_bind_mount,
run_args, compose_fragment, host_paths_for_app.
- Wired into the single-container deploy (_deploy_docker passes -v specs to
docker run) and new compose stacks (create_docker_app gains named_volumes ->
top-level volumes: block).
- API under /api/v1/apps/<id>/volumes (list/attach/detach?wipe=/convert), ACL
via ResourceGrantService.
- Frontend: Storage panel in Service detail Settings (attach/detach with an
off-by-default "also delete data" checkbox, disabled while running).
- Tests: create + namespaced volume, dup-path refusal, wipe safety, single-
container -v proving test, compose top-level named-volume generation, convert,
API CRUD.
Scoping: volumes are backup-addressable (mountpoints exposed; protect via the
files target) rather than force-folded into the app's single-parent incremental
tar chain, which would break restore. Live compose files (esp. the WP multidev
stack) are not rewritten -- convert_bind_mount is the opt-in migration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Store per-registry credentials (Fernet-encrypted) and run `docker login`
before pulling a private image — GHCR, Docker Hub, GitLab, ECR, or any
generic registry. Public/anonymous pulls are unchanged.
- ContainerRegistry model (secret encrypted at rest; to_dict never returns it)
+ nullable applications.registry_id; migration 042.
- container_registry_service: CRUD + login/logout/test_connection (secret piped
via stdin, never on argv) + lazy ECR token exchange.
- DockerService.pull_image gains a registry= arg (login -> pull -> logout in a
try/finally). Wired into the deploy path (_deploy_docker) and the compose
path (start_app / apply_image_update): a docker login authenticates both
`docker pull` and `docker compose pull`.
- API under /api/v1/connections/registries (list=auth, writes=admin); registries
join the unified /connections list.
- Frontend: Connections hub "Container registry" card (add/test/remove with a
provider selector) + a per-app registry picker in Container Ops.
- Tests: encrypt/decrypt, stdin-not-argv, login->pull->logout ordering incl.
finally-on-failure, deploy + compose auth-before-pull, API CRUD/masking/admin.
Deviation from the plan's "single injection point": docker apps run via compose,
so login was wired into both the compose and run_container paths; remote-agent
compose is a follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shared variable groups were resolved/displayed but never reached the container.
EnvService.get_effective_env(app) now merges them in: shared groups
(workspace < project < environment < direct) form the base layer and the app's
own local env vars override them, so a key set in both yields the local value —
matching the "Set locally — local value applies" UI hint. Secrets are decrypted
for injection; shared resolution is best-effort and never blocks a deploy.
The docker deploy path (_deploy_docker) now injects this merged env via
run_container, which also fixes a latent bug there (it called .get('success')
on get_env_vars()'s list result, so env was effectively never applied). Updated
the shared-variables panel note to say shared vars now apply at deploy with
local taking precedence.
Proving test test_effective_env.py (6 cases) covers the merge precedence, the
best-effort fallback, and that _deploy_docker hands run_container the merged env.
Full backend suite: 941 passed, 14 skipped; frontend builds clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Build packs / zero-Dockerfile deploys (C1): new BuildpackService detects the
stack from a cloned repo (Node/Python/Go/PHP/Ruby/Rust/static), produces a
transparent build plan and a generated Dockerfile, and persists the plan +
overrides on the Application. POST /api/v1/buildpacks/{detect,generate}; a
BuildpackPreview shows the detected plan + editable overrides + Dockerfile in
the create wizard and the app Build tab. Builds on the existing build_method
plumbing instead of duplicating it.
- Deployment config snapshots + diff engine (C4): ConfigurationService captures
an immutable, hashed snapshot of resolved config (env keys + masked values,
domains, image/tag, build method, volumes) at every deploy and rollback; new
DeploymentSnapshot model + /api/v1/apps/<id>/snapshots[/diff|/restore]. A
DeploymentTimeline + ConfigDiffModal surface "what changed" and one-click
restore in the app Deploy tab. Secret values are never serialized.
- Template catalog certification (C5 cont.): a whole-catalog validation test
proves all 105 declarative templates conform to the documented schema; fixed
the stale index.json (was 63/105 entries) to list every bundled template.
Migration 030 adds the buildpack columns + deployment_snapshots table. Full
backend suite: 820 passed, 14 skipped; frontend builds clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce safer system interactions and improved logging fallbacks across services. Replace direct sudo/subprocess calls with run_privileged/privileged_cmd, add command availability checks (is_command_available) and Linux guards for user management. Enhance LogService with fallback readers (python file I/O, grep fallback, syslog and Windows Event Log support) and standardize responses via sourced_result. Update frontend Terminal to display log source and adjust UI when journalctl is unavailable. Misc: delegate ProcessService log retrieval to LogService, add PackageManager usage in PHP/SSL installs, and add warning logging in DeploymentService when diff generation fails.
Refactor services to use centralized system helpers (ServiceControl, PackageManager, run_privileged, is_command_available) instead of ad-hoc subprocess/os calls. Updated deployment, firewall, ftp, nginx, php, python, security, and ssl services to call the new utilities and added error/timeout handling improvements and package manager abstraction. Added backend/app/utils/system.py implementing the helpers and backend/tests/test_utils_system.py with unit tests. This centralizes privileged command execution and service/package management, simplifying tests and platform differences.
Introduces backend/app/paths.py to centralize all key directory paths, allowing overrides via environment variables for local development. Updates all backend services and APIs to use these centralized paths instead of hardcoded values. Adds dev path override examples to backend/.env.example and updates .gitignore to exclude dev-data. Also changes frontend dev server port to 5274 and updates related scripts and CORS settings for consistency.
Replaced all uses of app.path with app.root_path in backend API and service layers for consistency and correctness. Renamed the 'metadata' field to 'extra_data' in the Deployment model to avoid SQLAlchemy reserved word conflicts and updated related methods. Also updated frontend to import Toast context from the correct location and added new border and font size variables in styles.
Introduces REST API endpoints for build and deployment management, including build configuration, triggering, logs, deployment history, rollback, and diff functionality. Adds new models for Deployment and DeploymentDiff, and implements BuildService and DeploymentService for orchestrating build and deploy workflows. Updates application initialization to register the new API blueprint and adjusts model exports accordingly. Frontend and style changes are not included in this summary.
Juan Denis