ServerKit/docs/ARCHITECTURE.md

ServerKit Architecture

Deep dive into how ServerKit connects domains, apps, containers, databases, and extensions.

Diagrams. Each section leads with an image and keeps an ASCII fallback in a collapsible fold underneath. Images live in docs/images/architecture/; if you change one, update the ASCII block so the doc still reads in plain text.

---

Table of Contents

• System Overview
• Request Flow
• Backend Layers
• Extension Platform
• Template System
• Port Allocation
• Service Linking & Env Injection
• Jobs & Scheduling
• Notifications Bus
• Agent Fleet
• Environment Pipeline
• File Paths

---

System Overview

ASCII diagram

                                    ┌─────────────────────────────────────────────────────────────┐
                                    │                        INTERNET                             │
                                    └─────────────────────────────────────────────────────────────┘
                                                              │
                                                              ▼
┌─────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                         YOUR SERVER                                                  │
│  ┌───────────────────────────────────────────────────────────────────────────────────────────────┐  │
│  │                                    NGINX (Reverse Proxy)                                      │  │
│  │                                      Port 80 / 443                                            │  │
│  │                                                                                               │  │
│  │   ┌──────────────┐    ┌──────────────┐    ┌──────────────┐    ┌──────────────┐              │  │
│  │   │ app1.com     │    │ app2.com     │    │ api.app3.com │    │ Panel API    │              │  │
│  │   │    :443      │    │    :443      │    │    :443      │    │  /api/v1/    │              │  │
│  │   └──────┬───────┘    └──────┬───────┘    └──────┬───────┘    └──────┬───────┘              │  │
│  └──────────┼───────────────────┼───────────────────┼───────────────────┼────────────────────────┘  │
│             │                   │                   │                   │                           │
│             │ proxy_pass        │ proxy_pass        │ proxy_pass        │ proxy_pass                │
│             ▼                   ▼                   ▼                   ▼                           │
│  ┌────────────────────────────────────────────────────────────┐  ┌─────────────────────────────┐   │
│  │                    DOCKER CONTAINERS                       │  │      SERVERKIT PANEL        │   │
│  │                                                            │  │   Flask + Gunicorn (-w 1)   │   │
│  │   ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐  │  │                             │   │
│  │   │WordPress │  │  Flask   │  │ Node.js  │  │  Custom  │  │  │  api/  services/  models/   │   │
│  │   │  :8001   │  │  :8002   │  │  :8003   │  │  :8004   │  │  │  jobs/ notifications/       │   │
│  │   │          │  │          │  │          │  │          │  │  │  plugins/  agent_gateway    │   │
│  │   │ Apache   │  │ Gunicorn │  │   PM2    │  │  Your    │  │  │                             │   │
│  │   │ PHP-FPM  │  │ Python   │  │ Express  │  │  App     │  │  │  Socket.IO (threading)      │   │
│  │   └────┬─────┘  └──────────┘  └──────────┘  └──────────┘  │  └──────────────┬──────────────┘   │
│  └────────┼───────────────────────────────────────────────────┘                 │                  │
│           │                                                                     │                  │
│           ▼                                                                     ▼                  │
│  ┌────────────────────────────────────────────────┐   ┌──────────────────────────────────────┐   │
│  │                  DATABASES                     │   │          PANEL STATE                 │   │
│  │  ┌────────┐ ┌──────────┐ ┌───────┐ ┌────────┐ │   │  SQLite (default) or PostgreSQL      │   │
│  │  │ MySQL  │ │Postgres  │ │ Redis │ │Mongo   │ │   │  Alembic migrations                  │   │
│  │  │ :3306  │ │  :5432   │ │ :6379 │ │ :27017 │ │   └──────────────────────────────────────┘   │
│  │  └────────┘ └──────────┘ └───────┘ └────────┘ │                                               │
│  └────────────────────────────────────────────────┘                                               │
│                                                                                                    │
└────────────────────────────────────────────────────────────────────────────────────────────────────┘
                                          │
                                          │ Socket.IO /agent namespace (HMAC) + HTTP long-poll
                                          ▼
                       ┌──────────────────────────────────────────────┐
                       │   REMOTE SERVERS (managed fleet)             │
                       │   serverkit-agent (Go) — separate repo       │
                       └──────────────────────────────────────────────┘

The numbered zones in the diagram map to the rest of this document:

#	Zone	Deep dive
1	Clients — admin users on the panel, public visitors on hosted sites	Request Flow
2	Edge layer — nginx terminating TLS, splitting panel traffic from public traffic	Request Flow
3	ServerKit panel — the Flask app and everything inside it	Backend Layers, Extension Platform, Jobs, Notifications
4	Runtime layer — app containers, databases, panel state, on-disk paths	Port Allocation, File Paths
5	Remote agent fleet — Go agents on other servers	Agent Fleet

Read as tiers rather than zones, that is:

1. Frontend — React 18 SPA (Vite + SCSS), built into Flask's static folder in production.
2. Backend — Flask REST API + Socket.IO managing Docker, nginx, databases, and system services.
3. Agent — a Go binary running on remote managed servers. Its source lives in the separate serverkit-agent repo, not in this one, so panel↔agent protocol changes are not atomic in a single commit.

Two flows cross the diagram in opposite directions: control and management (panel → edge → containers, and panel → agents as outgoing commands) and public traffic / telemetry (visitors → edge → containers, and agents → panel as incoming heartbeat and metrics).

---

Request Flow

ASCII diagram

User Request                    What Happens
─────────────────────────────────────────────────────────────────────────────────

  Browser                 1. DNS resolves app1.com to your server IP
     │
     ▼
┌─────────┐              2. Request hits Nginx on port 80/443
│  Nginx  │                 Nginx checks server_name directives
│ :80/443 │                 Matches "app1.com" → proxy_pass http://127.0.0.1:8001
└────┬────┘
     │
     ▼
┌─────────┐              3. Nginx forwards request to Docker container
│ Docker  │                 Container receives request on internal port
│ :8001   │                 App processes and returns response
└────┬────┘
     │
     ▼
┌─────────┐              4. Response flows back through Nginx
│ Response│                 SSL termination handled by Nginx
│  200 OK │                 User sees the page
└─────────┘

Panel requests take the same path but terminate at Flask. Everything under /api/v1/ is JSON and auth-protected (JWT by default; a route accepts API keys only by opting in through the shared policy decorator); the Flask 404 handler serves index.html so client-side SPA routing works on deep links, while API routes return typed JSON error envelopes carrying a request-correlation id, shaped by one global handler (app/exceptions.py).

Client IP. Behind the bundled nginx, set TRUST_PROXY_HEADERS=true and TRUSTED_PROXY_HOPS=1 so Werkzeug's ProxyFix derives the real client IP; leave it off for a directly-exposed dev server. All IP reads go through a single get_client_ip() helper. See SECURITY.md.

---

Backend Layers

The Flask app factory is create_app() in backend/app/__init__.py. Three layers:

Layer	Path	Role
API	app/api/	Flask Blueprints, one file per feature. All routes under /api/v1/, guarded by the policy decorators in app/middleware/rbac.py (auth_required() / role checks — JWT-only is the default policy, API-key access is per-route opt-in; the caller's identity always resolves through rbac.get_current_user()).
Services	app/services/	Business logic. Stateless modules; all shell-outs, Docker API calls, and file writes live here.
Models	app/models/	SQLAlchemy ORM. Schema managed by Alembic migrations. Shared shapes live in mixins (TimestampMixin, JsonColumnMixin, SoftDeleteMixin, SerializableMixin, RunLifecycleMixin) and run/status fields use the canonical status vocabulary in app/models/status.py.

Cross-cutting subsystems that sit beside those three:

• app/jobs/ — background work and scheduling (see Jobs & Scheduling)
• app/notifications/ — the notification bus (see Notifications Bus)
• app/plugins/ + app/plugins_sdk/ — the extension runtime and its SDK
• app/sockets.py — the Socket.IO channel registry (one declarative subscribe/unsubscribe door per channel) over a shared room grammar (app/sockets_rooms.py), plus the generalized run envelope (run_log/run_status events in run_<kind>_<id> rooms); two legacy raw log handlers remain, frozen by test until migrated
• app/agent_gateway.py — the /agent Socket.IO namespace for the remote fleet
• app/middleware/security.py — security headers
• app/paths.py — the single source of truth for on-disk locations

---

Extension Platform

Most non-core functionality ships as an extension. The panel stays lean and operators install only what they need.

ASCII diagram

┌───────────────────────────────────────────────────────────────────────────────┐
│                       REGISTRY (serverkit-extensions repo)                    │
│                       curated index.json — schema v2                          │
│   SERVERKIT_REGISTRY_URL (unset → GitHub raw; set-but-empty → disabled)        │
└───────────────────────────────┬───────────────────────────────────────────────┘
                                │ fetch (TTL 1h)
                                │ fallback: last-good cache → app/data/registry_index.json
                                ▼
┌───────────────────────────────────────────────────────────────────────────────┐
│                          MARKETPLACE (panel UI)                               │
│   preview → show permissions + sha256 → operator consents → install           │
└───────────────────────────────┬───────────────────────────────────────────────┘
                                │ download zip, verify sha256 BEFORE extract
                                ▼
        ┌───────────────────────┴───────────────────────┐
        │                                               │
        ▼  (a) COPY-INSTALLED                           ▼  (b) FLAGSHIP / IN-PLACE
┌────────────────────────────────┐          ┌────────────────────────────────────┐
│ backend/**  → app/plugins/<s>/ │          │ builtin-extensions/<slug>/         │
│ frontend/** → src/plugins/<s>/ │          │   loaded via importlib spec,       │
│                                │          │   NO file copy; seeded every boot  │
└───────────────┬────────────────┘          └───────────────┬────────────────────┘
                │                                           │
                └─────────────────┬─────────────────────────┘
                                  ▼
                    ┌──────────────────────────────┐
                    │  import app.plugins.<slug>   │
                    │  register blueprint          │
                    │  + models / jobs / sockets   │
                    │  + before_request 503 guard  │  ← makes disable take effect
                    └──────────────────────────────┘

Manifest

Every extension ships a plugin.json. Required: name (slug), display_name, version. Notable optional keys:

• entry_point — "module:blueprint_attr", mounted at url_prefix (default /api/v1/<slug>)
• models, lifecycle (install/upgrade/uninstall hooks)
• jobs[] and schedules[] — declaratively wired into the job system
• permissions[], config_schema, templates[]
• contributions{ nav, routes, tabs, command_palette, widgets, layouts, ai }

The canonical spec is served live at GET /api/v1/plugins/manifest-spec.

Two install classes

(a) Flagship / in-place — source stays in builtin-extensions/<slug>/ and is imported directly via an importlib spec injected as app.plugins.<slug>. No copy. Re-seeded on every boot; "uninstall" writes a marker so the seeder skips it.

(b) Copy-installed — the zip is extracted into backend/app/plugins/<slug>/ and frontend/src/plugins/<slug>/. This covers registry installs and non-flagship bundled extensions.

Working on a copy-installed builtin? Edit both the source in builtin-extensions/<slug>/ and the live copy under app/plugins/<slug>/, or the next reinstall reverts you. Sync frontends with sync-builtin-frontends.mjs.

In-repo extensions

Source lives in this repo under builtin-extensions/:

Slug	Class	Notes
serverkit-cloudflare-ops	Flagship, in-place	Route-only, no sidebar entry
serverkit-ftp	Copy-installed builtin	Auto-installs on upgrade
serverkit-cloud-provision	Copy-installed builtin	Auto-installs on upgrade
serverkit-remote-access	Copy-installed builtin	Auto-installs on upgrade
serverkit-status	Copy-installed builtin	Auto-installs on upgrade
serverkit-email	Copy-installed builtin	Gated — auto-installs only if mail rows exist
serverkit-git	Marketplace one-click	Frontend-only, no backend
serverkit-localkit	Marketplace one-click	Backend-only, no frontend

serverkit-wordpress left the tree in plan 52 Phase 5: it is a standalone repo distributed through the registry (like serverkit-k8s/serverkit-tramo), offered during onboarding from the bundled registry index. Its database schema stays core forever — the core data seam (plan 52 D1, see EXTENSIONS.md "Data models & policy").

The security suite left core the same way (plan 47 Ph3b-4 / plan 55 Phase 3): serverkit-clamav (ClamAV+YARA scanning, quarantine), serverkit-fail2ban (management surface — status probing + the per-site jail write-half stay core on Fail2banJailService), serverkit-lynis, serverkit-auto-updates, and the headless serverkit-image-scan (CVE/SBOM; image_vulnerability_scans/sbom_artifacts stay core tables). Each mounts its routes back at the /api/v1/security prefix and contributes tabs into the Security tab group (groupId="security"), so an installed extension is indistinguishable from the pre-split page. The core Security page keeps only the zero-host-package baseline: overview, firewall, SSH keys, IP lists, integrity, audit, events, settings. Upgrade parity: extension_migration.run_registry_auto_install() installs each from the registry once when the matching host tool is present.

Standalone extension repos

These have their own repos and install from the registry as versioned, sha256-pinned zips (bundled: false):

Extension	Repo
Analytics	jhd3197/serverkit-analytics
Automations (tramo)	jhd3197/serverkit-tramo
CrowdSec	jhd3197/serverkit-crowdsec
DNS Server	jhd3197/serverkit-dns-server
Faro	jhd3197/serverkit-faro
GPU Monitor	jhd3197/serverkit-gpu
Agent GUI (beta)	jhd3197/serverkit-gui
Kubernetes	jhd3197/serverkit-k8s
Mail Server	jhd3197/serverkit-mail

Supporting repos:

Repo	Role
jhd3197/serverkit-extensions	The curated registry index.json
jhd3197/serverkit-agent	The Go fleet agent
jhd3197/Tramo	The automation engine the Automations extension embeds

Retired. The old drag-and-drop Workflow Builder (serverkit-workflows) no longer exists. It was replaced by the Automations extension, which runs workflows in a managed tramo container rather than parsing a graph in-panel; the panel proxies runs, approvals, and inbound webhooks. /workflow redirects to /automations, and the retired slug is swept on upgrade.

Permissions

Declared in plugin.json, enforced by plugins_sdk.permissions.require(slug, cap). Known capabilities: docker, filesystem, shell, network, db, plus namespaced agent.command:<action>. Unknown strings are surfaced in the consent UI.

This is a consent gate, not a sandbox. An extension that imports a host module directly bypasses it. The security posture is: curated registry + sha256 pinning + install-time consent + auditable source — not isolation.

SDK

Backend (app.plugins_sdk): db, JWT helpers, current_user(), logger, audit, config(slug), plus ai, permissions, sockets, queue, notify, and jobs façades.

Frontend (frontend/src/plugins/sdk): versioned (SDK_VERSION), pinned by each extension's sdk_version. Exports api, design-system primitives (KpiBand, MetricCard, DataTable, ResourceList, Drawer, PageTopbar, …), hooks (useToast, useAuth, useTheme, useServerkitAI), and router helpers.

AI is core, not an extension. The assistant lives in app/services/ai_service.py

• app/api/ai.py + contexts/AIContext.jsx. Extensions extend it by registering tools and context through plugins_sdk.ai and the manifest's contributions.ai.

---

Template System

ASCII diagram

┌──────────────────────────────┐        ┌──────────────────────────────┐
│  SHIPPED CATALOG (in repo)   │        │  OPERATOR TEMPLATES          │
│  backend/templates/*.yaml    │        │  /etc/serverkit/templates/   │
│  100+ templates              │        │  installer-populated + local │
└──────────────┬───────────────┘        └──────────────┬───────────────┘
               │                                       │
               └──────────────────┬────────────────────┘
                                  ▼
                     ┌────────────────────────┐
                     │    TemplateService     │
                     │  merges both dirs      │
                     └───────────┬────────────┘
                                 │
                 ┌───────────────┴───────────────┐
                 ▼                               ▼
        ┌─────────────────┐            ┌──────────────────┐
        │  kind: compose  │            │   kind: repo     │
        │  docker-compose │            │  build from git  │
        └────────┬────────┘            └────────┬─────────┘
                 └──────────────┬───────────────┘
                                ▼
                   ┌─────────────────────────┐
                   │  allocate port          │
                   │  render compose         │
                   │  create app + nginx     │
                   │  vhost + optional SSL   │
                   └─────────────────────────┘

TemplateService reads from two directories and merges them: the shipped catalog in backend/templates/ (the source of truth, 100+ YAML files) and the operator-writable /etc/serverkit/templates/ that the installer populates.

Two template kinds: kind: compose (the vast majority — a rendered docker-compose stack) and kind: repo (build from a git repository). Schema reference: TEMPLATE_CATALOG_SCHEMA.md.

---

Port Allocation

TemplateService._find_available_port() resolves the starting port in priority order:

1. The global managed_app_base_port setting, when set to a non-zero value
2. The template's own default port
3. 8000 as the fallback

From there it scans upward, skipping any port that is already assigned in the panel database, already published by a Docker container, or fails a live bind() test on 127.0.0.1. Ports below 1024 are always skipped. If no port is found within the attempt budget, it falls back to a random high port.

---

Service Linking & Env Injection

There is no fixed set of injected variable names. Database env var names are chosen by the template or manifest author, because they have to match what the image actually expects — wordpress.yaml emits WORDPRESS_DB_HOST / WORDPRESS_DB_USER, a Postgres stack emits POSTGRES_*, and so on.

What ServerKit provides is reference resolution at injection time (env_reference_service.py). A manifest declares a reference and ServerKit resolves it when the container starts:

• fromSecret — pull the value out of a vault entry
• fromService — pull a field off a linked service: connectionString, host, port, database, username, password

environment:
  MY_APP_DB_URL:
    fromService: { name: app-postgres, field: connectionString }
  MY_APP_API_KEY:
    fromSecret: { vault: prod, key: stripe_api_key }

So the value is managed by ServerKit; the name is yours. See SERVERKIT_YAML.md.

---

Jobs & Scheduling

ASCII diagram

  producer                                                     handler registry
     │                                                         ┌───────────────┐
     │ JobService.enqueue(kind, payload)                       │ kind → fn(job)│
     ▼                                                         └───────┬───────┘
┌──────────┐   persist    ┌──────────────┐   publish {job_id}          │
│ Job row  │◀────────────▶│  QUEUE BUS   │────────────────────┐        │
│ pending  │   mirror     │ serverkit-   │                    │        │
│ running  │   outcome    │ system/jobs  │  retry / backoff   │        │
│ succeeded│              └──────────────┘  dead-letter       ▼        ▼
│ failed   │                                          ┌──────────────────────┐
└──────────┘                                          │    JobConsumer       │
                                                      │  (daemon thread)     │
     ▲                                                └──────────────────────┘
     │ enqueue due work
┌──────────────────┐
│  JobScheduler    │  15s tick — reads ScheduledJob rows (cron / interval)
│  (daemon thread) │  cadence lives in the DB, not in code
└──────────────────┘

All background work funnels through one abstraction. A producer calls JobService.enqueue(kind, payload), which persists a Job row (pending → running → succeeded | failed | cancelled, with attempts, priority, owner_type/owner_id, correlation_id) and publishes a thin {'job_id': ...} message onto the Queue Bus. Retry, backoff, and dead-lettering are inherited from the bus; the Job row mirrors the outcome so there is a single place to observe every background operation.

A single JobConsumer daemon thread polls and dispatches by kind through an in-process handler registry. A single JobScheduler (15s tick) replaced the old per-domain while True: sleep threads — it enqueues a Job for every due ScheduledJob row, so adding or pausing periodic work is a database change, not a code change.

Extensions plug in via jobs.register(kind, handler) / jobs.enqueue() / jobs.schedule(), or declaratively through the manifest's jobs[] and schedules[] blocks — which are paused and resumed alongside the extension itself. The whole system is a no-op under the testing config so the suite can drive handlers directly.

Don't force real-time streams (live metrics, log tailing, terminal) into jobs — those belong on Socket.IO.

---

Deploy Console & run logs

Every install and deploy is a DeploymentJob run by the single JobConsumer, and its live output is surfaced by the full-page Deploy Console at /deployments/<jobId>. All deploy-path logging funnels through one seam, app/services/run_log_service.py (RunLogStream):

• Batched persistence — one DB commit + one socket emit per flush (50 lines, 300 ms, a step change, or close) instead of a commit per line; a hard 5000-row cap per job. Every line is a DeploymentJobLog row.
• Truthful failure tail — an 80-line in-memory ring buffer persisted to the job's result JSON on failure, alongside step_timings and a matched plain-language hint (no schema migration).
• Clean text — ANSI escapes and \r progress overwrites are stripped at the seam; builds run with --ansi never --progress plain (DockerService.compose_up_streaming).
• Crash-proof — RunLogStream.log() never raises; a flush failure degrades to a buffered-drop + one telemetry event. Job failure marking never depends on the log path.

The read side stays the source of truth: GET /deployment-jobs/<id>/logs?after_id= for incremental polling, plus the run envelope's run_log / run_status Socket.IO events (room run_deploy_<job_id>) as an accelerator — the legacy deploy_log / deploy_status pair (room deploy_{job_id}) is still dual-emitted for compatibility until the Deploy Console migrates. The generic frontend hook is useServerStream; the deploy-specific useDeployJobStream hook de-dupes by row id and re-syncs with after_id on reconnect, with a 2-second poll fallback — so the console works with sockets disabled. Emits are in-process only, consistent with the single-worker gateway constraint below. See DEPLOY_CONSOLE.md.

---

Notifications Bus

ASCII diagram

  notify.send('backup.completed', to='admins', data={...})   ← non-blocking
     │
     ▼
┌──────────────────────────────────────────┐
│  Notification  +  NotificationDelivery   │   one delivery per (recipient × channel)
└──────────────────┬───────────────────────┘
                   │ publish
                   ▼
        ┌──────────────────────────┐
        │        QUEUE BUS         │  serverkit-system/notifications
        │  retry / backoff / DLQ   │
        └────────────┬─────────────┘
                     ▼
        ┌──────────────────────────┐        ┌────────────────────────┐
        │  NotificationConsumer    │───────▶│   CATALOG              │
        │  (daemon thread)         │        │ event_key → title,     │
        │  render → dispatch       │        │ template, severity,    │
        └────────────┬─────────────┘        │ preference category    │
                     │                      └────────────────────────┘
     ┌───────────────┼───────────────┬───────────────┬──────────────┐
     ▼               ▼               ▼               ▼              ▼
 ┌────────┐    ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐
 │ inapp  │    │  email   │   │ discord  │   │ telegram │   │ webhook  │  + slack
 └────────┘    └──────────┘   └──────────┘   └──────────┘   └──────────┘

  digestable / quiet-hours deliveries ──▶ status: queued_digest (never enqueued)
                                              │
                                              ▼  hourly ScheduledJob
                                    one grouped, branded email → sent

notify.send(event, to, data) is non-blocking: it writes a durable Notification plus one NotificationDelivery per (recipient × channel), enqueues them, and returns. A background consumer renders each delivery and hands it to the matching channel adapter.

Channels are a pluggable key→adapter registry (inapp, email, discord, slack, telegram, webhook); register_adapter() lets an extension add SMS or web-push and have it delivered exactly like a core channel. The catalog maps each event_key to its presentation defaults — title, template, severity, and preference category (system | security | backups | apps) — and catalog.register() lets a plugin event render through the identical pipeline.

Digests are the per-user cadence layer. Digestable deliveries and quiet-hours catch-ups are parked at queued_digest and never enqueued; an hourly scheduled job groups a user's held rows into one branded email. Cadence is off | daily | weekly per user. Contract: NOTIFICATIONS_CONTRACT.md.

---

Agent Fleet

ASCII diagram

┌──────────────────────────────────────────────────────────────────┐
│                        SERVERKIT PANEL                           │
│                                                                  │
│   app/agent_gateway.py                app/api/agent_poll.py      │
│   Socket.IO /agent namespace          HTTP long-poll fallback    │
│              │                                  │                │
│              └──────────────┬───────────────────┘                │
│                             ▼                                    │
│              ┌──────────────────────────────┐                    │
│              │   agent_registry (in-memory) │  ⚠ SINGLE PROCESS  │
│              │   live agents, socket↔server │                    │
│              │   index, session tokens,     │                    │
│              │   in-flight command queues   │                    │
│              └──────────────────────────────┘                    │
└─────────────────────────────┬────────────────────────────────────┘
                              │ HMAC auth, heartbeat + metrics, command routing
          ┌───────────────────┼───────────────────┐
          ▼                   ▼                   ▼
   ┌────────────┐      ┌────────────┐      ┌────────────┐
   │  server A  │      │  server B  │      │  server C  │
   │ Go agent   │      │ Go agent   │      │ Go agent   │
   └────────────┘      └────────────┘      └────────────┘

The panel manages remote servers through a fleet of Go agents. The agent binary is maintained in the separate serverkit-agent repo — agent-side capability changes ship there, so panel↔agent protocol changes are not atomic in one commit. Coordinate with FLEET_CONTRACT.md.

Agents connect over the Socket.IO /agent namespace (HMAC-authenticated, heartbeat + metrics + command routing), with an HTTP long-poll fallback for networks where WebSocket is blocked.

⚠️ Deployment constraint — single WebSocket worker

The agent gateway keeps all connected-agent state — the live-agent registry, the socket↔server index, session tokens, and in-flight command queues — in memory in one process.

Run the panel with a single gunicorn worker using a plain threaded worker: -w 1 --threads N. WebSocket is served by simple-websocket to match async_mode='threading'; the gevent-websocket worker class double-answers the WS handshake and breaks WebSocket.

Scaling to multiple workers without a shared backplane (e.g. a Redis message queue) will silently misroute or drop commands for agents connected to a different worker. See HORIZONTAL_SCALING_SPEC.md and SECURITY.md.

---

Environment Pipeline

Dev/staging/production workflows for managed WordPress sites.

ASCII diagram

┌──────────────┐ promotion  ┌──────────────┐ promotion  ┌──────────────┐
│     DEV      │───────────▶│   STAGING    │───────────▶│  PRODUCTION  │
│ (Standalone) │            │ (Standalone) │            │ (Production) │
└──────┬───────┘            └──────┬───────┘            └──────┬───────┘
       │                           │                           │
       └─────────── sync ──────────┴─────────── sync ──────────┘

• Promotion — push code (git) and/or database from a lower environment to a higher one: promote_code, promote_database, promote_full.
• Syncing — sync_from_production pulls the latest production database and media down to dev/staging for testing.
• Sanitization — sensitive user data is stripped automatically during sync.

The engine is environment_pipeline_service.py in core, driven by a job handler; the HTTP routes live in the serverkit-wordpress extension. Details: MULTI_ENVIRONMENT.md.

---

File Paths

Every path below is defined in backend/app/paths.py and each is overridable by the matching environment variable — the values shown are the production defaults.

/var/serverkit/                     # SERVERKIT_DIR — data root
├── apps/                           # deployed applications
└── deployments/                    # deployment working dirs

/etc/serverkit/                     # SERVERKIT_CONFIG_DIR
├── templates/                      # operator template library (YAML)
├── email/                          # mail server config
├── install-state.json              # written by the installer
└── ssl-mode                        # nginx SSL mode flag

/var/backups/serverkit/             # SERVERKIT_BACKUP_DIR
├── databases/
├── wordpress/
└── snapshots/

/var/log/serverkit/                 # SERVERKIT_LOG_DIR
└── builds/                         # build logs

/var/cache/serverkit/               # SERVERKIT_CACHE_DIR
├── builds/
└── wp-plugins/

/var/quarantine/                    # SERVERKIT_QUARANTINE_DIR
/var/vmail/                         # VMAIL_DIR — mail storage

<install-dir>/nginx/ssl/            # nginx TLS material

Notes:

• The panel has no config.yaml. Panel configuration is backend/config.py plus environment variables / .env. The only config.yaml in the ecosystem belongs to the agent, at /etc/serverkit-agent/config.yaml.
• Backups live under /var/backups/serverkit, not under /var/serverkit/.
• Always import from app.paths rather than hardcoding — the env overrides are what let the test suite and local dev run off-root.

---

Troubleshooting

See the Deployment Guide for 502 errors, container failures, and networking issues.

---

See Also

• Installation Guide
• Local Development
• API Reference
• Deployment Guide
• Extensions · Registry
• Fleet Contract
• serverkit.yaml Reference