feat(agents): permission levels, signed report endpoint, health alerts

Agent-reported monitor/protect/full level (never settable from the panel),
POST /api/v1/agent/report for report-only agents, silence-based offline
detection, service-check alerts via MetricAlert, a panel-side command gate,
docs, and a demo seed script. Migration 112.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Juan DenisandClaude Sonnet 5.5 committed 2026-11-03 00:04:58 -05:00
1 parent 2ef0a8f370
commit a4d2ea37ef
26 files changed
+861 -12

No files matched your search

+9
View File
@@ -22,6 +22,15 @@ release history; historical `agent-v*` tags are not panel releases.
### Added
- Agent permission levels (Monitor / Protect / Full), enforced by the agent and
reported to the panel. Adds a signed `POST /api/v1/agent/report` endpoint for
report-only Monitor agents, per-server level and service-check state (server
list badge and health, server detail panel), offline / service-down alerts
through the existing fleet alerts and notification channels, audited level
changes, and a panel-side command gate. The panel cannot set the level; it is
changed on the server with `serverkit-agent permissions set`. See
[docs/AGENT_PERMISSION_LEVELS.md](docs/AGENT_PERMISSION_LEVELS.md). Migration
112. The Add Server commands now pass `--level` / `-Level`.
- `GET /api/v1/servers` now accepts an API key (`X-API-Key`) carrying the
`servers:read` scope, in addition to the usual JWT, so external tools can
poll the server list read-only without a short-lived token. JWT callers are
+6
View File
@@ -47,6 +47,12 @@ trust boundaries:
(`system:exec`) and interactive PTY sessions are controlled by the agent's
`Features.Exec` flag, which is **off by default**. Enable it only on servers
where you intend the panel to run shell commands.
- **Permission levels bound what a compromised panel can do.** An agent runs as
Monitor (report-only, no inbound command channel), Protect (security allowlist
only) or Full. The level is read only from the agent's local config and can
only be changed on the server (`serverkit-agent permissions set`); the panel
records and audits the reported level but has no API to set it. See
[docs/AGENT_PERMISSION_LEVELS.md](docs/AGENT_PERMISSION_LEVELS.md).
- **Transport & connection controls.** Agents authenticate to the panel with
per-connection HMAC-SHA256 (with nonce/replay protection and a timestamp-skew
check), and the panel enforces a per-server IP allowlist. Use `wss://`
+4 -1
View File
@@ -136,7 +136,10 @@ class AgentNamespace(Namespace):
return
metrics = data.get('metrics', {})
agent_registry.ingest_agent_state(agent, metrics=metrics)
agent_registry.ingest_agent_state(
agent, metrics=metrics,
permission_level=data.get('permission_level'),
service_checks=data.get('service_checks'))
emit('heartbeat_ack', {'type': 'heartbeat_ack'})
+3 -1
View File
@@ -108,7 +108,9 @@ def poll():
# Shared transport-neutral ingest: capability
# side-effects (e.g. tunnel reconcile) fire exactly as on the WS path.
agent_registry.ingest_agent_state(agent, metrics=metrics, sysinfo=sysinfo,
capabilities=caps)
capabilities=caps,
permission_level=body.get('permission_level'),
service_checks=body.get('service_checks'))
# Long-poll up to 25s for any queued command. Below the typical
# tunnel idle-timeout (Cloudflare ~100s, ngrok ~60s) so we never
+53
View File
@@ -0,0 +1,53 @@
"""Monitor-mode agent report endpoint.
``POST /api/v1/agent/report`` is the ONLY channel a monitor-level agent uses:
a signed, outbound-only HTTPS POST. The reply never carries commands.
Auth (no JWT/CSRF; the HMAC is the credential):
X-Agent-Key: <api key> -> server looked up by its key prefix
X-Timestamp: <unix seconds> -> |skew| <= 300s
X-Nonce: <hex, >=32 chars> -> single use (TTL cache, 5 min)
X-Signature: hex(HMAC_SHA256(api_secret, ts + "." + nonce + "." + sha256hex(body)))
"""
from flask import Blueprint, jsonify, request
from app import limiter
from app.services import agent_health_service, permission_levels
from app.utils.ip_utils import is_ip_allowed
agent_report_bp = Blueprint('agent_report', __name__)
MAX_BODY_BYTES = 64 * 1024
MAX_SKEW_SECONDS = 300
def _rate_key():
return f"{request.remote_addr}:{(request.headers.get('X-Agent-Key') or '')[:12]}"
@agent_report_bp.route('/report', methods=['POST'])
@limiter.limit("30 per minute", key_func=_rate_key)
def report():
if (request.content_length or 0) > MAX_BODY_BYTES:
return jsonify({'error': 'payload too large'}), 413
raw = request.get_data(cache=True)
if len(raw) > MAX_BODY_BYTES:
return jsonify({'error': 'payload too large'}), 413
server = agent_health_service.authenticate_report(raw, request.headers, MAX_SKEW_SECONDS)
if not server:
return jsonify({'error': 'authentication failed'}), 401
if server.allowed_ips and not is_ip_allowed(request.remote_addr, server.allowed_ips):
return jsonify({'error': 'IP not allowed'}), 403
body = request.get_json(silent=True)
if not isinstance(body, dict):
return jsonify({'error': 'JSON object required'}), 400
level = body.get('permission_level')
if level not in permission_levels.LEVELS:
return jsonify({'error': 'permission_level must be monitor, protect or full'}), 400
interval = agent_health_service.ingest_report(server, level, body)
return jsonify({'ok': True, 'interval_seconds': interval})
+8
View File
@@ -13,6 +13,7 @@ from flask_jwt_extended import jwt_required, get_jwt_identity
from sqlalchemy.orm import joinedload
from app import db, limiter
from app.services import permission_levels
from app.services.server_metrics_service import latest_metrics_by_server
from app.api._query import apply_query, QueryParseError
from app.api._request import json_object
@@ -440,6 +441,8 @@ def create_server():
registered_by=user_id,
workspace_id=ws_id,
registration_token_expires=expires_at,
# What the install command asks for; the agent's report is the truth.
requested_permission_level=permission_levels.normalize(data.get('requested_level')),
)
server.set_registration_token(registration_token)
@@ -718,6 +721,7 @@ def regenerate_token(server_id):
registration_token = Server.generate_registration_token()
server.set_registration_token(registration_token)
server.registration_token_expires = expires_at
server.requested_permission_level = permission_levels.normalize(data.get('requested_level'))
server.status = 'pending'
server.agent_id = None
@@ -792,6 +796,10 @@ def register_agent():
server.total_disk = system_info.get('total_disk', server.total_disk)
server.agent_version = data.get('agent_version')
# Level is whatever the agent says it was installed with; the panel never
# sets it. A mismatch with the requested level is surfaced in the UI.
from app.services import agent_health_service
agent_health_service.record_level(server, data.get('permission_level'))
# Replace the panel-issued placeholder name with whatever the agent
# reports — preferring the OS hostname over the agent's optional
+8 -4
View File
@@ -5,6 +5,7 @@ import os
import re
import requests
from app.models.server import AgentVersion, Server
from app.services import permission_levels
from flask import Response, current_app, jsonify, redirect, request
from app.middleware.rbac import admin_required, developer_required
from app.exceptions import agent_offline, field_required
@@ -178,8 +179,11 @@ def get_install_instructions(server_id):
# Get base URL
base_url = _get_external_base_url()
api_url = f"{base_url}/api/v1/servers"
level = (permission_levels.normalize(request.args.get('level'))
or server.requested_permission_level or 'monitor')
return jsonify({
'permission_level': level,
'linux': {
# Download-then-run rather than `curl … | sudo bash`: in a pipeline
# the shell reports bash's exit status, so a failed download exits 0
@@ -187,7 +191,7 @@ def get_install_instructions(server_id):
'one_liner': (
f'curl -fsSL {api_url}/install.sh -o /tmp/serverkit-agent-install.sh '
f'&& sudo bash /tmp/serverkit-agent-install.sh '
f'--token "YOUR_TOKEN" --server "{base_url}"'
f'--token "YOUR_TOKEN" --server "{base_url}" --level {level}'
),
'manual': [
f'# Download the script',
@@ -195,17 +199,17 @@ def get_install_instructions(server_id):
f'chmod +x install.sh',
f'',
f'# Run installation',
f'sudo ./install.sh --token "YOUR_TOKEN" --server "{base_url}"'
f'sudo ./install.sh --token "YOUR_TOKEN" --server "{base_url}" --level {level}'
]
},
'windows': {
'one_liner': f'irm {api_url}/install.ps1 | iex; Install-ServerKitAgent -Token "YOUR_TOKEN" -Server "{base_url}"',
'one_liner': f'irm {api_url}/install.ps1 | iex; Install-ServerKitAgent -Token "YOUR_TOKEN" -Server "{base_url}" -Level {level}',
'manual': [
f'# Download the script (run in PowerShell as Administrator)',
f'Invoke-WebRequest -Uri "{api_url}/install.ps1" -OutFile install.ps1',
f'',
f'# Run installation',
f'.\\install.ps1 -Token "YOUR_TOKEN" -Server "{base_url}"'
f'.\\install.ps1 -Token "YOUR_TOKEN" -Server "{base_url}" -Level {level}'
]
},
'note': 'Replace YOUR_TOKEN with the registration token shown in the UI'
+1
View File
@@ -46,6 +46,7 @@ class BlueprintSpec:
CORE_BLUEPRINTS: tuple[BlueprintSpec, ...] = (
BlueprintSpec('app.api.auth', 'auth_bp', '/api/v1/auth'),
BlueprintSpec('app.api.agent_poll', 'agent_poll_bp', '/api/v1/agent'),
BlueprintSpec('app.api.agent_report', 'agent_report_bp', '/api/v1/agent'),
BlueprintSpec('app.api.linked_panel', 'linked_panel_bp', '/api/v1/linked-panel'),
BlueprintSpec('app.api.apps', 'apps_bp', '/api/v1/apps'),
BlueprintSpec('app.api.apps', 'apps_bp', '/api/v1/services', name='services'),
+4 -2
View File
@@ -534,7 +534,7 @@ def _notify_fleet_alerts(opened, resolved):
'metric': alert.metric,
'value': alert.value,
'threshold': alert.threshold,
'message': (
'message': alert.detail or (
f'{hostname}: {alert.metric} averaged {alert.value} over '
f'{alert.duration_seconds}s, past the '
f'{alert.severity} threshold of {alert.threshold}.'),
@@ -555,7 +555,9 @@ def _notify_fleet_alerts(opened, resolved):
'hostname': hostname,
'alert_type': f'{alert.metric} recovered',
'metric': alert.metric,
'message': (f'{hostname}: {alert.metric} is back under its '
'message': (f'{hostname}: {alert.metric} recovered.'
if alert.detail else
f'{hostname}: {alert.metric} is back under its '
f'threshold.'),
},
)
+3
View File
@@ -49,6 +49,8 @@ class MetricAlert(db.Model):
threshold = db.Column(db.Float) # the threshold exceeded
duration_seconds = db.Column(db.Integer) # how long it was exceeded
detail = db.Column(db.Text) # human message for offline / service-check alerts
status = db.Column(db.String(20), default='active', index=True) # active, acknowledged, resolved
acknowledged_by = db.Column(db.Integer, db.ForeignKey('users.id'), nullable=True, index=True)
@@ -67,6 +69,7 @@ class MetricAlert(db.Model):
'value': self.value,
'threshold': self.threshold,
'duration_seconds': self.duration_seconds,
'detail': self.detail,
'status': self.status,
'acknowledged_by': self.acknowledged_by,
'created_at': self.created_at.isoformat() if self.created_at else None,
+17
View File
@@ -126,6 +126,17 @@ class Server(TimestampMixin, JsonColumnMixin, db.Model):
allow_agent_update_observed = db.Column(db.Boolean, nullable=False,
default=False, server_default='0')
# Agent permission level (monitor|protect|full). REPORTED BY THE AGENT
# only -- no panel API writes it. NULL = unknown/legacy, treated as full.
# requested_permission_level is what the install command asked for, kept
# only to flag a mismatch with what the agent reports.
permission_level = db.Column(db.String(16), nullable=True)
permission_level_updated_at = db.Column(db.DateTime, nullable=True)
requested_permission_level = db.Column(db.String(16), nullable=True)
last_report_at = db.Column(db.DateTime, nullable=True)
report_interval_seconds = db.Column(db.Integer, nullable=True)
service_checks = db.Column(db.JSON, nullable=True) # latest state per check
@property
def is_managed(self):
"""False only for observed boxes."""
@@ -490,6 +501,12 @@ class Server(TimestampMixin, JsonColumnMixin, db.Model):
'last_error': self.last_error,
'management_mode': self.management_mode or 'managed',
'allow_agent_update_observed': bool(self.allow_agent_update_observed),
'permission_level': self.permission_level,
'permission_level_effective': self.permission_level or 'full',
'permission_level_updated_at': self.permission_level_updated_at.isoformat() if self.permission_level_updated_at else None,
'requested_permission_level': self.requested_permission_level,
'last_report_at': self.last_report_at.isoformat() if self.last_report_at else None,
'service_checks': self.service_checks or [],
'onboarding_state': self.onboarding_state,
'onboarding_updated_at': self.onboarding_updated_at.isoformat() if self.onboarding_updated_at else None,
'agent_version': self.agent_version,
@@ -0,0 +1,223 @@
"""Agent-reported health: permission level, service checks, silence, alerts.
Everything here consumes what the agent REPORTS (registration, auth, heartbeat,
or the monitor-mode ``/agent/report`` POST). Alerts reuse ``MetricAlert`` and the
fleet notification path; one active row per (server, metric) de-duplicates.
"""
import hashlib
import hmac
import logging
import time
from datetime import datetime, timedelta
from app import db
from app.models.audit_log import AuditLog
from app.models.metric_alert import MetricAlert
from app.models.server import Server
from app.services import permission_levels
from app.services.nonce_service import nonce_service
logger = logging.getLogger(__name__)
MIN_SILENCE_SECONDS = 180 # monitor agents: offline after max(3 min, 3x interval)
DEFAULT_INTERVAL_SECONDS = 60
DEFAULT_INTERVAL = DEFAULT_INTERVAL_SECONDS
OFFLINE_METRIC = 'offline'
OPEN = ('active', 'acknowledged')
BAD = ('down',)
def _notify(opened=(), resolved=()):
if not opened and not resolved:
return
try:
from app.jobs.builtin_handlers import _notify_fleet_alerts
_notify_fleet_alerts(list(opened), list(resolved))
except Exception:
logger.exception('Could not notify agent health alerts')
def _open_alert(server, metric, detail):
"""Open an alert unless one is already open. Returns it if newly opened."""
if MetricAlert.query.filter(MetricAlert.server_id == server.id,
MetricAlert.metric == metric,
MetricAlert.status.in_(OPEN)).first():
return None
alert = MetricAlert(server_id=server.id, metric=metric, severity='critical',
detail=detail, status='active')
db.session.add(alert)
return alert
def _resolve_alerts(server, metric):
rows = MetricAlert.query.filter(MetricAlert.server_id == server.id,
MetricAlert.metric == metric,
MetricAlert.status.in_(OPEN)).all()
for row in rows:
row.status = 'resolved'
row.resolved_at = datetime.utcnow()
return rows
def record_level(server, reported):
"""Persist a level reported by the agent; audit when it changes.
Invalid/missing values are ignored. Escalations (monitor->protect->full) are
logged like any change but flagged, since they mean someone re-ran the
local confirmation on the box.
"""
level = permission_levels.normalize(reported)
if level is None:
return
old = server.permission_level
if old == level:
return
server.permission_level = level
server.permission_level_updated_at = datetime.utcnow()
AuditLog.log(
'server.permission_level_changed', target_type='server', target_id=server.id,
details={'server_id': server.id, 'from': old, 'to': level,
'escalation': permission_levels.is_escalation(old, level),
'source': 'agent_report'})
logger.warning('Server %s reported permission level %s -> %s', server.id, old, level)
def record_service_checks(server, checks):
"""Store the latest per-check state and open/resolve alerts on transitions."""
if not isinstance(checks, list):
return
previous = {c.get('name'): c.get('status') for c in (server.service_checks or [])}
clean, opened, resolved = [], [], []
for raw in checks[:50]:
if not isinstance(raw, dict) or not isinstance(raw.get('name'), str):
continue
item = {k: (str(raw.get(k))[:200] if raw.get(k) is not None else None)
for k in ('name', 'kind', 'status', 'detail', 'checked_at')}
item['name'] = item['name'][:60]
if item['status'] not in ('up', 'down', 'degraded', 'unknown'):
item['status'] = 'unknown'
clean.append(item)
metric = f"svc:{item['name']}"[:20]
if item['status'] == 'down':
alert = _open_alert(
server, metric,
f"{server.name}: service '{item['name']}' is down since "
f"{datetime.utcnow():%Y-%m-%d %H:%M} UTC"
+ (f" ({item['detail']})" if item['detail'] else ''))
if alert:
opened.append(alert)
elif item['status'] == 'up' or (item['status'] == 'degraded'
and previous.get(item['name']) == 'down'):
resolved += _resolve_alerts(server, metric)
server.service_checks = clean
db.session.flush()
_notify(opened, resolved)
def mark_offline(server, reason):
"""Open the (de-duplicated) offline alert. Caller commits."""
since = server.last_report_at or server.last_seen or datetime.utcnow()
alert = _open_alert(
server, OFFLINE_METRIC,
f"{server.name} is offline since {since:%Y-%m-%d %H:%M} UTC: {reason}")
db.session.flush()
_notify([alert] if alert else [])
def mark_online(server):
resolved = _resolve_alerts(server, OFFLINE_METRIC)
db.session.flush()
_notify(resolved=resolved)
def silence_limit(server):
interval = server.report_interval_seconds or DEFAULT_INTERVAL_SECONDS
return timedelta(seconds=max(MIN_SILENCE_SECONDS, 3 * interval))
def sweep_silent_reporters(now=None):
"""Flip report-only (monitor) servers offline once their reports stop.
Reports are not a socket, so the registry's heartbeat reaper never sees
them. Only servers that have ever reported are considered.
"""
now = now or datetime.utcnow()
flipped = []
for server in Server.query.filter(Server.permission_level == 'monitor',
Server.last_report_at.isnot(None),
Server.status != 'offline').all():
if now - server.last_report_at > silence_limit(server):
server.status = 'offline'
server.last_error = 'Agent stopped reporting'
mark_offline(server, 'no report received')
flipped.append(server.id)
if flipped:
db.session.commit()
return flipped
def authenticate_report(raw, headers, max_skew):
"""Return the Server for a validly signed report request, else None."""
key = headers.get('X-Agent-Key') or ''
ts = headers.get('X-Timestamp') or ''
nonce = headers.get('X-Nonce') or ''
signature = headers.get('X-Signature') or ''
if not (key and ts and signature) or len(nonce) < 32:
return None
try:
if abs(time.time() - int(ts)) > max_skew:
return None
except ValueError:
return None
prefix = key[:12]
server = Server.query.filter(
(Server.api_key_prefix == prefix) | (Server.api_key_pending_prefix == prefix)
).first()
if not server:
return None
pending = (server.api_key_pending_prefix == prefix and server.api_key_prefix != prefix
and server.api_key_rotation_expires
and datetime.utcnow() <= server.api_key_rotation_expires)
secret = server.get_pending_api_secret() if pending else server.get_api_secret()
if not secret:
return None
message = f"{ts}.{nonce}.{hashlib.sha256(raw).hexdigest()}"
expected = hmac.new(secret.encode(), message.encode(), hashlib.sha256).hexdigest()
if not hmac.compare_digest(signature, expected):
return None
# Consume the nonce only after the signature proves valid.
if not nonce_service.check_and_record(server.id, nonce):
return None
return server
def ingest_report(server, level, body):
"""Apply a verified report to the server row; returns the interval in use."""
interval = body.get('interval_seconds')
interval = interval if isinstance(interval, int) and 5 <= interval <= 3600 else DEFAULT_INTERVAL
now = datetime.utcnow()
was_offline = server.status != 'online'
server.status = 'online'
server.last_error = None
server.last_seen = now
server.last_report_at = now
server.report_interval_seconds = interval
if isinstance(body.get('agent_version'), str):
server.agent_version = body['agent_version'][:20]
record_level(server, level)
record_service_checks(server, body.get('service_checks'))
if was_offline:
mark_online(server)
db.session.commit()
metrics = body.get('metrics')
if isinstance(metrics, dict):
from app.services.server_metrics_service import ServerMetricsService
ServerMetricsService.record_metrics(server.id, {
**{k: metrics.get(k) for k in ('cpu_percent', 'memory_percent', 'disk_percent')},
'extra': {k: metrics.get(k) for k in ('uptime_seconds', 'load1')},
})
return interval
+36 -1
View File
@@ -22,6 +22,7 @@ from collections import defaultdict
from app import db
from app.models.server import Server, ServerCommand, AgentSession
from app.services import permission_levels
# ---------------------------------------------------------------------------
# Per-IP auth rate limiter — transport-neutral state shared by the WS
@@ -245,6 +246,12 @@ class AgentRegistry:
self._agents.pop(server_id, None)
self._socket_to_server.pop(socket_id, None)
# Report-only (monitor) agents hold no socket; sweep them too.
if self._app is not None:
with self._app.app_context():
from app.services import agent_health_service
agent_health_service.sweep_silent_reporters()
except Exception as e:
logger.error("Error in heartbeat checker: %s", e)
@@ -279,6 +286,8 @@ class AgentRegistry:
if server:
server.status = 'offline'
server.last_error = 'Agent heartbeat timeout'
from app.services import agent_health_service
agent_health_service.mark_offline(server, 'heartbeat timeout')
db.session.commit()
# Mark session as inactive
@@ -392,6 +401,8 @@ class AgentRegistry:
# even for healthy connected agents.
if ip_address and ip_address != 'unknown':
server.ip_address = ip_address
from app.services import agent_health_service
agent_health_service.mark_online(server)
db.session.commit()
# Create session record
@@ -519,6 +530,8 @@ class AgentRegistry:
server.last_seen = now
if server.status != 'online':
server.status = 'online'
from app.services import agent_health_service
agent_health_service.mark_online(server)
db.session.commit()
# Update session latency
@@ -589,6 +602,12 @@ class AgentRegistry:
# Check permissions
server = Server.query.get(server_id)
# Defence in depth: never send what the agent's reported level refuses.
refused = permission_levels.refusal(
server.permission_level if server else None, action)
if refused:
return {'success': False, 'error': refused, 'code': 'LEVEL_DENIED'}
if server and not server.has_permission(action):
return {
'success': False,
@@ -863,13 +882,15 @@ class AgentRegistry:
# in-memory state — the adapter must not report success.
if not session_token:
return None, server, 'registration_failed'
self.record_agent_health(server.id, payload.get('permission_level'))
logger.info("Agent %s authenticated successfully from %s (%s)",
agent_id, ip_address, transport)
return session_token, server, None
def ingest_agent_state(self, agent, metrics=None, sysinfo=None,
capabilities=None):
capabilities=None, permission_level=None,
service_checks=None):
"""One ingest path for agent-reported state on BOTH transports.
Owns the capability side-effects (the WireGuard tunnel reconcile
@@ -882,6 +903,20 @@ class AgentRegistry:
self.update_system_info(agent.server_id, sysinfo)
if capabilities:
self.update_capabilities(agent.server_id, capabilities)
if permission_level is not None or service_checks is not None:
self.record_agent_health(agent.server_id, permission_level, service_checks)
def record_agent_health(self, server_id, permission_level=None, service_checks=None):
from app.services import agent_health_service
try:
server = Server.query.get(server_id)
if server:
agent_health_service.record_level(server, permission_level)
agent_health_service.record_service_checks(server, service_checks)
db.session.commit()
except Exception:
logger.exception("Error recording agent health")
db.session.rollback()
def verify_agent_auth(
self,
+56
View File
@@ -0,0 +1,56 @@
"""Agent permission levels and the one command-gate table.
The level is chosen on the server (``permissions.level`` in the agent's
config.yaml) and REPORTED to the panel. The panel never sets it. This module
is only the panel's defence in depth: it refuses to send a command the agent
would refuse anyway. ``PROTECT_ACTIONS`` must stay identical to the agent's
allowlist (docs/AGENT_PERMISSION_LEVELS.md; checked by a test).
"""
LEVELS = ('monitor', 'protect', 'full')
_RANK = {level: i for i, level in enumerate(LEVELS)}
LABELS = {'monitor': 'Monitor', 'protect': 'Protect', 'full': 'Full'}
PROTECT_ACTIONS = frozenset({
'firewall:allow_port', 'firewall:deny_port', 'firewall:status',
'firewall:allow_ip', 'firewall:deny_ip',
'fail2ban:status', 'fail2ban:ban', 'fail2ban:unban',
'crowdsec:decisions:list', 'crowdsec:ban', 'crowdsec:unban',
'cloudflare:origin_lock:status', 'cloudflare:origin_lock:apply',
'cloudflare:origin_lock:remove',
})
# Signature-verified key rotation is not control; allowed at every level.
ALWAYS_ALLOWED = frozenset({'credential_update'})
def normalize(level):
"""Return a valid level string, or None for missing/unknown values."""
level = level.strip().lower() if isinstance(level, str) else None
return level if level in _RANK else None
def effective_level(level):
"""Unknown/legacy (None) means a pre-levels agent, i.e. full."""
return normalize(level) or 'full'
def is_escalation(old, new):
return old is not None and _RANK[new] > _RANK[old]
def refusal(level, action):
"""Message explaining why ``action`` is refused at ``level``, else None."""
level = effective_level(level)
if level == 'full' or action in ALWAYS_ALLOWED:
return None
if level == 'protect' and action in PROTECT_ACTIONS:
return None
if level == 'monitor':
return (f"server is in Monitor mode: '{action}' refused. Monitor agents "
"are report-only and accept no commands. The level can only be "
"changed on the server itself (serverkit-agent permissions set).")
return (f"server is in Protect mode: '{action}' refused. Protect agents only "
"run firewall, fail2ban, CrowdSec and Cloudflare origin-lock actions. "
"The level can only be changed on the server itself "
"(serverkit-agent permissions set).")
@@ -52,6 +52,7 @@ _AUDIT_DETAIL_KEYS = frozenset({
'application_id',
'code',
'endpoint',
'escalation',
'from',
'kind',
'label',
@@ -0,0 +1,36 @@
"""Agent-reported permission level, report bookkeeping and service checks."""
import sqlalchemy as sa
from alembic import op
revision = '112_agent_permission_levels'
down_revision = '111_preview_runtime'
branch_labels = None
depends_on = None
SERVER_COLUMNS = [
('permission_level', sa.String(16)),
('permission_level_updated_at', sa.DateTime()),
('requested_permission_level', sa.String(16)),
('last_report_at', sa.DateTime()),
('report_interval_seconds', sa.Integer()),
('service_checks', sa.JSON()),
]
def upgrade():
inspector = sa.inspect(op.get_bind())
existing = {c['name'] for c in inspector.get_columns('servers')}
for name, kind in SERVER_COLUMNS:
if name not in existing:
op.add_column('servers', sa.Column(name, kind, nullable=True))
existing = {c['name'] for c in inspector.get_columns('metric_alerts')}
if 'detail' not in existing:
op.add_column('metric_alerts', sa.Column('detail', sa.Text(), nullable=True))
def downgrade():
with op.batch_alter_table('metric_alerts') as batch:
batch.drop_column('detail')
with op.batch_alter_table('servers') as batch:
for name, _ in reversed(SERVER_COLUMNS):
batch.drop_column(name)
+80
View File
@@ -0,0 +1,80 @@
"""Seed five demo servers showing every permission-level / health state.
Dev sandbox only; never run automatically. Idempotent: demo rows are keyed by
agent_id 'demo-perm-*' and are rebuilt on each run.
cd backend && python scripts/seed_permission_demo.py
"""
import os
import sys
from datetime import datetime, timedelta
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from app import create_app, db # noqa: E402
from app.models.metric_alert import MetricAlert # noqa: E402
from app.models.server import Server, ServerMetrics # noqa: E402
UP, DOWN, DEG = 'up', 'down', 'degraded'
def checks(now, web=UP, php=UP, database=UP, disk=UP):
stamp = now.isoformat() + 'Z'
rows = [('nginx', 'web', web, ''), ('php8.2-fpm', 'php_fpm', php, 'slow pool' if php == DEG else ''),
('mysql', 'database', database, 'connection refused on 3306' if database == DOWN else ''),
('disk /', 'disk', disk, ''), ('cpu', 'cpu', UP, ''), ('memory', 'memory', UP, '')]
return [{'name': n, 'kind': k, 'status': s, 'detail': d, 'checked_at': stamp} for n, k, s, d in rows]
# (name, level, status, minutes since last report, check overrides, cpu)
DEMOS = [
('demo-monitor-healthy', 'monitor', 'online', 1, {}, 14.0),
('demo-monitor-degraded', 'monitor', 'online', 1, {'database': DOWN, 'php': DEG}, 71.0),
('demo-protect-healthy', 'protect', 'online', 0, {}, 22.0),
('demo-full-healthy', 'full', 'online', 0, {}, 35.0),
('demo-monitor-offline', 'monitor', 'offline', 20, {'database': DOWN}, 9.0),
]
def seed():
now = datetime.utcnow()
for name, level, status, age, overrides, cpu in DEMOS:
agent_id = f'demo-perm-{name}'
old = Server.query.filter_by(agent_id=agent_id).first()
if old:
MetricAlert.query.filter_by(server_id=old.id).delete()
ServerMetrics.query.filter_by(server_id=old.id).delete()
db.session.delete(old)
db.session.flush()
reported = now - timedelta(minutes=age)
srv = Server(name=name, agent_id=agent_id, hostname=name, status=status,
os_type='linux', permission_level=level, requested_permission_level=level,
permission_level_updated_at=reported, last_seen=reported,
last_report_at=reported, report_interval_seconds=60,
service_checks=checks(reported, **overrides))
db.session.add(srv)
db.session.flush()
for i in range(12):
db.session.add(ServerMetrics(
server_id=srv.id, timestamp=reported - timedelta(minutes=i),
cpu_percent=cpu + i % 3, memory_percent=48.0, disk_percent=61.0))
for check in srv.service_checks:
if check['status'] == DOWN:
db.session.add(MetricAlert(
server_id=srv.id, metric=f"svc:{check['name']}"[:20], severity='critical',
status='active', created_at=reported,
detail=f"{name}: service '{check['name']}' is down since "
f"{reported:%Y-%m-%d %H:%M} UTC ({check['detail']})"))
if status == 'offline':
db.session.add(MetricAlert(
server_id=srv.id, metric='offline', severity='critical', status='active',
created_at=now, detail=f"{name} is offline since {reported:%Y-%m-%d %H:%M} UTC: "
"no report received"))
db.session.commit()
print(f'Seeded {len(DEMOS)} demo servers.')
if __name__ == '__main__':
app = create_app(os.environ.get('FLASK_ENV') or 'development')
with app.app_context():
seed()
@@ -0,0 +1,208 @@
"""Agent permission levels: report endpoint, level reporting, dispatch gate,
silence sweep, service-check alerts and install commands."""
import hashlib
import hmac
import json
import re
import secrets
import time
from datetime import datetime, timedelta
from pathlib import Path
import pytest
from app import db
from app.models.audit_log import AuditLog
from app.models.metric_alert import MetricAlert
from app.models.server import Server
from app.services import agent_health_service, permission_levels
from app.services.agent_registry import ConnectedAgent, agent_registry
REPORT = '/api/v1/agent/report'
SECRET = 'test-secret'
@pytest.fixture
def server(app, monkeypatch):
monkeypatch.setattr(Server, 'get_api_secret', lambda self: SECRET)
key, _ = Server.generate_api_credentials()
srv = Server(name='web-1', agent_id=secrets.token_hex(8))
srv.set_api_key(key)
db.session.add(srv)
db.session.commit()
srv.test_key = key
return srv
def _post(client, srv, body=None, ts=None, nonce=None, secret=SECRET, raw=None):
body = body if body is not None else {
'permission_level': 'monitor', 'interval_seconds': 60,
'metrics': {'cpu_percent': 12.5, 'memory_percent': 40.0, 'disk_percent': 55.0,
'uptime_seconds': 99, 'load1': 0.3},
'service_checks': []}
raw = raw if raw is not None else json.dumps(body).encode()
ts = str(int(time.time())) if ts is None else str(ts)
nonce = nonce or secrets.token_hex(16)
sig = hmac.new(secret.encode(),
f'{ts}.{nonce}.{hashlib.sha256(raw).hexdigest()}'.encode(),
hashlib.sha256).hexdigest()
return client.post(REPORT, data=raw, content_type='application/json', headers={
'X-Agent-Key': srv.test_key, 'X-Timestamp': ts, 'X-Nonce': nonce,
'X-Signature': sig})
def test_good_signature_updates_server(client, server):
resp = _post(client, server)
assert resp.status_code == 200
assert resp.get_json() == {'ok': True, 'interval_seconds': 60}
srv = db.session.get(Server, server.id)
assert srv.status == 'online' and srv.last_report_at is not None
assert srv.permission_level == 'monitor'
assert 'commands' not in resp.get_json()
def test_bad_signature_rejected(client, server):
assert _post(client, server, secret='wrong').status_code == 401
def test_stale_timestamp_rejected(client, server):
assert _post(client, server, ts=int(time.time()) - 400).status_code == 401
def test_replayed_nonce_rejected(client, server):
nonce = secrets.token_hex(16)
assert _post(client, server, nonce=nonce).status_code == 200
assert _post(client, server, nonce=nonce).status_code == 401
def test_oversize_body_rejected(client, server):
assert _post(client, server, raw=b'{"x":"' + b'a' * 70000 + b'"}').status_code == 413
def test_invalid_level_rejected(client, server):
assert _post(client, server, body={'permission_level': 'root'}).status_code == 400
def test_level_persisted_and_escalation_audited(client, server):
_post(client, server, body={'permission_level': 'monitor'})
_post(client, server, body={'permission_level': 'full'})
rows = AuditLog.query.filter_by(action='server.permission_level_changed').order_by(AuditLog.id).all()
details = [r.get_details() for r in rows]
assert [(d['from'], d['to'], d['escalation']) for d in details] == [
(None, 'monitor', False), ('monitor', 'full', True)]
assert db.session.get(Server, server.id).permission_level == 'full'
def test_no_api_sets_level(client, server, auth_headers):
resp = client.put(f'/api/v1/servers/{server.id}', headers=auth_headers,
json={'permission_level': 'full', 'name': 'x'})
assert resp.status_code in (200, 400)
assert db.session.get(Server, server.id).permission_level is None
writers = [r.rule for r in client.application.url_map.iter_rules()
if 'level' in r.rule and r.methods & {'POST', 'PUT', 'PATCH'}]
assert writers == []
def _connect(server_id):
agent_registry._agents[server_id] = ConnectedAgent(
server_id=server_id, socket_id='s', session_token='t',
connected_at=datetime.utcnow(), last_heartbeat=datetime.utcnow(),
ip_address='1.1.1.1', agent_version='1', transport='poll')
@pytest.mark.parametrize('level,action,refused', [
('monitor', 'firewall:status', True),
('monitor', 'docker:container:list', True),
('protect', 'docker:container:list', True),
('protect', 'system:exec', True),
('protect', 'firewall:status', False),
('protect', 'cloudflare:origin_lock:apply', False),
('protect', 'credential_update', False),
('full', 'system:exec', False),
(None, 'system:exec', False),
])
def test_refusal_table(level, action, refused):
assert bool(permission_levels.refusal(level, action)) is refused
@pytest.mark.parametrize('level', ['monitor', 'protect'])
def test_send_command_gate(server, level):
server.permission_level = level
db.session.commit()
_connect(server.id)
try:
result = agent_registry.send_command(server.id, 'system:exec', {})
finally:
agent_registry._agents.pop(server.id, None)
assert result['code'] == 'LEVEL_DENIED'
assert result['error'].startswith(f'server is in {level.title()} mode')
def test_protect_allowlist_matches_documented_contract():
doc = Path(__file__).parents[2] / 'docs/AGENT_PERMISSION_LEVELS.md'
block = re.search(r'```text allowlist\n(.*?)```', doc.read_text(), re.S).group(1)
assert set(block.split()) == set(permission_levels.PROTECT_ACTIONS)
def test_silent_monitor_goes_offline_with_one_alert(server):
server.permission_level = 'monitor'
server.status = 'online'
server.last_report_at = datetime.utcnow() - timedelta(minutes=20)
db.session.commit()
assert agent_health_service.sweep_silent_reporters() == [server.id]
assert agent_health_service.sweep_silent_reporters() == []
assert db.session.get(Server, server.id).status == 'offline'
assert MetricAlert.query.filter_by(metric='offline', status='active').count() == 1
def test_recent_monitor_stays_online(server):
server.permission_level, server.status = 'monitor', 'online'
server.last_report_at = datetime.utcnow() - timedelta(minutes=2)
db.session.commit()
assert agent_health_service.sweep_silent_reporters() == []
def _checks(status):
return {'permission_level': 'monitor', 'service_checks': [
{'name': 'mysql', 'kind': 'database', 'status': status, 'detail': 'refused',
'checked_at': '2026-01-01T00:00:00Z'}]}
def test_service_down_alert_dedupes_and_resolves(client, server):
_post(client, server, body=_checks('up'))
assert MetricAlert.query.count() == 0
_post(client, server, body=_checks('down'))
_post(client, server, body=_checks('down'))
alert = MetricAlert.query.one()
assert alert.status == 'active' and 'mysql' in alert.detail and 'web-1' in alert.detail
_post(client, server, body=_checks('up'))
assert MetricAlert.query.one().status == 'resolved'
def test_degraded_to_down_alerts_and_report_recovers_offline(client, server):
_post(client, server, body=_checks('degraded'))
assert MetricAlert.query.count() == 0
_post(client, server, body=_checks('down'))
assert MetricAlert.query.filter_by(status='active').count() == 1
agent_health_service.mark_offline(server, 'test')
db.session.commit()
server.status = 'offline'
db.session.commit()
_post(client, server, body=_checks('down'))
assert MetricAlert.query.filter_by(metric='offline', status='resolved').count() == 1
def test_install_instructions_include_level(client, server, auth_headers):
server.set_registration_token('sk_reg_x')
db.session.commit()
data = client.get(f'/api/v1/servers/install-instructions/{server.id}?level=protect',
headers=auth_headers).get_json()
assert data['permission_level'] == 'protect'
assert '--level protect' in data['linux']['one_liner']
assert '-Level protect' in data['windows']['one_liner']
def test_token_records_requested_level(client, server, auth_headers):
client.post(f'/api/v1/servers/{server.id}/regenerate-token', headers=auth_headers,
json={'requested_level': 'protect'})
assert db.session.get(Server, server.id).requested_permission_level == 'protect'
@@ -7,9 +7,9 @@ from pathlib import Path
from app.core_blueprints import CORE_BLUEPRINTS
EXPECTED_CORE_BLUEPRINT_COUNT = 116
EXPECTED_CORE_BLUEPRINT_COUNT = 117
EXPECTED_MANIFEST_SHA256 = (
'25d28d92ccddddc246d5b2081d84b11f1df10cb662f33c256ebcbe390d2813d5'
'52abbc7429d079c7cf6ea61a638f5dc4dbd991e4f02ec31c81563e01442aa7fb'
)
+1
View File
@@ -145,6 +145,7 @@ PUBLIC_ALT_AUTH = {
# Public by design; the value names the mechanism that authenticates the call.
'agent_poll.connect': 'agent HMAC handshake + per-IP throttle',
'agent_poll.disconnect': 'X-Session-Token lookup',
'agent_report.report': 'HMAC over timestamp+nonce+body hash, skew/replay/rate limits',
'agent_poll.poll': 'X-Session-Token resolves the ConnectedAgent',
'agent_poll.result': 'X-Session-Token resolves the ConnectedAgent',
'auth.connect_session': 'loopback-only + Cloud grant verified against JWKS, 30/min',
+1
View File
@@ -221,6 +221,7 @@ VIEWER_WRITABLE = {
'docker.get_containers_stats',
# PUBLIC transport — agent long-poll fallback; user JWT is ignored here.
'agent_poll.disconnect',
'agent_report.report',
# PUBLIC authentication challenge only; credential verification and UV are
# required by passkey_authenticate before any session tokens are issued.
'auth.passkey_auth_options',
@@ -0,0 +1,17 @@
import importlib.util
from pathlib import Path
from app.models.metric_alert import MetricAlert
from app.models.server import Server
def test_seed_is_idempotent(app):
path = Path(__file__).parents[1] / 'scripts/seed_permission_demo.py'
spec = importlib.util.spec_from_file_location('seed_demo', path)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
mod.seed()
mod.seed()
assert Server.query.filter(Server.agent_id.like('demo-perm-%')).count() == 5
assert Server.query.filter_by(status='offline').count() == 1
assert MetricAlert.query.filter_by(metric='offline').count() == 1
+81
View File
@@ -0,0 +1,81 @@
# Agent permission levels
Each ServerKit agent runs at one of three levels. The level is chosen **on the
server**, enforced **by the agent**, and merely *reported* to the panel.
| Level | What it can do | What it cannot do |
|-------|----------------|-------------------|
| **Monitor** (default for new installs) | Send CPU / memory / disk / uptime / load and service checks (web server, PHP-FPM, database, disk, CPU, memory) to the panel with signed outbound HTTPS POSTs. Opens no inbound command channel. | Run any command, open a terminal, touch files, Docker, packages or services. The panel cannot send it anything. |
| **Protect** | Everything in Monitor, plus a fixed security allowlist: firewall (ufw) rules, fail2ban, CrowdSec decisions, Cloudflare origin lock (restrict 80/443 to Cloudflare ranges). | Docker, shell, files, terminal, packages, systemd, cron, SSH keys, WireGuard, runtimes, agent updates. |
| **Full** | Today's behaviour: the whole agent feature set. Legacy installs with no level configured are treated as Full. | — |
## Install
The Add Server drawer shows the level picker and puts the choice in the command:
```bash
curl -fsSL https://panel.example.com/api/v1/servers/install.sh -o /tmp/serverkit-agent-install.sh \
&& sudo bash /tmp/serverkit-agent-install.sh --token "TOKEN" --server "https://panel.example.com" --level monitor
```
```powershell
irm https://panel.example.com/api/v1/servers/install.ps1 | iex; Install-ServerKitAgent -Token "TOKEN" -Server "https://panel.example.com" -Level monitor
```
The installer scripts under `scripts/` are vendored from the agent repo
(`scripts/sync-agent-installers.sh`); `--level` / `-Level` arrives with the next sync.
## Changing the level
Only on the server itself, as root, with a typed confirmation:
```bash
serverkit-agent permissions show
sudo serverkit-agent permissions set protect
```
There is no panel button or API for this, by design. Re-running `install.sh` with a
different `--level` on an installed box refuses unless `--reconfigure` is passed.
## What the panel does
- Records the level the agent reports (registration, auth, heartbeat, report). Every
change, escalations included, is written to the audit log and shown in the server
timeline. The Add Server flow remembers the *requested* level so a mismatch with
the reported one is visible.
- Monitor agents POST to `/api/v1/agent/report` (HMAC over timestamp, nonce and body
hash; 300 s skew; single-use nonces; 64 KB cap; per-IP+key rate limit). A monitor
server with no report for max(3 minutes, 3 x its interval) is marked offline.
- Opens an alert (existing fleet alerts + notification channels) when a server goes
offline or stops reporting and when a service check goes down; it resolves on
recovery and does not repeat while the condition persists.
- Refuses to dispatch a command the reported level forbids (`LEVEL_DENIED`,
"server is in Monitor mode: ..."). This is defence in depth; the agent enforces
the same table. The table lives in `backend/app/services/permission_levels.py`.
Protect allowlist (checked against the panel table by a test):
```text allowlist
firewall:allow_port firewall:deny_port firewall:status
firewall:allow_ip firewall:deny_ip
fail2ban:status fail2ban:ban fail2ban:unban
crowdsec:decisions:list crowdsec:ban crowdsec:unban
cloudflare:origin_lock:status cloudflare:origin_lock:apply cloudflare:origin_lock:remove
```
`credential_update` (signature-verified key rotation) is allowed at every level.
## Threat model
The level is read only from the agent's local config at startup; no message from
the panel can change it. A compromised panel therefore **cannot escalate** a Monitor
agent: it has no channel to send commands to, and the report reply never carries any.
The panel's own gate only narrows what it will send. Someone with root on the box
can still change the level, which is the intended re-confirmation path, and the
panel audits the resulting change. A compromised panel can lie in the UI, but
cannot make a Monitor agent act.
## Demo data
`python backend/scripts/seed_permission_demo.py` (never run automatically) creates five
demo servers for the dev sandbox. It is idempotent.
+1
View File
@@ -755,6 +755,7 @@ Regenerate (backend/):
- `POST /agent/connect`
- `POST /agent/disconnect`
- `POST /agent/poll`
- `POST /agent/report`
- `POST /agent/result`
- `POST /ai/chat`
- `POST /ai/chat/cancel`
+1 -1
View File
@@ -16,7 +16,7 @@ debt. Regenerate with `python scripts/generate-migration-inventory.py`.
| Routes on bare @jwt_required() | auth_required() / role decorators | 598 | 598 | REGISTERED EXCEPTION: JWT-only is the deliberate default; conversion grants API-key access and happens per route, on decision |
| HTTP statuses chosen by sniffing error text | typed errors from app.exceptions | 0 | 0 | INVARIANT at 0 - migration completed 2026-08-19 |
| API crashes swallowed without recording | app.error_reporting | 0 | 0 | INVARIANT at 0 |
| Hand-shaped {'error': ...} bodies in app/api | typed errors + the global handler | 849 | 850 | migrate when touched; new endpoints raise |
| Hand-shaped {'error': ...} bodies in app/api | typed errors + the global handler | 855 | 850 | migrate when touched; new endpoints raise |
| Raw subprocess calls outside the runners | app/utils/system.py runners | 23 | 23 | migrate when touched |
| Controller-boundary violations (routes doing service work) | service layer extraction | 453 | 453 | migrate when touched (first-wave ratchet) |
| raw api.* calls in pages/ | E1: useServerQuery/useServerMutation | 374 | 398 | migrate when touched |
+1
View File
@@ -11,6 +11,7 @@ Welcome to the ServerKit documentation. This guide will help you install, config
| [Video: template to live app](https://youtu.be/TXfXFz6bVjs) | Watch a 4-minute AgentSite deployment through the template catalog and live Deploy Console |
| [Video: deploy a Git repository](https://youtu.be/Fv28qMERYDU) | Connect a repository, review detected settings and deploy through the three-step New Service wizard |
| [Installation Guide](INSTALLATION.md) | Complete setup instructions for Docker and manual installation |
| [Agent permission levels](AGENT_PERMISSION_LEVELS.md) | Monitor / Protect / Full: what each allows and how to change it |
| [Architecture](ARCHITECTURE.md) | System design, request flow, and diagrams |
| [Deployment Guide](DEPLOYMENT.md) | Production deployment, CLI commands, and configuration |
| [Local Development](LOCAL_DEVELOPMENT.md) | Run the panel locally for development |