mirror of
https://github.com/jhd3197/ServerKit.git
synced 2026-11-09 05:24:49 +09:00
feat(agents): permission levels, signed report endpoint, health alerts
Agent-reported monitor/protect/full level (never settable from the panel), POST /api/v1/agent/report for report-only agents, silence-based offline detection, service-check alerts via MetricAlert, a panel-side command gate, docs, and a demo seed script. Migration 112. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
2ef0a8f370
commit
a4d2ea37ef
26 files changed
+861
-12
No files matched your search
@@ -22,6 +22,15 @@ release history; historical `agent-v*` tags are not panel releases.
|
||||
|
||||
### Added
|
||||
|
||||
- Agent permission levels (Monitor / Protect / Full), enforced by the agent and
|
||||
reported to the panel. Adds a signed `POST /api/v1/agent/report` endpoint for
|
||||
report-only Monitor agents, per-server level and service-check state (server
|
||||
list badge and health, server detail panel), offline / service-down alerts
|
||||
through the existing fleet alerts and notification channels, audited level
|
||||
changes, and a panel-side command gate. The panel cannot set the level; it is
|
||||
changed on the server with `serverkit-agent permissions set`. See
|
||||
[docs/AGENT_PERMISSION_LEVELS.md](docs/AGENT_PERMISSION_LEVELS.md). Migration
|
||||
112. The Add Server commands now pass `--level` / `-Level`.
|
||||
- `GET /api/v1/servers` now accepts an API key (`X-API-Key`) carrying the
|
||||
`servers:read` scope, in addition to the usual JWT, so external tools can
|
||||
poll the server list read-only without a short-lived token. JWT callers are
|
||||
|
||||
@@ -47,6 +47,12 @@ trust boundaries:
|
||||
(`system:exec`) and interactive PTY sessions are controlled by the agent's
|
||||
`Features.Exec` flag, which is **off by default**. Enable it only on servers
|
||||
where you intend the panel to run shell commands.
|
||||
- **Permission levels bound what a compromised panel can do.** An agent runs as
|
||||
Monitor (report-only, no inbound command channel), Protect (security allowlist
|
||||
only) or Full. The level is read only from the agent's local config and can
|
||||
only be changed on the server (`serverkit-agent permissions set`); the panel
|
||||
records and audits the reported level but has no API to set it. See
|
||||
[docs/AGENT_PERMISSION_LEVELS.md](docs/AGENT_PERMISSION_LEVELS.md).
|
||||
- **Transport & connection controls.** Agents authenticate to the panel with
|
||||
per-connection HMAC-SHA256 (with nonce/replay protection and a timestamp-skew
|
||||
check), and the panel enforces a per-server IP allowlist. Use `wss://`
|
||||
|
||||
@@ -136,7 +136,10 @@ class AgentNamespace(Namespace):
|
||||
return
|
||||
|
||||
metrics = data.get('metrics', {})
|
||||
agent_registry.ingest_agent_state(agent, metrics=metrics)
|
||||
agent_registry.ingest_agent_state(
|
||||
agent, metrics=metrics,
|
||||
permission_level=data.get('permission_level'),
|
||||
service_checks=data.get('service_checks'))
|
||||
|
||||
emit('heartbeat_ack', {'type': 'heartbeat_ack'})
|
||||
|
||||
|
||||
@@ -108,7 +108,9 @@ def poll():
|
||||
# Shared transport-neutral ingest: capability
|
||||
# side-effects (e.g. tunnel reconcile) fire exactly as on the WS path.
|
||||
agent_registry.ingest_agent_state(agent, metrics=metrics, sysinfo=sysinfo,
|
||||
capabilities=caps)
|
||||
capabilities=caps,
|
||||
permission_level=body.get('permission_level'),
|
||||
service_checks=body.get('service_checks'))
|
||||
|
||||
# Long-poll up to 25s for any queued command. Below the typical
|
||||
# tunnel idle-timeout (Cloudflare ~100s, ngrok ~60s) so we never
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
"""Monitor-mode agent report endpoint.
|
||||
|
||||
``POST /api/v1/agent/report`` is the ONLY channel a monitor-level agent uses:
|
||||
a signed, outbound-only HTTPS POST. The reply never carries commands.
|
||||
|
||||
Auth (no JWT/CSRF; the HMAC is the credential):
|
||||
X-Agent-Key: <api key> -> server looked up by its key prefix
|
||||
X-Timestamp: <unix seconds> -> |skew| <= 300s
|
||||
X-Nonce: <hex, >=32 chars> -> single use (TTL cache, 5 min)
|
||||
X-Signature: hex(HMAC_SHA256(api_secret, ts + "." + nonce + "." + sha256hex(body)))
|
||||
"""
|
||||
|
||||
from flask import Blueprint, jsonify, request
|
||||
|
||||
from app import limiter
|
||||
from app.services import agent_health_service, permission_levels
|
||||
from app.utils.ip_utils import is_ip_allowed
|
||||
|
||||
|
||||
agent_report_bp = Blueprint('agent_report', __name__)
|
||||
|
||||
MAX_BODY_BYTES = 64 * 1024
|
||||
MAX_SKEW_SECONDS = 300
|
||||
|
||||
|
||||
def _rate_key():
|
||||
return f"{request.remote_addr}:{(request.headers.get('X-Agent-Key') or '')[:12]}"
|
||||
|
||||
|
||||
@agent_report_bp.route('/report', methods=['POST'])
|
||||
@limiter.limit("30 per minute", key_func=_rate_key)
|
||||
def report():
|
||||
if (request.content_length or 0) > MAX_BODY_BYTES:
|
||||
return jsonify({'error': 'payload too large'}), 413
|
||||
raw = request.get_data(cache=True)
|
||||
if len(raw) > MAX_BODY_BYTES:
|
||||
return jsonify({'error': 'payload too large'}), 413
|
||||
|
||||
server = agent_health_service.authenticate_report(raw, request.headers, MAX_SKEW_SECONDS)
|
||||
if not server:
|
||||
return jsonify({'error': 'authentication failed'}), 401
|
||||
if server.allowed_ips and not is_ip_allowed(request.remote_addr, server.allowed_ips):
|
||||
return jsonify({'error': 'IP not allowed'}), 403
|
||||
|
||||
body = request.get_json(silent=True)
|
||||
if not isinstance(body, dict):
|
||||
return jsonify({'error': 'JSON object required'}), 400
|
||||
level = body.get('permission_level')
|
||||
if level not in permission_levels.LEVELS:
|
||||
return jsonify({'error': 'permission_level must be monitor, protect or full'}), 400
|
||||
|
||||
interval = agent_health_service.ingest_report(server, level, body)
|
||||
return jsonify({'ok': True, 'interval_seconds': interval})
|
||||
@@ -13,6 +13,7 @@ from flask_jwt_extended import jwt_required, get_jwt_identity
|
||||
|
||||
from sqlalchemy.orm import joinedload
|
||||
from app import db, limiter
|
||||
from app.services import permission_levels
|
||||
from app.services.server_metrics_service import latest_metrics_by_server
|
||||
from app.api._query import apply_query, QueryParseError
|
||||
from app.api._request import json_object
|
||||
@@ -440,6 +441,8 @@ def create_server():
|
||||
registered_by=user_id,
|
||||
workspace_id=ws_id,
|
||||
registration_token_expires=expires_at,
|
||||
# What the install command asks for; the agent's report is the truth.
|
||||
requested_permission_level=permission_levels.normalize(data.get('requested_level')),
|
||||
)
|
||||
server.set_registration_token(registration_token)
|
||||
|
||||
@@ -718,6 +721,7 @@ def regenerate_token(server_id):
|
||||
registration_token = Server.generate_registration_token()
|
||||
server.set_registration_token(registration_token)
|
||||
server.registration_token_expires = expires_at
|
||||
server.requested_permission_level = permission_levels.normalize(data.get('requested_level'))
|
||||
server.status = 'pending'
|
||||
server.agent_id = None
|
||||
|
||||
@@ -792,6 +796,10 @@ def register_agent():
|
||||
server.total_disk = system_info.get('total_disk', server.total_disk)
|
||||
|
||||
server.agent_version = data.get('agent_version')
|
||||
# Level is whatever the agent says it was installed with; the panel never
|
||||
# sets it. A mismatch with the requested level is surfaced in the UI.
|
||||
from app.services import agent_health_service
|
||||
agent_health_service.record_level(server, data.get('permission_level'))
|
||||
|
||||
# Replace the panel-issued placeholder name with whatever the agent
|
||||
# reports — preferring the OS hostname over the agent's optional
|
||||
|
||||
@@ -5,6 +5,7 @@ import os
|
||||
import re
|
||||
import requests
|
||||
from app.models.server import AgentVersion, Server
|
||||
from app.services import permission_levels
|
||||
from flask import Response, current_app, jsonify, redirect, request
|
||||
from app.middleware.rbac import admin_required, developer_required
|
||||
from app.exceptions import agent_offline, field_required
|
||||
@@ -178,8 +179,11 @@ def get_install_instructions(server_id):
|
||||
# Get base URL
|
||||
base_url = _get_external_base_url()
|
||||
api_url = f"{base_url}/api/v1/servers"
|
||||
level = (permission_levels.normalize(request.args.get('level'))
|
||||
or server.requested_permission_level or 'monitor')
|
||||
|
||||
return jsonify({
|
||||
'permission_level': level,
|
||||
'linux': {
|
||||
# Download-then-run rather than `curl … | sudo bash`: in a pipeline
|
||||
# the shell reports bash's exit status, so a failed download exits 0
|
||||
@@ -187,7 +191,7 @@ def get_install_instructions(server_id):
|
||||
'one_liner': (
|
||||
f'curl -fsSL {api_url}/install.sh -o /tmp/serverkit-agent-install.sh '
|
||||
f'&& sudo bash /tmp/serverkit-agent-install.sh '
|
||||
f'--token "YOUR_TOKEN" --server "{base_url}"'
|
||||
f'--token "YOUR_TOKEN" --server "{base_url}" --level {level}'
|
||||
),
|
||||
'manual': [
|
||||
f'# Download the script',
|
||||
@@ -195,17 +199,17 @@ def get_install_instructions(server_id):
|
||||
f'chmod +x install.sh',
|
||||
f'',
|
||||
f'# Run installation',
|
||||
f'sudo ./install.sh --token "YOUR_TOKEN" --server "{base_url}"'
|
||||
f'sudo ./install.sh --token "YOUR_TOKEN" --server "{base_url}" --level {level}'
|
||||
]
|
||||
},
|
||||
'windows': {
|
||||
'one_liner': f'irm {api_url}/install.ps1 | iex; Install-ServerKitAgent -Token "YOUR_TOKEN" -Server "{base_url}"',
|
||||
'one_liner': f'irm {api_url}/install.ps1 | iex; Install-ServerKitAgent -Token "YOUR_TOKEN" -Server "{base_url}" -Level {level}',
|
||||
'manual': [
|
||||
f'# Download the script (run in PowerShell as Administrator)',
|
||||
f'Invoke-WebRequest -Uri "{api_url}/install.ps1" -OutFile install.ps1',
|
||||
f'',
|
||||
f'# Run installation',
|
||||
f'.\\install.ps1 -Token "YOUR_TOKEN" -Server "{base_url}"'
|
||||
f'.\\install.ps1 -Token "YOUR_TOKEN" -Server "{base_url}" -Level {level}'
|
||||
]
|
||||
},
|
||||
'note': 'Replace YOUR_TOKEN with the registration token shown in the UI'
|
||||
|
||||
@@ -46,6 +46,7 @@ class BlueprintSpec:
|
||||
CORE_BLUEPRINTS: tuple[BlueprintSpec, ...] = (
|
||||
BlueprintSpec('app.api.auth', 'auth_bp', '/api/v1/auth'),
|
||||
BlueprintSpec('app.api.agent_poll', 'agent_poll_bp', '/api/v1/agent'),
|
||||
BlueprintSpec('app.api.agent_report', 'agent_report_bp', '/api/v1/agent'),
|
||||
BlueprintSpec('app.api.linked_panel', 'linked_panel_bp', '/api/v1/linked-panel'),
|
||||
BlueprintSpec('app.api.apps', 'apps_bp', '/api/v1/apps'),
|
||||
BlueprintSpec('app.api.apps', 'apps_bp', '/api/v1/services', name='services'),
|
||||
|
||||
@@ -534,7 +534,7 @@ def _notify_fleet_alerts(opened, resolved):
|
||||
'metric': alert.metric,
|
||||
'value': alert.value,
|
||||
'threshold': alert.threshold,
|
||||
'message': (
|
||||
'message': alert.detail or (
|
||||
f'{hostname}: {alert.metric} averaged {alert.value} over '
|
||||
f'{alert.duration_seconds}s, past the '
|
||||
f'{alert.severity} threshold of {alert.threshold}.'),
|
||||
@@ -555,7 +555,9 @@ def _notify_fleet_alerts(opened, resolved):
|
||||
'hostname': hostname,
|
||||
'alert_type': f'{alert.metric} recovered',
|
||||
'metric': alert.metric,
|
||||
'message': (f'{hostname}: {alert.metric} is back under its '
|
||||
'message': (f'{hostname}: {alert.metric} recovered.'
|
||||
if alert.detail else
|
||||
f'{hostname}: {alert.metric} is back under its '
|
||||
f'threshold.'),
|
||||
},
|
||||
)
|
||||
|
||||
@@ -49,6 +49,8 @@ class MetricAlert(db.Model):
|
||||
threshold = db.Column(db.Float) # the threshold exceeded
|
||||
duration_seconds = db.Column(db.Integer) # how long it was exceeded
|
||||
|
||||
detail = db.Column(db.Text) # human message for offline / service-check alerts
|
||||
|
||||
status = db.Column(db.String(20), default='active', index=True) # active, acknowledged, resolved
|
||||
acknowledged_by = db.Column(db.Integer, db.ForeignKey('users.id'), nullable=True, index=True)
|
||||
|
||||
@@ -67,6 +69,7 @@ class MetricAlert(db.Model):
|
||||
'value': self.value,
|
||||
'threshold': self.threshold,
|
||||
'duration_seconds': self.duration_seconds,
|
||||
'detail': self.detail,
|
||||
'status': self.status,
|
||||
'acknowledged_by': self.acknowledged_by,
|
||||
'created_at': self.created_at.isoformat() if self.created_at else None,
|
||||
|
||||
@@ -126,6 +126,17 @@ class Server(TimestampMixin, JsonColumnMixin, db.Model):
|
||||
allow_agent_update_observed = db.Column(db.Boolean, nullable=False,
|
||||
default=False, server_default='0')
|
||||
|
||||
# Agent permission level (monitor|protect|full). REPORTED BY THE AGENT
|
||||
# only -- no panel API writes it. NULL = unknown/legacy, treated as full.
|
||||
# requested_permission_level is what the install command asked for, kept
|
||||
# only to flag a mismatch with what the agent reports.
|
||||
permission_level = db.Column(db.String(16), nullable=True)
|
||||
permission_level_updated_at = db.Column(db.DateTime, nullable=True)
|
||||
requested_permission_level = db.Column(db.String(16), nullable=True)
|
||||
last_report_at = db.Column(db.DateTime, nullable=True)
|
||||
report_interval_seconds = db.Column(db.Integer, nullable=True)
|
||||
service_checks = db.Column(db.JSON, nullable=True) # latest state per check
|
||||
|
||||
@property
|
||||
def is_managed(self):
|
||||
"""False only for observed boxes."""
|
||||
@@ -490,6 +501,12 @@ class Server(TimestampMixin, JsonColumnMixin, db.Model):
|
||||
'last_error': self.last_error,
|
||||
'management_mode': self.management_mode or 'managed',
|
||||
'allow_agent_update_observed': bool(self.allow_agent_update_observed),
|
||||
'permission_level': self.permission_level,
|
||||
'permission_level_effective': self.permission_level or 'full',
|
||||
'permission_level_updated_at': self.permission_level_updated_at.isoformat() if self.permission_level_updated_at else None,
|
||||
'requested_permission_level': self.requested_permission_level,
|
||||
'last_report_at': self.last_report_at.isoformat() if self.last_report_at else None,
|
||||
'service_checks': self.service_checks or [],
|
||||
'onboarding_state': self.onboarding_state,
|
||||
'onboarding_updated_at': self.onboarding_updated_at.isoformat() if self.onboarding_updated_at else None,
|
||||
'agent_version': self.agent_version,
|
||||
|
||||
@@ -0,0 +1,223 @@
|
||||
"""Agent-reported health: permission level, service checks, silence, alerts.
|
||||
|
||||
Everything here consumes what the agent REPORTS (registration, auth, heartbeat,
|
||||
or the monitor-mode ``/agent/report`` POST). Alerts reuse ``MetricAlert`` and the
|
||||
fleet notification path; one active row per (server, metric) de-duplicates.
|
||||
"""
|
||||
import hashlib
|
||||
import hmac
|
||||
import logging
|
||||
import time
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from app import db
|
||||
from app.models.audit_log import AuditLog
|
||||
from app.models.metric_alert import MetricAlert
|
||||
from app.models.server import Server
|
||||
from app.services import permission_levels
|
||||
from app.services.nonce_service import nonce_service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
MIN_SILENCE_SECONDS = 180 # monitor agents: offline after max(3 min, 3x interval)
|
||||
DEFAULT_INTERVAL_SECONDS = 60
|
||||
DEFAULT_INTERVAL = DEFAULT_INTERVAL_SECONDS
|
||||
OFFLINE_METRIC = 'offline'
|
||||
OPEN = ('active', 'acknowledged')
|
||||
BAD = ('down',)
|
||||
|
||||
|
||||
def _notify(opened=(), resolved=()):
|
||||
if not opened and not resolved:
|
||||
return
|
||||
try:
|
||||
from app.jobs.builtin_handlers import _notify_fleet_alerts
|
||||
_notify_fleet_alerts(list(opened), list(resolved))
|
||||
except Exception:
|
||||
logger.exception('Could not notify agent health alerts')
|
||||
|
||||
|
||||
def _open_alert(server, metric, detail):
|
||||
"""Open an alert unless one is already open. Returns it if newly opened."""
|
||||
if MetricAlert.query.filter(MetricAlert.server_id == server.id,
|
||||
MetricAlert.metric == metric,
|
||||
MetricAlert.status.in_(OPEN)).first():
|
||||
return None
|
||||
alert = MetricAlert(server_id=server.id, metric=metric, severity='critical',
|
||||
detail=detail, status='active')
|
||||
db.session.add(alert)
|
||||
return alert
|
||||
|
||||
|
||||
def _resolve_alerts(server, metric):
|
||||
rows = MetricAlert.query.filter(MetricAlert.server_id == server.id,
|
||||
MetricAlert.metric == metric,
|
||||
MetricAlert.status.in_(OPEN)).all()
|
||||
for row in rows:
|
||||
row.status = 'resolved'
|
||||
row.resolved_at = datetime.utcnow()
|
||||
return rows
|
||||
|
||||
|
||||
def record_level(server, reported):
|
||||
"""Persist a level reported by the agent; audit when it changes.
|
||||
|
||||
Invalid/missing values are ignored. Escalations (monitor->protect->full) are
|
||||
logged like any change but flagged, since they mean someone re-ran the
|
||||
local confirmation on the box.
|
||||
"""
|
||||
level = permission_levels.normalize(reported)
|
||||
if level is None:
|
||||
return
|
||||
old = server.permission_level
|
||||
if old == level:
|
||||
return
|
||||
server.permission_level = level
|
||||
server.permission_level_updated_at = datetime.utcnow()
|
||||
AuditLog.log(
|
||||
'server.permission_level_changed', target_type='server', target_id=server.id,
|
||||
details={'server_id': server.id, 'from': old, 'to': level,
|
||||
'escalation': permission_levels.is_escalation(old, level),
|
||||
'source': 'agent_report'})
|
||||
logger.warning('Server %s reported permission level %s -> %s', server.id, old, level)
|
||||
|
||||
|
||||
def record_service_checks(server, checks):
|
||||
"""Store the latest per-check state and open/resolve alerts on transitions."""
|
||||
if not isinstance(checks, list):
|
||||
return
|
||||
previous = {c.get('name'): c.get('status') for c in (server.service_checks or [])}
|
||||
clean, opened, resolved = [], [], []
|
||||
for raw in checks[:50]:
|
||||
if not isinstance(raw, dict) or not isinstance(raw.get('name'), str):
|
||||
continue
|
||||
item = {k: (str(raw.get(k))[:200] if raw.get(k) is not None else None)
|
||||
for k in ('name', 'kind', 'status', 'detail', 'checked_at')}
|
||||
item['name'] = item['name'][:60]
|
||||
if item['status'] not in ('up', 'down', 'degraded', 'unknown'):
|
||||
item['status'] = 'unknown'
|
||||
clean.append(item)
|
||||
metric = f"svc:{item['name']}"[:20]
|
||||
if item['status'] == 'down':
|
||||
alert = _open_alert(
|
||||
server, metric,
|
||||
f"{server.name}: service '{item['name']}' is down since "
|
||||
f"{datetime.utcnow():%Y-%m-%d %H:%M} UTC"
|
||||
+ (f" ({item['detail']})" if item['detail'] else ''))
|
||||
if alert:
|
||||
opened.append(alert)
|
||||
elif item['status'] == 'up' or (item['status'] == 'degraded'
|
||||
and previous.get(item['name']) == 'down'):
|
||||
resolved += _resolve_alerts(server, metric)
|
||||
server.service_checks = clean
|
||||
db.session.flush()
|
||||
_notify(opened, resolved)
|
||||
|
||||
|
||||
def mark_offline(server, reason):
|
||||
"""Open the (de-duplicated) offline alert. Caller commits."""
|
||||
since = server.last_report_at or server.last_seen or datetime.utcnow()
|
||||
alert = _open_alert(
|
||||
server, OFFLINE_METRIC,
|
||||
f"{server.name} is offline since {since:%Y-%m-%d %H:%M} UTC: {reason}")
|
||||
db.session.flush()
|
||||
_notify([alert] if alert else [])
|
||||
|
||||
|
||||
def mark_online(server):
|
||||
resolved = _resolve_alerts(server, OFFLINE_METRIC)
|
||||
db.session.flush()
|
||||
_notify(resolved=resolved)
|
||||
|
||||
|
||||
def silence_limit(server):
|
||||
interval = server.report_interval_seconds or DEFAULT_INTERVAL_SECONDS
|
||||
return timedelta(seconds=max(MIN_SILENCE_SECONDS, 3 * interval))
|
||||
|
||||
|
||||
def sweep_silent_reporters(now=None):
|
||||
"""Flip report-only (monitor) servers offline once their reports stop.
|
||||
|
||||
Reports are not a socket, so the registry's heartbeat reaper never sees
|
||||
them. Only servers that have ever reported are considered.
|
||||
"""
|
||||
now = now or datetime.utcnow()
|
||||
flipped = []
|
||||
for server in Server.query.filter(Server.permission_level == 'monitor',
|
||||
Server.last_report_at.isnot(None),
|
||||
Server.status != 'offline').all():
|
||||
if now - server.last_report_at > silence_limit(server):
|
||||
server.status = 'offline'
|
||||
server.last_error = 'Agent stopped reporting'
|
||||
mark_offline(server, 'no report received')
|
||||
flipped.append(server.id)
|
||||
if flipped:
|
||||
db.session.commit()
|
||||
return flipped
|
||||
|
||||
|
||||
def authenticate_report(raw, headers, max_skew):
|
||||
"""Return the Server for a validly signed report request, else None."""
|
||||
key = headers.get('X-Agent-Key') or ''
|
||||
ts = headers.get('X-Timestamp') or ''
|
||||
nonce = headers.get('X-Nonce') or ''
|
||||
signature = headers.get('X-Signature') or ''
|
||||
if not (key and ts and signature) or len(nonce) < 32:
|
||||
return None
|
||||
try:
|
||||
if abs(time.time() - int(ts)) > max_skew:
|
||||
return None
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
prefix = key[:12]
|
||||
server = Server.query.filter(
|
||||
(Server.api_key_prefix == prefix) | (Server.api_key_pending_prefix == prefix)
|
||||
).first()
|
||||
if not server:
|
||||
return None
|
||||
pending = (server.api_key_pending_prefix == prefix and server.api_key_prefix != prefix
|
||||
and server.api_key_rotation_expires
|
||||
and datetime.utcnow() <= server.api_key_rotation_expires)
|
||||
secret = server.get_pending_api_secret() if pending else server.get_api_secret()
|
||||
if not secret:
|
||||
return None
|
||||
|
||||
message = f"{ts}.{nonce}.{hashlib.sha256(raw).hexdigest()}"
|
||||
expected = hmac.new(secret.encode(), message.encode(), hashlib.sha256).hexdigest()
|
||||
if not hmac.compare_digest(signature, expected):
|
||||
return None
|
||||
# Consume the nonce only after the signature proves valid.
|
||||
if not nonce_service.check_and_record(server.id, nonce):
|
||||
return None
|
||||
return server
|
||||
|
||||
|
||||
def ingest_report(server, level, body):
|
||||
"""Apply a verified report to the server row; returns the interval in use."""
|
||||
interval = body.get('interval_seconds')
|
||||
interval = interval if isinstance(interval, int) and 5 <= interval <= 3600 else DEFAULT_INTERVAL
|
||||
|
||||
now = datetime.utcnow()
|
||||
was_offline = server.status != 'online'
|
||||
server.status = 'online'
|
||||
server.last_error = None
|
||||
server.last_seen = now
|
||||
server.last_report_at = now
|
||||
server.report_interval_seconds = interval
|
||||
if isinstance(body.get('agent_version'), str):
|
||||
server.agent_version = body['agent_version'][:20]
|
||||
record_level(server, level)
|
||||
record_service_checks(server, body.get('service_checks'))
|
||||
if was_offline:
|
||||
mark_online(server)
|
||||
db.session.commit()
|
||||
|
||||
metrics = body.get('metrics')
|
||||
if isinstance(metrics, dict):
|
||||
from app.services.server_metrics_service import ServerMetricsService
|
||||
ServerMetricsService.record_metrics(server.id, {
|
||||
**{k: metrics.get(k) for k in ('cpu_percent', 'memory_percent', 'disk_percent')},
|
||||
'extra': {k: metrics.get(k) for k in ('uptime_seconds', 'load1')},
|
||||
})
|
||||
return interval
|
||||
@@ -22,6 +22,7 @@ from collections import defaultdict
|
||||
|
||||
from app import db
|
||||
from app.models.server import Server, ServerCommand, AgentSession
|
||||
from app.services import permission_levels
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-IP auth rate limiter — transport-neutral state shared by the WS
|
||||
@@ -245,6 +246,12 @@ class AgentRegistry:
|
||||
self._agents.pop(server_id, None)
|
||||
self._socket_to_server.pop(socket_id, None)
|
||||
|
||||
# Report-only (monitor) agents hold no socket; sweep them too.
|
||||
if self._app is not None:
|
||||
with self._app.app_context():
|
||||
from app.services import agent_health_service
|
||||
agent_health_service.sweep_silent_reporters()
|
||||
|
||||
except Exception as e:
|
||||
logger.error("Error in heartbeat checker: %s", e)
|
||||
|
||||
@@ -279,6 +286,8 @@ class AgentRegistry:
|
||||
if server:
|
||||
server.status = 'offline'
|
||||
server.last_error = 'Agent heartbeat timeout'
|
||||
from app.services import agent_health_service
|
||||
agent_health_service.mark_offline(server, 'heartbeat timeout')
|
||||
db.session.commit()
|
||||
|
||||
# Mark session as inactive
|
||||
@@ -392,6 +401,8 @@ class AgentRegistry:
|
||||
# even for healthy connected agents.
|
||||
if ip_address and ip_address != 'unknown':
|
||||
server.ip_address = ip_address
|
||||
from app.services import agent_health_service
|
||||
agent_health_service.mark_online(server)
|
||||
db.session.commit()
|
||||
|
||||
# Create session record
|
||||
@@ -519,6 +530,8 @@ class AgentRegistry:
|
||||
server.last_seen = now
|
||||
if server.status != 'online':
|
||||
server.status = 'online'
|
||||
from app.services import agent_health_service
|
||||
agent_health_service.mark_online(server)
|
||||
db.session.commit()
|
||||
|
||||
# Update session latency
|
||||
@@ -589,6 +602,12 @@ class AgentRegistry:
|
||||
|
||||
# Check permissions
|
||||
server = Server.query.get(server_id)
|
||||
# Defence in depth: never send what the agent's reported level refuses.
|
||||
refused = permission_levels.refusal(
|
||||
server.permission_level if server else None, action)
|
||||
if refused:
|
||||
return {'success': False, 'error': refused, 'code': 'LEVEL_DENIED'}
|
||||
|
||||
if server and not server.has_permission(action):
|
||||
return {
|
||||
'success': False,
|
||||
@@ -863,13 +882,15 @@ class AgentRegistry:
|
||||
# in-memory state — the adapter must not report success.
|
||||
if not session_token:
|
||||
return None, server, 'registration_failed'
|
||||
self.record_agent_health(server.id, payload.get('permission_level'))
|
||||
|
||||
logger.info("Agent %s authenticated successfully from %s (%s)",
|
||||
agent_id, ip_address, transport)
|
||||
return session_token, server, None
|
||||
|
||||
def ingest_agent_state(self, agent, metrics=None, sysinfo=None,
|
||||
capabilities=None):
|
||||
capabilities=None, permission_level=None,
|
||||
service_checks=None):
|
||||
"""One ingest path for agent-reported state on BOTH transports.
|
||||
|
||||
Owns the capability side-effects (the WireGuard tunnel reconcile
|
||||
@@ -882,6 +903,20 @@ class AgentRegistry:
|
||||
self.update_system_info(agent.server_id, sysinfo)
|
||||
if capabilities:
|
||||
self.update_capabilities(agent.server_id, capabilities)
|
||||
if permission_level is not None or service_checks is not None:
|
||||
self.record_agent_health(agent.server_id, permission_level, service_checks)
|
||||
|
||||
def record_agent_health(self, server_id, permission_level=None, service_checks=None):
|
||||
from app.services import agent_health_service
|
||||
try:
|
||||
server = Server.query.get(server_id)
|
||||
if server:
|
||||
agent_health_service.record_level(server, permission_level)
|
||||
agent_health_service.record_service_checks(server, service_checks)
|
||||
db.session.commit()
|
||||
except Exception:
|
||||
logger.exception("Error recording agent health")
|
||||
db.session.rollback()
|
||||
|
||||
def verify_agent_auth(
|
||||
self,
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Agent permission levels and the one command-gate table.
|
||||
|
||||
The level is chosen on the server (``permissions.level`` in the agent's
|
||||
config.yaml) and REPORTED to the panel. The panel never sets it. This module
|
||||
is only the panel's defence in depth: it refuses to send a command the agent
|
||||
would refuse anyway. ``PROTECT_ACTIONS`` must stay identical to the agent's
|
||||
allowlist (docs/AGENT_PERMISSION_LEVELS.md; checked by a test).
|
||||
"""
|
||||
|
||||
LEVELS = ('monitor', 'protect', 'full')
|
||||
_RANK = {level: i for i, level in enumerate(LEVELS)}
|
||||
LABELS = {'monitor': 'Monitor', 'protect': 'Protect', 'full': 'Full'}
|
||||
|
||||
PROTECT_ACTIONS = frozenset({
|
||||
'firewall:allow_port', 'firewall:deny_port', 'firewall:status',
|
||||
'firewall:allow_ip', 'firewall:deny_ip',
|
||||
'fail2ban:status', 'fail2ban:ban', 'fail2ban:unban',
|
||||
'crowdsec:decisions:list', 'crowdsec:ban', 'crowdsec:unban',
|
||||
'cloudflare:origin_lock:status', 'cloudflare:origin_lock:apply',
|
||||
'cloudflare:origin_lock:remove',
|
||||
})
|
||||
|
||||
# Signature-verified key rotation is not control; allowed at every level.
|
||||
ALWAYS_ALLOWED = frozenset({'credential_update'})
|
||||
|
||||
|
||||
def normalize(level):
|
||||
"""Return a valid level string, or None for missing/unknown values."""
|
||||
level = level.strip().lower() if isinstance(level, str) else None
|
||||
return level if level in _RANK else None
|
||||
|
||||
|
||||
def effective_level(level):
|
||||
"""Unknown/legacy (None) means a pre-levels agent, i.e. full."""
|
||||
return normalize(level) or 'full'
|
||||
|
||||
|
||||
def is_escalation(old, new):
|
||||
return old is not None and _RANK[new] > _RANK[old]
|
||||
|
||||
|
||||
def refusal(level, action):
|
||||
"""Message explaining why ``action`` is refused at ``level``, else None."""
|
||||
level = effective_level(level)
|
||||
if level == 'full' or action in ALWAYS_ALLOWED:
|
||||
return None
|
||||
if level == 'protect' and action in PROTECT_ACTIONS:
|
||||
return None
|
||||
if level == 'monitor':
|
||||
return (f"server is in Monitor mode: '{action}' refused. Monitor agents "
|
||||
"are report-only and accept no commands. The level can only be "
|
||||
"changed on the server itself (serverkit-agent permissions set).")
|
||||
return (f"server is in Protect mode: '{action}' refused. Protect agents only "
|
||||
"run firewall, fail2ban, CrowdSec and Cloudflare origin-lock actions. "
|
||||
"The level can only be changed on the server itself "
|
||||
"(serverkit-agent permissions set).")
|
||||
@@ -52,6 +52,7 @@ _AUDIT_DETAIL_KEYS = frozenset({
|
||||
'application_id',
|
||||
'code',
|
||||
'endpoint',
|
||||
'escalation',
|
||||
'from',
|
||||
'kind',
|
||||
'label',
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
"""Agent-reported permission level, report bookkeeping and service checks."""
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = '112_agent_permission_levels'
|
||||
down_revision = '111_preview_runtime'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
SERVER_COLUMNS = [
|
||||
('permission_level', sa.String(16)),
|
||||
('permission_level_updated_at', sa.DateTime()),
|
||||
('requested_permission_level', sa.String(16)),
|
||||
('last_report_at', sa.DateTime()),
|
||||
('report_interval_seconds', sa.Integer()),
|
||||
('service_checks', sa.JSON()),
|
||||
]
|
||||
|
||||
|
||||
def upgrade():
|
||||
inspector = sa.inspect(op.get_bind())
|
||||
existing = {c['name'] for c in inspector.get_columns('servers')}
|
||||
for name, kind in SERVER_COLUMNS:
|
||||
if name not in existing:
|
||||
op.add_column('servers', sa.Column(name, kind, nullable=True))
|
||||
existing = {c['name'] for c in inspector.get_columns('metric_alerts')}
|
||||
if 'detail' not in existing:
|
||||
op.add_column('metric_alerts', sa.Column('detail', sa.Text(), nullable=True))
|
||||
|
||||
|
||||
def downgrade():
|
||||
with op.batch_alter_table('metric_alerts') as batch:
|
||||
batch.drop_column('detail')
|
||||
with op.batch_alter_table('servers') as batch:
|
||||
for name, _ in reversed(SERVER_COLUMNS):
|
||||
batch.drop_column(name)
|
||||
@@ -0,0 +1,80 @@
|
||||
"""Seed five demo servers showing every permission-level / health state.
|
||||
|
||||
Dev sandbox only; never run automatically. Idempotent: demo rows are keyed by
|
||||
agent_id 'demo-perm-*' and are rebuilt on each run.
|
||||
|
||||
cd backend && python scripts/seed_permission_demo.py
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from app import create_app, db # noqa: E402
|
||||
from app.models.metric_alert import MetricAlert # noqa: E402
|
||||
from app.models.server import Server, ServerMetrics # noqa: E402
|
||||
|
||||
UP, DOWN, DEG = 'up', 'down', 'degraded'
|
||||
|
||||
|
||||
def checks(now, web=UP, php=UP, database=UP, disk=UP):
|
||||
stamp = now.isoformat() + 'Z'
|
||||
rows = [('nginx', 'web', web, ''), ('php8.2-fpm', 'php_fpm', php, 'slow pool' if php == DEG else ''),
|
||||
('mysql', 'database', database, 'connection refused on 3306' if database == DOWN else ''),
|
||||
('disk /', 'disk', disk, ''), ('cpu', 'cpu', UP, ''), ('memory', 'memory', UP, '')]
|
||||
return [{'name': n, 'kind': k, 'status': s, 'detail': d, 'checked_at': stamp} for n, k, s, d in rows]
|
||||
|
||||
|
||||
# (name, level, status, minutes since last report, check overrides, cpu)
|
||||
DEMOS = [
|
||||
('demo-monitor-healthy', 'monitor', 'online', 1, {}, 14.0),
|
||||
('demo-monitor-degraded', 'monitor', 'online', 1, {'database': DOWN, 'php': DEG}, 71.0),
|
||||
('demo-protect-healthy', 'protect', 'online', 0, {}, 22.0),
|
||||
('demo-full-healthy', 'full', 'online', 0, {}, 35.0),
|
||||
('demo-monitor-offline', 'monitor', 'offline', 20, {'database': DOWN}, 9.0),
|
||||
]
|
||||
|
||||
|
||||
def seed():
|
||||
now = datetime.utcnow()
|
||||
for name, level, status, age, overrides, cpu in DEMOS:
|
||||
agent_id = f'demo-perm-{name}'
|
||||
old = Server.query.filter_by(agent_id=agent_id).first()
|
||||
if old:
|
||||
MetricAlert.query.filter_by(server_id=old.id).delete()
|
||||
ServerMetrics.query.filter_by(server_id=old.id).delete()
|
||||
db.session.delete(old)
|
||||
db.session.flush()
|
||||
reported = now - timedelta(minutes=age)
|
||||
srv = Server(name=name, agent_id=agent_id, hostname=name, status=status,
|
||||
os_type='linux', permission_level=level, requested_permission_level=level,
|
||||
permission_level_updated_at=reported, last_seen=reported,
|
||||
last_report_at=reported, report_interval_seconds=60,
|
||||
service_checks=checks(reported, **overrides))
|
||||
db.session.add(srv)
|
||||
db.session.flush()
|
||||
for i in range(12):
|
||||
db.session.add(ServerMetrics(
|
||||
server_id=srv.id, timestamp=reported - timedelta(minutes=i),
|
||||
cpu_percent=cpu + i % 3, memory_percent=48.0, disk_percent=61.0))
|
||||
for check in srv.service_checks:
|
||||
if check['status'] == DOWN:
|
||||
db.session.add(MetricAlert(
|
||||
server_id=srv.id, metric=f"svc:{check['name']}"[:20], severity='critical',
|
||||
status='active', created_at=reported,
|
||||
detail=f"{name}: service '{check['name']}' is down since "
|
||||
f"{reported:%Y-%m-%d %H:%M} UTC ({check['detail']})"))
|
||||
if status == 'offline':
|
||||
db.session.add(MetricAlert(
|
||||
server_id=srv.id, metric='offline', severity='critical', status='active',
|
||||
created_at=now, detail=f"{name} is offline since {reported:%Y-%m-%d %H:%M} UTC: "
|
||||
"no report received"))
|
||||
db.session.commit()
|
||||
print(f'Seeded {len(DEMOS)} demo servers.')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
app = create_app(os.environ.get('FLASK_ENV') or 'development')
|
||||
with app.app_context():
|
||||
seed()
|
||||
@@ -0,0 +1,208 @@
|
||||
"""Agent permission levels: report endpoint, level reporting, dispatch gate,
|
||||
silence sweep, service-check alerts and install commands."""
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import re
|
||||
import secrets
|
||||
import time
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app import db
|
||||
from app.models.audit_log import AuditLog
|
||||
from app.models.metric_alert import MetricAlert
|
||||
from app.models.server import Server
|
||||
from app.services import agent_health_service, permission_levels
|
||||
from app.services.agent_registry import ConnectedAgent, agent_registry
|
||||
|
||||
REPORT = '/api/v1/agent/report'
|
||||
SECRET = 'test-secret'
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def server(app, monkeypatch):
|
||||
monkeypatch.setattr(Server, 'get_api_secret', lambda self: SECRET)
|
||||
key, _ = Server.generate_api_credentials()
|
||||
srv = Server(name='web-1', agent_id=secrets.token_hex(8))
|
||||
srv.set_api_key(key)
|
||||
db.session.add(srv)
|
||||
db.session.commit()
|
||||
srv.test_key = key
|
||||
return srv
|
||||
|
||||
|
||||
def _post(client, srv, body=None, ts=None, nonce=None, secret=SECRET, raw=None):
|
||||
body = body if body is not None else {
|
||||
'permission_level': 'monitor', 'interval_seconds': 60,
|
||||
'metrics': {'cpu_percent': 12.5, 'memory_percent': 40.0, 'disk_percent': 55.0,
|
||||
'uptime_seconds': 99, 'load1': 0.3},
|
||||
'service_checks': []}
|
||||
raw = raw if raw is not None else json.dumps(body).encode()
|
||||
ts = str(int(time.time())) if ts is None else str(ts)
|
||||
nonce = nonce or secrets.token_hex(16)
|
||||
sig = hmac.new(secret.encode(),
|
||||
f'{ts}.{nonce}.{hashlib.sha256(raw).hexdigest()}'.encode(),
|
||||
hashlib.sha256).hexdigest()
|
||||
return client.post(REPORT, data=raw, content_type='application/json', headers={
|
||||
'X-Agent-Key': srv.test_key, 'X-Timestamp': ts, 'X-Nonce': nonce,
|
||||
'X-Signature': sig})
|
||||
|
||||
|
||||
def test_good_signature_updates_server(client, server):
|
||||
resp = _post(client, server)
|
||||
assert resp.status_code == 200
|
||||
assert resp.get_json() == {'ok': True, 'interval_seconds': 60}
|
||||
srv = db.session.get(Server, server.id)
|
||||
assert srv.status == 'online' and srv.last_report_at is not None
|
||||
assert srv.permission_level == 'monitor'
|
||||
assert 'commands' not in resp.get_json()
|
||||
|
||||
|
||||
def test_bad_signature_rejected(client, server):
|
||||
assert _post(client, server, secret='wrong').status_code == 401
|
||||
|
||||
|
||||
def test_stale_timestamp_rejected(client, server):
|
||||
assert _post(client, server, ts=int(time.time()) - 400).status_code == 401
|
||||
|
||||
|
||||
def test_replayed_nonce_rejected(client, server):
|
||||
nonce = secrets.token_hex(16)
|
||||
assert _post(client, server, nonce=nonce).status_code == 200
|
||||
assert _post(client, server, nonce=nonce).status_code == 401
|
||||
|
||||
|
||||
def test_oversize_body_rejected(client, server):
|
||||
assert _post(client, server, raw=b'{"x":"' + b'a' * 70000 + b'"}').status_code == 413
|
||||
|
||||
|
||||
def test_invalid_level_rejected(client, server):
|
||||
assert _post(client, server, body={'permission_level': 'root'}).status_code == 400
|
||||
|
||||
|
||||
def test_level_persisted_and_escalation_audited(client, server):
|
||||
_post(client, server, body={'permission_level': 'monitor'})
|
||||
_post(client, server, body={'permission_level': 'full'})
|
||||
rows = AuditLog.query.filter_by(action='server.permission_level_changed').order_by(AuditLog.id).all()
|
||||
details = [r.get_details() for r in rows]
|
||||
assert [(d['from'], d['to'], d['escalation']) for d in details] == [
|
||||
(None, 'monitor', False), ('monitor', 'full', True)]
|
||||
assert db.session.get(Server, server.id).permission_level == 'full'
|
||||
|
||||
|
||||
def test_no_api_sets_level(client, server, auth_headers):
|
||||
resp = client.put(f'/api/v1/servers/{server.id}', headers=auth_headers,
|
||||
json={'permission_level': 'full', 'name': 'x'})
|
||||
assert resp.status_code in (200, 400)
|
||||
assert db.session.get(Server, server.id).permission_level is None
|
||||
writers = [r.rule for r in client.application.url_map.iter_rules()
|
||||
if 'level' in r.rule and r.methods & {'POST', 'PUT', 'PATCH'}]
|
||||
assert writers == []
|
||||
|
||||
|
||||
def _connect(server_id):
|
||||
agent_registry._agents[server_id] = ConnectedAgent(
|
||||
server_id=server_id, socket_id='s', session_token='t',
|
||||
connected_at=datetime.utcnow(), last_heartbeat=datetime.utcnow(),
|
||||
ip_address='1.1.1.1', agent_version='1', transport='poll')
|
||||
|
||||
|
||||
@pytest.mark.parametrize('level,action,refused', [
|
||||
('monitor', 'firewall:status', True),
|
||||
('monitor', 'docker:container:list', True),
|
||||
('protect', 'docker:container:list', True),
|
||||
('protect', 'system:exec', True),
|
||||
('protect', 'firewall:status', False),
|
||||
('protect', 'cloudflare:origin_lock:apply', False),
|
||||
('protect', 'credential_update', False),
|
||||
('full', 'system:exec', False),
|
||||
(None, 'system:exec', False),
|
||||
])
|
||||
def test_refusal_table(level, action, refused):
|
||||
assert bool(permission_levels.refusal(level, action)) is refused
|
||||
|
||||
|
||||
@pytest.mark.parametrize('level', ['monitor', 'protect'])
|
||||
def test_send_command_gate(server, level):
|
||||
server.permission_level = level
|
||||
db.session.commit()
|
||||
_connect(server.id)
|
||||
try:
|
||||
result = agent_registry.send_command(server.id, 'system:exec', {})
|
||||
finally:
|
||||
agent_registry._agents.pop(server.id, None)
|
||||
assert result['code'] == 'LEVEL_DENIED'
|
||||
assert result['error'].startswith(f'server is in {level.title()} mode')
|
||||
|
||||
|
||||
def test_protect_allowlist_matches_documented_contract():
|
||||
doc = Path(__file__).parents[2] / 'docs/AGENT_PERMISSION_LEVELS.md'
|
||||
block = re.search(r'```text allowlist\n(.*?)```', doc.read_text(), re.S).group(1)
|
||||
assert set(block.split()) == set(permission_levels.PROTECT_ACTIONS)
|
||||
|
||||
|
||||
def test_silent_monitor_goes_offline_with_one_alert(server):
|
||||
server.permission_level = 'monitor'
|
||||
server.status = 'online'
|
||||
server.last_report_at = datetime.utcnow() - timedelta(minutes=20)
|
||||
db.session.commit()
|
||||
assert agent_health_service.sweep_silent_reporters() == [server.id]
|
||||
assert agent_health_service.sweep_silent_reporters() == []
|
||||
assert db.session.get(Server, server.id).status == 'offline'
|
||||
assert MetricAlert.query.filter_by(metric='offline', status='active').count() == 1
|
||||
|
||||
|
||||
def test_recent_monitor_stays_online(server):
|
||||
server.permission_level, server.status = 'monitor', 'online'
|
||||
server.last_report_at = datetime.utcnow() - timedelta(minutes=2)
|
||||
db.session.commit()
|
||||
assert agent_health_service.sweep_silent_reporters() == []
|
||||
|
||||
|
||||
def _checks(status):
|
||||
return {'permission_level': 'monitor', 'service_checks': [
|
||||
{'name': 'mysql', 'kind': 'database', 'status': status, 'detail': 'refused',
|
||||
'checked_at': '2026-01-01T00:00:00Z'}]}
|
||||
|
||||
|
||||
def test_service_down_alert_dedupes_and_resolves(client, server):
|
||||
_post(client, server, body=_checks('up'))
|
||||
assert MetricAlert.query.count() == 0
|
||||
_post(client, server, body=_checks('down'))
|
||||
_post(client, server, body=_checks('down'))
|
||||
alert = MetricAlert.query.one()
|
||||
assert alert.status == 'active' and 'mysql' in alert.detail and 'web-1' in alert.detail
|
||||
_post(client, server, body=_checks('up'))
|
||||
assert MetricAlert.query.one().status == 'resolved'
|
||||
|
||||
|
||||
def test_degraded_to_down_alerts_and_report_recovers_offline(client, server):
|
||||
_post(client, server, body=_checks('degraded'))
|
||||
assert MetricAlert.query.count() == 0
|
||||
_post(client, server, body=_checks('down'))
|
||||
assert MetricAlert.query.filter_by(status='active').count() == 1
|
||||
agent_health_service.mark_offline(server, 'test')
|
||||
db.session.commit()
|
||||
server.status = 'offline'
|
||||
db.session.commit()
|
||||
_post(client, server, body=_checks('down'))
|
||||
assert MetricAlert.query.filter_by(metric='offline', status='resolved').count() == 1
|
||||
|
||||
|
||||
def test_install_instructions_include_level(client, server, auth_headers):
|
||||
server.set_registration_token('sk_reg_x')
|
||||
db.session.commit()
|
||||
data = client.get(f'/api/v1/servers/install-instructions/{server.id}?level=protect',
|
||||
headers=auth_headers).get_json()
|
||||
assert data['permission_level'] == 'protect'
|
||||
assert '--level protect' in data['linux']['one_liner']
|
||||
assert '-Level protect' in data['windows']['one_liner']
|
||||
|
||||
|
||||
def test_token_records_requested_level(client, server, auth_headers):
|
||||
client.post(f'/api/v1/servers/{server.id}/regenerate-token', headers=auth_headers,
|
||||
json={'requested_level': 'protect'})
|
||||
assert db.session.get(Server, server.id).requested_permission_level == 'protect'
|
||||
@@ -7,9 +7,9 @@ from pathlib import Path
|
||||
from app.core_blueprints import CORE_BLUEPRINTS
|
||||
|
||||
|
||||
EXPECTED_CORE_BLUEPRINT_COUNT = 116
|
||||
EXPECTED_CORE_BLUEPRINT_COUNT = 117
|
||||
EXPECTED_MANIFEST_SHA256 = (
|
||||
'25d28d92ccddddc246d5b2081d84b11f1df10cb662f33c256ebcbe390d2813d5'
|
||||
'52abbc7429d079c7cf6ea61a638f5dc4dbd991e4f02ec31c81563e01442aa7fb'
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -145,6 +145,7 @@ PUBLIC_ALT_AUTH = {
|
||||
# Public by design; the value names the mechanism that authenticates the call.
|
||||
'agent_poll.connect': 'agent HMAC handshake + per-IP throttle',
|
||||
'agent_poll.disconnect': 'X-Session-Token lookup',
|
||||
'agent_report.report': 'HMAC over timestamp+nonce+body hash, skew/replay/rate limits',
|
||||
'agent_poll.poll': 'X-Session-Token resolves the ConnectedAgent',
|
||||
'agent_poll.result': 'X-Session-Token resolves the ConnectedAgent',
|
||||
'auth.connect_session': 'loopback-only + Cloud grant verified against JWKS, 30/min',
|
||||
|
||||
@@ -221,6 +221,7 @@ VIEWER_WRITABLE = {
|
||||
'docker.get_containers_stats',
|
||||
# PUBLIC transport — agent long-poll fallback; user JWT is ignored here.
|
||||
'agent_poll.disconnect',
|
||||
'agent_report.report',
|
||||
# PUBLIC authentication challenge only; credential verification and UV are
|
||||
# required by passkey_authenticate before any session tokens are issued.
|
||||
'auth.passkey_auth_options',
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
from app.models.metric_alert import MetricAlert
|
||||
from app.models.server import Server
|
||||
|
||||
|
||||
def test_seed_is_idempotent(app):
|
||||
path = Path(__file__).parents[1] / 'scripts/seed_permission_demo.py'
|
||||
spec = importlib.util.spec_from_file_location('seed_demo', path)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
mod.seed()
|
||||
mod.seed()
|
||||
assert Server.query.filter(Server.agent_id.like('demo-perm-%')).count() == 5
|
||||
assert Server.query.filter_by(status='offline').count() == 1
|
||||
assert MetricAlert.query.filter_by(metric='offline').count() == 1
|
||||
@@ -0,0 +1,81 @@
|
||||
# Agent permission levels
|
||||
|
||||
Each ServerKit agent runs at one of three levels. The level is chosen **on the
|
||||
server**, enforced **by the agent**, and merely *reported* to the panel.
|
||||
|
||||
| Level | What it can do | What it cannot do |
|
||||
|-------|----------------|-------------------|
|
||||
| **Monitor** (default for new installs) | Send CPU / memory / disk / uptime / load and service checks (web server, PHP-FPM, database, disk, CPU, memory) to the panel with signed outbound HTTPS POSTs. Opens no inbound command channel. | Run any command, open a terminal, touch files, Docker, packages or services. The panel cannot send it anything. |
|
||||
| **Protect** | Everything in Monitor, plus a fixed security allowlist: firewall (ufw) rules, fail2ban, CrowdSec decisions, Cloudflare origin lock (restrict 80/443 to Cloudflare ranges). | Docker, shell, files, terminal, packages, systemd, cron, SSH keys, WireGuard, runtimes, agent updates. |
|
||||
| **Full** | Today's behaviour: the whole agent feature set. Legacy installs with no level configured are treated as Full. | — |
|
||||
|
||||
## Install
|
||||
|
||||
The Add Server drawer shows the level picker and puts the choice in the command:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://panel.example.com/api/v1/servers/install.sh -o /tmp/serverkit-agent-install.sh \
|
||||
&& sudo bash /tmp/serverkit-agent-install.sh --token "TOKEN" --server "https://panel.example.com" --level monitor
|
||||
```
|
||||
|
||||
```powershell
|
||||
irm https://panel.example.com/api/v1/servers/install.ps1 | iex; Install-ServerKitAgent -Token "TOKEN" -Server "https://panel.example.com" -Level monitor
|
||||
```
|
||||
|
||||
The installer scripts under `scripts/` are vendored from the agent repo
|
||||
(`scripts/sync-agent-installers.sh`); `--level` / `-Level` arrives with the next sync.
|
||||
|
||||
## Changing the level
|
||||
|
||||
Only on the server itself, as root, with a typed confirmation:
|
||||
|
||||
```bash
|
||||
serverkit-agent permissions show
|
||||
sudo serverkit-agent permissions set protect
|
||||
```
|
||||
|
||||
There is no panel button or API for this, by design. Re-running `install.sh` with a
|
||||
different `--level` on an installed box refuses unless `--reconfigure` is passed.
|
||||
|
||||
## What the panel does
|
||||
|
||||
- Records the level the agent reports (registration, auth, heartbeat, report). Every
|
||||
change, escalations included, is written to the audit log and shown in the server
|
||||
timeline. The Add Server flow remembers the *requested* level so a mismatch with
|
||||
the reported one is visible.
|
||||
- Monitor agents POST to `/api/v1/agent/report` (HMAC over timestamp, nonce and body
|
||||
hash; 300 s skew; single-use nonces; 64 KB cap; per-IP+key rate limit). A monitor
|
||||
server with no report for max(3 minutes, 3 x its interval) is marked offline.
|
||||
- Opens an alert (existing fleet alerts + notification channels) when a server goes
|
||||
offline or stops reporting and when a service check goes down; it resolves on
|
||||
recovery and does not repeat while the condition persists.
|
||||
- Refuses to dispatch a command the reported level forbids (`LEVEL_DENIED`,
|
||||
"server is in Monitor mode: ..."). This is defence in depth; the agent enforces
|
||||
the same table. The table lives in `backend/app/services/permission_levels.py`.
|
||||
|
||||
Protect allowlist (checked against the panel table by a test):
|
||||
|
||||
```text allowlist
|
||||
firewall:allow_port firewall:deny_port firewall:status
|
||||
firewall:allow_ip firewall:deny_ip
|
||||
fail2ban:status fail2ban:ban fail2ban:unban
|
||||
crowdsec:decisions:list crowdsec:ban crowdsec:unban
|
||||
cloudflare:origin_lock:status cloudflare:origin_lock:apply cloudflare:origin_lock:remove
|
||||
```
|
||||
|
||||
`credential_update` (signature-verified key rotation) is allowed at every level.
|
||||
|
||||
## Threat model
|
||||
|
||||
The level is read only from the agent's local config at startup; no message from
|
||||
the panel can change it. A compromised panel therefore **cannot escalate** a Monitor
|
||||
agent: it has no channel to send commands to, and the report reply never carries any.
|
||||
The panel's own gate only narrows what it will send. Someone with root on the box
|
||||
can still change the level, which is the intended re-confirmation path, and the
|
||||
panel audits the resulting change. A compromised panel can lie in the UI, but
|
||||
cannot make a Monitor agent act.
|
||||
|
||||
## Demo data
|
||||
|
||||
`python backend/scripts/seed_permission_demo.py` (never run automatically) creates five
|
||||
demo servers for the dev sandbox. It is idempotent.
|
||||
@@ -755,6 +755,7 @@ Regenerate (backend/):
|
||||
- `POST /agent/connect`
|
||||
- `POST /agent/disconnect`
|
||||
- `POST /agent/poll`
|
||||
- `POST /agent/report`
|
||||
- `POST /agent/result`
|
||||
- `POST /ai/chat`
|
||||
- `POST /ai/chat/cancel`
|
||||
|
||||
@@ -16,7 +16,7 @@ debt. Regenerate with `python scripts/generate-migration-inventory.py`.
|
||||
| Routes on bare @jwt_required() | auth_required() / role decorators | 598 | 598 | REGISTERED EXCEPTION: JWT-only is the deliberate default; conversion grants API-key access and happens per route, on decision |
|
||||
| HTTP statuses chosen by sniffing error text | typed errors from app.exceptions | 0 | 0 | INVARIANT at 0 - migration completed 2026-08-19 |
|
||||
| API crashes swallowed without recording | app.error_reporting | 0 | 0 | INVARIANT at 0 |
|
||||
| Hand-shaped {'error': ...} bodies in app/api | typed errors + the global handler | 849 | 850 | migrate when touched; new endpoints raise |
|
||||
| Hand-shaped {'error': ...} bodies in app/api | typed errors + the global handler | 855 | 850 | migrate when touched; new endpoints raise |
|
||||
| Raw subprocess calls outside the runners | app/utils/system.py runners | 23 | 23 | migrate when touched |
|
||||
| Controller-boundary violations (routes doing service work) | service layer extraction | 453 | 453 | migrate when touched (first-wave ratchet) |
|
||||
| raw api.* calls in pages/ | E1: useServerQuery/useServerMutation | 374 | 398 | migrate when touched |
|
||||
|
||||
@@ -11,6 +11,7 @@ Welcome to the ServerKit documentation. This guide will help you install, config
|
||||
| [Video: template to live app](https://youtu.be/TXfXFz6bVjs) | Watch a 4-minute AgentSite deployment through the template catalog and live Deploy Console |
|
||||
| [Video: deploy a Git repository](https://youtu.be/Fv28qMERYDU) | Connect a repository, review detected settings and deploy through the three-step New Service wizard |
|
||||
| [Installation Guide](INSTALLATION.md) | Complete setup instructions for Docker and manual installation |
|
||||
| [Agent permission levels](AGENT_PERMISSION_LEVELS.md) | Monitor / Protect / Full: what each allows and how to change it |
|
||||
| [Architecture](ARCHITECTURE.md) | System design, request flow, and diagrams |
|
||||
| [Deployment Guide](DEPLOYMENT.md) | Production deployment, CLI commands, and configuration |
|
||||
| [Local Development](LOCAL_DEVELOPMENT.md) | Run the panel locally for development |
|
||||
|
||||
Reference in new issue
Block a user