ServerKit/docs/AI_PROVIDERS.md

AI provider connections

In Settings → AI Assistant, add a named connection, choose a provider, fill in its configuration, and choose a default model. Discover models uses the current form values. Browse models opens a searchable picker; manual model entry remains available for gateways and providers without discovery. Save connection stores credentials encrypted. The first connection becomes the default automatically; another connection can be made default when saving it. The Enable AI assistant switch saves immediately and refreshes the chat UI. Disabling blocks new messages in both chat endpoints while retaining conversation history. Limits and protections have their own Save action; saving a connection preserves unsaved changes to those settings.

New chats offer a task, model role, and optional connection/model override. The selected connection and model are saved with the conversation. Changing the panel default does not reroute existing chats. Configured fallback or budget degradation may switch the saved binding before a turn produces output. You can rotate a connection's credentials, but changing a provider or endpoint used by existing chats requires a new connection. Delete those chats and remove task/routing/fallback references before deleting their connection, and select another default before deleting the current default.

Native providers and gateways

The catalog comes from the installed Prompture chat-provider descriptors, with aliases deduplicated. Anthropic/Claude, OpenAI, Google, Azure, Bedrock, Mistral, and other registered chat drivers appear together. Providers needing optional Python dependencies are labeled in the selector. The backend requirements include the OpenAI and Anthropic SDKs; other providers may require their SDKs to be installed in the same backend environment before they become available.

For OmniRoute, select OpenAI-Compatible, set the API base URL to your backend-reachable OmniRoute address (for example http://omniroute:20128/v1 on a shared Docker network), and enter the gateway key if authentication is enabled. Enter the exact model ID or routing alias from that instance. Slashes are preserved; ServerKit does not add an upstream-provider prefix. The endpoint preset selector also exposes Prompture's curated compatible-provider URLs. Prompture Hub has its own friendly entry and uses the same connection mechanism.

localhost means the machine or container running the ServerKit backend, not the browser. Endpoints accept HTTP or HTTPS but cannot embed credentials, query strings, or fragments. When changing an endpoint, re-enter credentials; ServerKit does not forward the previous endpoint's saved key to the new address.

Testing and capabilities

For compatible endpoints and OpenAI, Test connection makes a small chat completion request, which may consume tokens. For other native providers, it verifies model discovery and checks that the requested model is returned. An unsupported or failed discovery reports an error, never a successful connection based only on constructing a driver. Discovery does not prove streaming or tool support; these depend on the provider and model. Error responses omit raw SDK errors and credentials.

ServerKit pins Prompture 1.11.0 and uses its native OpenAI-compatible driver for custom gateways, Prompture Hub, and LM Studio. That driver handles text streaming, streamed tool calls, and ordinary tool calls without the OpenAI SDK. Explicit model IDs are preserved, and unauthenticated gateways omit the Authorization header even if a process-level compatible API key is set. Native Anthropic's installed driver does not accept an endpoint argument: ServerKit refuses a conflicting process-level ANTHROPIC_BASE_URL override instead of silently using it. Vertex Gemini requires an explicit API key; Vertex Claude requires a project, location, and access token. Ambient cloud credentials are not a connection option.

Existing assistant RBAC, redaction, and write-action confirmations remain active. The picker includes catalog prices per million input/output tokens, context and output limits, and tools, vision, reasoning, and structured-output capabilities. Capability filters show only confirmed support. Missing metadata stays unknown. Gateway aliases are not assigned an upstream model's price or capabilities. Discovering a saved connection caches its nonsecret catalog; changing its configuration clears that cache. Manual IDs remain supported.

Task models and execution

Task models and behavior assigns a saved connection and exact model ID to Utility, Standard, and Advanced. Standard inherits the panel default; Utility and Advanced inherit Standard when unset. The UI shows the inherited binding. Prompture's ModelResolver resolves these slots before execution.

Task	Default role	Runtime behavior
Chat	Standard	Existing assistant tools and confirmations, or simple text mode
Summarize	Utility	Summarizes supplied text and attached context without tools
Extract structured facts	Utility	Prompture validates an object with summary, facts, and warnings; no tools
Read-only diagnosis	Advanced	Only tools marked read-only and authorized for the caller; bounded tool rounds

An explicit connection/model override wins, followed by an explicit role. Without either, enabled automatic routing uses the approved pool; otherwise the task's role is used. Existing conversations retain their task and binding. Advanced models never grant additional permissions.

Routing uses Prompture's cost-optimized, balanced, quality-first, or fast strategy, with discovery restricted to administrator-approved saved bindings. Candidate capability tiers guide its heuristics; “fast” is not a measured latency guarantee. The routing preview runs against the unsaved draft without making a model call. Each run records the selection strategy and explanation.

Fallback is opt-in and tries the ordered list with each candidate's own endpoint and credentials. It only runs before any provider output has been delivered. The unused legacy fallback_models settings field is rejected; fallback candidates must include a saved connection ID in /ai/management. Once output or a tool response starts, a failure ends that turn; actions are never replayed on another connection. The adapter avoids Prompture's ambient credential fallback path. The actual selected model appears in run reporting.

Generation and spending controls

Generation controls are applied to actual conversations: maximum output tokens per response, lifetime conversation tokens, history messages, tool rounds, and tool-result characters. Temperature is forwarded only with catalog-confirmed support; reasoning effort is forwarded only to supported OpenAI reasoning models. Unsupported settings retain provider defaults. Arbitrary driver JSON and detached tool timeouts are not exposed.

The conversation cost ceiling defaults to $0.50; zero disables it. Select:

• Stop at the limit: check cumulative cost/tokens before every provider call.
• Warn and continue: retain usage and show the exceeded allowance in chat.
• Use a cheaper model near the limit, then stop: at 80% of the cost allowance, switch before a turn to the first configured fallback with lower known combined input/output rates, then enforce the hard ceiling. Unknown prices are not compared.

Monthly allowances can apply panel-wide, per user, and per workspace. Workspace attribution is captured from the active workspace when a chat starts and access is rechecked on resume. All-workspace chats still use panel/user allowances. Reservations prevent concurrent runs from all consuming the same remaining allowance in ServerKit's single worker. Each run stops further calls once its reservation is spent. Unknown-cost attempts conservatively consume that reservation. A restarted worker marks abandoned runs interrupted and retains their conservative charge.

Limits use reported/estimated usage; a final call can overshoot. Unknown gateway pricing cannot enforce a billing ceiling. Use provider/gateway limits for authoritative spending enforcement.

Usage and plugin integration

Each response shows its actual model, per-run tokens and cost source, duration, provider calls/errors, routing reason, and remaining conversation allowance. Reloading history retains those run details. AI usage provides administrator reports by period, role, model, connection, user, and workspace, with daily breakdowns, remaining monthly allowances, and a drawer for provider attempts. Unknown/partial pricing is labeled separately from known cost.

Prompture UsageSession collects each actual driver call; its cost calculation and BudgetState checks are reused. ServerKit stores one migration-backed AiRun per turn, with idempotent completion. Cumulative Conversation snapshots remain for lifetime limits and are never summed as per-run costs. Run records survive transcript deletion. Reporting starts with this migration; older transcripts are not backfilled with invented per-call costs.

The plugin SDK ask() and ask_stream() use the same enable state, role/routing policy, guardrails, limits, driver adapter, and ledger without creating chat transcripts. Both accept optional profile and workflow arguments:

from app.plugins_sdk import ai

summary = ai.ask(text, workflow='summarize')
facts = ai.ask(text, workflow='extract', profile='utility')

The implementation was checked against local Prompture d36872b and AgentSite db323d4. AgentSite informed the model-picker/reporting interactions. Prompture 1.11.0 already supplies the relevant resolver, router, structured output, budget, and usage machinery. ServerKit supplies credential binding, authorization, scope-filtered accounting, persistence, and UI. In this SDK version, router preferences are not an allowlist and tracker budget scopes do not filter usage by tenant; ServerKit explicitly enforces both boundaries. No upstream issue or external message was sent.

Upgrade behavior

Migration 098_ai_provider_connections creates encrypted connection records, copies the existing single-provider configuration into Existing connection, and attaches matching legacy conversations to it. Older conversations from a different provider cannot borrow the current connection's credentials: start a new chat using an explicitly configured connection. Resumed chats reconstruct their Prompture driver with their saved connection, and restore history and usage without exporting credentials or using global provider configuration.

Migration 099_ai_management adds cached model metadata, conversation task snapshots, and the durable run ledger. Apply normal Alembic upgrades before starting the updated backend. No SDK upgrade or manual schema creation is needed.