Skip to content
Kernel Service

Model Routing

Kernel Service uses inexpensive execution where appropriate and stronger models for complex reasoning. The model list, rates, runtime controls and call-type tier overrides are maintained in config/models.yaml.

Default routing

The cheap tier starts with Grok 4.1 Fast; the expensive tier starts with Kimi K3, followed by Grok 4.6 and Claude Opus 5. Role defaults and existing call-type tier overrides still apply. Discovery refreshes available families and preserves configured model choices.

GPT-6 Astra is marked opt_in. Default routing and an OpenAI provider preference do not include it. An administrator must explicitly select its model ID, either as the primary choice or as a named fallback. Exact configured model choices can cross tier boundaries; the original tier’s fallback chain remains available.

Administrator model allocation

Use Kernel Service Admin → Model Routing, choose the organisation and agent, then configure Task execution and Final synthesis independently. Changes apply to the next task. Tenant agent pages do not expose these controls. The API requires InternalAdminDep; a tenant owner or an operator-looking email address does not grant access.

Policies are stored in agent_model_routing, outside tenant-editable agent JSON. Legacy llm_override entries are copied once during migration. After migration they no longer influence runtime selection. Tenant configuration writes and agent tools cannot change model-routing keys.

Migration rollback restores the execution selection into the legacy override. The old schema cannot retain a separate synthesis selection or the new generation and budget controls.

Internal endpoints:

  • GET /api/v1/internal/model-routing/models
  • GET /api/v1/internal/tenants/{tenant_id}/model-routing
  • PUT /api/v1/internal/tenants/{tenant_id}/model-routing/{agent_id}

Example policy, set through the authenticated internal admin endpoint:

{
  "loop": {"model": "kimi-k3"},
  "synthesis": {
    "model": "gpt-6-astra",
    "reasoning_effort": "low",
    "max_output_tokens": 16384,
    "timeout_seconds": 180
  },
  "max_task_cost_cents": 100
}

A null selection restores default routing for that phase. A synthesis selection triggers the final synthesis pass after tool work even if execution and synthesis use the same tier. Callers that intentionally discard prose still suppress synthesis.

The Efficiency controls section sets prompt-cache policy, reference-read reuse, business-context rendering reuse, execution and result token budgets, focused/all tool schemas, and adaptive/always synthesis. These settings live in the same admin-only policy table. Defaults enable the efficiency features described below.

Premium policies default to a USD $1 task budget. The stricter of the admin budget and task constraint applies. This is checked between calls, including before synthesis; a single call can exceed the remaining budget. Output-token and timeout limits provide additional controls. It is not a guaranteed provider billing cap.

GPT-6 adapter

GPT-6 uses the Responses API with store=false. Native function-call IDs, tool results, assistant phase and encrypted reasoning items are replayed within the same model. Private provider state is removed for other adapters. Existing tool execution and approval gates still run in Kernel Service. Incomplete Responses output is never dispatched as a tool action.

Existing providers keep their adapters. Model invocation errors use the existing fallback chain. Successful Responses models are preferred on subsequent execution calls to preserve native reasoning replay. Other adapters return to the configured order once cooldowns expire, so a temporary failure does not keep a task on a more expensive fallback. Synthesis has its own independent routing. Router instances cool down exhausted-quota/401 endpoints for 120 seconds, models returning 403 for 120 seconds, rate-limited models for 30 seconds, and transient server/timeout failures for 15 seconds. Invalid-request errors do not disable other requests. If every candidate is cooling down, the router attempts the ordered chain again. Cooldowns are local to the router instance; they are not a distributed availability service. There is no automatic quality-based escalation or production experiment in this release.

Prompt caching

The prompt has up to three explicit prefix boundaries: platform/persona instructions, brand/writing guidance, and playbooks/business reference context. Product catalogues and published strategy precede timestamps, user details, changing memories and task data. A change to a later layer preserves earlier eligible boundaries. Tool definitions and schema object keys are sorted; semantically ordered arrays retain their order. Recalled data retains its untrusted-content framing.

Anthropic gets up to three system breakpoints (including the preceding tool definitions) and a fourth history breakpoint only for tool loops. Astra and GPT-5.6 Chat Completions get the same explicit layers and implicit growing-history caching for execution with tools. Single-pass synthesis uses only explicit prefixes, avoiding writes of one-off evidence. The admin can restrict execution to prefix-only caching or disable controllable caching. Other providers may still cache automatically.

OpenAI cache keys are scoped by tenant, agent and call type. Grok receives x-grok-conv-id, scoped to tenant, agent, conversation/task and phase. Cache keys help routing; they are not permission boundaries or guarantees of hits. Tool changes can invalidate subsequent system prefixes. Caches do not transfer between models or providers. See xAI cache routing.

Astra uses a 30-minute TTL and requires at least 1,024 cacheable tokens. Its standard rates are USD $10/M uncached input, $1/M cache reads, $12.50/M writes and $50/M output. Above 272K input tokens, input/cache rates double and output increases 50%. See OpenAI prompt caching and Astra pricing.

Execution efficiency

  • Reference reads: Only the explicitly declared brand, product and segment get/search tools are cached initially, for up to 60 seconds within one task. Each reuse still resolves the registered tool and checks its permitted tool list and approval requirements. Returned values are copied. Writes and undeclared operations clear reference results before execution; force_refresh bypasses reuse. Live inbox/status operations, failed results and approval receipts are not cached. Full write idempotency is separate from read caching and remains outside this change.
  • Context budget: Each result normally contributes at most 3,000 budgeting tokens; original outputs remain in the step audit. kernel.read_task_evidence retrieves omitted content using task-local IDs and paginated token offsets. Once execution exceeds 48,000 estimated tokens, older completed rounds become a factual ledger. The original objective, system prefix, tool schemas and newest tool-call/result pairs remain intact; result previews can shrink further while retaining full-evidence references. There is no repeated rewriting below the threshold. If the initial prompt alone exceeds the budget, the first model call proceeds with a warning; subsequent calls that still cannot fit stop explicitly as failed with context_budget_exceeded. Token estimates include reported replayed reasoning; image sizes use an approximation and the limit is not a provider-token guarantee.
  • Focused tools: For more than 40 permitted tools, execution starts with 24 ranked schemas plus any exact tool names mentioned in the objective. The remaining permitted tools stay available through kernel.find_task_tools and kernel.call_task_tool. Schemas remain fixed during the task. Delegated execution passes through the original tool executor and approval checks. all restores the complete schema list.
  • Adaptive synthesis: A single successful write returning only a matching status, record ID and optional success flag can use a deterministic acknowledgement for a simple create/update/rename/archive/delete/remove request. Multi-action work, reports, drafts, failures and approval queues keep the normal synthesis behaviour. always disables this optimisation without overriding headless callers that suppress synthesis.
  • Business context: Rendering is cached by tenant, agent and an exact canonical snapshot of the loaded business data, including stale-strategy flags. Edits change the key immediately. Database loads and permission checks still happen normally. This cache holds at most 128 entries/8 MB for ten minutes; it saves rendering work and stabilises prompts, not database reads.
  • Embeddings: Exact API inputs share a bounded tenant/model-scoped vector cache for one hour (at most 2,048 entries/16 MB). Concurrent duplicates in the same event loop share a request; misses are sent in batches of up to 32 inputs. Failed requests are not cached. Search results and permissions are always evaluated afresh. Business and embedding caches are process-local; restarts clear them and other worker processes warm independently.

Evidence and cost reporting

Synthesis receives an evidence log budgeted to 12,000 tokens using the o200k_base budgeting tokenizer. Short results are retained in full. Larger results use beginning/end and objective-relevant excerpts with explicit omission markers. This replaces the fixed 1,000-character truncation; excerpts do not establish exhaustive review of omitted records.

Every successful router response records total input, cache reads, cache writes, output and latency in llm_calls. Anthropic input is normalised by adding its separately reported read/write counts. OpenAI output already includes billed reasoning tokens; it is not added again. Admin usage analytics expose cache writes and hit rates. Context-length multipliers are configured for Astra, Luna, Terra, Grok 4.6 and Gemini 3.1 Pro.

Kernel Service Admin → Cache Efficiency groups measured calls by organisation, agent, model and phase. It shows input spend, write premiums, net savings against uncached input at the request’s recorded rates, latency, completed-task cost contribution, version counts, reused reference reads and compacted tokens. A negative saving means cache writes outweighed reads. Only hashes of prefixes and tool schemas are recorded in llm_calls.cache_metrics; prompt contents are not added to analytics. Version variation is not proof of why a provider cache missed.

Completed-task cost is each row’s average contribution to tasks marked completed, for calls in the selected period. A task can appear in several model/phase rows; these averages are not additive. The efficiency view excludes historical rows without the new measurements and failed provider attempts. It is not an invoice reconciliation or a measurement of end-to-end task quality.

Sonnet 5 is corrected to USD $2/M input and $10/M output, with $0.20/M cache reads and $2.50/M five-minute writes. Anthropic pricing

Historical cost rows are not rewritten. In particular, older Anthropic input denominators and OpenAI cache writes remain incomplete. Compare post-deployment periods when evaluating caching. Failed/time-out attempts, provider-hosted tool fees, Google explicit-cache storage fees and unconfigured context-price tiers are not included in these token estimates. Unverified provider rates remain as configured; family-default prices for newly discovered models are estimates and log a warning.

Rollout and validation

  1. Apply Alembic through revision b74f2d0a831c to the control-plane database before restarting API and worker processes. This includes the earlier a63e1c9f420b admin-policy migration and the new nullable cache-metrics column. No historical costs are rewritten.
  2. Install the updated dependencies and deploy the backend and frontend together.
  3. Start with an administrator-selected synthesis trial; leave execution defaults unchanged.
  4. Run python scripts/check_openai_astra.py for model catalogue access. Add --smoke to run at most two billable requests with a synthetic echo tool and no customer data or external side effects.
  5. Compare successful task completion, corrections, retries, latency, cost and cache reuse against Kimi, Opus and Sonnet. Model catalogue access alone does not establish that inference is funded or available.

During development on 7 September 2026, catalogue access succeeded but live generation returned credit_balance_exhausted. Restore OpenAI credits before the live trial. Batch/Flex processing and automatic quality escalation remain separate follow-up work.

Provider wire behaviour is tested with mocked SDK transports. Context budgets, full Unicode evidence retrieval, read-cache invalidation, tenant scope, tool approvals, embedding reuse and admin access have regression coverage. The analytics query and cache-metrics upgrade/downgrade are exercised against disposable PostgreSQL 16 using tests/unit/test_cache_efficiency_database.py; its opt-in KERNEL_EFFICIENCY_TEST_DATABASE_URL must point to a localhost database named kernel_efficiency_test with no existing application tables.