MiniCPM Agent Gateway is a quality-first, local-first Agent routing system built on NVIDIA-NeMo/Switchyard. It uses local MiniCPM models for frequent, well-bounded work and transfers difficult tasks or validated local failures to an external strong model only when the selected Route's data policy permits cloud access.

The gateway keeps OpenAI Chat, OpenAI Responses, and Anthropic Messages clients on their native protocols. Its inherited Switchyard engine handles protocol translation, routing, health-aware execution, logging, and metrics; the MiniCPM routing layer adds deterministic capability gates, policy-safe cloud takeover, validated control outputs, and an optional local Runtime Supervisor.

Core capabilities

• MiniCPM-first routing: MiniCPM5-1B handles classification, simple text and tools, and context compression; MiniCPM-V 4.6 handles images, OCR, documents, ordered video frames, and visual tools.
• Policy-controlled cloud takeover: every Agent Route declares local_only, cloud_allowed, or cloud_redacted; reliability fallback never expands that permission.
• Quality-preserving execution: difficult or uncertain tasks default to a configured external strong model, while malformed local control output is rejected before it can be used.
• Safe streaming boundaries: ordinary text remains streamed; classification, tools, structured output, and summaries are buffered and validated. Two models are never spliced into one stream.
• Optional local supervision: switchyard-runtime can verify approved model artifacts, keep MiniCPM5 resident, lazily load MiniCPM-V, and unload the vision model after an idle timeout.
• Protocol and telemetry compatibility: clients can use OpenAI or Anthropic APIs while the gateway exports request, error, latency, token, and routing metrics.

Maturity

MiniCPM Agent Gateway 0.1.0 is an alpha-quality Developer Preview. The MiniCPM routing path has passed its offline gates, the six-platform build matrix, and an Apple Silicon live evaluation with LM Studio. Pure-CPU and NVIDIA runtime paths are implemented and covered by deterministic tests, but still require equivalent hardware qualification before they should be described as fully validated deployments. It is not yet recommended for unattended production use.

Measured results

The final 80-sample paired evaluation reached 95% routed task success versus 75% for the cloud-only baseline, with 57.8% of routed model requests handled by MiniCPM and zero data-policy violations. TTFT and peak memory were not captured, so this release makes no cost-savings claim.

Architecture

Request routing and cloud takeover

The gateway makes deterministic policy and capability decisions before asking a model to classify task complexity. Local control output must pass validation, and every path to the cloud crosses the Route's data-policy boundary. A request flows as follows:

1. An Agent / SDK client (OpenAI Chat, Responses, or Anthropic Messages) reaches switchyard-server, which handles protocol translation and route selection.
2. The Route data-policy gate (local_only / cloud_allowed / cloud_redacted) applies, followed by request requirement extraction (modality, tools, schema, context).
3. A deterministic capability gate (profile, health, capacity) dispatches the request: multimodal requests go straight to MiniCPM-V 4.6 Q8; text candidates are first classified by MiniCPM5-1B — simple tasks run on MiniCPM5-1B F16, hard or uncertain ones go to the cloud-takeover decision.
4. Plain-text local output is streamed to the client. Controlled output (classification, tools, schema, summary) is buffered and validated; valid output is returned, and output that fails validation before anything is visible goes to the cloud-takeover decision.
5. Cloud-takeover decision: local_only returns a policy-safe local error; cloud_allowed sends the request to the external strong model; cloud_redacted runs deterministic redaction and revalidation, passing to the cloud only on success and failing closed otherwise.
6. A failure after visible output issues a task-scoped cloud lease, valid for the next turn only; the next request then goes through the cloud-takeover decision.

A failure after visible text has been streamed ends that response with a stream error. The task lease affects the next request; it does not replay side-effecting tools or append cloud text to the partial local answer. If the original context also exceeds the cloud profile, the strong model performs validated cloud compression before answering.

Local deployment and runtime supervision

Use either a user-managed OpenAI-compatible runtime such as LM Studio or the optional Supervisor-managed llama.cpp path for a given local target. The Supervisor does not manage LM Studio, vLLM, or SGLang processes.

• User-managed runtime mode: the gateway talks to LM Studio or a compatible endpoint serving MiniCPM5-1B F16 and MiniCPM-V 4.6 Q8.
• Supervisor-managed llama.cpp mode: switchyard-runtime handles hardware detection, the approved manifest, and health, and fronts a resident MiniCPM5 sidecar and a lazy, idle-unloaded MiniCPM-V sidecar through an authenticated loopback proxy.
• The external strong model sits outside the workstation trust boundary and is called only when Route policy permits.

Model roles

Role	Approved model	Profile	Residency	Primary work
Text and control	MiniCPM5-1B	F16, 131,072-token context, 4,096-token control budget	Resident	Classification, simple text and tools, context compression
Multimodal	MiniCPM-V 4.6	Q8, 131,072-token configured context	Lazy when supervised	Images, OCR, documents, ordered video frames, visual tools
Difficult work	Operator-selected strong model	Provider-specific	Cloud	Difficult or uncertain tasks, policy-permitted takeover and compression

MiniCPM-o 4.5, MiniCPM4.1-8B, and MiniCPM5 Q4 are outside the 0.1 release. The reviewed LM Studio configuration requires the explicit minicpm5-1b@f16 model ID; the shorter minicpm5-1b alias resolved to an unapproved Q8 artifact in the tested environment.

Quick start

The GitHub Release contains Python integration wheels and a source distribution. The standalone native server is currently installed from the repository with Cargo.

1. Install the native server

Install Rust with Cargo, then run:

Bash

git clone https://github.com/pzc163/MiniCPM-Agent-Gateway.git
cd MiniCPM-Agent-Gateway
cargo install --locked --path crates/switchyard-server
switchyard-server --help

2. Configure the MiniCPM Agent Route

Start from the reviewed configuration instead of assembling capability profiles by hand:

Bash

cp config/minicpm-agent.example.toml minicpm-agent.toml
# Edit the local/cloud URLs and exact model IDs first.
export MINICPM_AGENT_CLOUD_API_KEY="your-provider-key"  # pragma: allowlist secret
switchyard-server --config minicpm-agent.toml --dry-run
switchyard-server --config minicpm-agent.toml --host 127.0.0.1 --port 4000

For a user-managed LM Studio endpoint, configure the server-root URL and exact tested model IDs:

Toml

[llm_clients.local_lmstudio]
format = "openai_chat"
base_url = "http://<lm-studio-host>:1234/v1"
use_legacy_max_tokens = true
max_retries = 0

[targets.minicpm5]
id = "minicpm5-1b@f16"
llm_client = "local_lmstudio"

[targets.minicpm_v46]
id = "minicpm-v-4.6"
llm_client = "local_lmstudio"

No local api_key_env is required when the endpoint does not require authentication. Configure the external strong model with an environment variable rather than a literal secret, then verify the gateway from another terminal:

Bash

curl --fail http://127.0.0.1:4000/health

3. Send a request through a policy Route

Clients select the data boundary through the Route model ID:

JSON

{
  "model": "switchyard/minicpm-agent-cloud",
  "messages": [{"role": "user", "content": "Classify this task and complete it."}]
}

The same server accepts OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages requests.

Library path

switchyard-libsy embeds the inherited routing engine in a Rust application. It decides which target to use but delegates model calls to the host; pair it with switchyard-llm-client when the library should perform translated HTTP calls.

Toml

[dependencies]
switchyard-libsy = { git = "https://github.com/pzc163/MiniCPM-Agent-Gateway.git", tag = "gateway-v0.1.0" }
switchyard-protocol = { git = "https://github.com/pzc163/MiniCPM-Agent-Gateway.git", tag = "gateway-v0.1.0" }

Route data policies

Every capability_router Route declares exactly one policy. Local failures, queue pressure, and validation errors never override it.

Policy	Cloud behavior	Recommended use
local_only	Request content never goes to the cloud; unsupported or failed work returns a local error	Sensitive attachments, media, documents, or air-gapped work
cloud_allowed	The original request may go to the strong model for difficult work or safe fallback	Non-sensitive tasks where answer quality is the priority
cloud_redacted	Deterministic text redaction and revalidation run before cloud access; uncertainty fails closed	Text containing known, testable secret patterns

cloud_redacted is not a visual-region redactor. Sensitive images, video, and documents must use local_only.

Failure and streaming semantics

• A local failure before the first visible streamed event may transfer the current request to the cloud when policy permits.
• Once visible text has been emitted, a failure ends the stream. The next turn can use the task-scoped cloud lease; the gateway never joins output from two models.
• Classification, context compression, tool calls, and user-visible structured output are fully buffered and validated before use or delivery.
• A classifier truncated with finish_reason=length receives one compact recovery attempt. Other truncated controlled outputs fail closed.
• If the request also exceeds the cloud context profile, the cloud target compresses and validates the context before producing the final answer.

Other routing strategies

MiniCPM Agent Gateway retains Switchyard's provider-neutral algorithms for deployments that need different routing behavior.

Strategy	Use it when	Route type
LLM Classifier	Request content should choose a weak or strong tier	llm_classifier
Stage Router	Conversation signals should route turns without another model call	stage_router
Escalation Router	A judge should review a weak-tier answer before optional escalation	llm_classifier with mode = "escalation"
Random	A fixed traffic split is needed for A/B tests or baselines	random

Links

• GitHub: pzc163/MiniCPM-Agent-Gateway
• Release: gateway-v0.1.0
• Upstream: NVIDIA-NeMo/Switchyard

Author: pzc163. Licensed under Apache 2.0; NVIDIA's upstream copyright and NOTICE are preserved.