Production routing is not picking one model for an entire application. It is a repeatable decision process that maps a request to a model profile, a budget, a response-time target, and a failure policy. Done well, it reduces spend without silently lowering the quality that users notice.
Route by task risk and complexity
Start with a small number of model classes, not a different rule for every prompt. A fast route can handle bounded extraction, classification, and short transformations. A balanced route fits conversational or user-facing work. A reasoning route belongs to complex coding, long-context analysis, and high-impact decisions. The important question is not “which model is best?” but “what does failure cost for this request?”
Build a routing contract
For each workflow, write an explicit contract: expected input, output schema, maximum acceptable latency, quality threshold, budget, and acceptable fallback. This turns routing into something engineers can test rather than a hard-coded hunch.
if request.is_high_risk: use deep_reasoning
else if request.is_bounded and schema_is_simple: use fast_extract
else: use balanced_chat
if transient_failure: retry with backoff
if route_unavailable and fallback_is_approved: use fallback
otherwise: return a clear recoverable errorEscalate on evidence
The most useful escalations come from deterministic signals: JSON validation failure, low retrieval confidence, a second failed tool call, a document above a measured complexity threshold, or an explicit user request for a deeper review. Avoid routing based only on prompt length; long text can be easy, while a short request can be high stakes.
Separate fallback from escalation
Escalation moves a request upward because it needs more capability. Fallback preserves availability after a failure. They should not be the same policy. For a low-risk summary, a compatible fallback may be acceptable. For a financial or compliance workflow, a weaker substitute can be worse than queuing work or asking the user to retry.
Measure the route, not just the model
Log the selected alias, resolved model, prompt version, input and output tokens, latency, retries, schema validity, and outcome score. Compare routes by cost per accepted result. This exposes cases where an inexpensive route costs more after retries or manual correction.
Keep decisions reversible
Store aliases and thresholds in configuration, and keep a stable evaluation set. When a model changes, rerun the same cases before changing production traffic. An OpenAI-compatible, multi-model API lets the application preserve its integration while routing policy evolves.