LiteLLM Auto Router: Configure Tiers and Test Routing

Updated 5 November 2026 · Standard formula, shown below

To configure LiteLLM’s auto router, define model groups at different capability or cost tiers, then send requests through the router so it can choose an eligible model. Test the setup with representative prompts and inspect the selected model and response; routing behavior depends on the router type, configuration, and LiteLLM version.

What LiteLLM routing does

LiteLLM is a proxy and Python library that can present multiple language models through a common interface. Its routing features can distribute requests among configured deployments, while an auto-routing strategy can use request characteristics to choose between model groups. This is different from ordinary load balancing: load balancing spreads requests among deployments that are already considered interchangeable, whereas tiered routing aims to select a suitable class of model for the request.

There is not one universal “auto router” configuration that applies to every LiteLLM release. The available router strategies and their configuration fields can change. Check the documentation for the exact LiteLLM version you run, and distinguish built-in routing from a custom router or an external classification step that decides which model group to call.

Plan model tiers before configuring

Start by deciding what the tiers mean in your application. A common design has a lower-cost group for straightforward requests and a stronger or more capable group for complex work. A tier is a policy choice, not a guaranteed ranking: model names, context limits, supported inputs, latency, and cost vary by provider and deployment.

Define the criteria: Decide which request features should affect routing, such as task type, required context length, tool use, or a quality requirement.
Choose eligible models: Confirm each model supports the API features and input types your requests need. Do not put a text-only deployment in a group expected to handle images.
Set a default and fallback: Specify what should happen when classification is uncertain, a group is unavailable, or a model returns an error. Fallbacks should be compatible with the request.
Set operational limits: Consider timeouts, retries, rate limits, and per-model spend controls separately from the tier-selection rule.

Do not assume that naming a model “cheap” or “advanced” automatically teaches the router how to classify requests. The routing strategy needs a supported policy, rules, classifier, or other explicit signal.

Configure the router and model groups

In a proxy deployment, configuration usually has two conceptual parts: model deployments that LiteLLM can call, and router settings that determine how requests are dispatched. The precise YAML keys depend on the version and routing feature, so use the matching configuration reference rather than copying an example from an unrelated release.

Register deployments: Give each deployment a model alias or group name and provide its provider/model identifier and required credentials through the supported configuration mechanism. Keep secrets in environment variables or a secret manager, not in a shared configuration file.
Group candidates by tier: Make it clear which deployments are eligible for each tier. If the chosen router supports named groups or model lists, use those documented fields; otherwise implement the tier decision in the supported calling layer.
Enable the intended routing strategy: Configure the documented auto-routing option and its required inputs, such as a classifier model, routing rules, or metadata. Do not guess a field name: unsupported keys may be ignored or rejected.
Define fallback behavior: Decide whether a failed choice should retry another deployment in the same tier or move to a fallback tier. Keep retry counts bounded to avoid latency, duplicate work, or unexpected cost.
Validate before serving traffic: Start the proxy or initialize the router with the configuration, then resolve any schema, authentication, or connectivity errors before directing production requests to it.

For the Python library, the same principles apply, but configuration is expressed through the router and completion-call interfaces supported by that release. Avoid mixing proxy YAML examples with Python constructor arguments; they are not necessarily interchangeable.

Test routing with controlled requests

A good test checks both the answer and the routing decision. Use a small set of prompts whose expected tier is unambiguous, then verify which deployment handled each request using the response’s model information, proxy logs, or supported callbacks and tracing. Do not infer a successful route solely because the returned text looks plausible.

Send a simple, low-complexity request that should qualify for the lower-cost tier.
Send a request that clearly requires the higher-capability tier under your policy, such as a task involving a longer context or a supported tool workflow.
Test an ambiguous prompt and confirm that the default or fallback policy is used as intended.
Simulate an unavailable deployment or invalid credentials in a non-production environment, and verify bounded retries and the documented failure response.
Repeat tests with different request metadata and input types to detect accidental dependence on a single prompt format.

Record the request, intended tier, selected deployment, latency, outcome, and any retry. If a classifier drives routing, review its decision separately from the final model’s answer. A classifier can choose the wrong tier even when the selected model responds correctly.

Monitor quality, latency, and cost

After functional tests, evaluate routing over a representative, privacy-appropriate sample of real workloads. Compare quality and operational measures by tier, not just in aggregate: an overall average can hide a poor fallback path or a category that is routinely sent to an unsuitable model. Track selection rates, errors, timeouts, retries, latency, and available cost data.

Revisit thresholds and rules when model deployments change. Provider aliases can point to different models over time, and model capabilities or pricing can change independently of your routing configuration. Keep a record of the configuration version used for each evaluation so that a routing change can be compared fairly with the previous behavior.

Common problems and misconceptions

Every request reaches the same model: Check that the request is actually passing through the router and that the auto-routing strategy is enabled. A direct model name in the request may bypass a group-selection step.
Unknown configuration fields: Verify the LiteLLM version and the corresponding docs. Examples for another version or for another router mode may use different fields.
Fallbacks do not work: Confirm the fallback target is configured, reachable, and compatible with the request. Also check whether the failure is eligible for retry; not every error should be retried.
Unexpectedly high spend: A classifier call, retries, or routing most requests to a higher tier can add cost. Measure the full request path, not only the final completion.
“Auto” means no policy is needed: Automatic selection still relies on configured candidate models and a decision mechanism. It does not guarantee the cheapest or best answer for every prompt.

Frequently Asked Questions

Does LiteLLM automatically create cheap and powerful model tiers?
No. You must configure available deployments and define how requests qualify for each group. The router cannot infer your application’s quality, budget, or compatibility requirements from model names alone.

How can I tell which model handled a request?
Check the response metadata when it exposes the model, or use the proxy’s supported logs, callbacks, or tracing. Confirm the recorded deployment against the route you expected.

Can I use different providers in different tiers?
Yes, when the configured deployments and request features are supported. Check credentials, compatible API parameters, input types, and error behavior for each provider.

Why is the auto router choosing the wrong tier?
Review the router’s decision inputs and policy, then test clear, ambiguous, and edge-case requests. If a classifier is involved, inspect its classification rather than judging only the final response.

Should I retry with a higher tier whenever a request fails?
Not automatically. Some failures are transient, while others indicate invalid input, permissions, or configuration problems. Use bounded retries and explicit fallback rules, and avoid repeating requests that could incur duplicate work or charges.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.