Image Model Routing: Shadow Tests vs Live Switching

Updated 6 November 2026 · Standard formula, shown below

Image model routing can absolutely separate shadow tests from live switching, and most production systems do exactly that. You run a candidate model in parallel with your current model, send it real traffic, compare the outputs, and only flip the live switch once the shadow test shows the new model is safe and better. This article explains how shadow testing and live switching work, why they are kept separate, and how to set up a routing pipeline that uses both.

What Is Image Model Routing?

Image model routing is the practice of sending each image-related request (classification, object detection, segmentation, generation, etc.) to one of several possible models, based on cost, latency, accuracy, or other criteria. A router can be a simple rule-based system (e.g., “use the small model for thumbnails, the large model for full-res images”) or a learned algorithm that predicts which model will perform best for a given input.

The key point is that routing is not the same as training a model. You are not changing the weights of the models; you are deciding which model gets which request. This is why shadow testing is so valuable: you can evaluate a new model without affecting live traffic.

Shadow Testing: How It Works

Shadow testing (also called shadow deployment or dark launch) sends a copy of real production requests to a candidate model, but the candidate’s responses are not returned to the user. Instead, they are logged and compared to the responses from the current production model.

The typical setup:

Intercept a percentage (or all) of incoming requests
Forward the same request to both the baseline model and the candidate model
Log both responses, along with metadata (request ID, timestamp, model version, latency, confidence scores)
Run offline evaluation: compare accuracy, error rates, latency, cost, and any other metrics
If the candidate passes your thresholds, you can then promote it to live traffic

Shadow testing is safe because the candidate model’s outputs never reach users. Even if the candidate crashes or returns garbage, the user still gets the baseline response. This makes it ideal for testing new architectures, fine-tuned versions, or even entirely different model families.

Live Switching: How It Works

Live switching (also called canary deployment or progressive rollout) is the process of gradually shifting real user traffic from the baseline model to the candidate model. This is done after shadow testing has shown the candidate to be acceptable.

Common strategies:

Percentage-based: Start with 1% of traffic, then 5%, 10%, 25%, 50%, 100% over hours or days
Segment-based: Route only certain user types or request types to the new model first
Feature-flag based: Use a configuration flag to instantly switch all traffic, but with the ability to roll back immediately

The key is that live switching is reversible. You should always have a rollback plan: if the candidate model shows degraded performance (higher error rate, slower latency, user complaints), you can instantly switch back to the baseline.

Why Keep Shadow Tests and Live Switching Separate?

They are separate because they serve different purposes and have different risk profiles.

Shadow tests are for evaluation; they have zero user impact. You can run them for as long as you want, with any amount of traffic, without worrying about user experience.
Live switching is for deployment; it has direct user impact. You need to be much more careful, using gradual rollouts and monitoring.
Mixing them would mean either risking user experience during evaluation (bad) or delaying deployment until you have enough shadow data (slow).

In practice, you run shadow tests first, then live switching. But some teams also run a “shadow” phase during live switching: they route a small percentage of traffic to the candidate, but also continue to log the baseline’s response for comparison. This is a hybrid, but the core separation remains: shadow testing is non-production, live switching is production.

How to Choose Between Shadow Testing and Live Switching

You don’t choose one over the other; you use both in sequence. But you might adjust the emphasis based on your situation:

If you are evaluating a model with a known different behavior (e.g., a new architecture), spend more time in shadow testing.
If you are just updating to a newer version of the same model (e.g., a retrained checkpoint), you might shorten the shadow test and move quickly to a small canary.
If your application is high-risk (medical imaging, autonomous driving), you might require a long shadow test with a high bar for statistical significance, and a very slow canary.
If your application is low-risk (e.g., image alt-text generation), you might skip shadow testing entirely and rely on a canary with automatic rollback.

In all cases, you need a clear set of metrics to compare: accuracy, latency, cost, and any domain-specific metrics (e.g., false positive rate for a safety filter).

Common Problems and Misconceptions

One common misconception is that shadow testing and live switching are mutually exclusive. They are not; they are two phases of the same deployment pipeline.

Another issue is data leakage: if you use the same requests for shadow testing that you later use for fine-tuning, you might overfit to the test set. Keep your evaluation data separate.

Also, shadow testing can be expensive because you are running two models for every request. You can mitigate this by sampling a subset of traffic (e.g., 10%) for shadow testing.

Finally, beware of comparing models on different distributions. If your shadow test only runs on a certain type of image (e.g., daytime photos), the results may not generalize to all traffic. Make sure your shadow test covers a representative sample.

Cost and Safety Considerations

Running shadow tests doubles your inference cost for the sampled traffic. If you shadow 100% of traffic, you pay for two inferences per request. You can reduce this by:

Sampling a percentage of requests
Using a cheaper candidate model for shadowing (but then you are not testing the real candidate)
Running shadow tests only during off-peak hours

Safety-wise, shadow testing is inherently safe because it does not affect users. The main risks are:

Data privacy: you are sending user data to a third-party model (if the candidate is hosted externally). Ensure you have proper data processing agreements.
Resource exhaustion: running two models can overload your infrastructure. Monitor CPU/GPU usage.
Logging sensitive data: be careful not to log raw images or user IDs unnecessarily.

For live switching, the safety measures are more about monitoring and rollback. Set up alerts for error rate, latency, and cost. Have a manual rollback button ready.

Alternatives to Shadow Testing

If shadow testing is too costly or complex, there are alternatives:

Offline evaluation: use a labeled test set to compare models before deploying. This is cheaper but may not reflect real-world distribution.
A/B testing: serve the candidate model to a small, random subset of users and compare their behavior (e.g., click-through rate, task completion). This is live but limited to a small segment.
Champion/challenger: similar to A/B testing, but the challenger model is only used for a small percentage of traffic, and the results are compared periodically.
Model distillation: instead of switching to a new model, you can distill the candidate into the baseline architecture, avoiding the need for a separate deployment.

Each has trade-offs in cost, risk, and speed. Shadow testing remains the gold standard for evaluating a model before any user impact.

Frequently Asked Questions

Can I run shadow tests on live traffic without affecting users?

Yes, that is the whole point of shadow testing. You send a copy of the request to the candidate model, but the response is discarded or logged. The user only sees the baseline model’s response. The only impact is on your infrastructure and cost, not on user experience.

How long should a shadow test last?

It depends on your traffic volume and the magnitude of the difference you want to detect. As a rule of thumb, you want enough samples to detect a meaningful difference in error rate with statistical confidence. For example, to detect a 1% difference in error rate with 95% confidence, you might need tens of thousands of requests.

What metrics should I compare during shadow testing?

At minimum, compare error rate (or accuracy), latency (p50, p95, p99), and cost per request. If your application has domain-specific metrics (e.g., false positive rate for a content filter, or BLEU score for image captioning), include those as well. Also monitor for crashes or timeouts.

Is it safe to switch all traffic to a new model after a successful shadow test?

Shadow testing reduces risk, but it does not eliminate it. There could be rare edge cases that only appear under specific conditions. It is safer to do a gradual rollout after shadow testing, starting with a small percentage of traffic and monitoring for a period before increasing.

Can I use shadow testing for cost optimization?

Yes, you can shadow a cheaper model to see if it meets your accuracy requirements. If it does, you can switch to it to reduce costs. However, be careful to test on a representative sample, as a cheaper model might perform poorly on certain types of images.

What is the difference between shadow testing and canary deployment?

Shadow testing sends traffic to the candidate model but does not use the responses. Canary deployment sends a small percentage of real traffic to the candidate model and uses the responses. Canary is a form of live switching, while shadow is a pre-live evaluation.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.