Keep an AI Serving Harness Compatible

Updated 7 November 2026 · Standard formula, shown below

This guide shows you how to keep an AI serving harness—the code that wraps a model and exposes it via an API—compatible as models and runtimes evolve. You’ll learn a practical, step-by-step approach to isolate changes, use versioning, and test continuously, so your service doesn’t break when you upgrade a model or the underlying runtime.

What You’ll Need

A version control system (e.g., Git) for your harness code.
A CI/CD pipeline (e.g., GitHub Actions, GitLab CI) to run tests automatically.
Access to the model artifacts and runtime versions you plan to support (e.g., ONNX, TensorFlow, PyTorch).
Basic familiarity with containerization (Docker) and API design (REST/gRPC).

Step-by-Step

Define a stable API contract. Start by writing down the exact input and output schema your harness exposes. Use a format like OpenAPI (Swagger) or a protobuf definition. This contract should be independent of the model internals—it defines what the client sends and receives, not how the model works.
Abstract the model interface. Create a thin wrapper class or function that your harness calls to run inference. This wrapper should have a fixed signature (e.g., predict(input: dict) -> dict) and handle all model-specific details (preprocessing, tensor conversion, postprocessing) inside. This way, the rest of your harness doesn’t care which model or runtime is behind the wrapper.
Version your models and runtimes. Tag each model artifact with a semantic version (e.g., v1.2.0). Similarly, pin the runtime version (e.g., tensorflow==2.13.0) in your dependency file. Store this metadata in a manifest file (e.g., model.yaml) that your harness reads at startup to know what to load.
Use a model registry. If possible, store your model artifacts in a registry (like MLflow, or a simple S3 bucket with versioning). This lets you roll back to a previous version easily and keeps a history of what was deployed.
Containerize your harness. Build a Docker image that includes the harness code, the runtime, and the model artifact (or a way to download it at runtime). Tag the image with both the harness version and the model version (e.g., my-harness:1.2.0-model-1.2.0). This makes deployments reproducible.
Write contract tests. Create a set of test cases that send sample inputs to your harness and assert the outputs match expected values. These tests should be independent of the model—they test that the API contract is honored, not that the model is accurate. Run these tests in CI on every change.
Test for compatibility across versions. In your CI, build a matrix of jobs that test your harness against multiple model versions and runtime versions. For example, run tests with model v1.0.0, v1.1.0, and v1.2.0, each with the runtime versions you support. This catches breaking changes early.
Use a feature flag for new models. When you introduce a new model version, don’t replace the old one immediately. Instead, deploy both and use a flag (e.g., an environment variable or a request parameter) to switch between them. This allows you to A/B test and roll back if needed.
Monitor and log. Add logging to capture the model version, runtime version, and input/output shapes for each request. This helps you trace issues back to a specific version combination.
Document the upgrade path. Maintain a CHANGELOG or migration guide that describes what changed between model versions and what adjustments the harness needs. This is especially important when the model’s input/output schema changes.

Common Mistakes

Hardcoding model paths or versions. If you hardcode a model path or version in your code, you’ll have to edit code to upgrade. Use environment variables or a config file.
Ignoring the runtime version. A model that works with TensorFlow 2.12 might break with 2.13 due to API changes. Always pin the runtime version and test upgrades.
Not testing the full request path. Unit tests on the model wrapper alone don’t catch issues like serialization errors or HTTP header problems. Test the actual API endpoint with representative requests.
Forgetting about preprocessing/postprocessing. Models often expect normalized inputs and produce raw outputs. If you change the model, the preprocessing steps (e.g., tokenization for NLP) may change. Keep these steps inside the wrapper and update them with the model version.
Overlooking backward compatibility. If you change the API contract (e.g., add a new field), old clients may break. Use versioning in the API path (e.g., /v1/predict) or make new fields optional.

Tips

Use a schema validation library (like Pydantic or Cerberus) to validate inputs and outputs against your contract. This catches mismatches early and gives clear error messages.
Keep the harness as thin as possible. The more logic you put in the harness, the more you have to maintain. Push complex logic into the model wrapper or a separate preprocessing library.
Automate the build of the manifest. If you use a model registry, you can generate the manifest file automatically from the registry metadata, reducing manual error.
Run a smoke test after deployment. After you deploy a new model version, send a few known-good requests to the live endpoint to confirm it works before routing real traffic.
Document the expected output format. Even if your contract is in OpenAPI, add comments in the code that show an example input and output. This helps future maintainers.

Troubleshooting

If your harness breaks after a model or runtime upgrade, follow these steps:

Check the logs. Look for stack traces or error messages. They often point to the exact line where the failure occurs—whether it’s a tensor shape mismatch, a missing attribute, or a serialization error.
Verify the model artifact is correct. Confirm that the model file was downloaded/loaded correctly. Check the file hash or size against the registry.
Test the model wrapper in isolation. Write a small script that loads the model using the same runtime and runs a dummy input through the wrapper. This isolates whether the issue is in the wrapper or in the harness’s request handling.
Compare with the previous version. If you have a working version, diff the code and configuration between the old and new. Often a change in a preprocessing step or a default parameter causes the issue.
Check for runtime API changes. Look up the release notes for the runtime version you upgraded to. For example, TensorFlow 2.x deprecated some APIs. If you find a breaking change, adjust the wrapper code accordingly.
Roll back. If you can’t fix it quickly, roll back to the last known-good model/runtime combination. Use your container tags or model registry to redeploy the previous version.

Frequently Asked Questions

How do I handle a model that changes its input size?

If the new model expects a different input size (e.g., image resolution), you have two options: either update the preprocessing to resize inputs to the new size, or keep the old input size and add a resizing step in the wrapper. The choice depends on your use case; resizing may lose information, so test accordingly.

Should I use a separate endpoint for each model version?

It’s a good practice to version your API (e.g., /v1/predict and /v2/predict) if the input/output schema changes. If only the model internals change but the schema stays the same, you can keep the same endpoint and just switch the model behind it.

What if I can’t use a container registry?

You can still achieve versioning by storing the model artifact in a shared filesystem with versioned directories (e.g., /models/v1.2.0/) and pointing your harness to the correct path via a config file. The key is to have a clear, reproducible way to reference a specific model version.

How do I test compatibility without a full CI pipeline?

You can write a simple shell script that runs your test suite against different model versions by setting an environment variable that the harness reads to load the model. Run the script manually each time you upgrade.

Is it necessary to pin the runtime version?

Yes, because even minor runtime updates can change behavior or deprecate APIs. Pinning the version in your requirements.txt or Dockerfile ensures that your harness runs the same way every time. Use a lock file (e.g., pip-tools or poetry.lock) for full reproducibility.

What should I do if the new model’s output format is different?

If the output format changes (e.g., from a list to a dict), you need to update the postprocessing step in the wrapper. To maintain backward compatibility, you could add a transformation that converts the new format to the old one, but that’s only a temporary fix. Eventually, you should update the API contract and inform clients.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.