AI Build: Define What Right Looks Like First

Updated 4 November 2026 · Standard formula, shown below

Before you build anything with AI, define what “right” looks like. This means specifying the measurable outcome, the acceptable error rate, and the success criteria before writing a single line of code or training a model. Without this definition, you cannot evaluate whether your AI system works, improve it systematically, or know when to stop.

Why Defining Success Matters

AI systems are optimization engines. They will optimize for whatever objective you give them, even if that objective is incomplete or misleading. If you don’t define what “right” looks like, the system will find a shortcut that scores well on your proxy metric but fails in the real world. For example, a spam filter trained to maximize accuracy might learn to delete all emails, achieving 99% accuracy if spam is rare, but also deleting legitimate messages. Defining “right” upfront forces you to articulate the trade-offs between precision and recall, between speed and thoroughness, and between automation and human oversight.

This definition also serves as the contract between stakeholders. Product managers, engineers, domain experts, and end users often have different visions of what success means. Writing down the definition surfaces disagreements early, when they are cheap to resolve, rather than after months of development. It also provides the foundation for testing, monitoring, and ongoing maintenance, because you can’t measure drift or degradation without a baseline.

How to Define “Right” for Your Project

Start with the business or user problem, not the technology. Ask what decision the AI will inform or automate, and what the consequence of being wrong is. Then translate that into concrete, measurable criteria using these steps:

Identify the core outcome: What observable change should happen? Examples: reduce response time, increase conversion, decrease error rate, or improve user satisfaction.
Set a baseline: Measure the current performance without AI. You need a comparison point to know if the AI is actually better.
Define the metric(s): Choose one primary metric and a few guardrail metrics. For classification, that might be F1 score or AUC; for regression, MAE or RMSE; for generation, human evaluation or task-specific scoring.
Set thresholds: What is the minimum acceptable performance? What is the target? What is the stretch goal? Be specific: “at least 90% precision and 80% recall on the held-out test set.”
Specify failure modes: What errors are unacceptable, even if the metric is high? For a medical triage system, missing a critical case is worse than a false alarm. Define these constraints explicitly.
Include human factors: How will people interact with the output? Define the workflow for when the AI is uncertain or wrong, and the process for human override.

Common Mistakes in Defining Success

The most common mistake is choosing a metric that is easy to compute but not aligned with the actual goal. For example, using accuracy for an imbalanced dataset (99% non-spam, 1% spam) makes a model that always predicts “non-spam” look great. Similarly, optimizing for click-through rate might increase engagement but also increase misinformation if the system recommends sensational content. Another mistake is defining success only on average, ignoring edge cases or minority groups. A facial recognition system that works well for light-skinned subjects but poorly for darker skin tones may have high overall accuracy but fail the “right” definition for a large segment of users.

Another frequent error is failing to define the counterfactual. What would happen if you did nothing? If the current process already has a 5% error rate, then an AI with a 4% error rate is an improvement, but if the AI costs more and slows down the process, the net benefit may be negative. The definition of “right” should include the full context of cost, latency, and user experience, not just model accuracy.

How to Measure and Validate

Once you have defined the target, you need a rigorous evaluation plan. This involves splitting your data into training, validation, and test sets, and ensuring the test set represents the real-world distribution. Use the following practices:

Hold out a test set: Never tune hyperparameters or make decisions on the test set. Use a separate validation set for that.
Use multiple metrics: Report the primary metric and also secondary ones like latency, memory usage, and fairness across subgroups.
Human evaluation: For natural language or creative tasks, automated metrics like BLEU or ROUGE are weak. Have human raters score outputs against a rubric that matches your definition of “right.”
Test in production: A/B test the AI system against the baseline with a small percentage of traffic before full rollout. Monitor both the metric and any unintended consequences.
Monitor over time: Data distributions shift. Set up alerts for when the model’s performance on live data drops below a threshold, and retrain or roll back.

Costs and Trade-offs

Defining “right” is not free. It requires time from domain experts, data analysis, and stakeholder alignment. The cost of getting it wrong, however, is far higher. A poorly defined objective can lead to months of wasted development, a system that is deployed but not trusted, or even active harm. For example, an AI hiring tool that was trained on historical resumes learned to penalize women because the historical data reflected past bias. The developers had defined “right” as “predict who was hired,” not “predict who would be a good employee.” The fix required redefining the outcome and collecting new data.

There is also the cost of over-specification. If you set thresholds too high, you may never deploy anything. If you set them too low, you may deploy a system that is worse than the baseline. The key is to set a minimum viable threshold that is better than the status quo, with a clear path to improve.

Alternatives to Full Automation

Defining “right” also means deciding how much autonomy the AI should have. In many cases, the best system is a human-in-the-loop design where the AI makes a recommendation and a human makes the final decision. This is common in medical diagnosis, legal review, and content moderation. The definition of “right” then includes the human’s ability to override, and the metric might be the accuracy of the final human-AI decision, not just the AI’s raw output.

Another alternative is to use AI to augment rather than replace. For example, instead of automatically rejecting a loan application, the AI could flag risky cases for a human reviewer. This changes the success metric: instead of maximizing classification accuracy, you might want to maximize the number of true positives caught while minimizing the number of false alarms that waste the reviewer’s time. The definition of “right” must reflect this hybrid workflow.

Frequently Asked Questions

What if I don’t have historical data to set a baseline?

If you don’t have data, you can still define “right” by using expert judgment, manual processes, or a small pilot study. Run the current process manually for a few weeks and measure the outcome. Or use a rule-based baseline (e.g., always predict the majority class) as a starting point. The key is to have some quantitative reference point before you build the AI.

How do I handle multiple stakeholders with conflicting definitions of “right”?

Facilitate a structured workshop where each stakeholder writes down their success criteria, then rank them by priority. Identify the non-negotiable constraints (e.g., regulatory, safety) and the trade-offs that can be adjusted. The final definition should be a written document that everyone signs off on, with a clear owner for each metric.

Can I change the definition of “right” after the model is built?

Yes, but you should treat it as a formal change request. Update the documentation, re-evaluate the model against the new criteria, and possibly retrain. If the change is significant, you may need to go back to data collection. It’s better to iterate on the definition early, before the model is in production.

What is the difference between a metric and a success criterion?

A metric is a quantitative measure (e.g., accuracy, precision, latency). A success criterion is a threshold or condition that determines whether the AI system is acceptable (e.g., “precision must be above 90%”). The metric is what you measure; the criterion is what you compare it against to decide if you are done.

How do I define “right” for a generative AI system like a chatbot?

For generative AI, define “right” in terms of safety, helpfulness, and faithfulness. Use a rubric with specific criteria: does the response answer the question? Is it factually correct? Does it avoid harmful content? Then have human raters score a sample of responses, and also track user feedback like thumbs up/down or follow-up questions. Automated metrics like BLEU are not sufficient.

What if the AI performs well in testing but fails in production?

This usually happens because the production data differs from the training data. To mitigate, define “right” to include a monitoring plan: track the distribution of inputs and outputs, set up alerts for drift, and have a rollback plan. Also, start with a shadow deployment where the AI runs in parallel with the existing system without affecting decisions, so you can compare performance before full rollout.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.