AI Decision Records: Defending Choices After 6 Months

AI decision records (ADRs) are structured logs that capture why an AI-related choice was made, what alternatives were considered, and the context at the time. If you maintain them properly, you can defend the choice six months later by pointing to the recorded rationale, data, and trade-offs. Without them, you are left with memory and guesswork, which rarely holds up under scrutiny.

Formula

The decision quality score is a weighted composite: Score = (Time/60) × 0.3 + Risk × 0.5 + Confidence × 0.2. Time is measured in minutes (converted to hours), risk and confidence are on 1–10 scales. The weights reflect that risk assessment and data confidence matter more than the time spent. This is not a scientific metric, but a simple heuristic to compare decisions retrospectively.

Worked Example

Using the defaults: Time = 30 min, Risk = 5, Confidence = 7. First, convert time: 30/60 = 0.5 hours. Then multiply: 0.5 × 0.3 = 0.15. Risk: 5 × 0.5 = 2.5. Confidence: 7 × 0.2 = 1.4. Sum: 0.15 + 2.5 + 1.4 = 4.05, rounded to 4.1. The score is 4.1 out of a possible 10, suggesting a moderately considered decision. This is a rough tool, not a judgment.

Why ADRs Matter for AI Projects

AI systems are not static; models get retrained, data pipelines change, and business requirements shift. A decision that made sense in January may look wrong in July if the underlying assumptions changed. ADRs preserve the original reasoning so that when someone asks “why did you use this model?” or “why did you exclude that feature?”, you can give a factual answer, not a defensive one.

Common scenarios where ADRs are essential:

• Choosing a model architecture (e.g., transformer vs. CNN)
• Selecting training data sources or exclusion criteria
• Setting thresholds for alerts or classifications
• Deciding to deploy or roll back a model
• Choosing a vendor or open-source library

Anatomy of a Good AI Decision Record

A useful ADR includes more than just the decision. It should have:

• Context: The problem being solved, business goals, and constraints (time, budget, regulatory).
• Decision: What was chosen, in clear terms.
• Alternatives: Options considered and why they were rejected.
• Rationale: Data, benchmarks, or expert opinions that supported the choice.
• Consequences: Expected positive and negative outcomes, including risks.
• Date and author: Who made the call and when.
• Status: Proposed, accepted, superseded, or deprecated.

This structure mirrors the classic architecture decision record but adapts it to AI-specific concerns like data versioning and model evaluation metrics.

How to Write an ADR That Survives Six Months

Writing a record that holds up over time requires discipline. Follow these steps:

1. Write it when the decision is made, not weeks later. Memory fades and biases creep in.
2. Quantify the rationale. Instead of “the model performed better,” write “accuracy improved from 91% to 94% on the validation set, but recall dropped by 2%.”
3. Record the data version and code commit. This ensures reproducibility.
4. Note what you did not know. Explicitly list uncertainties and assumptions.
5. Set a review date. Schedule a check to see if the decision still holds.
6. Link related ADRs. If a later decision depends on this one, reference it.

Common Mistakes That Undermine Defensibility

Even with good intentions, people make errors that weaken their ability to defend a choice later:

• Vague language: “We used X because it’s better” is not defensible. Better than what, by how much, and at what cost?
• Ignoring counterevidence: If you considered a strong alternative but omitted it from the record, it looks like you hid it.
• No data versioning: If you can’t reproduce the exact dataset, the decision is hard to verify.
• Confusing correlation with causation: If you chose a feature because it correlated with the target, note that it may not be causal.
• Overlooking ethical or legal constraints: A decision that is technically sound but violates privacy norms will fail later.

Tools and Practices for Managing ADRs

You don’t need expensive software. Many teams keep ADRs as Markdown files in a Git repository, named with a date and a short description (e.g., 2024-01-15-model-threshold.md). This allows version control, peer review via pull requests, and easy searching. Some teams use dedicated wiki pages or issue trackers. The key is consistency: pick a location and format, and make it part of the development workflow.

When a decision is challenged, the ADR serves as the first line of defense. You can point to the file, show the rationale, and explain how the context has or hasn’t changed. If the decision was wrong, the ADR helps you learn from it rather than assign blame.

Frequently Asked Questions

What if I didn’t write an ADR at the time?

You can write a retrospective record, but you must be honest about the timing. Label it as “retrospective” and reconstruct the context from emails, chat logs, and code history. It won’t be as strong as a contemporaneous record, but it’s better than nothing.

How long should an ADR be?

One to two pages is typical. If it’s longer, you’re probably including too much implementation detail. The goal is to capture the decision and rationale, not the entire project.

Do ADRs apply to non-technical AI decisions?

Yes. Decisions about data collection, annotation guidelines, or even whether to deploy an AI feature at all can be recorded in the same format. The structure is universal.

Can an ADR be changed after the fact?

Yes, but you should append a new entry or update the status, not rewrite history. Use a “Superseded by” link to the new decision. This preserves the original context and shows the evolution of thinking.

What is the difference between an ADR and a design doc?

A design doc describes how a system works; an ADR explains why a specific decision was made. They complement each other. A design doc might say “the model uses a threshold of 0.8,” while the ADR explains why 0.8 was chosen over 0.7 or 0.9.

Is a decision log the same as a changelog?

No. A changelog lists what changed in the codebase; a decision log records why changes were made. They serve different purposes, though both are useful for auditing.

Related

• E-Waste Recycling: How to Reuse, Donate, or Recycle Old Devices
• Prevent Crashes Enshrouded? Game Stability Fixes
• Dual Channel Memory RAM: What It Is & How It Works

in t | Time to decide (min) | min | 30 | 1
in r | Risk level (1-10) | - | 5 | 1
in c | Confidence in data (1-10) | - | 7 | 1
out score | Decision quality score | - | (t/60)*0.3 + r*0.5 + c*0.2 | 1