Reproducing AI-Discovered Scientific Results

Updated 8 November 2026 · Standard formula, shown below

Reproducing an AI-discovered result means independently checking whether the reported finding follows from the data, methods, and assumptions—not merely rerunning the same software. This protocol helps research teams plan a fair test, document deviations, and distinguish a reproducible computation from a result that also holds in new experiments or data.

What You’ll Need

The paper, supplement, and any public protocol or preregistration describing the claim.
Access to the original data and code, if available, plus permission to use them and any required training or approvals for human, animal, or hazardous-material work.
A separate, documented computing environment and the relevant instruments, materials, or datasets for the claimed test.
A written reproduction plan, versioned records, and a way to preserve raw data, analysis outputs, and software versions.

Step-by-Step

Translate the claim into a testable statement. Write down what the AI system is said to have discovered, the population or system it concerns, the outcome, and the conditions. Separate the core claim from secondary findings. For example, distinguish “method A predicts outcome B better than baseline C on dataset D” from broader claims that it works in other settings. Identify the exact result—such as an effect size, predictive metric, or experimentally observed property—that would count as support.
Trace how the claim was produced. Read the main paper, supplementary methods, model card or technical notes, and cited data sources. Record which parts were generated or selected by an AI system: a hypothesis, candidate molecule, parameter setting, image label, or analysis. Check what human choices followed, including filtering and exclusions. An AI-generated suggestion is not itself evidence; identify the measurements and comparisons that support it.
Resolve missing details before running anything. List the data version, preprocessing, model and checkpoint, prompts or configuration, random seeds, hardware-sensitive settings, statistical tests, exclusion rules, and software dependencies. Ask the authors or inspect archived materials when these are unavailable. Mark each item as confirmed, inferred, or unknown. Do not silently fill important gaps; choose a defensible rule in advance and record it.
Choose the level of reproduction. A computational rerun checks whether the published pipeline produces comparable outputs from the same inputs. An independent reanalysis checks the claim using independently written or verified code. An external validation tests it on a genuinely separate dataset, sample, laboratory, or experimental batch. These answer different questions. State which you are doing, and do not describe a rerun on the original data as independent validation.
Write and register the plan before examining validation outcomes. Define the primary outcome, comparison or baseline, sample size or stopping rule, exclusions, missing-data handling, statistical analysis, and acceptable tolerance for numerical differences. If the study is exploratory, label it as such and separate exploratory analyses from confirmatory tests. For physical experiments, specify controls, randomization, blinding where feasible, calibration, and replication across independent runs. Obtain ethics and safety approvals before regulated work.
Rebuild the environment and protect the data split. Use the documented code and dependencies where possible, but keep an independent copy of the raw inputs. Record operating system, package versions, model weights, hardware, and configuration. For predictive work, ensure that training, tuning, and test data are separated at the correct level: related samples from one person, device, site, or time period may need to stay in the same split. Check for duplicated records and information leakage before scoring.
Run controls and the primary analysis. First confirm that the pipeline works on a small test case and that positive, negative, or baseline controls behave as expected. Then run the prespecified analysis without changing thresholds to improve the result. Preserve raw outputs and logs. For stochastic systems, run the planned number of seeds or repeats and report their spread; do not report only the best run. For laboratory work, retain instrument settings, batch details, and control measurements.
Compare results on the same scale. Match the paper’s metric, units, population, and evaluation conditions. Report estimates with uncertainty, not just a pass/fail label or a rounded headline number. Check whether differences are explained by documented software, data, hardware, or protocol changes. A small numerical mismatch may be immaterial, while a similar headline score can conceal a changed subgroup or outcome.
Test generalization separately. If the claim extends beyond the original dataset or setup, evaluate it on held-out or newly collected evidence that was not used to select the model or tune the method. Keep the validation set sealed until the plan is fixed. Where practical, repeat experiments across independent batches or sites. Report negative findings and failures alongside successes; changing the model after seeing validation results makes that set part of development, not an untouched test.
Publish an auditable record. Share the protocol, analysis code, environment details, data or access instructions, exclusions, deviations, and results in an appropriate repository, subject to privacy, consent, safety, and licensing limits. Include enough information for another team to distinguish exact reproduction from adaptation. State what could not be reproduced and what evidence would resolve the uncertainty.

Common Mistakes

Confusing code execution with confirmation of the scientific claim. A pipeline can run perfectly and still rely on biased data or an invalid assumption.
Using the test set repeatedly to adjust settings, thresholds, or prompts. This leaks information and makes final performance optimistic.
Comparing different metrics, units, data versions, or populations as if they were equivalent.
Ignoring preprocessing, exclusions, failed runs, or human decisions because they are not in the headline method.
Calling a result “reproduced” based on one random seed, one batch, or a rounded score without uncertainty.

Tips

Make a claim-to-evidence table: each claim, its supporting result, data source, analysis, and reproduction status.
Use a read-only copy of raw data and save checksums or other identifiers when appropriate, so accidental edits can be detected.
Keep a dated decision log. Record deviations as they happen, including why they were necessary and whether they affect interpretation.
Choose acceptance criteria based on scientific relevance and measurement precision before seeing results; exact numerical identity is not always a sensible standard.
Invite someone not involved in the original analysis to review the split, controls, and interpretation for avoidable bias.

Troubleshooting

The code will not run: Compare dependency and hardware requirements, check the exact data format, and test a minimal example. Save the error and environment details. If a modern dependency changes outputs, document the change and compare against an archived environment rather than quietly substituting it.

The reported result does not match: Check data versions, preprocessing order, units, random seeds, checkpoint files, and exclusion rules. Reproduce intermediate tables or predictions to locate where results diverge. If the cause remains unknown, report the discrepancy and avoid tuning until it disappears.

The original result appears, but external validation fails: Verify that the new data measure the same outcome and that the protocol was applied consistently. Then examine uncertainty, sample composition, and possible distribution shifts. A failed external test may limit the claim rather than invalidate the original computation; report both conclusions clearly.

There is no usable code or enough detail: Contact the authors with specific questions, search for archived supplements or data records, and document assumptions needed to proceed. If critical choices cannot be recovered, label the work a partial reconstruction, not an exact reproduction.

Frequently Asked Questions

Does reproduction require the original AI model?
For an exact computational rerun, usually yes, including the relevant model version and configuration. Independent validation may use a different method, but it answers a broader or different question.

How many times should a stochastic model be run?
Set the number of runs in advance based on the study design and expected variability. Report all planned runs and their distribution, not only the strongest result.

What is the difference between reproducibility and replicability?
Usage varies across fields. A common distinction is rerunning an analysis with the same data and methods versus obtaining compatible evidence with new data or experiments. Define the terms in your report.

Can a result be reproduced if the exact data are private?
Possibly, if authorized researchers can access them under suitable controls. Otherwise, document the access barrier and use permitted alternatives without implying that they are the same dataset.

What if the reproduction gives a different answer?
Check procedural and data differences, preserve both outcomes, and report uncertainty and deviations. Do not change the protocol after seeing results and present the revised analysis as prespecified confirmation.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.