Forensic Metascience for Policymakers: How AI Can Clean Up the Scientific Record

Summary: The Metascience Observatory’s research has demonstrated that the replication crisis is a science-wide phenomenon, not just limited to psychology and the social sciences. Across many fields, 40% or more of results fail to replicate. Fraud is a non-trivial contributor to the replication crisis. James Heathers estimates that 15% of peer-reviewed papers are contaminated with some form of deliberate fabrication. Whether the true number is 5% or 25%, the problem of fraud in science must be grappled with by policymakers. Science’s ongoing fraud problem reduces trust in science and slows scientific progress. In addition to fraud, many papers contain simple mistakes that AI systems can catch before publication. Research in forensic metascience using tools like statcheck and the GRIM test shows that calculation mistakes are common, with about half of papers in psychology having at least one error in the calculation of p-values or averages. Most errors are harmless, but some change key results, contaminating meta-analyses and causing downstream problems. The Metascience Observatory is developing the Forensic Metascience AI Toolkit to address these problems. The toolkit can be used to stress test existing papers or audit new papers before they are published. Using AI to check papers is not a new idea, and many startups are already selling “AI for peer review” services. However, auditing the text and images of a single paper using commercial services currently costs $50 or more. The Metascience Observatory’s toolkit does a thorough audit of methods, calculations, images, and data. When used with subscription services like Claude Code on the backend, the cost is only $0.50 -$1.00 per paper. I’ll make the case for why funding agencies should invest in and promote forensic metascience, encourage AI peer review, clean up the scientific record with the help of AI, and crack down on fraud by cutting off funding. The very future of science is at stake. Bio: Dan Elton is the Fo