AI tools in science and engineering: what has been checked independently

For five AI tools used in science and engineering, what has a named independent check, what is only reported by its developers, and what limits do sources name?

The table sets out a headline result, and where an open source allows, an independent check, an official statement of routine use and a named limit, for four AI tools. Most headline figures come from the developers' own evaluations. Empty cells mark where no figure could be checked from an open source.

AI tools in science and engineering: what has been checked independently. Each cell shows a value, how well it is evidenced, and its sources.
Option	Headline published resultNamed paper or official benchmark, year, and the main figure its authors report (units: count, percent, score out of 42, or grid degrees and days).	Independent check or replicationNamed study or grading by people outside the developing team, with the figure it reports and the conditions it used.	Operational or routine useWhether an official source states the system is in routine or operational use, and the start date given.	Known failure, dispute or limitA failure, dispute or limit a cited source names, with its figure or detail where one is given.
AI weather forecasting	GraphCast (2022 preprint): significantly outperforms the most accurate operational deterministic systems on 90% of 1,380 verification targets (authors' evaluation; forecasts over 10 days at 0.25 degree resolution).Demonstrated[1]	No reliable public figure	ECMWF took AIFS Single into operations on 25 February 2025 (ECMWF announcement).Demonstrated[2]	ECMWF's gains of up to 20% and about 1,000 times less energy have no stated baseline, forecast period or verification method.Reported, unverified[2]
Protein structure prediction	AlphaFold-Multimer (2021 preprint): 14 targets reached at least medium accuracy (DockQ of 0.49 or higher), against 9 for the previous state of the art (authors' benchmark).Reported, unverified[3]	No reliable public figure	The AlphaFold Protein Structure Database, developed by Google DeepMind and EMBL-EBI, states it gives open access to over 260 million protein structure predictions (site figure, no publication date; the page gives over 200 million elsewhere).Demonstrated[4]	No reliable public figure
AI mathematical reasoning	Gemini Deep Think (IMO 2025): 35 of 42 points, five of six problems solved end to end in natural language (company announcement).Demonstrated[5]	IMO 2025: the solutions were officially graded and certified by IMO coordinators under the same criteria as student solutions, according to the developer's announcement.Reported, unverified[5]	No reliable public figure	The 2024 system (silver level, 28 of 42) needed competition problems translated by hand into formal mathematical language before it could attempt them.Demonstrated[6]
AI coding agents	SWE-bench (2023): 2,294 problems drawn from real GitHub issues; the paper's abstract reports that Claude 2 solved 1.96% of them (benchmark, 2023).Demonstrated[7]	METR 2025 randomised trial, 16 developers, 246 tasks: allowing AI tools increased completion time by 19% (confidence interval +2% to +39%). METR's February 2026 update says its later data gives only very weak evidence for the size of this increase.Demonstrated[8][9]	No reliable public figure	Liang and colleagues (2025): up to 76% file-path accuracy from issue text alone; the authors question the validity of existing SWE-bench results.Disputed[10]

AI tools in science and engineering: what has been checked independently. Each cell shows a value, how well it is evidenced, and its sources.

Option

Headline published resultNamed paper or official benchmark, year, and the main figure its authors report (units: count, percent, score out of 42, or grid degrees and days).

Independent check or replicationNamed study or grading by people outside the developing team, with the figure it reports and the conditions it used.

Operational or routine useWhether an official source states the system is in routine or operational use, and the start date given.

Known failure, dispute or limitA failure, dispute or limit a cited source names, with its figure or detail where one is given.

AI weather forecasting

GraphCast (2022 preprint): significantly outperforms the most accurate operational deterministic systems on 90% of 1,380 verification targets (authors' evaluation; forecasts over 10 days at 0.25 degree resolution).Demonstrated[1]

No reliable public figure

ECMWF took AIFS Single into operations on 25 February 2025 (ECMWF announcement).Demonstrated[2]

ECMWF's gains of up to 20% and about 1,000 times less energy have no stated baseline, forecast period or verification method.Reported, unverified[2]

Protein structure prediction

AlphaFold-Multimer (2021 preprint): 14 targets reached at least medium accuracy (DockQ of 0.49 or higher), against 9 for the previous state of the art (authors' benchmark).Reported, unverified[3]

No reliable public figure

The AlphaFold Protein Structure Database, developed by Google DeepMind and EMBL-EBI, states it gives open access to over 260 million protein structure predictions (site figure, no publication date; the page gives over 200 million elsewhere).Demonstrated[4]

No reliable public figure

AI mathematical reasoning

Gemini Deep Think (IMO 2025): 35 of 42 points, five of six problems solved end to end in natural language (company announcement).Demonstrated[5]

IMO 2025: the solutions were officially graded and certified by IMO coordinators under the same criteria as student solutions, according to the developer's announcement.Reported, unverified[5]

No reliable public figure

The 2024 system (silver level, 28 of 42) needed competition problems translated by hand into formal mathematical language before it could attempt them.Demonstrated[6]

AI coding agents

SWE-bench (2023): 2,294 problems drawn from real GitHub issues; the paper's abstract reports that Claude 2 solved 1.96% of them (benchmark, 2023).Demonstrated[7]

METR 2025 randomised trial, 16 developers, 246 tasks: allowing AI tools increased completion time by 19% (confidence interval +2% to +39%). METR's February 2026 update says its later data gives only very weak evidence for the size of this increase.Demonstrated[8][9]

No reliable public figure

Liang and colleagues (2025): up to 76% file-path accuracy from issue text alone; the authors question the validity of existing SWE-bench results.Disputed[10]

What the table shows

Most headline figures come from the developers' own papers or announcements, and the cells say so. The coding headline comes from an academic benchmark.
Independent checks are recorded for mathematics (IMO 2025 certification by IMO coordinators) and for coding agents (METR's 2025 trial, with METR's own 2026 caveat), but not for weather forecasting or protein structure prediction.
An official source states routine use for weather forecasting (ECMWF AIFS, from 25 February 2025) and for protein structures (the AlphaFold Protein Structure Database).
Named limits appear for weather (no stated baseline for ECMWF's figures), mathematics (manual translation in 2024) and coding (disputed validity of SWE-bench results).

Read with care

Headline figures are mostly the developers' own results on benchmarks or competitions they chose, so figures in different rows measure different things and should not be read side by side as a comparison of quality.
Conditions differ by row: the coding figure is a 2023 benchmark score, the mathematics figures are 2024 and 2025 competition scores, and the weather figure is a 2022 preprint evaluation.
Autonomous laboratories have no cell values in this version because the primary sources could not be opened from this environment, so the empty row reflects a gap in checking rather than an assessment of the field.
Benchmark scores and competition results do not show real-world reliability, productivity or safety in routine use.
Autonomous laboratories were dropped from this table because no open source gave a checkable figure for any column.

Sources

preprint
GraphCast: Learning skillful medium-range global weather forecasting
Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson et al.. arXiv, Dec 2022. Accessed 21 Oct 2026.
other
ECMWF's AI forecasts become operational
ECMWF, 25 Feb 2025. Accessed 21 Oct 2026.
preprint
Protein complex prediction with AlphaFold-Multimer
Richard Evans, Michael O'Neill, Alexander Pritzel et al.. bioRxiv, Oct 2021. Accessed 21 Oct 2026.
other
AlphaFold Protein Structure Database
EMBL-EBI, Oct 2026. Accessed 21 Oct 2026.
company announcement
Advanced version of Gemini with Deep Think officially achieves gold-medal standard at the International Mathematical Olympiad
Google DeepMind, 21 Jul 2025. Accessed 21 Oct 2026.
company announcement
AI achieves silver-medal standard solving International Mathematical Olympiad problems
Google DeepMind, 25 Jul 2024. Accessed 21 Oct 2026.
preprint
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Carlos E. Jimenez, John Yang, Alexander Wettig et al.. arXiv, Oct 2023. Accessed 21 Oct 2026.
preprint
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
Joel Becker, Nate Rush, Elizabeth Barnes et al.. arXiv, Jul 2025. Accessed 21 Oct 2026.
other
We are Changing our Developer Productivity Experiment Design
Joel Becker, Nate Rush, Tom Cunningham et al.. METR, 24 Feb 2026. Accessed 21 Oct 2026.
preprint
The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason
Shanchao Liang, Spandan Garg, Roshanak Zilouchian Moghaddam. arXiv, Jun 2025. Accessed 21 Oct 2026.