Can AI Review Scientific Literature?

AI is making strides in the field of research synthesis, but experts are divided over its ability to fully analyze and summarize the vast landscape of scientific literature, as reported on Wednesday in a literature review published on Nature.

The paper by Helen Pearson explores how this technology seems to hold immense potential for streamlining scientific reviews, yet significant challenges remain.

Sam Rodriques, a former neurobiology student and director at the U.S. startup FutureHouse, is among those pushing for AI to tackle scientific literature, as reported on Nature.

In September, his team launched an AI-based system that, within minutes, created summaries on thousands of human genes, filling in details that previously went unrecorded.

This tool, called PaperQA2, has shown promise in generating summaries that sometimes surpass human-written content in accuracy, according to early tests,  as reported on Nature.

The appeal of AI for literature review is clear. Traditional research reviews, often lengthy and time-consuming, can take years to complete and risk being outdated by the time they are published.

AI, particularly large language models (LLMs) like ChatGPT, offers the possibility of quickly compiling data and summarizing information from vast databases, easing the burden on researchers, as noted on Nature.

Despite these advancements, AI-based reviews remain far from fully replacing human-led systematic reviews, which involve stringent criteria for assessing studies and synthesizing results, notes Pearson.

Tools such as Consensus and Elicit, AI-powered search engines, allow researchers to filter and summarize academic papers, providing a first layer of insights.

However, they are limited in their ability to conduct thorough, gold-standard reviews. Paul Glasziou, a specialist in systematic reviews at Bond University, suggests that full automation of these reviews could still be decades away, as repored by Pearson.

The article says how AI’s limitations raise concerns about accuracy and transparency. LLMs, for instance, can generate content that lacks context or misrepresents data, often drawing from unreliable sources without weighing the quality of information.

Additionally, they are prone to “hallucinating” errors—creating references or data points that don’t exist.

To mitigate this, some systems allow users to upload specific papers to an LLM, which can then base its analysis only on the uploaded sources, reducing inaccuracies but not fully eliminating them, says Pearson.

Critics warn that AI could flood the scientific landscape with low-quality or even misleading summaries. James Thomas of University College London cautions that poorly executed reviews could undermine years of evidence-based practices, as noted on Nature.

Ultimately, while AI offers a promising tool for accelerating the review process, experts stress the need for rigorous oversight and transparency if it is to genuinely enhance scientific understanding.