Scherbakov et al. 2025 — LLMs as tools in literature reviews (LLM-assisted SR)

A JAMIA review that is both (a) a systematic review of LLM-assisted review automation and (b) an applied demonstration — the authors used an LLM-augmented Covidence workflow to perform the review itself. Published May 2025 (advance access) — the most recent of the four LLM-era papers in this batch.

Why it matters here

Supplies a landscape map of which SR stages are being automated and how well, plus a self-demonstrating pipeline (Covidence + GPT-4o plugin) that operationalises the patterns advocated by sources/2023-qureshi-chatgpt-sr-automation (always keep a human), sources/2024-dennstaedt-llm-title-abstract-screening (screening as Likert classifier) and sources/2024-agarwal-litllms-are-we-there-yet (retrieval + generation).

Coverage of the field (172 eligible studies)

By review type automated: Systematic Review (68.6%), Literature/Narrative Review (21.5%), Meta-Analysis (11.0%), Scoping Review (4.7%), Umbrella (1.2%).

By stage automated: Searching 34.9%, Data extraction 31.4%, Title/abstract screening 25.0%, Evidence synthesis 18.6%, Drafting 12.8%, Full-text screening 8.1%, Quality/bias assessment 7.0%.

By model family: GPT/ChatGPT 73.2%, BERT-family 18.6%, LLaMA/Alpaca 4.7%, Claude 4.1%, Gemini 2.9%.

Demonstration pipeline

Covidence plugin wraps OpenAI GPT-4o via Azure; Python/R intermediary passes content between Covidence and the LLM.
Three automated stages: abstract screening, full-text screening, extraction — each with 2 calibrated human reviewers + LLM voting 3 inference runs for self-consistency (majority vote).
Human-LLM consensus via: 2 humans agree → that is human consensus → compared against LLM vote; disagreement reveals LLM false positives/negatives.
Extraction validated by single human; low-precision categories (<80%) reassigned to human.
LLM drafted ~40% of Introduction, ~90% of Results, ~30% of Discussion, subsequently edited.

Performance

GPT-4o extraction: precision 83.0% (SD 10.4), recall 86.0% (SD 9.8) against expert gold standard; outperforms BERT baselines from pre-LLM era.
Numeric extraction lower accuracy — flagged as a specific weakness.

Connections

methods/systematic-literature-review — covers all Kitchenham stages via LLM assistance.
concepts/llm-assisted-literature-review — hub; this paper consolidates the batch.
entities/dmitry-scherbakov — first author.
Applies patterns from sources/2024-dennstaedt-llm-title-abstract-screening (screening) and sources/2024-agarwal-litllms-are-we-there-yet (retrieval + writing).