Dennstädt et al. 2024 — LLM title/abstract screening (biomedical)

Systematic Reviews 2024 research article benchmarking four openly available LLMs on title-and-abstract screening, across ten published biomedical SLR datasets plus one newly created dataset. Probably the most rigorous single-stage SLR-automation benchmark published to date.

Why it matters here

Provides an empirical yardstick for when LLM-assisted screening is workable and which architectures to prefer. Complements the sceptical commentary of sources/2023-qureshi-chatgpt-sr-automation with real sensitivity/specificity numbers. Mixtral's 81.93%/75.19% on published datasets suggests an MoE open model can hit usable screening balance without proprietary APIs.

Method

Prompt template: [Instruction] + "Title: " + title + ", Abstract: " + abstract + [Relevant Criteria]. Relevant Criteria is hand-written per SLR.
Output: LLM is asked to emit a single integer on a Likert scale (default 1–5). A regex extracts the number.
Thresholding: a score cutoff (e.g. ≥3 = include) converts Likert output into a binary classifier.
Models tested: FlanT5-XXL, OpenHermes-2.5-Neural-Chat-7B, Mixtral-8×7B-Instruct, Platypus 2.
Datasets: 10 published biomedical SLR corpora + 1 new dataset constructed for this study.

Results

Model	Sensitivity (pub)	Specificity (pub)	Best of four on new set
FlanT5-XXL	94.48%	31.78%	12.58% @ 100% sens.
OpenHermes-NeuralChat	97.58%	19.12%	4.54%
Mixtral-8×7B	81.93%	75.19%	62.47%
Platypus 2	97.58%	38.34%	24.74%

Most models give high sensitivity at the cost of low specificity — i.e. they err toward inclusion, which is the right direction for a screening tool but still imposes reviewer burden. Mixtral is the exception with balanced performance.

Caveats

Prompt sensitivity — changing "3." to "The relevance is 3." and changing Likert range changes outcomes considerably; systematic prompt engineering is mandatory.
No LLM reached production-grade specificity on most datasets; use as first-pass, not sole reviewer.

Connections

methods/systematic-literature-review — screening is stage 6.2 (Kitchenham §6.2).
concepts/llm-assisted-literature-review — hub.
entities/fabio-dennstaedt — first author.
Contrasts with sources/2023-qureshi-chatgpt-sr-automation (qualitative scepticism) and sources/2025-scherbakov-llms-as-tools-literature-reviews (meta-empirical mapping).