Agarwal et al. 2024 — LitLLMs: LLMs for literature review, are we there yet?

TMLR 2024 paper from ServiceNow Research + Mila that decomposes literature-review writing into retrieval + generation and evaluates a pipeline of LLM-based tricks over each sub-task. The project page ships code and a demo at litllm.github.io.

Why it matters here

The most operationally concrete of the LLM-for-literature-review papers: produces a pipeline specification (keyword extraction → dual retrieval → re-ranking → plan → generation) with ablations showing which tricks pay off. The plan-then-generate pattern is the strongest hallucination-reduction finding so far.

Pipeline

Keyword extraction — LLM takes an abstract or research-idea paragraph and emits search keywords.
Retrieval — two-track: LLM-generated keywords into Google Scholar / Semantic Scholar, plus embedding-based similarity over abstracts. Combining both beats either alone.
Re-ranking — LLM scores candidate papers against the query abstract and attributes its reasoning to specific excerpts; debate-style aggregation between multiple LLM passes is also explored.
Plan — optional sentence-level outline ("cite paper X at sentence 3 to justify Y") generated by LLM, user, or hybrid.
Generation — LLM writes the related-work section conditioned on query abstract, top-k papers, and plan.

Findings

Keyword-from-abstract extraction outperforms using the raw abstract as a query.
Attribution-prompted re-ranking improves ROUGE and human-judged quality.
Plan-based generation reduces hallucinated citations by 18-26% vs. plan-free baselines — the headline result for scientific writing use.
Rolling arXiv protocol (most-recent-month papers) is proposed as a contamination-resistant evaluation design.

Limits

Abstract-only input means the method gives early-stage preliminary references, not a final curated bibliography. Evaluation via ROUGE and human judgement on arXiv ML papers — transfer to other domains (medicine, BPM) untested. Does not address the Kitchenham inclusion/exclusion-criteria rigour or quality assessment.

Connections