大语言模型用于系统综述表现良好但少有可验证保障:横断面研究
该横断面元流行病学研究检索9个数据库至2025年12月1日,纳入229项评估生成式大语言模型用于系统综述核心任务的实证研究(共440项任务)。报告透明度中等(平均0.52/1),方案信息和模型细节等可复现性关键领域存在明显缺口,防止信息泄漏的保障措施极少报告,作者对性能和实践就绪度的声明多持谨慎措辞。
要点
- LLM用于系统综述的评估研究报告透明度中等,可复现性关键细节常缺失。
- 防止过拟合和数据泄漏的保障措施极少报告,如锁定测试集在99.8%任务中未报告。
- 作者常声明性能优越和实践就绪,但多使用限定性措辞。
核心结论
- 应用LLM于系统综述任务的研究报告透明度中等,但常省略评估泄漏和过拟合所需的可复现性关键细节。
- 作者常对性能和实践就绪度做出声明,但通常表达谨慎;在推荐常规使用LLM于证据合成前,亟需改进报告标准和明确保障措施。
研究设计与样本
- 横断面元流行病学研究,检索PubMed、Embase、Web of Science核心合集、IEEE Xplore等9个数据库从建库至2025年12月1日的同行评议文章和预印本。
- 纳入评估生成式Transformer大语言模型(如Gemini)用于核心系统综述任务(如筛选、数据提取)并与参考标准对照的实证研究。
- 采用改编的Chatbot评估报告工具评估报告透明度,采用改编的诊断准确性研究质量评估工具2评估方法学质量,并分析作者声明和注意事项的频率与强度。
- 研究已在Open Science Framework注册(https://osf.io/8edhb)。
主要发现
- 共纳入229项研究,包含440项实证任务。
- 报告透明度中等,平均条目得分为0.52(标准差0.30),采用0-1量表,得分越高表示报告越完整。
- 可复现性关键领域存在明显缺口:方案信息平均得分0.12,模型细节平均得分0.30。
- 尽管60.6%的评估被评为低偏倚风险,但针对过拟合和数据泄漏的关键保障措施极少报告;例如,在提示优化前锁定测试集这一防信息泄漏的基本保护措施,在99.8%的任务中未报告。
- 共识别837条声明和693条注意事项;作者弱框架声明多于强框架声明(66.8% vs 33.2%);性能优于对照是最常见声明(64.8%的任务);47.5%的任务声称具备实践就绪度,其中几乎均使用限定性措辞(93.3%)。
临床怎么用
- 在将LLM用于系统综述的证据合成前,应关注研究是否报告了防止数据泄漏和过拟合的保障措施,如提示优化前锁定测试集。
- 评估LLM在系统综述中的表现时,需核查方案信息、模型细节等可复现性关键领域的报告完整性。
- 对LLM性能优越和实践就绪的声明应结合其措辞强度谨慎解读,避免过度外推。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVES: The application of large language models (LLMs) to systematic review tasks is rapidly expanding, yet the transparency and methodological rigor of these evaluations remain unclear. We aimed to assess reporting transparency, methodological quality, and how authors frame claims and caveats in studies applying LLMs to systematic review tasks. STUDY DESIGN AND SETTING: We conducted a cross-sectional meta-epidemiological study by searching PubMed, Embase, Web of Science Core Collection, IEEE Xplore, and five other databases from inception to December 1, 2025, for peer-reviewed articles and preprints. We included empirical studies evaluating generative transformer-based LLMs (eg, Gemini) for core systematic review tasks (eg, screening, data extraction) against a reference standard. We assessed reporting transparency using an adapted Chatbot Assessment Reporting Tool and methodological quality using an adapted Quality Assessment of Diagnostic Accuracy Studies 2 tool. We also analyzed the frequency and strength of claims and caveats mentioned by the authors. The study is registered with the Open Science Framework (https://osf.io/8edhb). RESULTS: We identified and included 229 studies comprising 440 empirical tasks. Reporting transparency was moderate, with a mean item score of 0.52 (standard deviation 0.30) on a 0-1 scale, where higher values indicate more complete reporting. We observed substantial gaps in reproducibility-essential domains, including protocol information (mean score 0.12) and model details (0.30). Although 60.6% of assessments were rated as having a low risk, key safeguards against overfitting and data leakage were rarely reported; for example, locking the test set before prompt optimization, a basic protection against information leakage, was not reported in 99.8% of tasks. We identified 837 claims and 693 caveats. Authors framed claims weakly more often than strongly (66.8% vs 33.2%). Performance superiority over a comparator was the most common claim (64.8% of tasks). Readiness for practical use was claimed in 47.5% of tasks, almost always in qualified terms (93.3%). CONCLUSION: Studies applying LLMs to systematic review tasks are reported with moderate transparency but often omit reproducibility-critical details necessary to assess leakage and overfitting. Although authors frequently make claims about performance and practice readiness, these are typically expressed cautiously. Improved reporting standards and clearer safeguards are urgently needed before routine use of LLMs in evidence synthesis can be recommended. PLAIN LANGUAGE SUMMARY: LLMs, such as ChatGPT, are increasingly used to help carry out parts of systematic reviews, which summarize evidence to inform healthcare decisions. We examined 229 studies that tested LLMs on tasks such as screening articles, extracting data, and assessing study quality, covering 440 evaluations in total. On average, these studies reported their methods with moderate clarity, but often omitted information needed to repeat the work or judge whether the results were trustworthy. Notably, almost none described safeguards to ensure that the test data had not already influenced how the model was set up, a key step for avoiding overly optimistic results. Authors frequently described LLMs as performing well and nearly ready for practical use, though usually in cautious terms. Clearer reporting standards and stronger safeguards are needed before LLMs can be routinely relied upon in evidence synthesis.
引用
- Lai H, Sousa-Pinto B, Cao C, et al. Large language models for systematic reviews were reported to perform well but rarely with verifiable safeguards: a cross-sectional study. J Clin Epidemiol. 2026. PMID: 42309363. doi:10.1016/j.jclinepi.2026.112383
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。