文献速递眼底病与葡萄膜炎

大型语言模型能否完全自动化或部分辅助系统综述中的论文筛选?

Can large language models fully automate or partially assist paper selection in systematic reviews?
PubMed 文献 · Br J Ophthalmol · 2025-07-22

本研究以三项已发表的糖尿病视网膜病变患病率系统综述(共98篇论文)为基准,测试了基于GPT-4的全自动(LLM-FA)与半自动(LLM-SA)筛选方法。结果显示全自动方法中Consensus GPT仅正确识别32.7%的论文,而半自动方法成功纳入82.7%的论文并正确排除92.2%的不相关论文。结论认为LLM尚不能自主完成系统综述的论文筛选,但有望作为辅助工具提高筛选效率。

要点

  • 全自动方法中Consensus GPT正确识别32.7%(32/98)的论文,Scholar GPT为19.4%(19/98),GPT4网页浏览模式为6.1%(6/98)。
  • 半自动方法成功纳入82.7%(81/98)的论文,并正确排除92.2%的4497篇不相关论文。
  • LLM尚不能自主识别和筛选系统综述的相关论文,但有望作为辅助工具提高筛选效率。

核心结论

  • 大型语言模型尚不能自主识别和筛选系统综述中的相关论文,但有望作为辅助工具提高论文筛选过程的效率。

研究设计与样本

  • 研究引入两种基于大型语言模型的系统综述方法:全自动方法(LLM-FA)使用三种GPT-4插件(Consensus GPT、Scholar GPT和GPT网页浏览模式),半自动方法(LLM-SA)使用GPT-4的应用程序编程接口(API)。
  • 以三项已发表的、报告不同人群(一般人群、孕妇和儿童)糖尿病视网膜病变患病率的系统综述为基准进行测试,三项综述共包含98篇论文。

主要发现

  • 在LLM-FA方法中,Consensus GPT正确识别32.7%(32/98)的论文,Scholar GPT正确识别19.4%(19/98),GPT4网页浏览模式正确识别6.1%(6/98)。
  • LLM-SA方法成功纳入82.7%(81/98)的论文,并正确排除4497篇不相关论文中的92.2%。

关键数值与推荐

  • 全自动方法中Consensus GPT正确识别32.7%(32/98)的论文,Scholar GPT为19.4%(19/98),GPT4网页浏览模式为6.1%(6/98)。
  • 半自动方法成功纳入82.7%(81/98)的论文,并正确排除4497篇不相关论文中的92.2%。

临床怎么用

  • 原文未据此给出推荐。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND/AIMS: Large language models (LLMs) have substantial potential to enhance the efficiency of academic research. The accuracy and performance of LLMs in a systematic review, a core part of evidence building, has yet to be studied in detail. METHODS: We introduced two LLM-based approaches of systematic review: an LLM-enabled fully automated approach (LLM-FA) utilising three different GPT-4 plugins (Consensus GPT, Scholar GPT and GPT web browsing modes) and an LLM-facilitated semi-automated approach (LLM-SA) using GPT4's Application Programming Interface (API). We benchmarked these approaches using three published systematic reviews that reported the prevalence of diabetic retinopathy across different populations (general population, pregnant women and children). RESULTS: The three published reviews consisted of 98 papers in total. Across these three reviews, in the LLM-FA approach, Consensus GPT correctly identified 32.7% (32 out of 98) of papers, while Scholar GPT and GPT4's web browsing modes only identified 19.4% (19 out of 98) and 6.1% (6 out of 98), respectively. On the other hand, the LLM-SA approach not only successfully included 82.7% (81 out of 98) of these papers but also correctly excluded 92.2% of 4497 irrelevant papers. CONCLUSIONS: Our findings suggest LLMs are not yet capable of autonomously identifying and selecting relevant papers in systematic reviews. However, they hold promise as an assistive tool to improve the efficiency of the paper selection process in systematic reviews.

来源:Br J Ophthalmol,查看原文。版权归原作者所有。

引用

  1. Chen H, Jiang Z, Liu X, et al. Can large language models fully automate or partially assist paper selection in systematic reviews? Br J Ophthalmol. 2025. PMID: 39814458. doi:10.1136/bjo-2024-326254

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。