文献速递医学人工智能与信息学流行病学与生物统计

基于大语言模型的分类器可减少筛选记录量且极少遗漏相关研究:一项案例研究

LLM-based classifiers can reduce the proportion of records to screen with minimal loss of relevant studies: a case study.
PubMed 文献 · J Clin Epidemiol · 2026-09-18

本研究评估Epistemonikos可持续知识平台中基于大语言模型(LLM)的分类器在摘要筛选中的表现,使用一项已完成但未发表的系统综述和网络荟萃分析作为参考标准,测试29种分类器组合在两种检索场景下自动排除记录的比例及遗漏相关研究的影响。结果显示分类器可排除5.8%至46.6%的原始检索记录和7.7%至34.3%的SKP检索记录,虽偶有遗漏研究,但网络荟萃分析效应估计变化极小(风险比变化0.0-0.04),且未改变证据确定性评级。结论认为LLM分类器是减少筛选量的有前景策略,但需跨医学主题进一步验证普适性。

要点

  • LLM分类器可自动排除5.8%-46.6%的原始检索记录和7.7%-34.3%的SKP检索记录。
  • 分类器遗漏相关研究时,网络荟萃分析风险比变化仅0.0-0.04,证据确定性不变。
  • 更特异的分类器排除更多记录,但可能遗漏更多相关研究,需权衡利弊。

核心结论

  • LLM分类器能减少摘要筛选记录量且遗漏相关研究的风险较低,但需多主题验证普适性。

研究设计与样本

  • 以一项已完成但未发表的系统综述和网络荟萃分析为参考标准,评估29种LLM分类器组合在两种检索场景下的表现。

主要发现

  • 原始检索场景中分类器排除5.8%至46.6%的1129条记录,SKP检索场景中排除7.7%至34.3%的1003条记录。
  • 29种组合中,10种错误排除了相同的两项相关研究;SKP场景中4种遗漏一项研究、10种遗漏两项研究。
  • 遗漏研究后网络荟萃分析风险比变化0.0-0.04,证据确定性评级未改变。

临床怎么用

  • LLM分类器可用于辅助摘要筛选,减少人工阅读量,但需监测遗漏相关研究的风险。
  • 应用时应根据研究问题权衡分类器的特异性,避免过度排除相关研究。

局限性

  • 仅基于单一系统综述案例,结果普适性需跨不同医学主题进一步评估。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND AND OBJECTIVE: The Epistemonikos Sustainable Knowledge Platform (SKP) integrates a suite of machine learning tools designed to make evidence synthesis more efficient, including access to the Epistemonikos Database of Trials (ED-Trials). One feature is the use of classifiers combined with large language models (LLMs) to support abstract screening during study selection. LLM-based classifiers can be applied to populations, interventions, and study design features. Their primary aim is to reduce the proportion of records needed to be screened by automatically excluding records with a high probability of being ineligible. We aimed to evaluate SKP's LLM-based classifiers on abstract screenings of records retrieved via (scenario 1) a traditional systematic literature search (original) and (scenario 2) a search conducted within ED-Trials and subsequently forwarded to SKP. METHODS: We used data from a completed but at the time unpublished systematic review and network meta-analysis (NMA) as reference standards. We evaluated 29 LLM-based classifier combinations in both scenarios and assessed the proportion of records excluded by automated screening and whether any relevant study was wrongly excluded and therefore missed. To determine the impact of missed studies, we reran the network meta-analyses of the original review, excluding missed studies from the model, and we evaluated how missed studies impact the certainty of evidence ratings. RESULTS: LLM-based classifiers excluded by automated screening, 5.8% to 46.6% of 1129 records identified by the original search and 7.7% to 34.3% of 1003 records based on the SKP search. More specific classifiers excluded more records than broader ones. Of the 29 combinations applied to the original search results, 10 incorrectly excluded the same two relevant studies. Based on the SKP review, four combinations missed one study, and 10 combinations missed two studies. However, exclusion of these studies resulted in minimal changes to NMA effect estimates (risk ratio changes of 0.0-0.04 in both scenarios) and did not alter certainty of evidence ratings. CONCLUSION: LLM-based classifiers are a promising strategy for reducing the amount of records to screen while keeping the risk of missing relevant studies low. However, further evaluations using multiple use cases across different medical topics are necessary to learn more about the generalizability of our results. PLAIN LANGUAGE SUMMARY: When conducting a systematic review, research teams typically have to read thousands of research summaries to find the studies relevant to their review question. We tested whether classifiers powered by large language models (a form of artificial intelligence) could help by automatically filtering out clearly irrelevant studies. We tested these classifiers within the Epistemonikos Sustainable Knowledge Platform, a software that supports systematic review conduct. We used a completed systematic review (that was not published at that time) as our benchmark. We then tested 29 classifiers in two situations: one where studies were gathered using a comprehensive literature search in multiple databases and sources and one where they came from Epistemonikos's own trial database. For each setup, we measured how many studies were filtered out and how many of them were wrongly excluded. When relevant studies were missed, we reran original meta-analysis without them to see if their absence would influence the overall results of the review. Classifiers removed between 6% and 47% of studies from the comprehensive search and 8% to 34% from the Epistemonikos database search. Even when relevant studies were excluded, the estimated treatment effects shifted only slightly and the confidence in the evidence stayed the same. These classifiers look like a useful way to cut down the proportion of research summaries to be read; however, more testing across different topics is needed before we can be confident that the approach works well.

来源:J Clin Epidemiol,查看原文。版权归原作者所有。

引用

  1. Nussbaumer-Streit B, Dobrescu A, Sharifan A, et al. LLM-based classifiers can reduce the proportion of records to screen with minimal loss of relevant studies: a case study. J Clin Epidemiol. 2026. PMID: 42759856. doi:10.1016/j.jclinepi.2026.112517

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。