文献速递医学人工智能与信息学流行病学与生物统计

基于标题和摘要自动识别物理治疗临床试验:商业大语言模型与人工方法的比较

Automated approaches to identifying clinical trials based on title and abstract in the field of physiotherapy: a comparative analysis.
PubMed 文献 · J Clin Epidemiol · 2026-04-30

本研究比较商业工具(ChatGPT与Copilot中的GPT-4o、GPT-4.5、GPT-4-turbo)与人工方法在10,793条新发表记录中基于标题和摘要筛选物理治疗试验的准确性、精确率、召回率、F1及耗时。商业工具准确率83%–86%、精确率44%–48%、召回率87%–88%、F1 59%–62%,耗时仅为人工筛选的37%。探索性分析显示API实现性能相当,但基于大语言模型的自动方法低于其他本地定制自动化方法(机器学习、自然语言处理)。

要点

  • 商业大语言模型筛选物理治疗试验的准确率83%–86%,召回率87%–88%,精确率44%–48%。
  • 使用商业工具加标注数据集的总耗时约为纯人工筛选过程的37%。
  • 基于大语言模型的自动方法性能低于机器学习或自然语言处理等本地定制方法。

核心结论

  • 商业网页版大语言模型可能具有足够准确性支持标题和摘要筛选,并大幅减少识别领域特定试验的时间。
  • 机器学习或自然语言处理等替代方法可获得相似或略高的筛选性能,但需一系列预处理步骤。

研究设计与样本

  • 比较研究,对10,793条新发表记录进行标题和摘要筛选。
  • 参考标准为人工方法:两名评价者独立按预设规则筛选,分歧由第三人解决。
  • 评估ChatGPT和Copilot中三种大语言模型(gpt-4o、gpt-4.5、gpt-4-turbo)。
  • 探索性分析比较商业工具与本地方法,包括本地大语言模型实现、机器学习和自然语言处理。

主要发现

  • ChatGPT vs Copilot:准确率83% vs 86%;精确率44% vs 48%;召回率88% vs 87%;F1 59% vs 62%。
  • 使用商业工具加标注数据集的总耗时相当于纯人工筛选过程的37%。
  • 基于API的实现性能相当:准确率82%;精确率42%;召回率93%;F1 58%。
  • 基于大语言模型的模型性能低于其他本地定制自动化方法(机器学习、自然语言处理)。

临床怎么用

  • 商业网页版大语言模型可用于辅助标题和摘要筛选,以减少筛选时间。
  • 若追求更高筛选性能,可考虑机器学习或自然语言处理等本地定制方法,但需预处理步骤。

局限性

  • 本研究为概念验证研究。
  • 探索性分析显示基于大语言模型的方法性能低于其他本地定制方法。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVES: To compare accuracy, precision, recall, F1, and time spent using commercial tools to identify physiotherapy trials based on title and abstract, compared with a human approach. STUDY DESIGN: This study compared two approaches for title and abstract screening of 10,793 newly published records. In the reference standard human approach, two reviewers independently screened records using prespecified rules to assess relevance to physiotherapy. A third person resolved disagreements. We evaluated three large language models (LLMs) (gpt-4o, gpt-4.5, and gpt-4-turbo) within two commercial, web-based tools (ChatGPT and Copilot). Outcomes were accuracy (proportion of records that model correctly identified as relevant or irrelevant), precision (proportion of records identified as relevant that were considered as relevant by human approach), recall (the proportion of all actual relevant records that the model successfully identified), F1 (harmonic mean of precision and recall), and time spent. Exploratory analyses compared the performance of the commercial tools with local approaches, including local LLMs implementation, machine learning, and natural language processing. RESULTS: Commercial tools showed comparable performance across all metrics (ChatGPT vs Copilot: accuracy: 83% vs 86%; precision: 44% vs 48%; recall: 88% vs 87%; F1: 59% vs 62%). The total time spent using commercial tools with a labeled dataset was equivalent to 37% of the time required for the human-only screening process. Exploratory analysis showed that the Application Programming Interface-based implementation has comparable performance (accuracy: 82%; precision: 42%; recall: 93%; F1: 58%). Yet, LLM-based models demonstrated lower performance compared with other local, custom-adapted automation approaches such as machine learning and natural language processing. CONCLUSION: This proof-of-concept study demonstrates that commercial web-based LLMs may have sufficient accuracy to support title and abstract screening and substantially reduce the time to identify field-specific trials. However, alternative approaches, including machine learning or natural language processing, could achieve screening performance similar to or slightly higher than that of commercial tools, yet they require a series of preprocessing steps for implementation.

来源:J Clin Epidemiol,查看原文。版权归原作者所有。

引用

  1. Kwok WS, Wallbank G, Hodgson P, et al. Automated approaches to identifying clinical trials based on title and abstract in the field of physiotherapy: a comparative analysis. J Clin Epidemiol. 2026. PMID: 42069070. doi:10.1016/j.jclinepi.2026.112309

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。