文献速递哮喘

评估大型语言模型回答哮喘多选题与客观结构化临床考试题的准确性

Evaluating the Accuracy of Large Language Models in Answering Asthma Multiple Choice and Objective Structured Clinical Examination Questions.
PubMed 文献 · Chest · 2025-12-08

本研究在116道成人哮喘多选题和3个哮喘OSCE上,对14个LLM聊天机器人各进行5次迭代测试,采用广义线性混合效应模型比较准确性。结果显示多选题准确率优秀(13/14模型超过85%),OSCE准确率多数达70%至86%;LLM在多选题上表现优于OSCE(92.8% vs 71.1%,P < .001),患者导向多选题优于临床医生导向(96.3% vs 85.6%,P < .01)。

要点

  • 13/14个模型哮喘多选题准确率超过85%,其中5个模型超过95%。
  • LLM多选题准确率显著高于OSCE(92.8% vs 71.1%,P < .001)。
  • 患者导向多选题准确率高于临床医生导向(96.3% vs 85.6%,P < .01)。

核心结论

  • 当前LLM在哮喘相关临床和患者导向问题上表现出高准确性,尤其是ChatGPT系列模型。
  • 这些模型可能有助于整合到未来的患者和临床医生教育干预中。

研究设计与样本

  • 研究测试了14个LLM聊天机器人(Asthma-GPT、ChatGPT-4o、ChatGPT-o1、ChatGPT-o3-mini、ChatGPT-o3-mini-high、ChatGPT-5、Claude-3.7-Sonnet、DeepSeek-V3、Gemini-2.0-Flash、Grok-3、Le-Chat、Llama-3、NICE-Asthma-Bot和Biomistral)。
  • 每个模型在116道成人哮喘多选题和3个哮喘OSCE上各进行5次迭代测试。
  • 使用广义线性混合效应模型比较准确性,采用对数优势尺度和Tukey调整进行两两比较。

主要发现

  • 多选题准确率优秀,14个模型中13个超过85%,ChatGPT-o1、Asthma-GPT、NICE-Asthma-Bot、ChatGPT-5和ChatGPT-4o超过95%。
  • OSCE准确率变异更大,但大多数模型仍达到70%至86%。
  • ChatGPT-o1在多选题中排名最高,ChatGPT-5在OSCE中排名最高。
  • LLM在多选题上表现优于OSCE(92.8% vs 71.1%,P < .001)。
  • LLM在患者导向多选题上表现优于临床医生导向多选题(96.3% vs 85.6%,P < .01)。

关键数值与推荐

  • 多选题准确率:13/14模型超过85%,5个模型超过95%。
  • OSCE准确率:大多数模型70%至86%。
  • 多选题 vs OSCE准确率:92.8% vs 71.1%,P < .001。
  • 患者导向 vs 临床医生导向多选题准确率:96.3% vs 85.6%,P < .01。

临床怎么用

  • LLM可作为哮喘患者教育和临床医生教育的辅助工具,但需注意OSCE类任务准确性较低。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: Large language models (LLMs) are demonstrating increasing promise across clinical applications, but their domain-specific knowledge in asthma has not been thoroughly explored. Additionally, state-of-the-art models released in 2025 (ChatGPT-5, ChatGPT-o3, Claude-3.7, DeepSeek-V3, and Grok-3) have yet to be studied in asthma. RESEARCH QUESTION: How accurate are LLMs at answering adult asthma multiple choice questions (MCQs) and Objective Structured Clinical Examinations (OSCEs)? STUDY DESIGN AND METHODS: Fourteen LLM chatbots (Asthma-GPT, ChatGPT-4o, ChatGPT-o1, ChatGPT-o3-mini, ChatGPT-o3-mini-high, ChatGPT-5, Claude-3.7-Sonnet, DeepSeek-V3, Gemini-2.0-Flash, Grok-3, Le-Chat, Llama-3, NICE-Asthma-Bot, and Biomistral) were tested in 5 iterations on 116 adult asthma MCQs and 3 asthma OSCEs. Using generalized linear mixed-effects models, we compared accuracy between the LLMs using log-odds scale and pairwise comparisons with Tukey adjustment, and assessed difference in MCQs vs OSCEs, generic vs medicine-specific models, open-source vs proprietary models, and patient- vs clinician-oriented questions. RESULTS: MCQ accuracy was excellent, with 13 of 14 models exceeding 85%, and ChatGPT-o1, Asthma-GPT, NICE-Asthma-Bot, ChatGPT-5, and ChatGPT-4o surpassing 95% accuracy. OSCE accuracy varied more significantly, but most models still achieved 70% to 86% accuracy. ChatGPT-o1 ranked highest in MCQs, and ChatGPT-5 ranked highest in OSCEs. Generalized linear mixed-effects model revealed that LLMs performed better at MCQs than OSCEs (92.8% vs 71.1%, P < .001), and that LLMs performed better at patient-oriented MCQs than clinician-oriented MCQs (96.3% vs 85.6%, P < .01). No significant differences were found between open-source and proprietary models in MCQs, or between generic and medicine-specific LLMs. INTERPRETATION: Our results indicate that current LLMs demonstrate high accuracy across asthma-related clinical and patient-oriented questions, particularly the ChatGPT family of models. These models may be useful tools to integrate into future patient and clinician education interventions.

来源:Chest,查看原文。版权归原作者所有。

引用

  1. Li PY, Gershon A, Kouri A, et al. Evaluating the Accuracy of Large Language Models in Answering Asthma Multiple Choice and Objective Structured Clinical Examination Questions. Chest. 2025. PMID: 41371491. doi:10.1016/j.chest.2025.11.034

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。