文献速递重症综合管理与镇静镇痛

大语言模型在专家级重症医学问题上的比较评估与表现:一项基准研究

Comparative evaluation and performance of large language models on expert level critical care questions: a benchmark study.
PubMed 文献 · Crit Care · 2025-02-10
一句话结论在1181道欧洲重症医学文凭考试级选择题中,GPT-4o准确率最高达93.3%,四种模型在77题练习测试中超过人类医生(61.9%),但所有模型均存在一致性错误。

本研究评估五种大语言模型(GPT-4o、GPT-4o-mini、GPT-3.5-turbo、Mistral Large 2407、Llama 3.1 70B)在来自gotheextramile.com数据库的1181道重症医学选择题上的表现,并与随机猜测及350名人类医生在77题练习测试上的表现比较。GPT-4o准确率最高(93.3%),Llama 3.1 70B(87.5%)、Mistral Large 2407(87.9%)、GPT-4o-mini(83.0%)、GPT-3.5-turbo(72.7%)次之,随机猜测为41.5%(p < 0.001)。练习测试中所有模型得分均超过人类医生(61.9%),但GPT-3.5-turbo未显著优于医生(p = 0.196)。所有模型虽一致性高,但会一致性地给出错误答案;GPT-4o成本最高,超过最便宜模型GPT-4o-mini的25倍。

要点

  • 在1181道重症医学选择题中,GPT-4o准确率最高达93.3%,Llama 3.1 70B为87.5%,Mistral Large 2407为87.9%,GPT-4o-mini为83.0%,GPT-3.5-turbo为72.7%,均高于随机猜测的41.5%(p < 0.001)。
  • 在77题练习测试中,五种模型得分分别为89.0%、80.9%、84.4%、80.3%、66.5%,均超过人类医生的61.9%;但GPT-3.5-turbo未显著优于医生(p = 0.196),而其他模型显著优于医生(p < 0.001)。
  • 尽管整体一致性高,所有模型均存在一致性地给出错误答案的情况。
  • GPT-4o成本最高,超过最便宜模型GPT-4o-mini的25倍以上,作为能耗代理指标引发关注。
  • 作者认为,尽管大语言模型在重症医学中有潜力,但一致性的错误答案提示需要更彻底和持续的评估,以指导临床环境中的负责任实施。

适用人群与场景

  • 评估对象为五种大语言模型(GPT-4o、GPT-4o-mini、GPT-3.5-turbo、Mistral Large 2407、Llama 3.1 70B),测试题目来自gotheextramile.com数据库,为欧洲重症医学文凭考试级别的重症医学选择题。
  • 人类对照为350名医生,在77题练习测试上进行比较。

研究设计与关键结果

  • 研究类型为基准研究,评估五种大语言模型在1181道重症医学选择题上的表现,并与随机猜测及350名人类医生在77题练习测试上的表现比较。
  • 在1181道题中,GPT-4o准确率最高为93.3%,Llama 3.1 70B为87.5%,Mistral Large 2407为87.9%,GPT-4o-mini为83.0%,GPT-3.5-turbo为72.7%;随机猜测为41.5%(p < 0.001)。
  • 在77题练习测试中,GPT-4o得分为89.0%,Llama 3.1 70B为80.9%,Mistral Large 2407为84.4%,GPT-4o-mini为80.3%,GPT-3.5-turbo为66.5%,人类医生为61.9%,随机猜测为42.7%(p < 0.001)。
  • 除GPT-3.5-turbo外,其他模型在练习测试中均显著优于人类医生(p < 0.001);GPT-3.5-turbo与人类医生的差异未达统计学显著(p = 0.196)。
  • 所有模型整体一致性高,但均存在一致性地给出错误答案的情况。
  • 成本分析显示,GPT-4o成本最高,超过最便宜模型GPT-4o-mini的25倍以上。

证据类型、等级与局限

  • 证据类型:基准研究(benchmark study)。
  • 局限:原文未报告局限性。

对床旁实践的改变

  • 作者认为,大语言模型在重症医学中表现出卓越的准确性和一致性,四种模型在欧洲级别练习测试中超过人类医生,但所有模型均会一致性地给出错误答案,提示需要更彻底和持续的评估以指导临床环境中的负责任实施。

提炼说明

  • 本条依据论文摘要整理;摘要未涉及「诊断标准与关键阈值」「处置流程与药物剂量」「红旗征与不能漏的情况」「指南或共识出处」,故未列出这些栏目;全文请点「阅读原文」。
英文原文(摘要)· 点击展开

BACKGROUND: Large language models (LLMs) show increasing potential for their use in healthcare for administrative support and clinical decision making. However, reports on their performance in critical care medicine is lacking.

METHODS: This study evaluated five LLMs (GPT-4o, GPT-4o-mini, GPT-3.5-turbo, Mistral Large 2407 and Llama 3.1 70B) on 1181 multiple choice questions (MCQs) from the gotheextramile.com database, a comprehensive database of critical care questions at European Diploma in Intensive Care examination level. Their performance was compared to random guessing and 350 human physicians on a 77-MCQ practice test. Metrics included accuracy, consistency, and domain-specific performance. Costs, as a proxy for energy consumption, were also analyzed.

RESULTS: GPT-4o achieved the highest accuracy at 93.3%, followed by Llama 3.1 70B (87.5%), Mistral Large 2407 (87.9%), GPT-4o-mini (83.0%), and GPT-3.5-turbo (72.7%). Random guessing yielded 41.5% (p < 0.001). On the practice test, all models surpassed human physicians, scoring 89.0%, 80.9%, 84.4%, 80.3%, and 66.5%, respectively, compared to 42.7% for random guessing (p < 0.001) and 61.9% for the human physicians. However, in contrast to the other evaluated LLMs (p < 0.001), GPT-3.5-turbo's performance did not significantly outperform physicians (p = 0.196). Despite high overall consistency, all models gave consistently incorrect answers. The most expensive model was GPT-4o, costing over 25 times more than the least expensive model, GPT-4o-mini.

CONCLUSIONS: LLMs exhibit exceptional accuracy and consistency, with four outperforming human physicians on a European-level practice exam. GPT-4o led in performance but raised concerns about energy consumption. Despite their potential in critical care, all models produced consistently incorrect answers, highlighting the need for more thorough and ongoing evaluations to guide responsible implementation in clinical settings.

来源:Crit Care,查看原文。版权归原作者所有。

参考来源

  1. Workum JD, Volkers BWS, van de Sande D, et al. Comparative evaluation and performance of large language models on expert level critical care questions: a benchmark study. Crit Care. 2025. PMID: 39930514. doi:10.1186/s13054-025-05302-0

供执业医师抢救与诊疗参考,不替代临床判断;剂量请结合体重、肝肾功能和本院药品规格复核,以指南原文为准。