文献速递质量管理与危急值临床生化临床免疫

先进大语言模型在检验医学住院医师考试中的性能评估

Evaluation of the Performance of Advanced Large Language Models in Laboratory Medicine Using Residency Examinations.
PubMed 文献 · Ann Lab Med · 2025-11-13

该研究使用320道韩国住院医师考试题(2021-2024年)评估了12个大语言模型在检验医学六个亚专业中的表现。Gemini 2.0 Pro准确率最高(80.0%),GPT-4.5为77.2%,Claude 3.7 Sonnet为74.1%。准确率随题目难度增加而下降(简单题78.0% vs 难题45.1%),输血医学表现最差(平均准确率38.8%)。错误主要源于推理错误,大多数模型重现性超过95%。

要点

  • Gemini 2.0 Pro在检验医学考试中准确率最高(80.0%),其次为GPT-4.5(77.2%)和Claude 3.7 Sonnet(74.1%)。
  • 模型准确率随题目难度增加而下降:简单题78.0%,难题45.1%。
  • 输血医学亚专业表现最差(平均准确率38.8%),主要因领域特定和区域知识表示不足。

核心结论

  • 大语言模型在检验医学整合中展现巨大潜力,尤其在临床化学和免疫学方面;但性能不一致(尤其高难度题)和知识缺口(尤其输血医学)需进一步开发(如领域微调、检索增强生成)和专家监督后方可临床应用。

研究设计与样本

  • 使用320道韩国住院医师考试题(2021-2024年)评估12个来自OpenAI、Anthropic和Google的大语言模型,覆盖检验医学六个亚专业;通过API在确定性设置(temperature=0)下提供标准化提示,每题施测三次以评估响应重现性;输出与验证答案比较,分析准确性、推理质量和错误类型。

主要发现

  • Gemini 2.0 Pro准确率最高(80.0%),其次GPT-4.5(77.2%)和Claude 3.7 Sonnet(74.1%)。
  • 准确率随题目难度增加而下降:简单题78.0%,难题45.1%。
  • 输血医学亚专业平均准确率38.8%,主要因领域特定和区域知识表示不足。
  • 错误答案主要源于推理错误。
  • 大多数模型重现性超过95%,但即使贪婪解码(temperature=0)仍存在残余非确定性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: Recent advancements in large language models (LLMs) have accelerated their integration into clinical domains, including laboratory medicine. The performance of LLMs in answering board-level laboratory medicine questions has not been comprehensively evaluated. Given the importance of diagnostic accuracy in this field, rigorous and objective evaluations of LLM capabilities are essential. METHODS: We assessed 12 LLMs from OpenAI, Anthropic, and Google using 320 Korean Residency Examination questions (2021-2024) spanning six laboratory medicine subspecialties. Standardized prompts were provided via their application programming interfaces under deterministic settings (temperature=0). Questions were administered thrice to assess response reproducibility. Outputs were compared with validated answers and analyzed for accuracy, reasoning quality, and error typology. RESULTS: Google's Gemini 2.0 Pro achieved the highest accuracy (80.0%), followed by OpenAI's GPT-4.5 (77.2%) and Anthropic's Claude 3.7 Sonnet (74.1%). Accuracy decreased as the difficulty of questions increased (78.0% for easy vs. 45.1% for challenging). Subspecialty performance varied. Al models underperformed on questions on transfusion medicine (mean accuracy: 38.8%), primarily because of limitations in domain-specific and regional knowledge representations. Incorrect answers primarily resulted from reasoning errors. Reproducibility exceeded 95% for most models; however, some residual non-determinism appeared even with greedy decoding (temperature=0). CONCLUSIONS: LLMs demonstrated substantial potential for integration into laboratory medicine, particularly in clinical chemistry and immunology. Performance inconsistencies (particularly for high-difficulty questions) and knowledge gaps (notably for transfusion medicine) highlight the necessity for further development-potentially including domain-specific fine-tuning and retrieval-augmented generation integration-and robust expert oversight before clinical application.

来源:Ann Lab Med,查看原文。版权归原作者所有。

引用

  1. Jung K, Kim HJ, Shin S, et al. Evaluation of the Performance of Advanced Large Language Models in Laboratory Medicine Using Residency Examinations. Ann Lab Med. 2025. PMID: 41224529. doi:10.3343/alm.2025.0200

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。