先进大语言模型在检验医学住院医师考试中的性能评估
该研究使用320道韩国住院医师考试题(2021-2024年)评估了12个大语言模型在检验医学六个亚专业中的表现。Gemini 2.0 Pro准确率最高(80.0%),GPT-4.5为77.2%,Claude 3.7 Sonnet为74.1%。准确率随题目难度增加而下降(简单题78.0% vs 难题45.1%),输血医学表现最差(平均准确率38.8%)。错误主要源于推理错误,大多数模型重现性超过95%。
要点
- Gemini 2.0 Pro在检验医学考试中准确率最高(80.0%),其次为GPT-4.5(77.2%)和Claude 3.7 Sonnet(74.1%)。
- 模型准确率随题目难度增加而下降:简单题78.0%,难题45.1%。
- 输血医学亚专业表现最差(平均准确率38.8%),主要因领域特定和区域知识表示不足。
核心结论
- 大语言模型在检验医学整合中展现巨大潜力,尤其在临床化学和免疫学方面;但性能不一致(尤其高难度题)和知识缺口(尤其输血医学)需进一步开发(如领域微调、检索增强生成)和专家监督后方可临床应用。
研究设计与样本
- 使用320道韩国住院医师考试题(2021-2024年)评估12个来自OpenAI、Anthropic和Google的大语言模型,覆盖检验医学六个亚专业;通过API在确定性设置(temperature=0)下提供标准化提示,每题施测三次以评估响应重现性;输出与验证答案比较,分析准确性、推理质量和错误类型。
主要发现
- Gemini 2.0 Pro准确率最高(80.0%),其次GPT-4.5(77.2%)和Claude 3.7 Sonnet(74.1%)。
- 准确率随题目难度增加而下降:简单题78.0%,难题45.1%。
- 输血医学亚专业平均准确率38.8%,主要因领域特定和区域知识表示不足。
- 错误答案主要源于推理错误。
- 大多数模型重现性超过95%,但即使贪婪解码(temperature=0)仍存在残余非确定性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND: Recent advancements in large language models (LLMs) have accelerated their integration into clinical domains, including laboratory medicine. The performance of LLMs in answering board-level laboratory medicine questions has not been comprehensively evaluated. Given the importance of diagnostic accuracy in this field, rigorous and objective evaluations of LLM capabilities are essential. METHODS: We assessed 12 LLMs from OpenAI, Anthropic, and Google using 320 Korean Residency Examination questions (2021-2024) spanning six laboratory medicine subspecialties. Standardized prompts were provided via their application programming interfaces under deterministic settings (temperature=0). Questions were administered thrice to assess response reproducibility. Outputs were compared with validated answers and analyzed for accuracy, reasoning quality, and error typology. RESULTS: Google's Gemini 2.0 Pro achieved the highest accuracy (80.0%), followed by OpenAI's GPT-4.5 (77.2%) and Anthropic's Claude 3.7 Sonnet (74.1%). Accuracy decreased as the difficulty of questions increased (78.0% for easy vs. 45.1% for challenging). Subspecialty performance varied. Al models underperformed on questions on transfusion medicine (mean accuracy: 38.8%), primarily because of limitations in domain-specific and regional knowledge representations. Incorrect answers primarily resulted from reasoning errors. Reproducibility exceeded 95% for most models; however, some residual non-determinism appeared even with greedy decoding (temperature=0). CONCLUSIONS: LLMs demonstrated substantial potential for integration into laboratory medicine, particularly in clinical chemistry and immunology. Performance inconsistencies (particularly for high-difficulty questions) and knowledge gaps (notably for transfusion medicine) highlight the necessity for further development-potentially including domain-specific fine-tuning and retrieval-augmented generation integration-and robust expert oversight before clinical application.
引用
- Jung K, Kim HJ, Shin S, et al. Evaluation of the Performance of Advanced Large Language Models in Laboratory Medicine Using Residency Examinations. Ann Lab Med. 2025. PMID: 41224529. doi:10.3343/alm.2025.0200
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。