大语言模型在放射学决策中的应用
研究评估了ChatGPT和Glass AI在1075个临床场景中预测最合适影像检查的能力,以ACR适宜性标准为基准。Glass AI得分显著高于ChatGPT(2.32±0.67 vs 2.08±0.74,P=.002),在11个专家组中的10个表现更优。大语言模型可用于预测影像检查,Glass AI的额外医学文本训练带来优势。
要点
- Glass AI在11个专家组中的10个优于ChatGPT,仅妇产科例外。
- 两者在多发伤、乳腺和血管组表现最佳,在神经、骨肌和心脏组最差。
- Glass AI得分2.32±0.67,ChatGPT得分2.08±0.74,差异有统计学意义(P=.002)。
核心结论
- 大语言模型可预测影像检查,Glass AI的优越表现表明额外医学文本训练有益,支持大语言模型在放射学决策中的潜力。
研究设计与样本
- 研究使用OpenAI的ChatGPT和Glass Health的Glass AI,对来自11个ACR专家组的1075个临床场景进行测试,以ACR适宜性标准为基准;每个临床场景生成两次回答并取平均作为最终评分,评分范围0至3分,非特异性回答给予部分分数。
诊断标准与关键数值
- 评分范围为0至3分,非特异性回答给予部分分数;每个临床场景两次回答取平均作为最终评分,每个主题领域内临床场景评分取平均作为该主题最终评分,每个专家组内主题评分取平均作为该专家组最终评分。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
BACKGROUND AND PURPOSE: Large language models (LLMs) have seen explosive growth, but their potential role in medical applications remains underexplored. Our study investigates the capability of LLMs to predict the most appropriate imaging study for specific clinical presentations in various subspecialty areas in radiology.
METHODS AND MATERIALS: Chat Generative Pretrained Transformer (ChatGPT), by OpenAI and Glass AI by Glass Health were tested on 1,075 clinical scenarios from 11 ACR expert panels to determine the most appropriate imaging study, benchmarked against the ACR Appropriateness Criteria. Two responses per clinical presentation were generated and averaged for the final clinical presentation score. Clinical presentation scores for each topic area were averaged as its final score. The average of the topic scores within a panel determined the final score of each panel. LLM responses were on a scale of 0 to 3. Partial scores were given for nonspecific answers. Pearson correlation coefficient (R-value) was calculated for each panel to determine a context-specific performance.
RESULTS: Glass AI scored significantly higher than ChatGPT (2.32 ± 0.67 versus 2.08 ± 0.74, P = .002). Both LLMs performed the best in the Polytrauma, Breast, and Vascular panels, and performed the worst in the Neurologic, Musculoskeletal, and Cardiac panels. Glass AI outperformed ChatGPT in 10 of 11 panels, except Obstetrics and Gynecology. Maximum agreement was in the Pediatrics, Neurologic, and Thoracic panels, and the most disagreement occurred in the Vascular, Breast, and Urologic panels.
CONCLUSION: LLMs can be used to predict imaging studies, with Glass AI's superior performance indicating the benefits of extra medical-text training. This supports the potential of LLMs in radiologic decision making.
引用
- Zaki HA, Aoun A, Munshi S, et al. The Application of Large Language Models for Radiologic Decision Making. J Am Coll Radiol. 2024. PMID: 38224925. doi:10.1016/j.jacr.2024.01.007
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。