GPT-4在美国放射学会培训考试中的表现:评估准确性、模型漂移和微调
本研究评估GPT-4在美国放射学会2022年诊断放射学培训考试(DXIT)中的表现,并进行多次实验以评估模型漂移及微调后的准确性差异。GPT-4总体准确率为58.5%,低于PGY-3平均(61.9%)但高于PGY-2平均(52.8%);图像题准确率显著较低(45.4% vs 文本题80.0%,p<0.001);重复提问时25.5%的题目答案改变但准确率无变化;微调未提升准确率。
要点
- GPT-4在2022年DXIT中总体准确率58.5%,介于PGY-2(52.8%)与PGY-3(61.9%)平均水平之间。
- 图像题准确率(45.4%)显著低于纯文本题(80.0%),逻辑调整后图像题准确率36.4%。
- 重复提问时GPT-4有25.5%的题目选择不同答案,但总体准确率无变化;微调未改善准确率。
核心结论
- GPT-4在2022年DXIT中表现介于PGY-2与PGY-3水平之间,图像题准确率显著较差,且不同时间点答案选择变异性大;探索性微调实验未提升性能。
研究设计与样本
- 研究将美国放射学会2022年诊断放射学培训考试(DXIT)题目按顺序以标准化提示输入GPT-4,记录答案并计算总体准确率、逻辑调整准确率和图像题准确率;数月后重复实验评估模型漂移,再进行微调后评估性能变化。
诊断标准与关键数值
- GPT-4总体准确率58.5%,低于PGY-3平均61.9%但高于PGY-2平均52.8%;逻辑调整准确率52.8%。
- 图像题准确率45.4%,显著低于纯文本题80.0%(p<0.001);图像题逻辑调整准确率36.4%。
- 正确答案置信度87.1%显著高于错误答案84.0%(p=0.012)。
临床应用与阅片要点
- 使用通用AI模型解读放射学图像作为诊断工具时,实施者应谨慎,因可能存在虚假但自信的回答。
- GPT-4在图像题上准确率显著较低,且重复提问时25.5%的题目答案改变,提示答案稳定性有限。
局限性
- 本研究为探索性实验,微调未提升性能,原文指出需进一步研究。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: In our study, we evaluate GPT-4's performance on the American College of Radiology (ACR) 2022 Diagnostic Radiology In-Training Examination (DXIT). We perform multiple experiments across time points to assess for model drift, as well as after fine-tuning to assess for differences in accuracy.
MATERIALS AND METHODS: Questions were sequentially input into GPT-4 with a standardized prompt. Each answer was recorded and overall accuracy was calculated, as was logic-adjusted accuracy, and accuracy on image-based questions. This experiment was repeated several months later to assess for model drift, then again after the performance of fine-tuning to assess for changes in GPT's performance.
RESULTS: GPT-4 achieved 58.5% overall accuracy, lower than the PGY-3 average (61.9%) but higher than the PGY-2 average (52.8%). Adjusted accuracy was 52.8%. GPT-4 showed significantly higher (p = 0.012) confidence for correct answers (87.1%) compared to incorrect (84.0%). Performance on image-based questions was significantly poorer (p < 0.001) at 45.4% compared to text-only questions (80.0%), with adjusted accuracy for image-based questions of 36.4%. When the questions were repeated, GPT-4 chose a different answer 25.5% of the time and there was no change in accuracy. Fine-tuning did not improve accuracy.
CONCLUSION: GPT-4 performed between PGY-2 and PGY-3 levels on the 2022 DXIT, significantly poorer on image-based questions, and with large variability in answer choices across time points. Exploratory experiments in fine-tuning did not improve performance. This study underscores the potential and risks of using minimally-prompted general AI models in interpreting radiologic images as a diagnostic tool. Implementers of general AI radiology systems should exercise caution given the possibility of spurious yet confident responses.
引用
- Payne DL, Purohit K, Borrero WM, et al. Performance of GPT-4 on the American College of Radiology In-training Examination: Evaluating Accuracy, Model Drift, and Fine-tuning. Acad Radiol. 2024. PMID: 38653599. doi:10.1016/j.acra.2024.04.006
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。