迈向放射学基础模型?GPT-4V多模态与多解剖区域能力的定量评估
本回顾性研究使用2024年2月至3月期间来自神经放射学、心胸放射学和肌肉骨骼放射学的515张单张代表图像(345张异常),通过应用程序编程接口让GPT-4V生成报告。GPT-4V能100%正确识别成像方式、99.2%正确识别解剖区域,但自由文本报告的诊断准确性在气胸为0%至脑肿瘤为90%之间;二分类任务敏感度56%至100%、特异度8%至52%,人类读者汇总敏感度和特异度均为97.2%。模型明显倾向于过度诊断异常。
要点
- GPT-4V识别成像方式准确率100%(515/515),识别解剖区域准确率99.2%(511/515)。
- 自由文本报告诊断准确性:气胸0%(0/33),脑肿瘤90%(45/50)。
- 二分类敏感度56%(缺血性卒中,14/25)至100%(脑出血,25/25);特异度8%(脑出血,2/25)至52%(气胸,13/25)。
核心结论
- GPT-4V最早版本能识别医学图像内容并可靠判定单张图像的成像方式和解剖区域,但无法检测、分类或排除图像解读中的异常。
研究设计与样本
- 回顾性研究,纳入470例患者的515张单张代表图像(异常345张,健康对照170张),来自神经放射学、心胸放射学和肌肉骨骼放射学(CT、MRI、X线摄影),2024年2月至3月通过应用程序编程接口用GPT-4V生成报告,并与一名非放射科培训第一年学员和四名委员会认证放射科医师比较二分类性能。
诊断标准与关键数值
- GPT-4V识别成像方式准确率100%(515/515),识别解剖区域准确率99.2%(511/515)。
- 自由文本报告诊断准确性范围:气胸(CT和X线摄影)0%(0/33)至脑肿瘤(MRI)90%(45/50)。
- 二分类任务敏感度范围:缺血性卒中56%(14/25)至脑出血100%(25/25)。
- 二分类任务特异度范围:脑出血8%(2/25)至气胸52%(13/25)。
- 人类读者在所有任务中汇总敏感度97.2%(1103/1135),汇总特异度97.2%(1084/1115)。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Background Large language models have already demonstrated potential in medical text processing. GPT-4V, a large vision-language model from OpenAI, has shown potential for medical imaging, yet a quantitative analysis is lacking. Purpose To quantitatively assess the performance of GPT-4V in interpreting radiologic images using unseen data. Materials and Methods This retrospective study included single representative abnormal and healthy control images from neuroradiology, cardiothoracic radiology, and musculoskeletal radiology (CT, MRI, radiography) to generate reports using GPT-4V via the application programming interface from February to March 2024. The factual correctness of free-text reports and the performance in detecting abnormalities in binary classification tasks were assessed using accuracy, sensitivity, and specificity. The binary classification performance was compared with that of a first-year nonradiologist in training and four board-certified radiologists. Results A total of 515 images in 470 patients (median age, 61 years [IQR, 44-71 years]; 267 male) were included, of which 345 images were abnormal. GPT-4V correctly identified the imaging modality and anatomic region in 100% (515 of 515) and 99.2% (511 of 515) of images, respectively. Diagnostic accuracy in free-text reports was between 0% (0 of 33 images) for pneumothorax (CT and radiography) and 90% (45 of 50 images) for brain tumor (MRI). In binary classification tasks, GPT-4V showed sensitivities between 56% (14 of 25 images) for ischemic stroke and 100% (25 of 25 images) for brain hemorrhage and specificities between 8% (two of 25 images) for brain hemorrhage and 52% (13 of 25 images) for pneumothorax, compared with a pooled sensitivity of 97.2% (1103 of 1135 images) and pooled specificity of 97.2% (1084 of 1115 images) for the human readers across all tasks. The model exhibited a clear tendency to overdiagnose abnormalities, with 86.5% (147 of 170 images) and 67.7% (151 of 223 images) false-positive rates for the free-text and binary classification tasks, respectively. Conclusion GPT-4V, in its earliest version, recognized medical image content and reliably determined the modality and anatomic region from single images. However, GPT-4V failed to detect, classify, or rule out abnormalities in image interpretation. © RSNA, 2024 Supplemental material is available for this article.
引用
- Strotzer QD, Nieberle F, Kupke LS, et al. Toward Foundation Models in Radiology? Quantitative Assessment of GPT-4V's Multimodal and Multianatomic Region Capabilities. Radiology. 2024. PMID: 39589253. doi:10.1148/radiol.240955
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。