比较放射科医师与GPT-4V及Gemini Pro Vision使用Diagnosis Please病例图像输入的诊断准确性
本回顾性研究纳入2008年1月至2023年10月发表的190例Diagnosis Please病例,比较GPT-4V和Gemini Pro Vision在不同温度参数下生成鉴别诊断的能力与8名亚专科放射科医师的准确性。GPT-4V和Gemini Pro Vision的总体准确性随温度升高而提高(GPT-4V:41%、45%、49%;Gemini Pro Vision:29%、36%、39%),但Bonferroni校正后无统计学显著差异。放射科医师总体准确性为61%,高于温度1时的Gemini Pro Vision(P < .001),与温度1时的GPT-4V无统计学显著差异(P = .02)。
要点
- GPT-4V总体准确性随温度升高而提高(41%→45%→49%),但校正后无统计学显著差异。
- 放射科医师总体准确性61%,高于温度1时Gemini Pro Vision,与温度1时GPT-4V无显著差异。
- 放射科医师在多数亚专科中表现优于温度1时的LLMs(医师45%-88% vs LLMs 24%-75%)。
核心结论
- 使用直接影像输入时,GPT-4V和Gemini Pro Vision的诊断准确性随温度设置升高而提高;GPT-4V虽略低于放射科医师,但作为诊断决策的支持工具显示出有希望的潜力。
研究设计与样本
- 回顾性研究,纳入2008年1月至2023年10月发表的190例Diagnosis Please病例,由8名接受过亚专科培训的放射科医师判读,并与GPT-4V和Gemini Pro Vision在温度0、0.5、1下各重复5次生成的三个鉴别诊断进行比较。
诊断标准与关键数值
- 若五次重复生成的鉴别诊断中包含最终诊断则判为正确;总体准确性:GPT-4V在温度0、0.5、1分别为41%(78/190)、45%(86/190)、49%(93/190),Gemini Pro Vision分别为29%(55/190)、36%(69/190)、39%(74/190);放射科医师总体准确性61%(115/190)。
临床应用与阅片要点
- 放射科医师总体准确性61%,高于温度1时Gemini Pro Vision(P < .001),与温度1时GPT-4V无统计学显著差异(P = .02);作者认为GPT-4V作为诊断决策的支持工具显示出有希望的潜力。
局限性
- GPT-4V温度0、0.5、1准确性差异经Bonferroni校正后无统计学显著证据(P = .12)。
- Gemini Pro Vision温度0、0.5、1准确性差异经Bonferroni校正后无统计学显著证据(P = .04)。
- 放射科医师与温度1时GPT-4V的差异经Bonferroni校正后无统计学显著差异(P = .02)。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Background The diagnostic abilities of multimodal large language models (LLMs) using direct image inputs and the impact of the temperature parameter of LLMs remain unexplored. Purpose To investigate the ability of GPT-4V and Gemini Pro Vision in generating differential diagnoses at different temperatures compared with radiologists using Radiology Diagnosis Please cases. Materials and Methods This retrospective study included Diagnosis Please cases published from January 2008 to October 2023. Input images included original images and captures of the textual patient history and figure legends (without imaging findings) from PDF files of each case. The LLMs were tasked with providing three differential diagnoses, repeated five times at temperatures 0, 0.5, and 1. Eight subspecialty-trained radiologists solved cases. An experienced radiologist compared generated and final diagnoses, considering the result correct if the generated diagnoses included the final diagnosis after five repetitions. Accuracy was assessed across models, temperatures, and radiology subspecialties, with statistical significance set at P < .007 after Bonferroni correction for multiple comparisons across the LLMs at the three temperatures and with radiologists. Results A total of 190 cases were included in neuroradiology (n = 53), multisystem (n = 27), gastrointestinal (n = 25), genitourinary (n = 23), musculoskeletal (n = 17), chest (n = 16), cardiovascular (n = 12), pediatric (n = 12), and breast (n = 5) subspecialties. Overall accuracy improved with increasing temperature settings (0, 0.5, 1) for both GPT-4V (41% [78 of 190 cases], 45% [86 of 190 cases], 49% [93 of 190 cases], respectively) and Gemini Pro Vision (29% [55 of 190 cases], 36% [69 of 190 cases], 39% [74 of 190 cases], respectively), although there was no evidence of a statistically significant difference after Bonferroni adjustment (GPT-4V, P = .12; Gemini Pro Vision, P = .04). The overall accuracy of radiologists (61% [115 of 190 cases]) was higher than that of Gemini Pro Vision at temperature 1 (T1) (P < .001), while no statistically significant difference was observed between radiologists and GPT-4V at T1 after Bonferroni adjustment (P = .02). Radiologists (range, 45%-88%) outperformed the LLMs at T1 (range, 24%-75%) in most subspecialties. Conclusion Using direct radiologic image inputs, GPT-4V and Gemini Pro Vision showed improved diagnostic accuracy with increasing temperature settings. Although GPT-4V slightly underperformed compared with radiologists, it nonetheless demonstrated promising potential as a supportive tool in diagnostic decision-making. © RSNA, 2024 See also the editorial by Nishino and Ballard in this issue.
引用
- Suh PS, Shim WH, Suh CH, et al. Comparing Diagnostic Accuracy of Radiologists versus GPT-4V and Gemini Pro Vision Using Image Inputs from Diagnosis Please Cases. Radiology. 2024. PMID: 38980179. doi:10.1148/radiol.240273
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。