GPT-3.5、GPT-4与Google Bard基于多语言乳腺影像报告进行BI-RADS分类的研究
本回顾性研究纳入三个转诊中心2000年1月至2023年10月以意大利语、英语或荷兰语书写的2400份乳腺影像报告,由乳腺放射科医师与GPT-3.5、GPT-4及Bard分别仅根据原始报告描述赋值BI-RADS分类。原始医师与复核医师间一致性几乎完美(AC1=0.91),与GPT-4、GPT-3.5、Bard仅为中等一致(AC1分别为0.52、0.48、0.42);可能导致临床管理改变及负面影响的分类不一致比例,人工阅片者均低于各LLM(P<.001)。
要点
- 原始与复核放射科医师间BI-RADS分类一致性几乎完美,AC1=0.91。
- GPT-4、GPT-3.5、Bard与原始医师一致性仅为中等,AC1分别为0.52、0.48、0.42。
- LLM导致的可能负面影响临床管理的分类不一致比例高于人工阅片者(P<.001)。
核心结论
- LLM在三种语言报告上与人工阅片者赋值的BI-RADS分类呈中等一致性,但产生了高比例可能负面影响临床管理的不一致BI-RADS分类。
研究设计与样本
- 回顾性研究,纳入三个转诊中心2000年1月至2023年10月以意大利语、英语或荷兰语书写、分类为BI-RADS 1-5的乳腺报告共2400份。
- 报告来自因乳腺癌筛查或诊断目的接受MRI、乳腺X线摄影和/或超声的女性。
- 由委员会认证的乳腺放射科医师与GPT-3.5、GPT-4及Bard(现称Gemini)仅根据原始放射科医师描述的所见赋值BI-RADS分类。
诊断标准与关键数值
- 原始与复核放射科医师间BI-RADS分类一致性AC1=0.91(几乎完美)。
- 原始医师与GPT-4、GPT-3.5、Bard的一致性AC1分别为0.52、0.48、0.42(中等)。
- 导致临床管理改变的BI-RADS升级或降级频率:人工阅片者118/2400(4.9%),Bard 611/2400(25.5%),GPT-3.5 573/2400(23.9%),GPT-4 435/2400(18.1%),P<.001。
- 可能负面影响临床管理的不一致频率:人工阅片者37/2400(1.5%),Bard 435/2400(18.1%),GPT-3.5 344/2400(14.3%),GPT-4 255/2400(10.6%),P<.001。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Background The performance of publicly available large language models (LLMs) remains unclear for complex clinical tasks. Purpose To evaluate the agreement between human readers and LLMs for Breast Imaging Reporting and Data System (BI-RADS) categories assigned based on breast imaging reports written in three languages and to assess the impact of discordant category assignments on clinical management. Materials and Methods This retrospective study included reports for women who underwent MRI, mammography, and/or US for breast cancer screening or diagnostic purposes at three referral centers. Reports with findings categorized as BI-RADS 1-5 and written in Italian, English, or Dutch were collected between January 2000 and October 2023. Board-certified breast radiologists and the LLMs GPT-3.5 and GPT-4 (OpenAI) and Bard, now called Gemini (Google), assigned BI-RADS categories using only the findings described by the original radiologists. Agreement between human readers and LLMs for BI-RADS categories was assessed using the Gwet agreement coefficient (AC1 value). Frequencies were calculated for changes in BI-RADS category assignments that would affect clinical management (ie, BI-RADS 0 vs BI-RADS 1 or 2 vs BI-RADS 3 vs BI-RADS 4 or 5) and compared using the McNemar test. Results Across 2400 reports, agreement between the original and reviewing radiologists was almost perfect (AC1 = 0.91), while agreement between the original radiologists and GPT-4, GPT-3.5, and Bard was moderate (AC1 = 0.52, 0.48, and 0.42, respectively). Across human readers and LLMs, differences were observed in the frequency of BI-RADS category upgrades or downgrades that would result in changed clinical management (118 of 2400 [4.9%] for human readers, 611 of 2400 [25.5%] for Bard, 573 of 2400 [23.9%] for GPT-3.5, and 435 of 2400 [18.1%] for GPT-4; P < .001) and that would negatively impact clinical management (37 of 2400 [1.5%] for human readers, 435 of 2400 [18.1%] for Bard, 344 of 2400 [14.3%] for GPT-3.5, and 255 of 2400 [10.6%] for GPT-4; P < .001). Conclusion LLMs achieved moderate agreement with human reader-assigned BI-RADS categories across reports written in three languages but also yielded a high percentage of discordant BI-RADS categories that would negatively impact clinical management. © RSNA, 2024 Supplemental material is available for this article.
引用
- Cozzi A, Pinker K, Hidber A, et al. BI-RADS Category Assignments by GPT-3.5, GPT-4, and Google Bard: A Multilanguage Study. Radiology. 2024. PMID: 38687216. doi:10.1148/radiol.232133
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。