利用大语言模型进行乳腺超声结构化报告:Open AI(GPT-4.0)与Microsoft Bing(GPT-4)的比较研究
本研究回顾性收集2023年1月至5月间接受手术的100份自由文本乳腺超声报告,评估Open AI(GPT-4.0)和Microsoft Bing(GPT-4)生成结构化报告、BI-RADS分类和管理建议的能力。GPT-4.0在结构化报告质量(88% vs. 55%)、正确BI-RADS分类(54% vs. 47%)和合理管理建议(81% vs. 63%)方面均优于Bing,且预测良恶性特征的AUC更高(0.9317 vs. 0.8177),但两者均显著低于高年资放射科医生(AUC 0.9763)。
要点
- GPT-4.0生成结构化报告的质量优于Bing(88% vs. 55%,p<0.001)。
- GPT-4.0提供正确BI-RADS分类的比例为54%,Bing为47%(p=0.013)。
- 两者预测良恶性的AUC均低于高年资放射科医生(0.9317/0.8177 vs. 0.9763)。
核心结论
- 大语言模型(尤其是GPT-4.0)有潜力将非结构化乳腺超声报告转换为结构化报告,并提供准确诊断和合理建议。
研究设计与样本
- 回顾性研究,纳入100份2023年1月至5月间接受手术患者的自由文本乳腺超声报告,比较Open AI(GPT-4.0)与Microsoft Bing(GPT-4)生成结构化报告、BI-RADS分类和管理建议的能力,由高年资放射科医生依据指南评估。
诊断标准与关键数值
- 结构化报告质量:GPT-4.0 88% vs. Bing 55%(p<0.001)。
- 正确BI-RADS分类:GPT-4.0 54% vs. Bing 47%(p=0.013)。
- 合理管理建议:GPT-4.0 81% vs. Bing 63%(p<0.001)。
- 预测良恶性特征的AUC:GPT-4.0 0.9317 vs. Bing 0.8177(p<0.001);两者均显著低于高年资放射科医生(AUC 0.9763,均p<0.001)。
临床应用与阅片要点
- 高年资放射科医生依据指南评估结构化报告、BI-RADS分类和管理建议的质量。
- GPT-4.0在结构化报告、BI-RADS分类、管理建议及良恶性预测方面均优于Bing。
- 两种大语言模型的良恶性预测能力均显著低于高年资放射科医生。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
OBJECTIVES: To assess the capabilities of large language models (LLMs), including Open AI (GPT-4.0) and Microsoft Bing (GPT-4), in generating structured reports, the Breast Imaging Reporting and Data System (BI-RADS) categories, and management recommendations from free-text breast ultrasound reports.
MATERIALS AND METHODS: In this retrospective study, 100 free-text breast ultrasound reports from patients who underwent surgery between January and May 2023 were gathered. The capabilities of Open AI (GPT-4.0) and Microsoft Bing (GPT-4) to convert these unstructured reports into structured ultrasound reports were studied. The quality of structured reports, BI-RADS categories, and management recommendations generated by GPT-4.0 and Bing were evaluated by senior radiologists based on the guidelines.
RESULTS: Open AI (GPT-4.0) was better than Microsoft Bing (GPT-4) in terms of performance in generating structured reports (88% vs. 55%; p < 0.001), giving correct BI-RADS categories (54% vs. 47%; p = 0.013) and providing reasonable management recommendations (81% vs. 63%; p < 0.001). As the ability to predict benign and malignant characteristics, GPT-4.0 performed significantly better than Bing (AUC, 0.9317 vs. 0.8177; p < 0.001), while both performed significantly inferior to senior radiologists (AUC, 0.9763; both p < 0.001).
CONCLUSION: This study highlights the potential of LLMs, specifically Open AI (GPT-4.0), in converting unstructured breast ultrasound reports into structured ones, offering accurate diagnoses and providing reasonable recommendations.
引用
- Liu C, Wei M, Qin Y, et al. Harnessing Large Language Models for Structured Reporting in Breast Ultrasound: A Comparative Study of Open AI (GPT-4.0) and Microsoft Bing (GPT-4). Ultrasound Med Biol. 2024. PMID: 39138026. doi:10.1016/j.ultrasmedbio.2024.07.007
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。