领域特异性多模态生成式人工智能模型生成胸部X线摄影报告的诊断准确性与临床价值
该研究基于42家医院2005至2023年连续收集的8838719对胸部X线摄影与报告训练领域特异性多模态生成式AI模型,在2145张测试影像上评估其对13种征象的诊断准确性,并由4名放射科医师比较模型、放射科医师与GPT-4Vision报告的主观质量。模型对气胸和皮下气肿敏感度高,报告接受率、一致性评分与质量评分均以模型报告最高、GPT-4Vision最低。
要点
- 模型对气胸敏感度95.3%(181/190)、对皮下气肿敏感度92.6%(138/149)。
- 模型报告接受率70.5%(6047/8680),放射科医师报告73.3%(6288/8580),GPT-4Vision报告29.6%(2536/8580)。
- 模型报告一致性评分中位数4(IQR 3-5)、质量评分中位数4(IQR 3-5),均高于GPT-4Vision。
核心结论
- 领域特异性多模态生成式AI模型在为放射科医师提供胸部X线摄影初步解读方面表现出高诊断准确性和临床价值的潜力。
研究设计与样本
- 训练数据为2005至2023年从42家医院回顾性连续收集的8838719对正位胸部X线摄影与报告配对。
- 测试集包括公共数据集(PadChest、Open-i、VinDr-CXR、MIMIC-CXR-JPG)及被排除于训练之外的胸部X线摄影,共2145张。
- 以放射科医师标注为参考标准,计算模型生成报告对13种胸部X线征象的敏感度和特异度(含95%CI)。
- 由4名放射科医师从可接受性、一致性评分、质量评分及报告比较排名四方面评估模型、放射科医师和通用大语言模型(GPT-4Vision)生成的报告。
诊断标准与关键数值
- 模型生成报告对气胸的敏感度为95.3%(181/190),对皮下气肿的敏感度为92.6%(138/149)。
- 可接受性定义为放射科医师是否愿意将报告作为自己的报告而不加修改地认可。
- 一致性评分采用RADPEER,从1分(有临床意义的差异)到5分(完全一致)。
- 质量评分采用5点Likert量表,从1分(很差)到5分(极好)。
临床应用与阅片要点
- 模型生成报告的可接受率为70.5%(6047/8680),放射科医师报告为73.3%(6288/8580),GPT-4Vision报告为29.6%(2536/8580)。
- 模型生成报告的一致性评分中位数为4(IQR 3-5),GPT-4Vision报告最低,中位数为1(IQR 1-3)(P < .001)。
- 模型生成报告的质量评分中位数为4(IQR 3-5),GPT-4Vision报告最低,中位数为2(IQR 1-3)(P < .001)。
- 在排名分析中,模型生成报告最常被排为最高(60.0%;5146/8580),GPT-4Vision报告最常被排为最低(73.6%;6312/8580)。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Background Generative artificial intelligence (AI) is anticipated to alter radiology workflows, requiring a clinical value assessment for frequent examinations like chest radiograph interpretation. Purpose To develop and evaluate the diagnostic accuracy and clinical value of a domain-specific multimodal generative AI model for providing preliminary interpretations of chest radiographs. Materials and Methods For training, consecutive radiograph-report pairs from frontal chest radiography were retrospectively collected from 42 hospitals (2005-2023). The trained domain-specific AI model generated radiology reports for the radiographs. The test set included public datasets (PadChest, Open-i, VinDr-CXR, and MIMIC-CXR-JPG) and radiographs excluded from training. The sensitivity and specificity of the model-generated reports for 13 radiographic findings, compared with radiologist annotations (reference standard), were calculated (with 95% CIs). Four radiologists evaluated the subjective quality of the reports in terms of acceptability, agreement score, quality score, and comparative ranking of reports from (a) the domain-specific AI model, (b) radiologists, and (c) a general-purpose large language model (GPT-4Vision). Acceptability was defined as whether the radiologist would endorse the report as their own without changes. Agreement scores from 1 (clinically significant discrepancy) to 5 (complete agreement) were assigned using RADPEER; quality scores were on a 5-point Likert scale from 1 (very poor) to 5 (excellent). Results A total of 8 838 719 radiograph-report pairs (training) and 2145 radiographs (testing) were included (anonymized with respect to sex and gender). Reports generated by the domain-specific AI model demonstrated high sensitivity for detecting two critical radiographic findings: 95.3% (181 of 190) for pneumothorax and 92.6% (138 of 149) for subcutaneous emphysema. Acceptance rate, evaluated by four radiologists, was 70.5% (6047 of 8680), 73.3% (6288 of 8580), and 29.6% (2536 of 8580) for model-generated, radiologist, and GPT-4Vision reports, respectively. Agreement scores were highest for the model-generated reports (median = 4 [IQR, 3-5]) and lowest for GPT-4Vision reports (median = 1 [IQR, 1-3]; P < .001). Quality scores were also highest for the model-generated reports (median = 4 [IQR, 3-5]) and lowest for the GPT-4Vision reports (median = 2 [IQR, 1-3]; P < .001). From the ranking analysis, model-generated reports were most frequently ranked the highest (60.0%; 5146 of 8580), and GPT-4Vision reports were most frequently ranked the lowest (73.6%; 6312 of 8580). Conclusion A domain-specific multimodal generative AI model demonstrated potential for high diagnostic accuracy and clinical value in providing preliminary interpretations of chest radiographs for radiologists. © RSNA, 2025 Supplemental material is available for this article. See also the editorial by Little in this issue.
引用
- Hong EK, Ham J, Roh B, et al. Diagnostic Accuracy and Clinical Value of a Domain-specific Multimodal Generative AI Model for Chest Radiograph Report Generation. Radiology. 2025. PMID: 40131111. doi:10.1148/radiol.241476
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。