大语言模型从半结构化冠状动脉CT血管造影报告中提取CAD-RADS 2.0的多机构研究
该多机构研究用319份合成半结构化CCTA报告评估6种大语言模型提取CAD-RADS 2.0三个组分(狭窄程度、斑块负荷、修饰符)的准确性,并比较提示策略的影响。狭窄程度最高准确率内部测试为0.980(48/49),外部测试为0.946(158/167);斑块负荷内部测试43/43完美准确,外部测试0.993(137/138);思维链提示显著提升部分模型准确率,GPT-4外部测试狭窄程度准确率提升0.192(P < 0.001)。
要点
- 大语言模型从半结构化CCTA报告提取CAD-RADS 2.0组分准确率高,斑块负荷与修饰符提取接近完美。
- 狭窄程度外部测试最高准确率0.946(158/167,GPT-4o和o1-mini),内部测试0.980(48/49)。
- 开源模型DeepSeek-R1-Distill-Qwen-14B狭窄程度准确率较低:内部0.898(44/49),外部0.820(137/167)。
核心结论
- 大语言模型从半结构化CCTA报告自动提取CAD-RADS 2.0组分准确率高,尤其在使用思维链提示时。
研究设计与样本
- 多机构研究,收集来自六家机构的319份合成半结构化CCTA报告以保护患者隐私并保持临床相关性。
- 数据集包括主要机构的150份报告(100份用于指令开发,50份用于内部测试)和五家外部机构的169份报告用于外部测试。
- 由委员会认证的放射科医生依据CAD-RADS 2.0指南为狭窄程度、斑块负荷和修饰符三个组分建立参考标准。
- 评估六种大语言模型(GPT-4、GPT-4o、Claude-3.5-Sonnet、o1-mini、Gemini-1.5-Pro和DeepSeek-R1-Distill-Qwen-14B),使用优化指令及零样本或少样本联合或不联合思维链提示策略。
- 准确率比较采用McNemar检验。
诊断标准与关键数值
- 狭窄程度最高准确率:内部测试0.980(48/49,Claude-3.5-Sonnet和o1-mini),外部测试0.946(158/167,GPT-4o和o1-mini)。
- 斑块负荷提取准确率:内部测试多个模型完美准确(43/43),外部测试0.993(137/138,GPT-4o和o1-mini)。
- 修饰符检测在多数模型中准确率持续高(≥0.990)。
- DeepSeek-R1-Distill-Qwen-14B狭窄程度准确率:内部0.898(44/49),外部0.820(137/167)。
- 思维链提示显著提升多个模型准确率,GPT-4外部测试狭窄程度准确率提升0.192(P < 0.001),斑块负荷准确率提升0.152(P < 0.001)。
临床应用与阅片要点
- 原文未据此给出推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
OBJECTIVE: To evaluate the accuracy of large language models (LLMs) in extracting Coronary Artery Disease-Reporting and Data System (CAD-RADS) 2.0 components from coronary CT angiography (CCTA) reports, and assess the impact of prompting strategies.
MATERIALS AND METHODS: In this multi-institutional study, we collected 319 synthetic, semi-structured CCTA reports from six institutions to protect patient privacy while maintaining clinical relevance. The dataset included 150 reports from a primary institution (100 for instruction development and 50 for internal testing) and 169 reports from five external institutions for external testing. Board-certified radiologists established reference standards following the CAD-RADS 2.0 guidelines for all three components: stenosis severity, plaque burden, and modifiers. Six LLMs (GPT-4, GPT-4o, Claude-3.5-Sonnet, o1-mini, Gemini-1.5-Pro, and DeepSeek-R1-Distill-Qwen-14B) were evaluated using an optimized instruction with prompting strategies, including zero-shot or few-shot with or without chain-of-thought (CoT) prompting. The accuracy was assessed and compared using McNemar's test.
RESULTS: LLMs demonstrated robust accuracy across all CAD-RADS 2.0 components. Peak stenosis severity accuracies reached 0.980 (48/49, Claude-3.5-Sonnet and o1-mini) in internal testing and 0.946 (158/167, GPT-4o and o1-mini) in external testing. Plaque burden extraction showed exceptional accuracy, with multiple models achieving perfect accuracy (43/43) in internal testing and 0.993 (137/138, GPT-4o, and o1-mini) in external testing. Modifier detection demonstrated consistently high accuracy (≥0.990) across most models. One open-source model, DeepSeek-R1-Distill-Qwen-14B, showed a relatively low accuracy for stenosis severity: 0.898 (44/49, internal) and 0.820 (137/167, external). CoT prompting significantly enhanced the accuracy of several models, with GPT-4 showing the most substantial improvements: stenosis severity accuracy increased by 0.192 (P < 0.001) and plaque burden accuracy by 0.152 (P < 0.001) in external testing.
CONCLUSION: LLMs demonstrated high accuracy in automated extraction of CAD-RADS 2.0 components from semi-structured CCTA reports, particularly when used with CoT prompting.
引用
- Min D, Jin KN, Bang S, et al. Large Language Models for CAD-RADS 2.0 Extraction From Semi-Structured Coronary CT Angiography Reports: A Multi-Institutional Study. Korean J Radiol. 2025. PMID: 40873373. doi:10.3348/kjr.2025.0293
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。