大语言模型生成肺结节随访建议能力的评估
回顾性研究以2023年9月1日至2024年4月30日资深放射科医师记录的肺结节CT随访报告为对象,将结节影像表现与最终提示词输入GPT-4o-mini或ERNIE-4.0-Turbo-8K生成随访建议。在996例患者(中位年龄50.0岁,IQR 39.0-60.0岁;511例男性)的1009份报告上,两种模型随访建议准确率(94.6% vs 92.8%,P=0.07)与有害率(2.9% vs 3.5%,P=0.48)表现相当。结论认为大语言模型在提供基于指南的肺结节随访建议方面显示出前景,但需要严格验证与监督以降低潜在临床风险。
要点
- ERNIE-4.0-Turbo-8K与GPT-4o-mini生成肺结节随访建议的准确率分别为94.6%与92.8%,差异无统计学意义(P=0.07)。
- 两种模型随访建议的有害率分别为2.9%与3.5%,差异无统计学意义(P=0.48)。
- 结论认为大语言模型在基于指南的肺结节随访建议方面显示出前景,但需要严格验证与监督。
核心结论
- 大语言模型在提供基于指南的肺结节随访建议方面显示出前景,但需要严格验证与监督以降低潜在临床风险。
研究设计与样本
- 回顾性研究,纳入2023年9月1日至2024年4月30日资深放射科医师记录的肺结节CT随访报告,共1009份报告来自996例患者(中位年龄50.0岁,IQR 39.0-60.0岁;511例男性);另收集60份报告用于基于少样本学习与思维链方法的提示词工程。
诊断标准与关键数值
- 以放射科医师定义的基于指南的标准为参考,通过二分类评估结节风险分类、随访间隔与有害性;ERNIE-4.0-Turbo-8K与GPT-4o-mini的结节分类准确率分别为99.8%与99.9%,敏感度分别为96.9%与100.0%,特异度均为99.9%,阳性预测值均为96.9%,阴性预测值分别为100.0%与99.9%,F1分数分别为96.9%与98.4%。
临床应用与阅片要点
- 原文建议对大语言模型生成的肺结节随访建议进行严格验证与监督,以降低潜在临床风险。
局限性
- 原文结论指出大语言模型需严格验证与监督以降低潜在临床风险。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: To evaluate the performance of large language models (LLMs) in generating clinically follow-up recommendations for pulmonary nodules by leveraging radiological report findings and management guidelines.
MATERIALS AND METHODS: This retrospective study included CT follow-up reports of pulmonary nodules documented by senior radiologists from September 1st, 2023, to April 30th, 2024. Sixty reports were collected for prompting engineering additionally, based on few-shot learning and the Chain of Thought methodology. Radiological findings of pulmonary nodules, along with finally prompt, were input into GPT-4o-mini or ERNIE-4.0-Turbo-8K to generate follow-up recommendations. The AI-generated recommendations were evaluated against radiologist-defined guideline-based standards through binary classification, assessing nodule risk classifications, follow-up intervals, and harmfulness. Performance metrics included sensitivity, specificity, positive/negative predictive values, and F1 score.
RESULTS: On 1009 reports from 996 patients (median age, 50.0 years, IQR, 39.0-60.0 years; 511 male patients), ERNIE-4.0-Turbo-8K and GPT-4o-mini demonstrated comparable performance in both accuracy of follow-up recommendations (94.6 % vs 92.8 %, P = 0.07) and harmfulness rates (2.9 % vs 3.5 %, P = 0.48). In nodules classification, ERNIE-4.0-Turbo-8K and GPT-4o-mini performed similarly with accuracy rates of 99.8 % vs 99.9 % sensitivity of 96.9 % vs 100.0 %, specificity of 99.9 % vs 99.9 %, positive predictive value of 96.9 % vs 96.9 %, negative predictive value of 100.0 % vs 99.9 %, f1-score of 96.9 % vs 98.4 %, respectively.
CONCLUSION: LLMs show promise in providing guideline-based follow-up recommendations for pulmonary nodules, but require rigorous validation and supervision to mitigate potential clinical risks. This study offers insights into their potential role in automated radiological decision support.
引用
- Wen J, Huang W, Yan H, et al. Evaluation of large language models in generating pulmonary nodule follow-up recommendations. Eur J Radiol Open. 2025. PMID: 40391069. doi:10.1016/j.ejro.2025.100655
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。