大规模评估机器学习模型在放射学报告中识别随访建议的表现
本回顾性研究在大型城市学术医疗中心分析49769份多模态放射学报告,评估32种文本分类方法基于所见和印象部分识别随访建议的性能;在所见部分Hybrid-google模型F1最高(0.835),在印象部分AttBiLSTM-random模型F1最高(0.979),GPT-4前缀提示在外部和时间测试中F1分别为0.969和0.973。
要点
- 基于印象部分识别随访建议的性能优于基于所见部分(F1 0.979 vs 0.835)。
- GPT-4前缀提示在外部MIMIC-CXR和时间测试中F1分别为0.969和0.973。
- 机器学习模型在自动化分类放射学报告随访建议方面显示出前景。
核心结论
- 机器学习模型在自动化分类放射学报告中的随访建议方面显示出前景。
研究设计与样本
- 回顾性研究,纳入来自大型城市学术医疗中心的49769份多模态放射学报告(35509例患者,平均年龄52.2岁±22.0,18477例女性),分为训练集37140份、验证集2584份和内部测试集10045份;另使用2000份MIMIC-CXR胸部X线报告进行外部测试和100份机构CT报告进行时间测试;评估32种文本分类方法,分别基于所见和印象部分。
诊断标准与关键数值
- 所见部分:Hybrid-google模型F1最高,为0.835(95%CI:0.825,0.845)。
- 印象部分:AttBiLSTM-random模型F1最高,为0.979(95%CI:0.976,0.982)。
- GPT-4前缀提示在MIMIC-CXR外部测试集F1为0.969(95%CI:0.961,0.977)。
- GPT-4前缀提示在机构CT时间测试集F1为0.973(95%CI:0.937,1.000)。
临床应用与阅片要点
- 基于印象部分识别随访建议的模型F1达0.979,高于基于所见部分的0.835。
- GPT-4前缀提示在外部MIMIC-CXR和时间测试中F1分别为0.969和0.973。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Background Radiology reports often contain follow-up recommendations vital for optimal patient care, prevention of complications, and mitigation of legal risk. However, there is a lack of comprehensive comparison methods for identifying these recommendations across a large volume of reports from various modalities, including open-source large language models. Purpose To evaluate the performance of machine learning (ML) models, including Meta's open-source LLAMA3 and OpenAI's Health Insurance Portability and Accountability Act-compliant Generative Pre-trained Transformer, in identifying follow-up recommendations in radiology reports. Materials and Methods In this retrospective study, three sets of radiology reports were analyzed across multiple imaging modalities from a large urban academic medical center: an expert annotated dataset (n = 11 901) from January 1 to January 10, 2015; a dataset (n = 32 959) extracted through regular expressions (ie, sequences of characters that define search patterns in text) from January 11, 2015, to January 1, 2017; and a dataset (n = 4909) annotated during dictation from September 8, 2018, to February 23, 2021. To assess generalization on impressions, two expertly annotated datasets were used: 2000 chest radiography reports from the publicly available MIMIC-CXR database for external testing and 100 institutional CT reports from January 1 to January 15, 2024, for temporal testing. Thirty-two text classification methods were evaluated separately based on the findings and impression sections of these reports. Performance metrics included precision, recall, accuracy, and F1 score; with 95% bootstrapped CIs and areas under the precision-recall curve. Statistical comparisons were performed by using the McNemar test. Results The study included 49 769 reports from 35 509 patients (mean age, 52.2 years ± 22.0 [SD]; 18 477 female patients) for training (n = 37 140), validation (n = 2584), and internal testing (n = 10 045). For the findings section, a generative-discriminative model initialized with Google's Word2vec embeddings (Hybrid-google) achieved the highest F1 score (0.835; 95% CI: 0.825, 0.845). For the impression section, an attention-based bidirectional long short-term memory (LSTM) with random initialization (AttBiLSTM-random) performed best, with an F1 score of 0.979 (95% CI: 0.976, 0.982). Prefixed prompting with GPT-4 demonstrated superior external and temporal generalization performance on the MIMIC-CXR and institutional CT datasets, achieving F1 scores of 0.969 (95% CI: 0.961, 0.977) and 0.973 (95% CI: 0.937, 1.000), respectively. Conclusion ML models showed promise for automating the classification of follow-up recommendations in radiology reports. © RSNA, 2025 Supplemental material is available for this article.
引用
- Xiao P, Yu X, Ha SM, et al. Large-Scale Evaluation of Machine Learning Models in Identifying Follow-Up Recommendations in Radiology Reports. Radiology. 2025. PMID: 41217283. doi:10.1148/radiol.242167
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。