开放权重大语言模型对病理性脑部MRI报告分类的比较评估:从提示词到微调模型
最佳模型Qwen3.5-35B-A3B的二分类宏F1达0.84,多标签分类达0.81;QLoRA微调后多标签宏F1为0.83,除宏召回率外,宏F1、微F1、宏精确率和完全匹配的提升经bootstrap 95%CI支持。作者认为该技术可作为内部索引与语义检索工具,但需外部多中心验证后方可临床部署。
要点
- 开放权重大语言模型在无需大量任务特定训练下即可对神经放射学报告进行语义分类。
- 最佳模型Qwen3.5-35B-A3B二分类宏F1为0.84,多标签分类宏F1为0.81,多数投票优于单次预测。
- 模型易在边界性标注病例出错并常漏掉次要标签,阅片报告结构化时需注意。
核心结论
- 开放权重大语言模型在无需大量任务特定训练下对神经放射学报告分类表现出较强能力;微调可进一步提升性能,支持整合到可扩展的语义放射学数据管道中。但现有证据来自单个意大利中心且无外部验证,微调增益在小型留出测试集上估计,宏F1提升虽具统计学显著性但置信区间下限接近零;因此该方法目前最适合作为内部索引与语义检索工具,支持更快的报告检索、结构化队列识别、内部审计和可扩展的表型分析,以及高效的主动学习策略,在临床部署前需额外外部多中心验证以确认更广泛的普适性。
关键影像表现/征象
- 二分类任务在最佳表现的Qwen3.5-35B-A3B模型上宏F1为0.84,多标签分类宏F1为0.81。
- 多数投票优于单个集合预测。
- QLoRA微调后多标签任务宏F1为0.83;相对零样本基线的提升在宏F1、微F1、宏精确率和完全匹配上bootstrap 95%置信区间排除零,但宏召回率未达统计学支持。
- 微调与少样本提示策略互补,无一方持续优于另一方。
- 误差分析显示模型在边界性标注病例上表现困难,且常遗漏次要标签。
诊断标准与关键数值
- 二分类任务:病理性 vs 非病理性,最佳模型Qwen3.5-35B-A3B宏F1为0.84。
- 多标签分类任务:五个诊断类别(新生物、血管性、炎症/感染、神经退行性、其他),最佳模型宏F1为0.81;QLoRA微调后宏F1为0.83。
- 微调相对零样本基线提升:宏F1、微F1、宏精确率、完全匹配的bootstrap 95%置信区间排除零;宏召回率未达统计学支持。
临床应用与阅片要点
- 该分类方法可作为内部索引与语义检索工具,支持更快的报告检索、结构化队列识别、内部审计和可扩展的表型分析。
- 模型易在边界性标注病例出错并常遗漏次要标签,用于报告分类时需注意这些陷阱。
- 在临床部署前需额外外部多中心验证以确认更广泛的普适性。
局限性
- 现有证据来自单个意大利中心,无外部验证。
- 微调增益在小型留出测试集上估计,宏F1提升虽具统计学显著性但置信区间下限接近零。
- 模型在边界性标注病例上表现困难且常遗漏次要标签。
- 临床部署前需额外外部多中心验证以确认更广泛的普适性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: Automated semantic classification of radiology reports can enhance clinical workflows and research, but often relies on language- and task-specific models, limiting scalability. This study aims to evaluate and improve the 0-shot baseline performance of open-weight large language models (LLMs) in the semantic classification of free-text magnetic resonance imaging (MRI) reports.
MATERIALS AND METHODS: A corpus of 1385 reports from a single neuroradiology center was used to assess four LLMs (GPT-OSS-20B, Magistral-Small-2509, Gemma-4-31B, Qwen3.5-35B-A3B) of comparable scale (∼20-35B parameters). The brain MRI reports were labeled as Pathological or Non-Pathological, with the Pathological ones further classified into five diagnostic categories (Neoformation, Vascular, Inflammation/Infection, Neurodegenerative, Other). The LLMs were evaluated using zero- and few-shot prompting with majority-vote aggregation. Quantized Low-Rank Adaptation (QLoRA) fine-tuning was then applied to the best-performing model across multiple configurations. Performance was assessed using Macro F1, Micro F1, Precision, Recall, and Exact Match Accuracy.
RESULTS: The binary classification task achieved a Macro F1 score of 0.84 on the best-performing Qwen3.5-35B-A3B model, while multilabel classification reached 0.81. Majority voting (MV) outperformed individual-set predictions. QLoRA fine-tuning yielded a Macro F1 score of 0.83 on the multilabel task; the improvement over the 0-shot baseline was statistically supported for Macro F1, Micro F1, Macro Precision, and Exact Match, whose bootstrap 95% confidence intervals excluded zero, but not for Macro Recall. Fine-tuning and few-shot prompting strategies are complementary, with neither consistently outperforming the other. Error analysis showed that the models struggled with borderline annotation cases and often missed secondary labels.
CONCLUSION: Open-weight LLMs demonstrate strong capability for neuroradiology report classification without extensive task-specific training. Fine-tuning further improves performance, supporting integration into scalable semantic radiology data pipelines. However, the present evidence comes from a single Italian center without external validation, and the fine-tuning gain was estimated on a small held-out test set, for which the Macro F1 improvement, though statistically significant, has a confidence interval with a lower bound close to zero. In this light, the approach is best positioned as an internal indexing and semantic-retrieval tool, supporting faster report retrieval, structured cohort identification, internal audit, and scalable phenotyping for neuroradiology research and data warehousing, as well as efficient active-learning strategies, with broader generalizability requiring additional external multicentre validation before clinical deployment.
引用
- Demaria C, Crema C, Verdicchio M, et al. Comparative Evaluation of Open-Weight Large Language Models for Pathological Brain MRI Reports Classification: From Prompts to a Fine-Tuned Model. Acad Radiol. 2026. PMID: 42875268. doi:10.1016/j.acra.2026.09.053
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。