Transformer与传统自然语言处理:自动放射学报告分类需要多少数据才足够?
该研究比较了BioBERT与传统机器学习模型在七项放射学报告自由文本分类任务上的性能,使用全部训练集及2.5%、10%、25%、50%、75%随机子集训练。结果显示,在较小训练样本量下,BioBERT表现不及非深度学习NLP模型,需约1000份报告才能达到相似或更优性能;训练样本约1250至1500份时所有模型性能开始进入平台期。
要点
- 小样本(少于1000份报告)且类别不平衡时,传统NLP模型分类性能优于Transformer模型BioBERT。
- BioBERT需约1000份训练报告才能达到与不依赖深度学习的NLP模型相似或更优的性能。
- 训练样本约1250至1500份时,所有模型测试性能进入平台期,增加数据带来的性能提升很小。
核心结论
- 样本量较大时,Transformer NLP模型在放射学报告二分类任务中表现更优;但样本量较小(少于1000)且训练数据更不平衡时,传统NLP技术表现更好。
研究设计与样本
- 比较研究,纳入7204份标注的放射学报告,涵盖腹部/盆腔CT、脑CT/MRI和膝关节MRI报告,比较BioBERT与三种传统机器学习模型(梯度提升树、随机森林、逻辑回归)在七项分类任务上的性能,训练集使用全量及2.5%、10%、25%、50%、75%随机子集。
诊断标准与关键数值
- 评估任务包括腹部/盆腔CT报告中的阑尾炎、憩室炎、肠梗阻、小肠炎/结肠炎检测,脑CT/MRI报告中的缺血性梗死检测,以及膝关节MRI报告中的内侧和外侧半月板撕裂检测;性能在留出测试集上比较。
临床应用与阅片要点
- 临床NLP研究人员可根据数据集规模选择模型:小样本(少于1000份报告)时传统NLP技术可能表现更好。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
OBJECTIVES: Current state-of-the-art natural language processing (NLP) techniques use transformer deep-learning architectures, which depend on large training datasets. We hypothesized that traditional NLP techniques may outperform transformers for smaller radiology report datasets.
METHODS: We compared the performance of BioBERT, a deep-learning-based transformer model pre-trained on biomedical text, and three traditional machine-learning models (gradient boosted tree, random forest, and logistic regression) on seven classification tasks given free-text radiology reports. Tasks included detection of appendicitis, diverticulitis, bowel obstruction, and enteritis/colitis on abdomen/pelvis CT reports, ischemic infarct on brain CT/MRI reports, and medial and lateral meniscus tears on knee MRI reports (7,204 total annotated reports). The performance of NLP models on held-out test sets was compared after training using the full training set, and 2.5%, 10%, 25%, 50%, and 75% random subsets of the training data.
RESULTS: In all tested classification tasks, BioBERT performed poorly at smaller training sample sizes compared to non-deep-learning NLP models. Specifically, BioBERT required training on approximately 1,000 reports to perform similarly or better than non-deep-learning models. At around 1,250 to 1,500 training samples, the testing performance for all models began to plateau, where additional training data yielded minimal performance gain.
CONCLUSIONS: With larger sample sizes, transformer NLP models achieved superior performance in radiology report binary classification tasks. However, with smaller sizes (<1000) and more imbalanced training data, traditional NLP techniques performed better.
ADVANCES IN KNOWLEDGE: Our benchmarks can help guide clinical NLP researchers in selecting machine-learning models according to their dataset characteristics.
引用
- Yang, Li, Raghavan, et al. Transformer versus traditional natural language processing: how much data is enough for automated radiology report classification? Br J Radiol. 2023. PMID: 37162253. doi:10.1259/bjr.20220769
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。