文献速递技术与安全CTMR

Transformer与传统自然语言处理:自动放射学报告分类需要多少数据才足够?

Transformer versus traditional natural language processing: how much data is enough for automated radiology report classification?
PubMed 文献 · Br J Radiol · 2023-05-25

该研究比较了BioBERT与传统机器学习模型在七项放射学报告自由文本分类任务上的性能,使用全部训练集及2.5%、10%、25%、50%、75%随机子集训练。结果显示,在较小训练样本量下,BioBERT表现不及非深度学习NLP模型,需约1000份报告才能达到相似或更优性能;训练样本约1250至1500份时所有模型性能开始进入平台期。

要点

  • 小样本(少于1000份报告)且类别不平衡时,传统NLP模型分类性能优于Transformer模型BioBERT。
  • BioBERT需约1000份训练报告才能达到与不依赖深度学习的NLP模型相似或更优的性能。
  • 训练样本约1250至1500份时,所有模型测试性能进入平台期,增加数据带来的性能提升很小。

核心结论

  • 样本量较大时,Transformer NLP模型在放射学报告二分类任务中表现更优;但样本量较小(少于1000)且训练数据更不平衡时,传统NLP技术表现更好。

研究设计与样本

  • 比较研究,纳入7204份标注的放射学报告,涵盖腹部/盆腔CT、脑CT/MRI和膝关节MRI报告,比较BioBERT与三种传统机器学习模型(梯度提升树、随机森林、逻辑回归)在七项分类任务上的性能,训练集使用全量及2.5%、10%、25%、50%、75%随机子集。

诊断标准与关键数值

  • 评估任务包括腹部/盆腔CT报告中的阑尾炎、憩室炎、肠梗阻、小肠炎/结肠炎检测,脑CT/MRI报告中的缺血性梗死检测,以及膝关节MRI报告中的内侧和外侧半月板撕裂检测;性能在留出测试集上比较。

临床应用与阅片要点

  • 临床NLP研究人员可根据数据集规模选择模型:小样本(少于1000份报告)时传统NLP技术可能表现更好。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

OBJECTIVES: Current state-of-the-art natural language processing (NLP) techniques use transformer deep-learning architectures, which depend on large training datasets. We hypothesized that traditional NLP techniques may outperform transformers for smaller radiology report datasets.

METHODS: We compared the performance of BioBERT, a deep-learning-based transformer model pre-trained on biomedical text, and three traditional machine-learning models (gradient boosted tree, random forest, and logistic regression) on seven classification tasks given free-text radiology reports. Tasks included detection of appendicitis, diverticulitis, bowel obstruction, and enteritis/colitis on abdomen/pelvis CT reports, ischemic infarct on brain CT/MRI reports, and medial and lateral meniscus tears on knee MRI reports (7,204 total annotated reports). The performance of NLP models on held-out test sets was compared after training using the full training set, and 2.5%, 10%, 25%, 50%, and 75% random subsets of the training data.

RESULTS: In all tested classification tasks, BioBERT performed poorly at smaller training sample sizes compared to non-deep-learning NLP models. Specifically, BioBERT required training on approximately 1,000 reports to perform similarly or better than non-deep-learning models. At around 1,250 to 1,500 training samples, the testing performance for all models began to plateau, where additional training data yielded minimal performance gain.

CONCLUSIONS: With larger sample sizes, transformer NLP models achieved superior performance in radiology report binary classification tasks. However, with smaller sizes (<1000) and more imbalanced training data, traditional NLP techniques performed better.

ADVANCES IN KNOWLEDGE: Our benchmarks can help guide clinical NLP researchers in selecting machine-learning models according to their dataset characteristics.

来源:Br J Radiol,查看原文。版权归原作者所有。

引用

  1. Yang, Li, Raghavan, et al. Transformer versus traditional natural language processing: how much data is enough for automated radiology report classification? Br J Radiol. 2023. PMID: 37162253. doi:10.1259/bjr.20220769

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。