文献速递骨肌神经技术与安全MRDR

自然语言处理方法在下腰痛相关腰椎影像表现分类中的比较

A Comparison of Natural Language Processing Methods for the Classification of Lumbar Spine Imaging Findings Related to Lower Back Pain.
PubMed 文献 · Acad Radiol · 2021-12-01

本研究基于2013年10月至2016年9月四个医疗系统的871份腰椎X线和MRI报告,标注26种下腰痛相关表现,比较四种自然语言处理表示方法结合弹性网逻辑回归的分类性能。结果显示n-grams在全部26种表现上平均AUC最佳(0.960),文档嵌入跨系统泛化性最稳定(AUC标准差0.010)。作者认为同一系统内开发部署首选n-grams,跨系统部署可考虑文档嵌入。

要点

  • n-grams在全部26种下腰痛相关表现上平均AUC最佳(0.960)。
  • 文档嵌入跨系统泛化平均性能最稳定(AUC标准差0.010)。
  • 8种临床重要表现中,n-grams平均AUC为0.954,文档嵌入泛化标准差为0.007。

核心结论

  • 若分类器开发与部署在同一系统,n-grams是首选方法;若需在开发系统外的多个系统部署,或系统内医师行为可能变化,应考虑文档嵌入,因其跨系统性能最稳定。

研究设计与样本

  • 研究使用2013年10月至2016年9月四个医疗系统前瞻性研究采样的871份X线和磁共振成像报告,每份报告标注26种下腰痛相关表现。
  • 框架应用四种不同自然语言处理方法将文本转换为特征集,每种表示对每种表现使用弹性网逻辑回归模型(26个二分类或「一对多」分类模型)。
  • 数据按80%(697/871)训练、20%(174/871)测试划分,训练集内用交叉验证确定最优超参数后在全训练集重训,以测试集AUC评估性能,重复25次不同随机划分以估计95%置信区间并评估表示间性能差异。
  • 泛化性评估中,在三个医疗系统数据上交叉验证训练后测试第四个系统,对每个系统重复该过程,计算跨系统AUC均值和标准差。

诊断标准与关键数值

  • 全部26种表现中,n-grams平均AUC为0.960,为各表示中最佳。
  • 跨系统泛化性方面,文档嵌入平均性能最稳定,AUC标准差为0.010。
  • 8种潜在临床重要表现(任何狭窄、中央狭窄、侧方狭窄、椎间孔狭窄、椎间盘突出、神经根移位压迫、终板水肿、滑脱2级)中,n-grams平均AUC为0.954,文档嵌入泛化标准差为0.007。

临床应用与阅片要点

  • 原文未据此给出推荐。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

RATIONALE AND OBJECTIVES: The use of natural language processing (NLP) in radiology provides an opportunity to assist clinicians with phenotyping patients. However, the performance and generalizability of NLP across healthcare systems is uncertain. We assessed the performance within and generalizability across four healthcare systems of different NLP representational methods, coupled with elastic-net logistic regression to classify lower back pain-related findings from lumbar spine imaging reports.

MATERIALS AND METHODS: We used a dataset of 871 X-ray and magnetic resonance imaging reports sampled from a prospective study across four healthcare systems between October 2013 and September 2016. We annotated each report for 26 findings potentially related to lower back pain. Our framework applied four different NLP methods to convert text into feature sets (representations). For each representation, our framework used an elastic-net logistic regression model for each finding (i.e., 26 binary or "one-vs.-rest" classification models). For performance evaluation, we split data into training (80%, 697/871) and testing (20%, 174/871). In the training set, we used cross validation to identify the optimal hyperparameter value and then retrained on the full training set. We then assessed performance based on area under the curve (AUC) for the test set. We repeated this process 25 times with each repeat using a different random train/test split of the data, so that we could estimate 95% confidence intervals, and assess significant difference in performance between representations. For generalizability evaluation, we trained models on data from three healthcare systems with cross validation and then tested on the fourth. We repeated this process for each system, then calculated mean and standard deviation (SD) of AUC across the systems.

RESULTS: For individual representations, n-grams had the best average performance across all 26 findings (AUC: 0.960). For generalizability, document embeddings had the most consistent average performance across systems (SD: 0.010). Out of these 26 findings, we considered eight as potentially clinically important (any stenosis, central stenosis, lateral stenosis, foraminal stenosis, disc extrusion, nerve root displacement compression, endplate edema, and listhesis grade 2) since they have a relatively greater association with a history of lower back pain compared to the remaining 18 classes. We found a similar pattern for these eight in which n-grams and document embeddings had the best average performance (AUC: 0.954) and generalizability (SD: 0.007), respectively.

CONCLUSION: Based on performance assessment, we found that n-grams is the preferred method if classifier development and deployment occur at the same system. However, for deployment at multiple systems outside of the development system, or potentially if physician behavior changes within a system, one should consider document embeddings since embeddings appear to have the most consistent performance across systems.

来源:Acad Radiol,查看原文。版权归原作者所有。

引用

  1. Jujjavarapu, Pejaver, Cohen, et al. A Comparison of Natural Language Processing Methods for the Classification of Lumbar Spine Imaging Findings Related to Lower Back Pain. Acad Radiol. 2021. PMID: 34862122. doi:10.1016/j.acra.2021.09.005

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。