基于规则与机器学习自然语言处理系统识别下背痛相关腰椎影像表现比较
研究从四个卫生系统的178333份腰椎影像报告随机抽取871份(413份X线、458份MR),由脊柱专家标注26种表现作为参考标准,开发基于规则和机器学习的自然语言处理系统。测试集中规则法与机器学习法平均特异度分别为0.97和0.95,机器学习平均敏感度0.94高于规则法0.83,总体AUC 0.98高于0.90。
要点
- 机器学习模型较规则法平均敏感度提高(0.94 vs 0.83),平均特异度略低(0.95 vs 0.97)。
- 专家标注的26种腰椎表现中25种组间一致性Cohen kappa>0.60,表现患病率3%至89%。
- 机器学习法总体AUC 0.98高于规则法0.90,在下背痛相关腰椎影像表现识别中表现良好。
核心结论
- 自然语言处理系统在识别26种腰椎影像表现方面表现良好,以医学专家参考标准标注为基准;机器学习模型大幅提高敏感度、特异度略降,总体AUC更高。
研究设计与样本
- 使用开源工具构建自然语言处理系统,数据来自四个卫生系统前瞻性 assembled 队列成人的腰椎影像报告,从178333份报告中随机抽取871份(413份X线、458份MR)形成参考标准数据集;由四名脊柱专家按标准化标准标注26种表现,其中71份由全部四名专家标注、800份各由两名专家标注;标注数据随机分为开发集80%和测试集20%,系统由基于规则和机器学习模型构建,并用敏感度、特异度、AUC等指标验证。
诊断标准与关键数值
- 多评分者标注数据集中26种表现有25种组间一致性Cohen kappa>0.60(实质一致),表现患病率范围3%至89%;测试样本中规则法与机器学习法平均特异度分别为0.97和0.95,机器学习法平均敏感度0.94高于规则法0.83,总体AUC机器学习法0.98高于规则法0.90。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: To evaluate a natural language processing (NLP) system built with open-source tools for identification of lumbar spine imaging findings related to low back pain on magnetic resonance and x-ray radiology reports from four health systems.
MATERIALS AND METHODS: We used a limited data set (de-identified except for dates) sampled from lumbar spine imaging reports of a prospectively assembled cohort of adults. From N = 178,333 reports, we randomly selected N = 871 to form a reference-standard dataset, consisting of N = 413 x-ray reports and N = 458 MR reports. Using standardized criteria, four spine experts annotated the presence of 26 findings, where 71 reports were annotated by all four experts and 800 were each annotated by two experts. We calculated inter-rater agreement and finding prevalence from annotated data. We randomly split the annotated data into development (80%) and testing (20%) sets. We developed an NLP system from both rule-based and machine-learned models. We validated the system using accuracy metrics such as sensitivity, specificity, and area under the receiver operating characteristic curve (AUC).
RESULTS: The multirater annotated dataset achieved inter-rater agreement of Cohen's kappa > 0.60 (substantial agreement) for 25 of 26 findings, with finding prevalence ranging from 3% to 89%. In the testing sample, rule-based and machine-learned predictions both had comparable average specificity (0.97 and 0.95, respectively). The machine-learned approach had a higher average sensitivity (0.94, compared to 0.83 for rules-based), and a higher overall AUC (0.98, compared to 0.90 for rules-based).
CONCLUSIONS: Our NLP system performed well in identifying the 26 lumbar spine findings, as benchmarked by reference-standard annotation by medical experts. Machine-learned models provided substantial gains in model sensitivity with slight loss of specificity, and overall higher AUC.
引用
- Tan WK, Hassanpour S, Heagerty PJ, et al. Comparison of Natural Language Processing Rules-based and Machine-learning Systems to Identify Lumbar Spine Imaging Findings Related to Low Back Pain. Acad Radiol. 2018. PMID: 29605561. doi:10.1016/j.acra.2018.03.008
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。