基于自然语言的机器学习模型用于临床放射学报告标注
本研究获取96303份头部CT报告,比较其与其他语料库的语言复杂度,并用词袋、词嵌入及潜在狄利克雷分配方法构建特征,对1004份报告由医师手动标注感兴趣征象,其中部分为关键征象。采用套索逻辑回归在602份报告上训练模型并在402份报告上验证,最佳模型识别任何关键征象的AUC为0.966,所有征象平均AUC为0.957。
要点
- 最佳模型(词袋含一元、二元、三元语法加平均词嵌入向量)识别任何关键头部CT征象的AUC为0.966。
- 识别任何关键征象的敏感度为92.59%(175/189),特异度为89.67%(191/213)。
- 头部CT语料的Yule I为34,明显低于路透社语料库的103和I2B2出院小结的271,提示语言复杂度较低。
核心结论
- 自动化方法可用于识别放射学报告中的征象,该方法的成功得益于这些报告的标准化语言,借此可生成大型标注语料库用于深度学习等应用。
研究设计与样本
- 研究获取96303份头部CT报告,比较其与其他语料库的语言复杂度;用词袋、词嵌入及潜在狄利克雷分配方法构建机器可分析特征;由医师手动标注1004份头部CT报告中的感兴趣征象(其中部分为关键征象);采用套索逻辑回归在602/1004份报告(60%)上训练模型,在留出的402/1004份报告(40%)上验证性能。
诊断标准与关键数值
- 最佳模型(词袋含一元、二元、三元语法加平均词嵌入向量)识别任何关键头部CT征象的留出AUC为0.966,所有头部CT征象平均AUC为0.957。
- 最佳模型识别任何关键征象的敏感度为92.59%(175/189),特异度为89.67%(191/213)。
- 所有征象平均敏感度为90.25%(1898/2103),平均特异度为91.72%(18351/20007)。
- 较简单的一元词袋模型识别任何关键征象的平均AUC为0.951,最佳模型为0.966。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Purpose To compare different methods for generating features from radiology reports and to develop a method to automatically identify findings in these reports. Materials and Methods In this study, 96 303 head computed tomography (CT) reports were obtained. The linguistic complexity of these reports was compared with that of alternative corpora. Head CT reports were preprocessed, and machine-analyzable features were constructed by using bag-of-words (BOW), word embedding, and Latent Dirichlet allocation-based approaches. Ultimately, 1004 head CT reports were manually labeled for findings of interest by physicians, and a subset of these were deemed critical findings. Lasso logistic regression was used to train models for physician-assigned labels on 602 of 1004 head CT reports (60%) using the constructed features, and the performance of these models was validated on a held-out 402 of 1004 reports (40%). Models were scored by area under the receiver operating characteristic curve (AUC), and aggregate AUC statistics were reported for (a) all labels, (b) critical labels, and (c) the presence of any critical finding in a report. Sensitivity, specificity, accuracy, and F1 score were reported for the best performing model's (a) predictions of all labels and (b) identification of reports containing critical findings. Results The best-performing model (BOW with unigrams, bigrams, and trigrams plus average word embeddings vector) had a held-out AUC of 0.966 for identifying the presence of any critical head CT finding and an average 0.957 AUC across all head CT findings. Sensitivity and specificity for identifying the presence of any critical finding were 92.59% (175 of 189) and 89.67% (191 of 213), respectively. Average sensitivity and specificity across all findings were 90.25% (1898 of 2103) and 91.72% (18 351 of 20 007), respectively. Simpler BOW methods achieved results competitive with those of more sophisticated approaches, with an average AUC for presence of any critical finding of 0.951 for unigram BOW versus 0.966 for the best-performing model. The Yule I of the head CT corpus was 34, markedly lower than that of the Reuters corpus (at 103) or I2B2 discharge summaries (at 271), indicating lower linguistic complexity. Conclusion Automated methods can be used to identify findings in radiology reports. The success of this approach benefits from the standardized language of these reports. With this method, a large labeled corpus can be generated for applications such as deep learning. © RSNA, 2018 Online supplemental material is available for this article.
引用
- Zech J, Pain M, Titano J, et al. Natural Language-based Machine Learning Models for the Annotation of Clinical Radiology Reports. Radiology. 2018. PMID: 29381109. doi:10.1148/radiol.2018171093
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。