机器学习分类器在两家大型学术放射学机构膝关节MRI报告中的性能:一种评估诊断产出的工具
本研究构建了基于自然语言处理(NLP)和支持向量机的机器学习分类器,用于对两家大型医疗机构的自由文本膝关节MRI报告进行分类。在同机构训练数据下,模型对临床显著概念类别的分类F1分数大于90%(95%CI 84.6-97.3%);跨机构应用时,两家机构的F1分数分别为77.6%(95%CI 69.5-85.7%)和90.2%(95%CI 84.5-95.9%)。结论支持该分类器在多机构影像文本报告分类中的可行性,且无需机构特异性训练数据。
要点
- 同机构训练后,NLP分类器对膝关节MRI报告临床显著概念类别的F1分数大于90%。
- 跨机构应用时,两家机构F1分数分别为77.6%和90.2%,无需机构特异性训练数据。
- 结果表明该分类器对自由文本膝关节MRI报告分类具有机构无关的可重复性。
核心结论
- NLP机器学习分类器对自由文本膝关节MRI报告分类具有优秀准确率,支持膝关节MRI报告分类的机构无关可重复性,并支持单一分类器无需机构特异性训练数据即可进行多机构影像文本报告分类的可行性。
研究设计与样本
- 研究构建了基于手动分类叙述性膝关节MRI报告中术语和模式的NLP系统,使用支持向量机框架训练分类器,在两家大型医疗机构的专家分类膝关节MRI报告上训练和测试,并评估同机构和跨机构性能。
诊断标准与关键数值
- 同机构训练数据后,模型对临床显著概念类别的分类F1分数大于90%(95%CI 84.6-97.3%)。
- 跨机构应用且无机构特异性训练数据时,两家机构的F1分数分别为77.6%(95%CI 69.5-85.7%)和90.2%(95%CI 84.5-95.9%)。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
OBJECTIVE: The purpose of this study is to evaluate the performance of a natural language processing (NLP) system in classifying a database of free-text knee MRI reports at two separate academic radiology practices.
MATERIALS AND METHODS: An NLP system that uses terms and patterns in manually classified narrative knee MRI reports was constructed. The NLP system was trained and tested on expert-classified knee MRI reports from two major health care organizations. Radiology reports were modeled in the training set as vectors, and a support vector machine framework was used to train the classifier. A separate test set from each organization was used to evaluate the performance of the system. We evaluated the performance of the system both within and across organizations. Standard evaluation metrics, such as accuracy, precision, recall, and F1 score (i.e., the weighted average of the precision and recall), and their respective 95% CIs were used to measure the efficacy of our classification system.
RESULTS: The accuracy for radiology reports that belonged to the model's clinically significant concept classes after training data from the same institution was good, yielding an F1 score greater than 90% (95% CI, 84.6-97.3%). Performance of the classifier on cross-institutional application without institution-specific training data yielded F1 scores of 77.6% (95% CI, 69.5-85.7%) and 90.2% (95% CI, 84.5-95.9%) at the two organizations studied.
CONCLUSION: The results show excellent accuracy by the NLP machine learning classifier in classifying free-text knee MRI reports, supporting the institution-independent reproducibility of knee MRI report classification. Furthermore, the machine learning classifier performed well on free-text knee MRI reports from another institution. These data support the feasibility of multiinstitutional classification of radiologic imaging text reports with a single machine learning classifier without requiring institution-specific training data.
引用
- Hassanpour S, Langlotz CP, Amrhein TJ, et al. Performance of a Machine Learning Classifier of Knee MRI Reports in Two Large Academic Radiology Practices: A Tool to Estimate Diagnostic Yield. AJR Am J Roentgenol. 2017. PMID: 28140627. doi:10.2214/AJR.16.16128
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。