使用自然语言处理与CLEVER术语词替换的胸部CT自动协议推荐系统的开发与评估
该研究从放射信息系统提取2017至2022年间322387条胸部CT历史检查申请,经预处理和CLEVER术语替换后训练多分类逻辑回归分类器构建协议推荐系统(PRS),并以4名阅片者对300例临床执行协议的选择及评分作为参考标准。PRS的准确率/临床可接受性为84.3%/98.6%,临床执行协议为83.0%/99.3%,阅片者间Fleiss Kappa为0.805,提示PRS达到与人工相似的准确率,可能有助于放射科医生应对日益增加的工作量。
要点
- PRS的准确率84.3%、临床可接受性98.6%,与人工协议(83.0%/99.3%)表现相似。
- 4名阅片者对PRS错误的判定率分别为1%、2.7%、1%和0.7%。
- 所有阅片者及所有协议的Fleiss Kappa为0.805。
核心结论
- PRS达到了与人工表现相似的准确率,可能有助于放射科医生应对日益增加的工作量。
研究设计与样本
- 开发与评估研究,从放射信息系统数据库提取2017至2022年间322387条连续胸部CT历史检查申请,含16个关联患者信息值,剔除缺失字段记录及出现次数少于100次的协议后保留18种协议用于训练。
诊断标准与关键数值
- PRS与临床执行协议的准确率/临床可接受性分别为84.3%/98.6%与83.0%/99.3%,临床可接受性定义为评分3分和4分。
临床应用与阅片要点
- 作者认为PRS可能有助于放射科医生应对日益增加的工作量。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Purpose: To evaluate the clinical performance of a Protocol Recommendation System (PRS) automatic protocolling of chest CT imaging requests. Materials and Methods: 322 387 consecutive historical imaging requests for chest CT between 2017 and 2022 were extracted from a radiology information system (RIS) database containing 16 associated patient information values. Records with missing fields and protocols with <100 occurrences were removed, leaving 18 protocols for training. After freetext pre-processing and applying CLEVER terminology word replacements, the features of a bag-of-words model were used to train a multinomial logistic regression classifier. Four readers protocolled 300 clinically executed protocols (CEP) based on all clinically available information. After their selection was made, the PRS and CEP were unblinded, and the readers were asked to score their agreement (1 = severe error, 2 = moderate error, 3 = disagreement but acceptable, 4 = agreement). The ground truth was established by the readers' majority selection, a judge helped break ties. For the PRS and CEP, the accuracy and clinical acceptability (scores 3 and 4) were calculated. The readers' protocolling reliability was measured using Fleiss' Kappa. Results: Four readers agreed on 203/300 protocols, 3 on 82/300 cases, and in 15 cases, a judge was needed. PRS errors were found by the 4 readers in 1%, 2.7%, 1%, and 0.7% of the cases, respectively. The accuracy/clinical acceptability of the PRS and CEP were 84.3%/98.6% and 83.0%/99.3%, respectively. The Fleiss' Kappa for all readers and all protocols was 0.805. Conclusion: The PRS achieved similar accuracy to human performance and may help radiologists master the ever-increasing workload.
引用
- Rogalla P, Fratesi J, Kandel S, et al. Development and Evaluation of an Automated Protocol Recommendation System for Chest CT Using Natural Language Processing With CLEVER Terminology Word Replacement. Can Assoc Radiol J. 2024. PMID: 39315514. doi:10.1177/08465371241280219
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。