不同自然语言处理文本预处理方法对ICU预测模型性能的影响
该研究利用加州大学旧金山分校(UCSF)1万例成人ICU患者临床记录文本构建机器学习模型预测院内死亡,比较不同文本预处理策略(原始文本、清洗、词干化、TF-IDF向量化、n-gram创建)对模型性能的影响。模型在UCSF数据上10折交叉验证训练和内部验证,随后在贝斯以色列女执事医疗中心(BIDMC)27058例患者数据上外部验证。UCSF和BIDMC的死亡率分别为7.4%和10.9%。内部验证中,TF-IDF模型AUROC为0.89(95%CI 0.85-0.94),优于原始文本模型的0.84(95%CI 0.80-0.89);外部验证中,TF-IDF模型AUROC为0.83,亦优于原始文本模型的0.72。结论认为预处理策略影响模型区分度,包含清洗、词干化和TF-IDF向量化的流程改善最大,但在临床实施自然语言处理算法前需进一步研究作者隐含偏倚的管理。
要点
- 在UCSF内部验证中,TF-IDF向量化模型预测院内死亡的AUROC为0.89(95%CI 0.85-0.94),优于原始文本模型的0.84(95%CI 0.80-0.89)。
- 在BIDMC外部验证中,TF-IDF模型AUROC为0.83,同样优于原始文本模型的0.72。
- 包含清洗、词干化和TF-IDF向量化的预处理流程带来最大模型性能提升。
- 在临床实施自然语言处理算法前,需进一步研究如何管理记录文本中作者隐含偏倚。
适用人群与场景
- 成人ICU患者,来自UCSF和BIDMC的ICU数据。
- 纳入标准:成人ICU患者;排除标准:原文未提及。
诊断标准与关键阈值
- 原文未提及
处置流程与药物剂量
- 原文未提及
红旗征与不能漏的情况
- 原文未提及
研究设计与关键结果
- 人群:成人ICU患者;干预:不同文本预处理策略(原始文本、清洗、词干化、TF-IDF向量化、n-gram创建);对照:原始文本。
- 主要终点:院内死亡预测模型性能(AUROC)。UCSF内部验证:原始文本模型AUROC 0.84(95%CI 0.80-0.89),TF-IDF模型AUROC 0.89(95%CI 0.85-0.94)。
- 外部验证(BIDMC):原始文本模型AUROC 0.72,TF-IDF模型AUROC 0.83。
- 关键次要终点和不良事件:原文未提及。
证据类型、等级与局限
- 证据类型:观察性研究(机器学习模型开发与验证);证据等级或推荐强度:原文未提及。
- 局限:单中心训练(UCSF)并外部验证(BIDMC);回顾性数据;未涉及作者隐含偏倚管理;需进一步研究临床实施。
对床旁实践的改变
- 对于ICU死亡预测的自然语言处理模型,采用清洗、词干化和TF-IDF向量化的预处理流程可能改善模型区分度,但需在临床实施前进一步验证并管理文本偏倚。
- 适用人群:成人ICU患者;不适用人群:原文未提及。
- 常见陷阱:原文未提及。
指南或共识出处
- 原文未提及
以上提炼依据论文摘要(全文未开放获取,或许可不允许翻译转载);全文请点「阅读原文」。
英文原文(摘要)· 点击展开
UNLABELLED: To evaluate whether different approaches in note text preparation (known as preprocessing) can impact machine learning model performance in the case of mortality prediction ICU.
DESIGN: Clinical note text was used to build machine learning models for adults admitted to the ICU. Preprocessing strategies studied were none (raw text), cleaning text, stemming, term frequency-inverse document frequency vectorization, and creation of n-grams. Model performance was assessed by the area under the receiver operating characteristic curve. Models were trained and internally validated on University of California San Francisco data using 10-fold cross validation. These models were then externally validated on Beth Israel Deaconess Medical Center data.
SETTING: ICUs at University of California San Francisco and Beth Israel Deaconess Medical Center.
SUBJECTS: Ten thousand patients in the University of California San Francisco training and internal testing dataset and 27,058 patients in the external validation dataset, Beth Israel Deaconess Medical Center.
INTERVENTIONS: None.
MEASUREMENTS AND MAIN RESULTS: Mortality rate at Beth Israel Deaconess Medical Center and University of California San Francisco was 10.9% and 7.4%, respectively. Data are presented as area under the receiver operating characteristic curve (95% CI) for models validated at University of California San Francisco and area under the receiver operating characteristic curve for models validated at Beth Israel Deaconess Medical Center. Models built and trained on University of California San Francisco data for the prediction of inhospital mortality improved from the raw note text model (AUROC, 0.84; CI, 0.80-0.89) to the term frequency-inverse document frequency model (AUROC, 0.89; CI, 0.85-0.94). When applying the models developed at University of California San Francisco to Beth Israel Deaconess Medical Center data, there was a similar increase in model performance from raw note text (area under the receiver operating characteristic curve at Beth Israel Deaconess Medical Center: 0.72) to the term frequency-inverse document frequency model (area under the receiver operating characteristic curve at Beth Israel Deaconess Medical Center: 0.83).
CONCLUSIONS: Differences in preprocessing strategies for note text impacted model discrimination. Completing a preprocessing pathway including cleaning, stemming, and term frequency-inverse document frequency vectorization resulted in the preprocessing strategy with the greatest improvement in model performance. Further study is needed, with particular emphasis on how to manage author implicit bias present in note text, before natural language processing algorithms are implemented in the clinical setting.
参考来源
- Mahendra M, Luo Y, Mills H, et al. Impact of Different Approaches to Preparing Notes for Analysis With Natural Language Processing on the Performance of Prediction Models in Intensive Care. Crit Care Explor. 2021. PMID: 34136824. doi:10.1097/CCE.0000000000000450
供执业医师抢救与诊疗参考,不替代临床判断;剂量请结合体重、肝肾功能和本院药品规格复核,以指南原文为准。