系统综述发现基于机器学习的预测模型研究中存在「spin」做法与报告标准不佳
该系统综述检索PubMed 2018年1月至2019年12月,纳入152项使用监督式机器学习的诊断与预后预测模型研究。研究发现,报告区分度时未给精度估计在摘要中占74.6%(53/71)、正文中占65.4%(53/81);推荐日常使用的摘要中95.2%(20/21)缺乏外部验证;正文中55.6%(74/133)在无外部验证下给出临床使用推荐;仅8.6%(13/152)引用报告指南。结论认为机器学习预测模型研究也存在「spin」做法与报告标准不佳。
要点
- 摘要中74.6%(53/71)报告区分度时未给精度估计,正文中为65.4%(53/81)。
- 推荐日常使用的21篇摘要中20篇(95.2%)缺乏外部验证。
- 正文中74/133(55.6%)在无外部验证下给出临床使用推荐。
核心结论
- 机器学习预测模型研究中也存在「spin」做法与报告标准不佳。
- 作者认为,针对「spin」识别的定制框架将提升预测模型研究的规范报告。
研究设计与样本
- 系统综述,系统检索PubMed 2018年1月至2019年12月,纳入使用监督式机器学习的诊断与预后预测模型研究,对数据来源、结局或临床专科无限制,共纳入152项研究。
主要发现
- 152项研究中,38%报告诊断模型,62%报告预后模型。
- 报告区分度时未给精度估计:摘要53/71(74.6% [95% CI 63.4-83.3]),正文53/81(65.4% [95% CI 54.6-74.9])。
- 21篇摘要推荐模型用于日常实践,其中20篇(95.2% [95% CI 77.3-99.8])缺乏对所开发模型的任何外部验证。
- 正文中74/133(55.6% [95% CI 47.2-63.8])在无任何外部验证下给出临床使用推荐。
- 13/152(8.6% [95% CI 5.1-14.1])研究引用了报告指南。
临床怎么用
- 原文未据此给出推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVES: We evaluated the presence and frequency of spin practices and poor reporting standards in studies that developed and/or validated clinical prediction models using supervised machine learning techniques. STUDY DESIGN AND SETTING: We systematically searched PubMed from 01/2018 to 12/2019 to identify diagnostic and prognostic prediction model studies using supervised machine learning. No restrictions were placed on data source, outcome, or clinical specialty. RESULTS: We included 152 studies: 38% reported diagnostic models and 62% prognostic models. When reported, discrimination was described without precision estimates in 53/71 abstracts (74.6% [95% CI 63.4-83.3]) and 53/81 main texts (65.4% [95% CI 54.6-74.9]). Of the 21 abstracts that recommended the model to be used in daily practice, 20 (95.2% [95% CI 77.3-99.8]) lacked any external validation of the developed models. Likewise, 74/133 (55.6% [95% CI 47.2-63.8]) studies made recommendations for clinical use in their main text without any external validation. Reporting guidelines were cited in 13/152 (8.6% [95% CI 5.1-14.1]) studies. CONCLUSION: Spin practices and poor reporting standards are also present in studies on prediction models using machine learning techniques. A tailored framework for the identification of spin will enhance the sound reporting of prediction model studies.
引用
- Andaur Navarro CL, Damen JAA, Takada T, et al. Systematic review finds "spin" practices and poor reporting standards in studies on machine learning-based prediction models. J Clin Epidemiol. 2023. PMID: 37024020. doi:10.1016/j.jclinepi.2023.03.024
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。