文献速递医学人工智能与信息学流行病学与生物统计

监督机器学习预测模型研究的偏倚风险:系统综述

Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review.
PubMed 文献 · BMJ · 2021-10-20

该系统综述评估2018至2019年PubMed收录的152项监督机器学习预测模型研究的方法学质量,使用PROBAST工具评价偏倚风险。结果显示,171项分析中148项(87%,95%CI 81%至91%)为高偏倚风险,分析领域问题最突出,小样本、缺失数据处理不当和过拟合处理不足是主要风险因素。

要点

  • 87%的机器学习预测模型分析存在高偏倚风险。
  • 分析领域是偏倚风险最高的领域。
  • 样本量小、缺失数据处理不当和过拟合处理不足是主要风险因素。

核心结论

  • 多数机器学习预测模型研究的方法学质量差且存在高偏倚风险。
  • 偏倚风险因素包括样本量小、缺失数据处理不当和过拟合处理不足。
  • 需要改进设计、实施、报告和验证以促进临床应用。

研究设计与样本

  • 纳入报告监督机器学习多变量预测模型开发(含或不含外部验证)的文章,无研究设计、数据来源或预测健康结局限制。
  • 共纳入152项研究,包括58项(38%)诊断预测模型和94项(62%)预后预测模型。
  • 使用PROBAST工具评估偏倚风险,应用于152个开发模型和19个外部验证,共171项分析。

主要发现

  • 171项分析中148项(87%,95%CI 81%至91%)被评为高偏倚风险。
  • 分析领域最常被评为高偏倚风险。
  • 152个模型中85个(56%,95%CI 48%至64%)每个候选预测变量的事件数不足。
  • 62个模型(41%,95%CI 33%至49%)缺失数据处理不当,59个模型(39%,95%CI 31%至47%)过拟合评估不当。
  • 多数模型使用了合适的数据源进行开发(73%,95%CI 66%至79%)和外部验证(74%,95%CI 51%至88%)。

临床怎么用

  • 临床应用机器学习预测模型时需警惕其高偏倚风险。
  • 应关注模型开发中的样本量、缺失数据处理和过拟合控制。

局限性

  • 仅检索PubMed,可能遗漏其他数据库的研究。
  • 仅纳入2018至2019年发表的研究。
  • 未报告对研究间异质性的评估。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVE: To assess the methodological quality of studies on prediction models developed using machine learning techniques across all medical specialties. DESIGN: Systematic review. DATA SOURCES: PubMed from 1 January 2018 to 31 December 2019. ELIGIBILITY CRITERIA: Articles reporting on the development, with or without external validation, of a multivariable prediction model (diagnostic or prognostic) developed using supervised machine learning for individualised predictions. No restrictions applied for study design, data source, or predicted patient related health outcomes. REVIEW METHODS: Methodological quality of the studies was determined and risk of bias evaluated using the prediction risk of bias assessment tool (PROBAST). This tool contains 21 signalling questions tailored to identify potential biases in four domains. Risk of bias was measured for each domain (participants, predictors, outcome, and analysis) and each study (overall). RESULTS: 152 studies were included: 58 (38%) included a diagnostic prediction model and 94 (62%) a prognostic prediction model. PROBAST was applied to 152 developed models and 19 external validations. Of these 171 analyses, 148 (87%, 95% confidence interval 81% to 91%) were rated at high risk of bias. The analysis domain was most frequently rated at high risk of bias. Of the 152 models, 85 (56%, 48% to 64%) were developed with an inadequate number of events per candidate predictor, 62 handled missing data inadequately (41%, 33% to 49%), and 59 assessed overfitting improperly (39%, 31% to 47%). Most models used appropriate data sources to develop (73%, 66% to 79%) and externally validate the machine learning based prediction models (74%, 51% to 88%). Information about blinding of outcome and blinding of predictors was, however, absent in 60 (40%, 32% to 47%) and 79 (52%, 44% to 60%) of the developed models, respectively. CONCLUSION: Most studies on machine learning based prediction models show poor methodological quality and are at high risk of bias. Factors contributing to risk of bias include small study size, poor handling of missing data, and failure to deal with overfitting. Efforts to improve the design, conduct, reporting, and validation of such studies are necessary to boost the application of machine learning based prediction models in clinical practice. SYSTEMATIC REVIEW REGISTRATION: PROSPERO CRD42019161764.

来源:BMJ,查看原文。版权归原作者所有。

引用

  1. Andaur Navarro CL, Damen JAA, Takada T, et al. Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review. BMJ. 2021. PMID: 34670780. doi:10.1136/bmj.n2281

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。