文献速递重症综合管理与镇静镇痛消化、代谢与肾脏急症

APPRAISE-AI工具:定量评估临床决策支持AI研究质量

APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support.
PubMed 文献 · JAMA Netw Open · 2023-09-05
一句话结论APPRAISE-AI工具在28项脓毒症机器学习研究中信度良好(组内相关系数0.91–0.98),总分与专家评分、引用率、QUADAS-2低偏倚域数及TRIPOD依从性均相关。

这项质量改进研究开发了APPRAISE-AI工具,从临床相关性、数据质量、方法学实施、结果稳健性、报告质量和可重复性6个域、24个条目对AI预测模型研究进行定量评分(总分100分)。将该工具应用于一项截至2019年9月13日发表的脓毒症机器学习系统评价中的28项研究,总分范围为33分(低质量)至67分(高质量),多数研究为中等质量。总分与专家评分(Spearman ρ=0.82)、3年引用率(ρ=0.69)、QUADAS-2低偏倚域数量(ρ=0.56)及TRIPOD声明依从性(ρ=0.87)相关;各条目、各域及总分的组内和组间相关系数分别为0.74–1.00、0.81–0.99和0.91–0.98。作者认为该工具可为研究者、审稿人、编辑和资助机构提供定量比较AI研究质量的方法。

要点

  • APPRAISE-AI共6个域、24个条目、总分100分,评分越高提示方法学或报告质量越强。
  • 在28项脓毒症机器学习研究中,APPRAISE-AI总分范围为33分(低质量)至67分(高质量),多数研究为中等质量。
  • 得分最低的5个条目为数据来源、样本量计算、偏倚评估、误差分析和透明度。
  • 总分与专家评分(Spearman ρ=0.82;95%CI 0.64–0.91;P<.001)、3年引用率(ρ=0.69;95%CI 0.43–0.85;P<.001)、QUADAS-2低偏倚域数量(ρ=0.56;95%CI 0.24–0.77;P=.002)及TRIPOD依从性(ρ=0.87;95%CI 0.73–0.94;P<.001)相关。
  • 组内和组间信度:各条目组内相关系数0.74–1.00,各域0.81–0.99,总分0.91–0.98。

适用人群与场景

  • 评估对象为临床决策支持的AI预测模型研究,涵盖模型开发期、静默期和临床试验期。
  • 工具应用于一项截至2019年9月13日发表的脓毒症机器学习系统评价中的28项研究。

诊断标准与关键阈值

  • APPRAISE-AI评分覆盖6个域:临床相关性、数据质量、方法学实施、结果稳健性、报告质量、可重复性。
  • 共24个条目,总分最高100分;每个条目分配分值,分值越高提示方法学或报告质量越强。

研究设计与关键结果

  • 研究类型为质量改进研究,纳入28项研究;总体APPRAISE-AI评分范围为33分(低质量)至67分(高质量),多数研究为中等质量。
  • 得分最低的5个条目依次为数据来源、样本量计算、偏倚评估、误差分析和透明度。
  • 总分与专家评分相关:Spearman ρ=0.82(95%CI 0.64–0.91),P<.001。
  • 总分与3年引用率相关:Spearman ρ=0.69(95%CI 0.43–0.85),P<.001。
  • 总分与QUADAS-2低偏倚域数量相关:Spearman ρ=0.56(95%CI 0.24–0.77),P=.002。
  • 总分与TRIPOD声明依从性相关:Spearman ρ=0.87(95%CI 0.73–0.94),P<.001。
  • 组内和组间信度:各条目组内相关系数为0.74–1.00,各域为0.81–0.99,总分为0.91–0.98。

证据类型、等级与局限

  • 证据类型:质量改进研究;原文未报告证据等级或推荐强度。
  • 局限:原文未报告局限性。

对床旁实践的改变

  • 作者认为,该工具可为研究者、审稿人、编辑和资助机构提供定量方法,用于比较临床决策支持AI研究的研究质量。

提炼说明

  • 本条依据论文摘要整理;摘要未涉及「处置流程与药物剂量」「红旗征与不能漏的情况」「指南或共识出处」,故未列出这些栏目;全文请点「阅读原文」。
英文原文(摘要)· 点击展开

IMPORTANCE: Artificial intelligence (AI) has gained considerable attention in health care, yet concerns have been raised around appropriate methods and fairness. Current AI reporting guidelines do not provide a means of quantifying overall quality of AI research, limiting their ability to compare models addressing the same clinical question.

OBJECTIVE: To develop a tool (APPRAISE-AI) to evaluate the methodological and reporting quality of AI prediction models for clinical decision support.

DESIGN, SETTING, AND PARTICIPANTS: This quality improvement study evaluated AI studies in the model development, silent, and clinical trial phases using the APPRAISE-AI tool, a quantitative method for evaluating quality of AI studies across 6 domains: clinical relevance, data quality, methodological conduct, robustness of results, reporting quality, and reproducibility. These domains included 24 items with a maximum overall score of 100 points. Points were assigned to each item, with higher points indicating stronger methodological or reporting quality. The tool was applied to a systematic review on machine learning to estimate sepsis that included articles published until September 13, 2019. Data analysis was performed from September to December 2022.

MAIN OUTCOMES AND MEASURES: The primary outcomes were interrater and intrarater reliability and the correlation between APPRAISE-AI scores and expert scores, 3-year citation rate, number of Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) low risk-of-bias domains, and overall adherence to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement.

RESULTS: A total of 28 studies were included. Overall APPRAISE-AI scores ranged from 33 (low quality) to 67 (high quality). Most studies were moderate quality. The 5 lowest scoring items included source of data, sample size calculation, bias assessment, error analysis, and transparency. Overall APPRAISE-AI scores were associated with expert scores (Spearman ρ, 0.82; 95% CI, 0.64-0.91; P < .001), 3-year citation rate (Spearman ρ, 0.69; 95% CI, 0.43-0.85; P < .001), number of QUADAS-2 low risk-of-bias domains (Spearman ρ, 0.56; 95% CI, 0.24-0.77; P = .002), and adherence to the TRIPOD statement (Spearman ρ, 0.87; 95% CI, 0.73-0.94; P < .001). Intraclass correlation coefficient ranges for interrater and intrarater reliability were 0.74 to 1.00 for individual items, 0.81 to 0.99 for individual domains, and 0.91 to 0.98 for overall scores.

CONCLUSIONS AND RELEVANCE: In this quality improvement study, APPRAISE-AI demonstrated strong interrater and intrarater reliability and correlated well with several study quality measures. This tool may provide a quantitative approach for investigators, reviewers, editors, and funding organizations to compare the research quality across AI studies for clinical decision support.

来源:JAMA Netw Open,查看原文。版权归原作者所有。

参考来源

  1. Kwong JCC, Khondker A, Lajkosz K, et al. APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support. JAMA Netw Open. 2023. PMID: 37747733. doi:10.1001/jamanetworkopen.2023.35377

供执业医师抢救与诊疗参考,不替代临床判断;剂量请结合体重、肝肾功能和本院药品规格复核,以指南原文为准。