文献速递泌尿生殖腹部MR

已发表前列腺MRI人工智能算法对CLAIM指南的依从性低

The Low Rate of Adherence to Checklist for Artificial Intelligence in Medical Imaging Criteria Among Published Prostate MRI Artificial Intelligence Algorithms.
PubMed 文献 · J Am Coll Radiol · 2022-07-31

该综述评估了53项前列腺MRI人工智能检测/分类研究对CLAIM清单的依从性,发现多数研究未满足关键条目(如去标识化方法、缺失数据处理、失败分析等)。CLAIM依从性百分比与模型AUC呈弱关联,建议补充CLAIM条目以改进出版标准。

要点

  • 多数前列腺MRI AI研究未报告去标识化方法、缺失数据处理和失败分析等关键条目。
  • CLAIM依从性百分比与AUC仅呈弱关联,依从性低不代表模型性能一定差。
  • 建议补充公开数据集使用、参考标准定义和样本量计算等条目以提高可重复性。

局限性

  • 仅纳入英文文献,可能遗漏非英语发表的研究。
  • CLAIM依从性评估基于文章方法/结果部分,可能受报告完整度影响。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

OBJECTIVE: To determine the rigor, generalizability, and reproducibility of published classification and detection artificial intelligence (AI) models for prostate cancer (PCa) on MRI using the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) guidelines, a 42-item checklist that is considered a measure of best practice for presenting and reviewing medical imaging AI research.

MATERIALS AND METHODS: This review searched English literature for studies proposing PCa AI detection and classification models on MRI. Each study was evaluated with the CLAIM checklist. The additional outcomes for which data were sought included measures of AI model performance (eg, area under the curve [AUC], sensitivity, specificity, free-response operating characteristic curves), training and validation and testing group sample size, AI approach, detection versus classification AI, public data set utilization, MRI sequences used, and definition of gold standard for ground truth. The percentage of CLAIM checklist fulfillment was used to stratify studies into quartiles. Wilcoxon's rank-sum test was used for pair-wise comparisons.

RESULTS: In all, 75 studies were identified, and 53 studies qualified for analysis. The original CLAIM items that most studies did not fulfill includes item 12 (77% no): de-identification methods; item 13 (68% no): handling missing data; item 15 (47% no): rationale for choosing ground truth reference standard; item 18 (55% no): measurements of inter- and intrareader variability; item 31 (60% no): inclusion of validated interpretability maps; item 37 (92% no): inclusion of failure analysis to elucidate AI model weaknesses. An AUC score versus percentage CLAIM fulfillment quartile revealed a significant difference of the mean AUC scores between quartile 1 versus quartile 2 (0.78 versus 0.86, P = .034) and quartile 1 versus quartile 4 (0.78 versus 0.89, P = .003) scores. Based on additional information and outcome metrics gathered in this study, additional measures of best practice are defined. These new items include disclosure of public dataset usage, ground truth definition in comparison to other referenced works in the defined task, and sample size power calculation.

CONCLUSION: A large proportion of AI studies do not fulfill key items in CLAIM guidelines within their methods and results sections. The percentage of CLAIM checklist fulfillment is weakly associated with improved AI model performance. Additions or supplementations to CLAIM are recommended to improve publishing standards and aid reviewers in determining study rigor.

来源:J Am Coll Radiol,查看原文。版权归原作者所有。

引用

  1. Belue, Harmon, Lay, et al. The Low Rate of Adherence to Checklist for Artificial Intelligence in Medical Imaging Criteria Among Published Prostate MRI Artificial Intelligence Algorithms. J Am Coll Radiol. 2022. PMID: 35922018. doi:10.1016/j.jacr.2022.05.022

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。