指南与共识技术与安全

关于开发和利用影像测试集以研究人工智能在健康筛查中检测性能的建议

Recommendations for the development and use of imaging test sets to investigate the test performance of artificial intelligence in health screening.
指南 / 共识 · Lancet Digit Health · 2022-01-01

本文通过快速文献综述(2012至2020年英文文献)与主题分析,结合筛查和人工智能专家评估,提出15条用于开发和利用影像测试集的循证原则,以支持筛查项目中人工智能准确性研究,确保测试集适用并减少偏倚。

要点

  • 15条原则涵盖人群、干预、对照或参考标准、结局和研究设计五个领域。
  • 其中五条涉及人群,一条干预,两条对照,一条参考标准,一条同时涉及参考标准和对照,四条适用于结局,一条适用于研究设计。
  • 文献原则未涵盖筛查特异性偏倚,本建议已将其纳入。

核心结论

  • 所提出的原则应用于支持筛查项目中评估人工智能准确性的测试集开发与使用,以确保测试集适用并减少偏倚。

研究设计与样本

  • 通过快速文献综述(2012至2020年英文文献)和主题分析,使用人群、干预、对照或参考标准、结局和研究设计框架映射主题并编码原则。
  • 由筛查和人工智能专家组评估循证原则的完整性并提供进一步考虑。

诊断标准与关键数值

  • 最终推荐15条原则,其中五条涉及人群,一条涉及干预,两条涉及对照,一条涉及参考标准,一条同时涉及参考标准和对照,四条适用于结局,一条适用于研究设计。

临床应用与阅片要点

  • 原则应作为开发和利用影像测试集的基础,用于评估人工智能在筛查项目中的准确性研究,以确保测试集适用并减少偏倚。

局限性

  • 文献中的原则未涵盖筛查特异性偏倚,本建议已将其纳入。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

Rigorous evaluation of artificial intelligence (AI) systems for image classification is essential before deployment into health-care settings, such as screening programmes, so that adoption is effective and safe. A key step in the evaluation process is the external validation of diagnostic performance using a test set of images. We conducted a rapid literature review on methods to develop test sets, published from 2012 to 2020, in English. Using thematic analysis, we mapped themes and coded the principles using the Population, Intervention, and Comparator or Reference standard, Outcome, and Study design framework. A group of screening and AI experts assessed the evidence-based principles for completeness and provided further considerations. From the final 15 principles recommended here, five affect population, one intervention, two comparator, one reference standard, and one both reference standard and comparator. Finally, four are appliable to outcome and one to study design. Principles from the literature were useful to address biases from AI; however, they did not account for screening specific biases, which we now incorporate. The principles set out here should be used to support the development and use of test sets for studies that assess the accuracy of AI within screening programmes, to ensure they are fit for purpose and minimise bias.

来源:Lancet Digit Health,查看原文。版权归原作者所有。

引用

  1. Chalkidou A, Shokraneh F, Kijauskaite G, et al. Recommendations for the development and use of imaging test sets to investigate the test performance of artificial intelligence in health screening. Lancet Digit Health. 2022. PMID: 36427951. doi:10.1016/S2589-7500(22)00186-8

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。