文献速递医学人工智能与信息学流行病学与生物统计基础与转化研究

人工智能与临床医生对比:深度学习研究设计、报告标准与声明之系统评价

Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies.
PubMed 文献 · BMJ · 2020-03-25

该系统评价检索2010年至2019年6月Medline、Embase、Cochrane CENTRAL及WHO试验注册库,纳入比较医学影像深度学习算法与专家临床医生表现的研究。仅发现10项深度学习随机临床试验记录(2项已发表,8项进行中);81项非随机试验中仅9项为前瞻性、6项在真实临床环境测试,58项偏倚风险高,数据与代码可及性严重不足。

要点

  • 医学影像深度学习随机试验极少,仅2项已发表、8项进行中。
  • 81项非随机试验中仅9项前瞻性、6项真实临床环境验证。
  • 58/81项研究整体偏倚风险高,报告标准依从性欠佳。

核心结论

  • 医学影像领域前瞻性深度学习研究与随机试验数量很少。
  • 多数非随机试验非前瞻性、偏倚风险高且偏离现有报告标准。
  • 多数研究缺乏数据与代码可及性,人类对照组建模偏小。
  • 未来研究应降低偏倚风险、增强真实临床相关性、改善报告透明度并适度克制结论。

研究设计与样本

  • 系统评价,检索Medline、Embase、Cochrane CENTRAL及WHO试验注册库2010年至2019年6月。
  • 纳入比较医学影像深度学习算法与一个或多个当代专家临床医生表现的随机试验注册和非随机研究。
  • 随机研究采用CONSORT评估报告依从性、Cochrane偏倚风险工具评估偏倚。
  • 非随机研究采用TRIPOD评估报告依从性、PROBAST评估偏倚风险。
  • 研究注册号PROSPERO CRD42019123605。

主要发现

  • 仅发现10项深度学习随机临床试验记录,其中2项已发表(除缺乏盲法外偏倚风险低、报告标准依从性高),8项进行中。
  • 识别81项非随机临床试验,仅9项为前瞻性、仅6项在真实世界临床环境测试。
  • 对照专家组中位人数仅4人(四分位距2-9)。
  • 全部数据集与代码完全可及性严重受限,分别有95%和93%的研究不可获取。
  • 81项研究中58项整体偏倚风险高;29项TRIPOD条目中有12项依从率低于50%。

关键数值与推荐

  • 81项研究中61项摘要声称人工智能表现至少与临床医生相当或更优。
  • 81项研究中仅31项(38%)声明需要进一步前瞻性研究或试验。
  • 81项非随机试验中仅9项为前瞻性、6项在真实世界临床环境测试。
  • 对照专家组中位人数为4人(四分位距2-9)。
  • 数据集不可获取见于95%的研究,代码不可获取见于93%的研究。

临床怎么用

  • 解读AI与临床医生对比研究时需关注是否前瞻性、真实世界验证及盲法。
  • 应核查研究数据和代码可及性及对照专家数量,警惕高偏倚风险。
  • 对摘要中AI不劣于或优于临床医生的声明应结合研究设计与报告质量审慎采信。
  • 临床采纳前优先参考已发表随机试验结果,并关注报告标准依从性。

局限性

  • 检索截止2019年6月,未涵盖此后发表的研究。
  • 仅纳入比较深度学习算法与专家临床医生的研究,可能遗漏其他设计。
  • 报告标准依从性采用CONSORT与TRIPOD评估,偏倚采用Cochrane工具与PROBAST评估。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVE: To systematically examine the design, reporting standards, risk of bias, and claims of studies comparing the performance of diagnostic deep learning algorithms for medical imaging with that of expert clinicians. DESIGN: Systematic review. DATA SOURCES: Medline, Embase, Cochrane Central Register of Controlled Trials, and the World Health Organization trial registry from 2010 to June 2019. ELIGIBILITY CRITERIA FOR SELECTING STUDIES: Randomised trial registrations and non-randomised studies comparing the performance of a deep learning algorithm in medical imaging with a contemporary group of one or more expert clinicians. Medical imaging has seen a growing interest in deep learning research. The main distinguishing feature of convolutional neural networks (CNNs) in deep learning is that when CNNs are fed with raw data, they develop their own representations needed for pattern recognition. The algorithm learns for itself the features of an image that are important for classification rather than being told by humans which features to use. The selected studies aimed to use medical imaging for predicting absolute risk of existing disease or classification into diagnostic groups (eg, disease or non-disease). For example, raw chest radiographs tagged with a label such as pneumothorax or no pneumothorax and the CNN learning which pixel patterns suggest pneumothorax. REVIEW METHODS: Adherence to reporting standards was assessed by using CONSORT (consolidated standards of reporting trials) for randomised studies and TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) for non-randomised studies. Risk of bias was assessed by using the Cochrane risk of bias tool for randomised studies and PROBAST (prediction model risk of bias assessment tool) for non-randomised studies. RESULTS: Only 10 records were found for deep learning randomised clinical trials, two of which have been published (with low risk of bias, except for lack of blinding, and high adherence to reporting standards) and eight are ongoing. Of 81 non-randomised clinical trials identified, only nine were prospective and just six were tested in a real world clinical setting. The median number of experts in the comparator group was only four (interquartile range 2-9). Full access to all datasets and code was severely limited (unavailable in 95% and 93% of studies, respectively). The overall risk of bias was high in 58 of 81 studies and adherence to reporting standards was suboptimal (<50% adherence for 12 of 29 TRIPOD items). 61 of 81 studies stated in their abstract that performance of artificial intelligence was at least comparable to (or better than) that of clinicians. Only 31 of 81 studies (38%) stated that further prospective studies or trials were required. CONCLUSIONS: Few prospective deep learning studies and randomised trials exist in medical imaging. Most non-randomised trials are not prospective, are at high risk of bias, and deviate from existing reporting standards. Data and code availability are lacking in most studies, and human comparator groups are often small. Future studies should diminish risk of bias, enhance real world clinical relevance, improve reporting and transparency, and appropriately temper conclusions. STUDY REGISTRATION: PROSPERO CRD42019123605.

来源:BMJ,查看原文。版权归原作者所有。

引用

  1. Nagendran M, Chen Y, Lovejoy CA, et al. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ. 2020. PMID: 32213531. doi:10.1136/bmj.m689

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。