文献速递眼底病与葡萄膜炎小儿眼科与斜弱视

人工智能用于早产儿视网膜病变:血管严重程度评分与国际专家诊断的验证

Artificial Intelligence for Retinopathy of Prematurity: Validation of a Vascular Severity Scale against International Expert Diagnosis.
PubMed 文献 · Ophthalmology · 2022-02-12

本研究以国际早产儿视网膜病变分类第三版(ICROP3)委员会的34名早产儿视网膜病变(ROP)专家为对象,将AI衍生的血管严重程度评分与专家对分期和附加病变的标注进行比较。结果显示该评分与附加病变分类高度相关(CC=0.90,P<0.001),与眼底镜分期诊断亦相关(P<0.001)。

要点

  • AI血管严重程度评分与专家平均附加病变分类高度相关(CC=0.90,P<0.001)。
  • 该评分与眼底镜分期诊断在所有专家中相关(方差分析P<0.001)。
  • 专家间对附加病变图像比较的平均加权kappa为0.67,平均相关系数为0.88。

核心结论

  • ROP血管严重程度评分与ICROP3委员会成员对附加病变和分期的标注相关性良好,而专家间标注存在显著差异。
  • 为ROP软件即医疗器械建立经验证的共识评分系统可促进全球创新和这些技术的监管授权。

研究设计与样本

  • 基于AI的ROP血管严重程度评分的验证研究。
  • 参与者为来自ICROP3委员会的34名ROP专家。
  • 使用两个独立数据集,每个数据集含30张眼底照片,分别用于分期(0-5)和附加病变(附加、前附加、无)标注。
  • 专家通过开源平台标注,平均结果生成每张图像附加病变(1-9)和分期(1-3)的连续标签。
  • 专家还被要求就附加病变的相对严重程度对每张图像进行两两比较。

主要发现

  • 所有观察者间对附加病变图像比较的平均加权kappa和相关系数分别为0.67和0.88。
  • 血管严重程度评分与所有专家平均附加病变分类高度相关(CC=0.90,P<0.001)。
  • 血管严重程度评分与所有专家眼底镜分期诊断相关(方差分析P<0.001)。

关键数值与推荐

  • 加权kappa和Pearson相关系数用于评估专家间分期和附加病变分类标签的一致性。
  • Elo算法将每位专家的两两比较转换为从最轻到最重的图像有序集合。

临床怎么用

  • 原文未据此给出推荐。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

PURPOSE: To validate a vascular severity score as an appropriate output for artificial intelligence (AI) Software as a Medical Device (SaMD) for retinopathy of prematurity (ROP) through comparison with ordinal disease severity labels for stage and plus disease assigned by the International Classification of Retinopathy of Prematurity, Third Edition (ICROP3), committee. DESIGN: Validation study of an AI-based ROP vascular severity score. PARTICIPANTS: A total of 34 ROP experts from the ICROP3 committee. METHODS: Two separate datasets of 30 fundus photographs each for stage (0-5) and plus disease (plus, preplus, neither) were labeled by members of the ICROP3 committee using an open-source platform. Averaging these results produced a continuous label for plus (1-9) and stage (1-3) for each image. Experts were also asked to compare each image to each other in terms of relative severity for plus disease. Each image was also labeled with a vascular severity score from the Imaging and Informatics in ROP deep learning system, which was compared with each grader's diagnostic labels for correlation, as well as the ophthalmoscopic diagnosis of stage. MAIN OUTCOME MEASURES: Weighted kappa and Pearson correlation coefficients (CCs) were calculated between each pair of grader classification labels for stage and plus disease. The Elo algorithm was also used to convert pairwise comparisons for each expert into an ordered set of images from least to most severe. RESULTS: The mean weighted kappa and CC for all interobserver pairs for plus disease image comparison were 0.67 and 0.88, respectively. The vascular severity score was found to be highly correlated with both the average plus disease classification (CC = 0.90, P < 0.001) and the ophthalmoscopic diagnosis of stage (P < 0.001 by analysis of variance) among all experts. CONCLUSIONS: The ROP vascular severity score correlates well with the International Classification of Retinopathy of Prematurity committee member's labels for plus disease and stage, which had significant intergrader variability. Generation of a consensus for a validated scoring system for ROP SaMD can facilitate global innovation and regulatory authorization of these technologies.

来源:Ophthalmology,查看原文。版权归原作者所有。

引用

  1. Campbell JP, Chiang MF, Chen JS, et al. Artificial Intelligence for Retinopathy of Prematurity: Validation of a Vascular Severity Scale against International Expert Diagnosis. Ophthalmology. 2022. PMID: 35157950. doi:10.1016/j.ophtha.2022.02.008

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。