文献速递眼底病与葡萄膜炎

自动化机器学习平台用于糖尿病视网膜病变筛查:与人工分级及IDx-DR的头对头多平台比较

Auto Machine Learning for Diabetic Retinopathy Screening: A Head-to-Head Multiplatform Comparison Against Human Graders and IDx-DR.
PubMed 文献 · Am J Ophthalmol · 2026-05-02

本研究在统一训练与评估框架下,比较多个自动化机器学习平台基于眼底照片筛查糖尿病视网膜病变(DR)的性能,以人工共识分级和FDA批准的自主系统IDx-DR为参考标准。结果显示Amazon SageMaker Canvas和AutoGluon整体判别能力最强,RDR的AUC达0.93-0.94,STDR的AUC最高0.96;Vertex AI性能较弱且不稳定。与人工分级的一致性普遍为低至中等(κ约0.3-0.6),提高概率阈值可改善一致性。

要点

  • Amazon SageMaker Canvas和AutoGluon在RDR和STDR上判别能力最强,AUC分别达0.93-0.94和最高0.96。
  • 50%阈值下Canvas对RDR的敏感度88.3%、特异度85.5%、准确率86.0%,表现最均衡。
  • AutoGluon偏重敏感度(any DR敏感度95.9%)但牺牲特异度;Vertex AI性能较弱且不稳定。

核心结论

  • AutoML平台可在DR筛查中达到有临床意义的表现,但不同工具和阈值间的差异反映其适应不同临床场景的能力,强调外部验证和阈值校准的重要性。

研究设计与样本

  • 回顾性诊断性能与基准比较研究,在自动图像质量过滤后使用APTOS(n=5590)、DDR(n=12524)和EyePACS(n=31557)公共数据集训练图像分类器。
  • 在独立收集的患者级测试队列(n=726)上评估,采用患者所有图像中的最高DR分级进行患者级分类,评估平台包括Google Vertex AI、Amazon Rekognition、Amazon SageMaker Canvas、AutoGluon、AutoKeras和Apple CreateML。
  • 评估any DR、需转诊DR(RDR)和威胁视力DR(STDR)三个筛查终点,概率阈值设为20%、50%和70%,主要终点为50%决策阈值下RDR的AUC和敏感度。

主要发现

  • 以人工共识分级为参考标准,Amazon SageMaker Canvas和AutoGluon整体判别能力最强,STDR的AUC最高达0.96,RDR的AUC为0.93至0.94。
  • 50%决策阈值下,Canvas对RDR表现最均衡(敏感度88.3%、特异度85.5%、准确率86.0%),AutoGluon偏重敏感度(any DR敏感度95.9%)但牺牲特异度。
  • Vertex AI表现持续较弱且不稳定(any DR AUC 0.58;RDR AUC 0.38)。
  • 与IDx-DR相比,Amazon Rekognition和Canvas一致性最高,尤其STDR(AUC最高0.88-0.90;κ最高约0.56)。
  • 与人工分级者的一致性普遍为低至中等(κ约0.3-0.6),且在较高概率阈值下增加。

关键数值与推荐

  • 训练数据集:APTOS n=5590、DDR n=12524、EyePACS n=31557,按国际临床糖尿病视网膜病变(ICDR)量表标注。
  • 测试队列:独立机构收集的患者级队列n=726。
  • 主要终点:50%决策阈值下RDR的AUC和敏感度;次要终点包括特异度、阳性预测值、阴性预测值、准确度和F1分数(含95%置信区间)。
  • 统计比较:AUC差异用bootstrap检验(n=1000),二分类结局用McNemar检验(50%阈值),均经Bonferroni校正(P<.0033)。
  • 评估阈值:20%、50%、70%概率阈值。

临床怎么用

  • AutoML平台可用于DR筛查并达到有临床意义的表现,但需根据临床场景选择工具和校准阈值。
  • 不同平台在敏感度和特异度间存在权衡,偏重敏感度的模型可能增加转诊量,需结合资源条件选择。
  • 部署前应进行外部验证和阈值校准,以确保在本地人群中的性能。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

PURPOSE: To benchmark multiple automated machine learning (AutoML) platforms for diabetic retinopathy (DR) screening from fundus photographs using a unified training and evaluationframework, with human consensus grading and an FDA-approved autonomous system (IDx-DR) as reference standards. DESIGN: Retrospective, diagnostic performance and benchmarking study. METHODS: Image classifiers were trained on large public datasets labeled according to the International Clinical Diabetic Retinopathy (ICDR) scale (APTOS, n = 5590; DDR, n = 12,524; EyePACS, n = 31,557) after automated image-quality filtering. Performance was evaluated on an independent, institutionally collected patient-level test cohort (n = 726) using the highest DR grade across all images for patient-level classification. The evaluated platforms included Google Vertex AI, Amazon Rekognition, Amazon SageMaker Canvas, AutoGluon, AutoKeras, and Apple CreateML. Models were assessed for 3 screening endpoints-any DR, referable DR (RDR), and sight-threatening DR (STDR)-across probability thresholds of 20%, 50%, and 70%. The primary endpoints were the Area Under the Receiver Operating Characteristic Curve (AUC) and sensitivity for RDR at a 50% decision threshold. Secondary endpoints included specificity, positive predictive value, negative predictive value, accuracy, and F1-score with 95% confidence intervals. Pairwise comparisons were performed using bootstrap testing (n = 1000) for AUC differences and McNemar's test (at a 50% threshold) for binary outcomes, both subjects to Bonferroni correction (P < .0033). Grad-CAM was applied to locally deployable convolutional neural network-based models. RESULTS: Using human consensus grading as the reference standard, Amazon SageMaker Canvas and AutoGluon demonstrated the strongest overall discrimination, achieving AUC values up to0.96 for STDR and 0.93 to 0.94 for RDR. At the 50% decision threshold, Canvas showed the most balanced performance for RDR (sensitivity 88.3%, specificity 85.5%, accuracy 86.0%), whereasAutoGluon favored sensitivity (any DR sensitivity 95.9%) at the expense of specificity. Vertex AI showed consistently weaker and unstable performance (any DR AUC 0.58; RDR AUC 0.38). Relative to IDx-DR, Amazon Rekognition and Canvas showed the highest agreement, particularly for STDR (AUC up to 0.88-0.90; κ up to ∼0.56). Agreement with human graders was generallylow to moderate (κ ≈ 0.3-0.6) and increased at higher probability thresholds. CONCLUSIONS: AutoML platforms can achieve clinically meaningful performance for DR screening. Differences across tools and thresholds reflect their adaptability to diverse clinical settings, underscoring the importance of external validation and threshold calibration.

来源:Am J Ophthalmol,查看原文。版权归原作者所有。

引用

  1. Krzywicki T, Engin CD, Grzybowski A, et al. Auto Machine Learning for Diabetic Retinopathy Screening: A Head-to-Head Multiplatform Comparison Against Human Graders and IDx-DR. Am J Ophthalmol. 2026. PMID: 42082068. doi:10.1016/j.ajo.2026.04.030

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。